Error running tune on k8s

Ray Clusters Kubernetes
1

Hi,

I deployed ray cluster to my k8s, following instructions on your website Deploying on Kubernetes — Ray v2.0.0.dev0.

the deployment succeeded without any errors

When I try to run simple rl app that uses tune on the cluster from my local machine, I get the following error:

 return self._call_schedule_for_task(task)
  File "/home/goranj/miniconda3/envs/rayenv/lib/python3.7/site-packages/ray/util/client/worker.py", line 318, in _call_schedule_for_task
    raise cloudpickle.loads(ticket.error)
ImportError: libgthread-2.0.so.0: cannot open shared object file: No such file or directory

I am getting the same error when deploying ray to my local cluster and to AKS.

App that I am trying to run:

ray.util.connect("127.0.0.1:10001")
env="CartPole-v0"
tune.run(
ppo.PPOTrainer,
config={
"env": env,
"timesteps_per_iteration": 300,
"monitor": False,
"log_level": "DEBUG",
"ignore_worker_failures": True
})
2

Thanks, tracking here:

3

Can you check which versions of Ray (and other dependencies) you are using on the client machine and the pods?

4

Any such helpful details appreciated on the issue :slight_smile:

5

Hi,

I used latest stable Ray version, 1.4 and the helm chart to deploy Ray ( ray/deploy/charts/ray at master · ray-project/ray (github.com)).