Hi there, Along with my k8s autoscaler work at the moment, I also found that my operator node would never fulfill the demands specified in my ray.remote decorator. I also tried this: @ray.remote(resources={'Custom':5, 'CPU'=5, 'num_cpus'=5, 'worker-node':5}) def f(x): time.sleep(0.01) retu…

Hey @Dmitri This seems to be an issue. Can you take a look at it? Also, @rico-ci which version of ray are you using?

I’d recommend using nightly versions of Ray everywhere, if you’re not already – so, freshly pulled rayproject/ray:nightly in the operator pod if using the Operator, local nightly build of ray if using the Ray Cluster Launcher, Ray nodes running Ray at least 1.2.0 .

Hi Dimitri! So nightly won’t work for us because we need a custom image for the head/worked nodes. Also, because this will eventually end-up on production, I would really rather have a fixed version number (last month we’ve lost a few days worth of work because another framework introduced a bug wi…

There was a critical fix for the operator that unfortunately didn’t make it into 1.2.0 ( [autoscaler][kubernetes] autoscaling hotfix by DmitriGekhtman · Pull Request #14024 · ray-project/ray · GitHub ). Just replacing the operator image with a newer one should improve the situation (everything else c…

Autoscaler doesn't scale workers on K8s

sangcho February 15, 2021, 10:39pm 3

Also can you address this at the same time? TypeError on Ray Cluster with Ray 1.2.0

Topic		Replies	Views
Autoscaler not scaling up the worker node when using image rayproject/ray:1.11.0-py38 Kubernetes	3	1022	July 2, 2022
Autoscaler SDK request_resoures fails on EKS Kubernetes	8	728	February 16, 2021
Autoscaler issues with the K8 Operator Kubernetes	8	778	March 2, 2021
Testing autoscaler Kubernetes	15	2013	March 16, 2021
Scale up from 0 Ray Clusters	7	701	July 15, 2021

Autoscaler doesn't scale workers on K8s

Related topics