# Autoscaling issue with Ray on SLURM? **URL:** \ **Category:** Ray Core\ **Created:** [August 27, 2021, 5:05pm UTC](https://discuss.ray.io/t/autoscaling-issue-with-ray-on-slurm/3370 "2021-08-27T17:05:13Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![tpouncy](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/tpouncy/32/755_2.png) [@tpouncy](https://discuss.ray.io/u/tpouncy)\ **Post date:** [August 27, 2021, 5:05pm UTC](https://discuss.ray.io/t/autoscaling-issue-with-ray-on-slurm/3370/1 "2021-08-27T17:05:14Z")
I have been using Ray 1.0.1 to run a distributed Python script on a SLURM system for the past few months. Recently I started having issues with Ray workers unexpectedly crashing and killing my job, so I upgraded to the latest version of Ray (1.6.0 at time of writing). Now when I run my job it intermittently fails with the following error: > 2021-08-27 12:20:02,445 WARNING worker.py:1215 – The autoscaler failed with the following error: > Traceback (most recent call last): > File “/lib/python3.7/site-packages/ray/autoscaler/\_private/monitor.py”, line 324, in run > self.\_run() > File “lib/python3.7/site-packages/ray/autoscaler/\_private/monitor.py”, line 214, in \_run > self.update\_load\_metrics() > File “/lib/python3.7/site-packages/ray/autoscaler/\_private/monitor.py”, line 177, in update\_load\_metrics > request, timeout=4) > File “/lib/python3.7/site-packages/grpc/\_channel.py”, line 946, in **call** > return \_end\_unary\_response\_blocking(state, call, False, None) > File “/lib/python3.7/site-packages/grpc/\_channel.py”, line 849, in \_end\_unary\_response\_blocking > raise \_InactiveRpcError(state) > grpc.\_channel.\_InactiveRpcError: \<\_InactiveRpcError of RPC that terminated with: > status = StatusCode.UNAVAILABLE > details = “failed to connect to all addresses” > debug\_error\_string = “{“created”:”@1630081202.423352369",“description”:“Failed to pick subchannel”,“file”:“src/core/ext/filters/client\_channel/client\_channel.cc”,“file\_line”:5419,“referenced\_errors”:[{“created”:“@1630081202.423350314”,“description”:“failed to connect to all addresses”,“file”:“src/core/ext/filters/client\_channel/lb\_policy/pick\_first/pick\_first.cc”,“file\_line”:397,“grpc\_status”:14}]}" > > > I’m a little confused by this error because it seems like Ray is crashing while trying to autoscale the number of workers, but because I am using SLURM, the number of worker nodes should be fixed. Here is the script I use to set up my ray cluster on SLURM (drawn from the Ray documentation [here](https://docs.ray.io/en/master/cluster/slurm.html) ): > #!/bin/bash > > # shellcheck disable=SC2206 > > # THIS FILE IS GENERATED BY AUTOMATION SCRIPT! PLEASE REFER TO ORIGINAL SCRIPT! > > # THIS FILE IS MODIFIED AUTOMATICALLY FROM TEMPLATE AND SHOULD BE RUNNABLE! > > #SBATCH --partition=shared > #SBATCH --job-name=test\_0426-1444 > #SBATCH --output=test\_%j.log > > ### This script works for any number of nodes, Ray will find and manage all resources > > #SBATCH --nodes=2 > #SBATCH --mem-per-cpu=2500 > > ### Give all resources to a single Ray task, ray can manage the resources internally > > #SBATCH --cpus-per-task=48 > #SBATCH --tasks-per-node=1 > #SBATCH -t 0-16:00:00 > > # Load modules or your own conda environment here > > # module load pytorch/v1.4.0-gpu > > # conda activate ${CONDA\_ENV} > > source activate MY\_CONDA\_ENV > > # ===== DO NOT CHANGE THINGS HERE UNLESS YOU KNOW WHAT YOU ARE DOING ===== > > # This script is a modification to the implementation suggest by gregSchwartz18 here: > > # [work with cluster managed by Slurm · Issue #826 · ray-project/ray · GitHub](https://github.com/ray-project/ray/issues/826#issuecomment-522116599) > > #redis\_password=$(uuidgen) > #export redis\_password > > nodes=$(scontrol show hostnames “$SLURM\_JOB\_NODELIST”) # Getting the node names > nodes\_array=($nodes) > > node\_1=${nodes\_array[0]} > ip=$(srun --nodes=1 --ntasks=1 -w “$node\_1” --mem-per-cpu=2500 hostname --ip-address) # making redis-address > > # if we detect a space character in the head node IP, we’ll > > # convert it to an ipv4 address. This step is optional. > > if [[“$ip” == _" "_]]; then > IFS=’ ’ read -ra ADDR \<\<\< “$ip” > if [[${#ADDR[0]} -gt 16 ]]; then > ip=${ADDR[1]} > else > ip=${ADDR[0]} > fi > echo “IPV6 address detected. We split the IPV4 address as $ip” > fi > > port=49160 > ip\_head=$ip:$port > export ip\_head > echo “IP Head: $ip\_head” > > echo “STARTING HEAD at $node\_1” > srun --nodes=1 --ntasks=1 -w “$node\_1” --mem-per-cpu=2500 > ray start --head --node-ip-address=“$ip” --port=$port --block & > sleep 30 > > worker\_num=$((SLURM\_JOB\_NUM\_NODES - 1)) #number of nodes other than the head node > for ((i = 1; i \<= worker\_num; i++)); do > node\_i=${nodes\_array[$i]} > echo “STARTING WORKER $i at $node\_i” > srun --nodes=1 --ntasks=1 -w “$node\_i” --mem-per-cpu=2500 > ray start --address “$ip\_head” --block & > sleep 5 > done > > # ===== Call your code below ===== > > python -u run\_agent.py What’s frustrating is that this error only occurs intermittently, and often only after the job has been running for several hours. Has anyone else encountered this issue, or have any idea how to correct it? Thanks in advance for any help! ---
**Author:** ![tpouncy](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/tpouncy/32/755_2.png) [@tpouncy](https://discuss.ray.io/u/tpouncy)\ **Post date:** [September 7, 2021, 1:09pm UTC](https://discuss.ray.io/t/autoscaling-issue-with-ray-on-slurm/3370/2 "2021-09-07T13:09:29Z")
I figured out what was going on here and wanted to post an update in case anyone else runs into the same issue. There was a bug in my code that was running the same block of code ~20 times, meaning that instead of running ~120 parallel tasks I was trying to run about 2000 on a cluster with ~100 CPUs. Removing this bug and scaling the number of tasks down to match the size of the cluster again allowed the task to run without any further problems. ---
**Author:** ![Chen\_Shen](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/chen_shen/32/1486_2.png) [@Chen\_Shen](https://discuss.ray.io/u/Chen_Shen)\ **Post date:** [September 8, 2021, 3:36am UTC](https://discuss.ray.io/t/autoscaling-issue-with-ray-on-slurm/3370/3 "2021-09-08T03:36:58Z")
That’s great to know!