# Issues with deploying Ray on Slurm:

**URL:** <https://discuss.ray.io/t/issues-with-deploying-ray-on-slurm/1090>\
**Category:** Ray Core\
**Created:** [March 3, 2021, 1:40am UTC](https://discuss.ray.io/t/issues-with-deploying-ray-on-slurm/1090 "2021-03-03T01:40:51Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![kipnisal](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kipnisal/32/534_2.png) [@kipnisal](https://discuss.ray.io/u/kipnisal)\
**Post date:** [March 3, 2021, 1:40am UTC](https://discuss.ray.io/t/issues-with-deploying-ray-on-slurm/1090/1 "2021-03-03T01:40:51Z")

</div>

I’ve been trying to run the following script on a slurm cluster (this script works fine on a local machine with ray installed).

```
import numpy as np
import pandas as pd
from evaluate_iteration import *
import ray

lr = 27
lb = 29
a = 0.8
x = 0.5
n = 1e5
nMonte = 1000

rr= np.linspace(0.1, 3, lr)
bb=np.linspace(0.45, 0.99, lb)
params = [(beta, r) for r in rr for beta in bb]

ray.init(address='auto')
res = [evaluate_iteration.remote(a, x, par[0], par[1], n,
                nMonte = nMonte, metric = 'power') for par in params]
return_value = ray.get(res)

df = pd.DataFrame()
for r in return_value :
    df = df.append(r, ignore_index=True)
df.to_csv('results.csv')

```

The submission script is as follows (based on [Deploying on Slurm — Ray v2.0.0.dev0](https://docs.ray.io/en/master/cluster/slurm.html)):

```
#!/bin/bash
#BATCH --job-name=test
#SBATCH --cpus-per-task=5
#SBATCH --mem-per-cpu=2GB
#SBATCH --nodes=5
#SBATCH --tasks-per-node 1
worker_num=3 # Must be one less that the total number of nodes
# module load Langs/Python/3.6.4 # This will vary depending on your environment
# source venv/bin/activate
nodes=$(scontrol show hostnames $SLURM_JOB_NODELIST) # Getting the node names
nodes_array=( $nodes )
node1=${nodes_array[0]}
ip_prefix=$(srun --nodes=1 --ntasks=1 -w $node1 hostname --ip-address) # Making address
suffix=':6379'
ip_head=$ip_prefix$suffix
redis_password=$(uuidgen)
export ip_head # Exporting for latter access by trainer.py
srun --nodes=1 --ntasks=1 -w $node1 ray start --block --head --redis-port=6379 --redis-password=$redis_password & # Starting the head
sleep 5
# Make sure the head successfully starts before any worker does, otherwise
# the worker will not be able to connect to redis. In case of longer delay,
# adjust the sleeptime above to ensure proper order.
for (( i=1; i<=$worker_num; i++ ))
do
  node2=${nodes_array[$i]}
  srun --nodes=1 --ntasks=1 -w $node2 ray start --block --address=$ip_head --redis-password=$redis_password & # Starting the workers
  # Flag --block will keep ray process alive on each compute node.
  sleep 5
done
python -u phase_diagram.py $redis_password 15 # Pass the total number of allocated CPUs

```

I get the following error in the .out file:  
Error: no such option: --redis-port  
srun: error: sh02-01n11: task 0: Exited with exit code 2

---

<div class="post-metadata">

**Author:** ![kipnisal](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kipnisal/32/534_2.png) [@kipnisal](https://discuss.ray.io/u/kipnisal)\
**Post date:** [March 3, 2021, 1:45am UTC](https://discuss.ray.io/t/issues-with-deploying-ray-on-slurm/1090/2 "2021-03-03T01:45:21Z")

</div>

Hi @richardliaw, @zhz,  
It would be great if you can take a look at this.

---

<div class="post-metadata">

**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\
**Post date:** [March 3, 2021, 2:59am UTC](https://discuss.ray.io/t/issues-with-deploying-ray-on-slurm/1090/3 "2021-03-03T02:59:47Z")

</div>

The argument `--redis-port` was renamed renamed `--port` at some point. If you leave it off, it defaults to 6379.

The change is reflected in the current docs, I think: [Deploying on Slurm — Ray v2.0.0.dev0](https://docs.ray.io/en/master/cluster/slurm.html#starting-the-ray-head-node)

---

<div class="post-metadata">

**Author:** ![kipnisal](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kipnisal/32/534_2.png) [@kipnisal](https://discuss.ray.io/u/kipnisal)\
**Post date:** [March 3, 2021, 4:27am UTC](https://discuss.ray.io/t/issues-with-deploying-ray-on-slurm/1090/4 "2021-03-03T04:27:21Z")

</div>

Thanks @Dmitri. I’ve used the `slurm-launch.py` script mentioned in [Deploying on Slurm](https://docs.ray.io/en/master/cluster/slurm.html#starting-the-ray-head-node).  
I now receive the following error:

```
[kipnisal@sh03-ln01 login ~/PhaseTrans]$ cat PT_0302-1947.log
/var/spool/slurmd/job19469069/slurm_script: line 19: None: command not found
IP Head: 10.19.1.1:6379
STARTING HEAD at sh03-01n01
srun: Job 19469069 step creation temporarily disabled, retrying (Requested nodes are busy)
srun: Step created for job 19469069
2021-03-02 19:48:56,326 INFO services.py:1174 -- View the Ray dashboard at http://localhost:8265
STARTING WORKER 1 at sh03-01n02
STARTING WORKER 2 at sh03-01n09
2021-03-02 19:49:35,348 INFO worker.py:655 -- Connecting to existing Ray cluster at address: 10.19.1.1:6379
Traceback (most recent call last):
  File "phase_diagram.py", line 22, in <module>
    ray.init(address='auto')
  File "/home/users/kipnisal/CJinstalled/miniconda/lib/python3.7/site-packages/ray/_private/client_mode_hook.py", line 47, in wrapper
    return func(*args, **kwargs)
  File "/home/users/kipnisal/CJinstalled/miniconda/lib/python3.7/site-packages/ray/worker.py", line 759, in init
    connect_only=True)
  File "/home/users/kipnisal/CJinstalled/miniconda/lib/python3.7/site-packages/ray/node.py", line 153, in __init__
    session_name = _get_with_retry(redis_client, "session_name")
  File "/home/users/kipnisal/CJinstalled/miniconda/lib/python3.7/site-packages/ray/node.py", line 39, in _get_with_retry
    result = redis_client.get(key)
  File "/home/users/kipnisal/CJinstalled/miniconda/lib/python3.7/site-packages/redis/client.py", line 1606, in get
    return self.execute_command('GET', name)
  File "/home/users/kipnisal/CJinstalled/miniconda/lib/python3.7/site-packages/redis/client.py", line 898, in execute_command
    conn = self.connection or pool.get_connection(command_name, **options)
  File "/home/users/kipnisal/CJinstalled/miniconda/lib/python3.7/site-packages/redis/connection.py", line 1192, in get_connection
    connection.connect()
  File "/home/users/kipnisal/CJinstalled/miniconda/lib/python3.7/site-packages/redis/connection.py", line 567, in connect
    self.on_connect()
  File "/home/users/kipnisal/CJinstalled/miniconda/lib/python3.7/site-packages/redis/connection.py", line 643, in on_connect
    auth_response = self.read_response()
  File "/home/users/kipnisal/CJinstalled/miniconda/lib/python3.7/site-packages/redis/connection.py", line 756, in read_response
    raise response
redis.exceptions.ResponseError: WRONGPASS invalid username-password pair
```

---

<div class="post-metadata">

**Author:** ![rkn](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rkn/32/8_2.png) [@rkn](https://discuss.ray.io/u/rkn)\
**Post date:** [March 3, 2021, 4:50am UTC](https://discuss.ray.io/t/issues-with-deploying-ray-on-slurm/1090/5 "2021-03-03T04:50:08Z")

</div>

Does the Redis password need to get passed into `ray.init`?

---

<div class="post-metadata">

**Author:** ![rliaw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rliaw/32/24_2.png) [@rliaw](https://discuss.ray.io/u/rliaw)\
**Post date:** [March 3, 2021, 8:56am UTC](https://discuss.ray.io/t/issues-with-deploying-ray-on-slurm/1090/6 "2021-03-03T08:56:48Z")

</div>

Yeah, I think one fix is to make sure `ray.init(address="auto", _redis_password=sys.argv[1])`.

The other fix (which I would probably recommend) is to remove all references to the redis password altogether in both your python and your sbatch script.