Error: Array of Jobs in SLURM

Ray Libraries (Data, Train, Tune, Serve)
1

Hello together,

for our current research project i run an array of jobs on a slurm cluster, each containing a ray tune experiment. But half of the jobs instantly fail with the following error:

2023-07-31 17:11:21,953 INFO worker.py:1636 -- Started a local Ray instance.
[2023-07-31 17:11:52,003 E 76057 76057] core_worker.cc:193: Failed to register worker 01000000ffffffffffffffffffffffffffffffffffffffffffffffff to Raylet. IOError: [RayletClient] Unable to register worker with raylet. No such file or directory

I provided a reduced example of what we are doing in the following GitHub Repository:

Thanks in advance,
Paul