How severe does this issue affect your experience of using Ray? High: It blocks me to complete my task. I slurm ran ray start --head, and the slurm gave me a node and showed the ray has started. As the picture blow shows: [image] but then I ran ray status, it showed me blow: [image] I wrot…

Thank you cade for helping me. [image] cade: Does the node on which you are running the slurm script (ray start --head and ray status) have multiple network interfaces? Yes, the node does has multiple network interfaces. [image] cade: To verify that this is the case, can you reproduce t…

Hi @zyc-bit , thanks for your question! Could you provide the relevant section of the bash script? It is interesting that ray stop works but ray status doesn’t work. As for the dashboard only working sometimes, could you provide more details? Can you retry and get the dashboard, or do you have to re…

Hi @cade , thank you very much for your reply. Let me describe in detail how the above situation happened. ( I am working with a project called Alpa which requires ray ) I install Ray by pip install ray. On the cluster I am using, there is one management node and many compute nodes. The management…

Hi @cade , do you know the error blow? [image]

Hi @zyc-bit , I’m sorry I dropped this! Were you able to get past the issue? Is it related to your other more recent post Ray dashboard can not start ?

or maybe Raylet errors some worker have not registered within the timeout ?

[image] cade: that ray stop works but ray status doesn’t work. Hi, cade Strictly speaking, I didn’t solve this problem completely. Occasionally there are times when I still run into this problem. When I run into this problem, I close the terminal and start over again, and it’s usually solved…

The problem about dashboard is not solved. On our Slurm, we have a management node and a compute node, the management node has no GPU, the compute node has GPU. on the management node I can start dashboard, but on the compute node I cannot start dashboard But since only the compute nodes have GPUs…

[image] Raylet errors some worker have not registered within the timeout Ray Core How severe does this issue affect your experience of using Ray? High: It blocks me to complete my task. It really bother me. I always get the error below. And just keep output the messag…

Does the node on which you are running the slurm script (ray start --head and ray status) have multiple network interfaces? I am guessing that would cause this issue. (this is being worked on here ) To verify that this is the case, can you reproduce the issue (so ray status crashes), then check each…

Yep, let’s work on that in the separate question.

“ray start --head” succeed but "ray status" could not find any running ray instance

Ray Clusters

zyc-bit June 16, 2022, 1:16am 9

This issue still blocks me.

Topic		Replies	Views
Ray dashboard can not start Ray Core	17	2178	February 14, 2024
Ray head isn't starting properly sometimes Ray Core	7	587	April 28, 2023
Ray status couldn't find the Ray cluster despite a working cluster Ray Clusters	2	1219	October 26, 2022
Ray dashboard won't start Dashboard, Monitoring & Debugging	1	339	February 20, 2025
(raylet) Some workers of the worker process(68497) have not registered within the timeout. The process is still alive, probably it's hanging during start Ray Clusters	4	2825	May 26, 2022

“ray start --head” succeed but "ray status" could not find any running ray instance

Related topics