Ray cluster worker nodes initialization issues

Ray Clusters
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

I’m trying to launch a Ray cluster on GCP following this and I’ve specified the initialization_commands and setup_commands but they are not run on the worker nodes. I verified that by looking at “monitor.out” log file and it says “No initialization commands to run.”. What could be going wrong here?

I’m constantly modifying the cluster YAML file but always made sure to save the file before creating the cluster. Additionally, I have been using “–no-config-cache” option.

2

Additionally, I have two worker nodes to be set up and I sometimes can see one of them running the initialization commands while the other one doesn’t.

3

@magichan33 is this still blocking you?

4

I am facing this same issue on Ray 2.37.0, only one or two worker nodes will be properly initialized, the others do not run anything requested for them in the configuration file. Any help would be much appreciated.