Hi, im a beginner of ray. I established a ray cluster with a head and two workers. After submitting ray job to cluster and loading the training data from local disk, the 6 processes on the head node immediately start and 6 cpus were used, but then all of the 18 processes get into status ray::IDLE and none of the cpus were used.
I turn to the log and find that it takes about 5 mins between two lines
(process_one_day pid=202346) and (process_one_day pid=119857, ip=10.50.12.45)
I think it suggests that the job start up too slow on the worker node, and same problem also happens on the other worker node 10.50.12.53, and it parallelize on 18 cpus very well after this two nodes get started.
How can I get the job start more quickly on the worker nodes? Please advise, thank you.