Ray up doesn't add worker resources to ray.status()

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

My problem is simple, I have a local PC and I have another pc, so I set the FIRST pc to the head_ip field in the cluster_config.yaml. But the ray doesn’t add all of the resources (head + worker) to the ray status.
Also after that I run ray submit {config} {script} (e.g. hyperparameter tuning using hyperopt). There I set the address field of init to ‘auto’, after that only one machine is executing the hyperparameter search but not the other! Do you know what to do at this point? I’m trying to set up a distributed hyperparameter search experiment, but it doesn’t seem to work. Help is much appreciated.
but after that the

2

Hey @Maxim_Afteniy this is most likely because the second node was never actually added to the cluster.

How exactly are you starting the ray cluster and what does your cluster yaml look like?

3

hi @amogkam , I’m starting just with the yaml file and the instructions that are given in the ray docs:
cluster.yaml

cluster_name: basic-ray

max_workers: 2 

provider:
  type: local
  head_ip: IP_COMPUTER_1
  worker_ips: [IP_COMPUTER_2]
  
auth:
   ssh_user: maxim

setup_commands: []

head_start_ray_commands:
  - ray stop
  - ulimit -c unlimited && ray start --head --port=6379

worker_start_ray_commands:
  - ray stop
  - ray start --address='IP_COMPUTER_1:6379' --redis-password='5241590000000000'

ray up -y cluster.yaml
this goes very well through all steps 1-7/7, and tells me the next steps. However I expect it to add all nodes specified in the yaml to the config. However ray status
Outputs the specs only of one machine ( I expect two, worker and head node )

Node status
---------------------------------------------------------------
Healthy:
 1 node_2a29ee6b0da91931eba96794a63f10a670452be2014d7449c14047a7
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/16.0 CPU
 0.0/1.0 GPU
 0.0/1.0 accelerator_type:G
 0.00/27.280 GiB memory
 0.00/13.640 GiB object_store_memory

Demands:
 (no resource demands)

Another thing that bugs me is that ray up -y cluster.yaml doesn’t work well if I specify the head node the same as the current pc I’m on. It performs some random ssh connections between the steps to my own computer which doesn’t make sense.

4

@Maxim_Afteniy sorry for the delay here.

Could you also share your Ray Tune code? What is the num_samples you have set? And what Ray version are you using?

The ray status you are seeing is expected since this is an autoscaling cluster. It will start with only 1 node and then will add another node if Ray has demand for more resources. Ray Tune is what triggers these resource requests.

If you want to start with 2 nodes off the bat, you can add min_workers: 2 to the cluster yaml (right before max_workers).