Ray cluster uses only Head node

Ray Clusters
1

image
image1896×304 33.4 KB

Hello, my cluster spawned successfully, but for some reason when running tasks it is utilizing head node only.

Didn’t notice any relevant errors in logs.

Head node been spawned with:
ulimit -n 65536; ray start --head --port=6379 --object-manager-port=8076 --autoscaling-config=~/ray_bootstrap_config.yaml --dashboard-host=$RAY_HEAD_IP
And workers with:
ulimit -n 65536; ray start --address=$RAY_HEAD_IP:6379 --object-manager-port=8076

Please point me what do I miss. Thanks.

2

Hi @kpavel, can you provide some more details about what tasks you’re running? If you’re running more than 2 tasks in parallel, i would expect the tasks beyond the 2nd task to be scheduled on the other two nodes. For example,

import ray
import time
ray.init(address="auto")
@ray.remote
def f(i):
    time.sleep(30)
    return i
futures = [f.remote(i) for i in range(6)]
print(ray.get(futures))

If you run this on the head node, do you see the tasks show up in the dashboard?

3

Hello @architkulkarni. Thanks, we can close the issue.
It appears that when you use ray.init() without address=“auto” it schedules tasks to head node only.
Didn’t see it documented anywhere.

4

Ah yeah, that’s correct: Starting Ray — Ray v2.0.0.dev0. We definitely want to make our documentation crystal-clear, so any edits or suggestions would be very welcome!