3 workers but only 1 available

1

Hello,

I have 3 nodes with GPU and ray cluster in kubernetes (containerd) environment.

I only can see 1 worker and 1 head in my Ray cluster dashboard. How can I add another worker and make that the head node works also as a worker?

image
image970×414 17.8 KB

NAME STATUS ROLES AGE VERSION
node1 Ready control-plane 71d v1.27.1
node2 Ready 4h42m v1.27.2
node3 Ready 4h42m v1.27.2

2

I am not very familiar with dashboard, cc: @sangcho, @rickyyx

3

Well, more than dashboard knowledge is about how to force a node to be a master.

The best approach I think is to have a (only CPU) node as a head node and then 3 (GPU) nodes as a workers but at the moment one of them is in kubernetes cluster but not in ray cluster.

4

cc @Kai-Hsun_Chen Looks like kuberay related questions? Can you follow up?

5

Hi @henryq3

I cannot get your point. Could you open a GitHub issue with more details on KubeRay repo?

How can I add another worker

If you are using KubeRay, you can specify or update the number of replicas for worker Pods in the RayCluster custom resource YAML file.

make that the head node works also as a worker?

Would you mind explaining more about “make that the head node works also as a worker”? I assumed that you want to schedule Ray Actors and Tasks on the Ray head Pod. If you do not specify num-cpus: "0" in Ray head’s rayStartParams, Ray will schedule Tasks and Actors on the head Pod.

3 workers but only 1 available

Based on the context, “3 workers” seems to indicate 3 Kubernetes nodes rather than Ray worker Pods.