Multiple head nodes on kubernetes

I am going through the example for deploying ray on kubernetes using the operator. I am getting multiple head nodes created when running tasks. My understanding is there should only ever be a single head node. Is this normal behavior?

NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-2ppkg 0/1 Pending 0 4m12s
example-cluster-ray-head-2xlmw 0/1 Pending 0 4m12s
example-cluster-ray-head-7lgbs 1/1 Running 0 9m59s
example-cluster-ray-head-jvvlk 1/1 Running 0 9m59s
example-cluster-ray-head-nbkgq 1/1 Running 0 9m59s
example-cluster-ray-head-pkr4s 1/1 Running 0 35m
example-cluster-ray-head-pn2sm 0/1 Pending 0 4m12s
example-cluster-ray-head-tssb6 0/1 Pending 0 4m12s
example-cluster-ray-head-wl7jz 1/1 Running 0 9m59s
example-cluster-ray-worker-kfvhx 1/1 Running 0 35m
example-cluster-ray-worker-s8sps 1/1 Running 0 35m
ray-operator-pod 1/1 Running 0 17h

Ray cluster should have only one head node. This looks weird to me, but I will delegate to the person who knows more about K8s operator; cc @Dmitri

Yeah, I think I see what’s happening here –
the bug was probably introduced in this PR:

and it should be fixed once this PR is merged

For now, you can apply the fix manually by setting maxWorkers: 0 here:

and similarly in example_cluster2.yaml.

1 Like