Multiple head nodes on kubernetes

[shoelesself]

I am going through the example for deploying ray on kubernetes using the operator. I am getting multiple head nodes created when running tasks. My understanding is there should only ever be a single head node. Is this normal behavior?

NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-2ppkg 0/1 Pending 0 4m12s
example-cluster-ray-head-2xlmw 0/1 Pending 0 4m12s
example-cluster-ray-head-7lgbs 1/1 Running 0 9m59s
example-cluster-ray-head-jvvlk 1/1 Running 0 9m59s
example-cluster-ray-head-nbkgq 1/1 Running 0 9m59s
example-cluster-ray-head-pkr4s 1/1 Running 0 35m
example-cluster-ray-head-pn2sm 0/1 Pending 0 4m12s
example-cluster-ray-head-tssb6 0/1 Pending 0 4m12s
example-cluster-ray-head-wl7jz 1/1 Running 0 9m59s
example-cluster-ray-worker-kfvhx 1/1 Running 0 35m
example-cluster-ray-worker-s8sps 1/1 Running 0 35m
ray-operator-pod 1/1 Running 0 17h

[sangcho]

Ray cluster should have only one head node. This looks weird to me, but I will delegate to the person who knows more about K8s operator; cc @Dmitri

[Dmitri]

Yeah, I think I see what’s happening here –
the bug was probably introduced in this PR:

[autoscaler] Max worker default infinity

and it should be fixed once this PR is merged

[autoscaler][draft][interface] Use multi node types in defaults.yaml and example-full.yaml

For now, you can apply the fix manually by setting maxWorkers: 0 here:

ray-project/ray/blob/6cd1359a11a1b471a1f9fdb385713b72397ce2e7/python/ray/autoscaler/kubernetes/operator_configs/example_cluster.yaml#L23

and similarly in example_cluster2.yaml.