Specify workerPodType in Helm chart values.yaml

Hi!
I’m trying to understand how to create a couple of different clusters with Ray Kubernetes Operator and Helm Charts. I understand I can specify headPodType in values.yaml. I use rayHeadType, as specified in the example, for my first cluster. Then I have another one, called rayHeadType2 that I can use. But what about workerPodType? There is an example in the end of the file, specifing rayWorkerType2, but I don’t understand where I can use it. I don’t even find anywhere where workerPodType is actually used!

Hope someone can help me. Thank you! :slightly_smiling_face:

cc @Dmitri for help.

Each Helm release you install gets you one Ray cluster (unless you install with the option operatorOnly, in which case you get the operator and no Ray clusters)

A single Ray cluster consists of 1 Ray head pod and any number of Ray worker pods.
Each Ray pod has a podType, which is a configuration template used to create the pod.
The field podTypes specifies these types.

The format for the podTypes field is a string-podType map. The string key is an arbitrary name you give the podType. The value is the configuration.

Since there is only 1 head node per cluster, it doesn’t make sense to have two podTypes named rayHeadType/rayHeadType2 in one values.yaml configuration.

One common pattern for mixed CPU/GPU workloads is to have three podTypes:

  1. Special configuration for a CPU head node.
  2. A CPU worker type.
  3. A GPU worker type (to be scaled up when there are GPU demands)

The configuration for that setup would look something like this:

headPodType: mySpecialHeadType

podTypes:
    mySpecialHeadType:
        CPU: 64
        ...
    myBoringCPUType:
        CPU: 8
        minWorkers: 10
        maxWorkers: 20
        ...
    myExtremelyExpensiveGPUType:
        GPU: 4
        CPU: 128
        minWorkers: 0
        maxWorkers: 2 
        ...

You can read more here about podTypes:
https://docs.ray.io/en/latest/cluster/kubernetes-advanced.html#ray-cluster-configuration

You can read more about how to run multiple Ray clusters here:
https://docs.ray.io/en/latest/cluster/kubernetes-advanced.html#running-multiple-ray-clusters

1 Like