Specify workerPodType in Helm chart values.yaml

Ray Clusters
1

Hi!
I’m trying to understand how to create a couple of different clusters with Ray Kubernetes Operator and Helm Charts. I understand I can specify headPodType in values.yaml. I use rayHeadType, as specified in the example, for my first cluster. Then I have another one, called rayHeadType2 that I can use. But what about workerPodType? There is an example in the end of the file, specifing rayWorkerType2, but I don’t understand where I can use it. I don’t even find anywhere where workerPodType is actually used!

Hope someone can help me. Thank you! :slightly_smiling_face:

2

cc @Dmitri for help.

3

Each Helm release you install gets you one Ray cluster (unless you install with the option operatorOnly, in which case you get the operator and no Ray clusters)

A single Ray cluster consists of 1 Ray head pod and any number of Ray worker pods.
Each Ray pod has a podType, which is a configuration template used to create the pod.
The field podTypes specifies these types.

The format for the podTypes field is a string-podType map. The string key is an arbitrary name you give the podType. The value is the configuration.

Since there is only 1 head node per cluster, it doesn’t make sense to have two podTypes named rayHeadType/rayHeadType2 in one values.yaml configuration.

One common pattern for mixed CPU/GPU workloads is to have three podTypes:

  1. Special configuration for a CPU head node.
  2. A CPU worker type.
  3. A GPU worker type (to be scaled up when there are GPU demands)

The configuration for that setup would look something like this:

headPodType: mySpecialHeadType

podTypes:
    mySpecialHeadType:
        CPU: 64
        ...
    myBoringCPUType:
        CPU: 8
        minWorkers: 10
        maxWorkers: 20
        ...
    myExtremelyExpensiveGPUType:
        GPU: 4
        CPU: 128
        minWorkers: 0
        maxWorkers: 2 
        ...

You can read more here about podTypes:
https://docs.ray.io/en/latest/cluster/kubernetes-advanced.html#ray-cluster-configuration

You can read more about how to run multiple Ray clusters here:
https://docs.ray.io/en/latest/cluster/kubernetes-advanced.html#running-multiple-ray-clusters

1 Like
4

Thank you @Dmitri !
I tried this, but realized all podtypes rely on the same image. Usually I want a very specialized image for myExtremlyExpensiveGPUType and another one for myBoringCPUType. I suppose I have to use different Helm charts for different purposes and put one podtype in each.

5

For simplicity, we’ve exposed only one image in values.yaml.
However, it should be straightforward to expose per-type images. You can do this by making a small modification to values.yaml and the chart’s Ray cluster template.

1 Like