How to ray job submit serve run a model application to use specific worker workgroup?

Ray Core
1

1. Severity of the issue: (select one)
None: I’m just curious or want clarification.
Low: Annoying but doesn’t hinder my work.
Medium: Significantly affects my productivity but can find a workaround.
High: Completely blocks me.

2. Environment:

  • Ray version: 2.44.0
  • Python version: 3.12
  • OS: Ubuntu
  • Cloud/Infrastructure: N/A
  • Other libs/tools (if relevant): N/A

3. What happened vs. what you expected:

  • Expected: Able to specify a worker workgroup in model application
  • Actual: Unable to specify a worker workgroup in model application

helm install raycluster kuberay/ray-cluster --version 1.3.0 -f values.yaml

values.yaml
worker:
groupName: workergroup-1

ray job submit --address=http://localhost:8265
– serve run config.yaml

config.yaml
applications:

  • args:
    llm_configs:
    • model_loading_config:

    how to specify to use workergroup-1 above?

2

Hi James!
Can you see if you can specify some rayStartParams in your values.yaml like this? kuberay/docs/guidance/rayStartParams.md at master · ray-project/kuberay · GitHub and set the groupName/resources to the one you want (workergroup-1)?

There was some more discussion about it here: Kuberay: run jobs in specific worker groups maybe some of the solutions discussed there will be helpful :slight_smile:

Lmk if that helps!
Christina

3

Thank you for your pointers! We had tried looking into the scheduling strategies and labels. We did end up using the same custom resources as pointed.