Hi everyone, we are using kuberay to submit RayJob to K8s cluster. The batchScheduler was set to false which means using the default-scheduler for batch schedulering. Now we want to set up volcano scheduler for our cluster in a graceful way. I noticed two related config about this:
Setting batchScheduler to volcano in helm/values.yaml of kuberay-operator
Setting label ray.io/scheduler-name:volcano in the yaml of RayJob
I found that setting ray.io/scheduler-name to volcano is just useless, the schdulering of RayJob depends on the batchScheduler config of helm for kuberay-operator.
Is that designed in this way? Is that possible for me to control the scheduler of RayJob? Some jobs uses volcano scheduelr in testing period, some uses default-scheduler. Setting ray.io/scheduler-name sounds like a good way but it just doesn’t work.
I also tried:
Set schedulerName: default-scheduler in headGroupSpec and workerGroupSpec with setting batchScheduler to volcano in helm. But the values will be overwritten to volcano when the RayJob submits.
Set schedulerName: volcano in headGroupSpec and workerGroupSpec without setting batchScheduler in helm. But it will try to submit the pod to default queue, because volcano.sh/queue-name: queue will be ignored in this case.
You are correct: as of recent KubeRay versions, the scheduler used for RayJob (and RayCluster) is determined by the KubeRay operator’s batchScheduler configuration (e.g., batchScheduler.name=volcano in Helm), not by the ray.io/scheduler-name label in the RayJob YAML. The label is no longer sufficient to select the scheduler; it is now just metadata and does not control scheduling behavior. This is by design after the adoption of the new scheduler framework in KubeRay, and the label is ignored for actual scheduling decisions. The operator-level setting applies globally to all RayJobs managed by that operator instance, so you cannot mix schedulers per-job using only the label or pod spec fields—setting schedulerName in pod templates is also overwritten by the operator when batch scheduling is enabled. See the discussion and documentation in KubeRay & Volcano integration and official docs.
Currently, there is no supported way to control the scheduler on a per-RayJob basis; all jobs managed by a given KubeRay operator will use the scheduler specified in the operator’s configuration. If you need to run some jobs with Volcano and others with the default scheduler, you must deploy separate KubeRay operator instances, each configured with the desired scheduler, and submit jobs to the appropriate operator. This limitation and the deprecation of the label-based approach are documented in the KubeRay-Volcano integration guide and related PRs.
Would you like more detail on how to set up multiple KubeRay operators for different schedulers?
To run multiple KubeRay operators with different schedulers (e.g., one with Volcano, one with the default scheduler) in the same Kubernetes cluster, you must deploy each operator in a separate namespace and configure each with its own batchScheduler.name (e.g., volcano or leave unset/false for default). Each operator will then manage RayJobs/RayClusters only in its namespace, allowing you to control the scheduler per namespace. This approach is implied in the KubeRay & Volcano integration PR and official docs.
You cannot mix schedulers within a single operator instance; the scheduler is set at the operator level. Deploying operators in different namespaces, each with its own configuration, is the supported way to achieve per-job scheduler selection.
Would you like a step-by-step example of how to deploy and use multiple operators?