Resources Labels from Flyte's PodTemplate Not Propagating to Ray-Managed Kubernetes Job

Ray Clusters Kubernetes
1

Description:

I’m using Flyte to manage workflows that include Ray tasks running on a Kubernetes cluster. When setting labels in a Flyte PodTemplate, I’ve noticed that these labels are not being applied to the Kubernetes Job resources managed by Ray. The labels do appear on the Pods but not on the Jobs.

Steps to Reproduce:

  1. Define labels in a Flyte PodTemplate.
  2. Launch a Ray task/job in Kubernetes through Flyte, utilizing the above PodTemplate.
  3. Examine the Kubernetes Job resources managed by Ray and observe that the labels from the PodTemplate are missing.

Expected Behavior:

I expect that when I set labels in Flyte’s PodTemplate, they should propagate to both the Pods and the Kubernetes Job resources that Ray manages.

Actual Behavior:

The labels set in Flyte’s PodTemplate are being applied to Pods but not to the Ray-managed Kubernetes Job resources.

Questions:

  1. Is this behaviour known or expected within the Ray community?
  2. Can this issue be addressed by Ray, or is this a limitation when using Flyte and Ray together?

Additional Information:

  • Flyte version: v1.9.2
  • KubeRay Helm Chart version: v0.6.0

Tags: Ray, Kubernetes, Flyte, PodTemplate, Labels