How to add Datadog annotation to Ray Pods

Ray Libraries (Data, Train, Tune, Serve)
1

Hi All,

I am trying to integrate Datadog to Ray Serve for the monitoring, In Datadog documentation they are asking to add annotation to ray pods, I have tried adding these annotation at service level

apiVersion: ray.io/v1alpha1
kind: RayService
metadata:
  name: {{ include "rayservice.fullname" . }}
  namespace: {{ .Values.namespace }}
  annotations:
    ray.io/ft-enabled: "true"
    ad.datadoghq.com/ray.checks: |-
      {
        "ray": {
          "instances": [
            {
              "openmetrics_endpoint": "http://%%host%%:8080"
            }
          ]
        }
      }

And also in the head node configuration,

 headGroupSpec:
      # The `rayStartParams` are used to configure the `ray start` command.
      # See https://github.com/ray-project/kuberay/blob/master/docs/guidance/rayStartParams.md for the default settings of `rayStartParams` in KubeRay.
      # See https://docs.ray.io/en/latest/cluster/cli.html#ray-start for all available options in `rayStartParams`.
      rayStartParams:
        dashboard-host: '0.0.0.0'
      #pod template
      template:
        metadata:
          annotations:
            ad.datadoghq.com/ray.checks: |-
              {
                "ray": {
                  "instances": [
                    {
                      "openmetrics_endpoint": "http://%%host%%:8080"
                    }
                  ]
                }
              }
        spec:

I might be missing something, or using incorrect way to add annotation to Ray pods, can some help me here in setting up the annotation in a correct way.

Note: The first method is not showing annotation in pod description, whereas the second method does.

Datadog Integration Documentation: Ray

2

The second method should be the correct way. You may be missing a spec.containers.name field.

spec:
    containers:
        name: ray

Note that you can name this anything. But in order for datadog’s auto discovery to pick it up, you need to ensure the annotation key also matches the naming. See this for the way the annotation and the container name are related.

A snippet from the doc:

apiVersion: v1
kind: Pod
# (...)
metadata:
  name: '<POD_NAME>'
  annotations:
    ad.datadoghq.com/<CONTAINER_IDENTIFIER>.checks: |
      {
        "<INTEGRATION_NAME>": {
          "init_config": <INIT_CONFIG>,
          "instances": [<INSTANCES_CONFIG>]
        }
      }      
    ad.datadoghq.com/<CONTAINER_IDENTIFIER>.logs: '[<LOGS_CONFIG>]'
    # (...)
spec:
  containers:
    - name: '<CONTAINER_IDENTIFIER>'
# (...)
1 Like
3

Thanks @Fmak95 , it worked.