# Kuberay sample RayService not launching serve apps

**URL:** <https://discuss.ray.io/t/kuberay-sample-rayservice-not-launching-serve-apps/14461>\
**Category:** Ray Serve\
**Created:** [April 19, 2024, 10:02pm UTC](https://discuss.ray.io/t/kuberay-sample-rayservice-not-launching-serve-apps/14461 "2024-04-19T22:02:01Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![shaikhismail](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/shaikhismail/32/5853_2.png) [@shaikhismail](https://discuss.ray.io/u/shaikhismail)\
**Post date:** [April 19, 2024, 10:02pm UTC](https://discuss.ray.io/t/kuberay-sample-rayservice-not-launching-serve-apps/14461/1 "2024-04-19T22:02:01Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- High: It blocks me to complete my task.

1. Installed Kuberay

```auto
helm repo add kuberay https://ray-project.github.io/kuberay-helm/
helm repo update
helm install kuberay-operator kuberay/kuberay-operator --version 1.1.0

```

1. Slightly updated `rayservice-sample.yaml` file (pasted later on) and packaging application code as a docker image instead of runtime env.

Dockerfile:

```auto
# File name: Dockerfile
FROM rayproject/ray:2.11.0

WORKDIR /app

COPY ./ray-test-app ./

```

1. `kubectl describe rayservice rayservice-sample`:

```auto
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal WaitForServeDeploymentReady 26m (x129 over 36m) rayservice-controller Fail to create / update Serve applications. If you observe this error consistently, please check "Issue 5: Fail to create / update Serve applications." in https://docs.ray.io/en/master/cluster/kubernetes/troubleshooting/rayservice-troubleshooting.html#kuberay-raysvc-troubleshoot for more details. err: Put "http://rayservice-sample-raycluster-7pwbq-head-svc.rubrik-spark.svc.cluster.local:8265/api/serve/applications/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

```

**Questions**

Q1. [RayService Quickstart — Ray 2.11.0](https://docs.ray.io/en/latest/cluster/kubernetes/getting-started/rayservice-quick-start.html#step-4-verify-the-kubernetes-cluster-status) says that:

```auto
When the Ray Serve applications are healthy and ready, KubeRay creates a head service and a Ray Serve service for the RayService custom resource. For example, rayservice-sample-head-svc and rayservice-sample-serve-svc

```

Q1. Can someone help me with why the “Ray Serve applications are healthy and ready” condition not true? Why are there no actors getting launched?

Q2. The troubleshooting docs at [RayService troubleshooting — Ray 3.0.0.dev0](https://docs.ray.io/en/master/cluster/kubernetes/troubleshooting/rayservice-troubleshooting.html#issue-5-fail-to-create-update-serve-applications) does not list the issue I’m seeing, can we add this issue over that as well?

Q3. I do not see the `/tmp/ray/session_latest/logs/serve/` in my head pod?

**Other logs/artifacts**

1. Full `rayservice-sample.yaml`:

```auto
# Make sure to increase resource requests and limits before using this example in production.
# For examples with more realistic resource configuration, see
# ray-cluster.complete.large.yaml and
# ray-cluster.autoscaler.large.yaml.
apiVersion: ray.io/v1
kind: RayService
metadata:
  name: rayservice-sample
spec:
  # serveConfigV2 takes a yaml multi-line scalar, which should be a Ray Serve multi-application config. See https://docs.ray.io/en/latest/serve/multi-app.html.
  serveConfigV2: |
    applications:
      - name: math_app
        import_path: conditional_dag.serve_dag
        route_prefix: /calc
        deployments:
          - name: Adder
            user_config:
              increment: 3
            max_ongoing_requests: 2
            autoscaling_config:
              target_ongoing_requests: 1
              metrics_interval_s: 0.2
              min_replicas: 0
              initial_replicas: 0
              max_replicas: 100
              look_back_period_s: 2
              downscale_delay_s: 10
              upscale_delay_s: 0
            graceful_shutdown_timeout_s: 5
            ray_actor_options:
              num_cpus: 0.5
          - name: Multiplier
            user_config:
              factor: 5
            max_ongoing_requests: 2
            autoscaling_config:
              target_ongoing_requests: 1
              metrics_interval_s: 0.2
              min_replicas: 0
              initial_replicas: 0
              max_replicas: 100
              look_back_period_s: 2
              downscale_delay_s: 10
              upscale_delay_s: 0
            graceful_shutdown_timeout_s: 5
            ray_actor_options:
              num_cpus: 1.5
          - name: Router
            max_ongoing_requests: 2
            autoscaling_config:
              target_ongoing_requests: 1
              metrics_interval_s: 0.2
              min_replicas: 1
              max_replicas: 100
              look_back_period_s: 2
              downscale_delay_s: 10
              upscale_delay_s: 0
            graceful_shutdown_timeout_s: 5
            ray_actor_options:
                  num_cpus: 1
  rayClusterConfig:
    rayVersion: '2.11.0' # should match the Ray version in the image of the containers
# enableInTreeAutoscaling: true
    ######################headGroupSpecs#################################
    # Ray head pod template.
    headGroupSpec:
      # The `rayStartParams` are used to configure the `ray start` command.
      # See https://github.com/ray-project/kuberay/blob/master/docs/guidance/rayStartParams.md for the default settings of `rayStartParams` in KubeRay.
      # See https://docs.ray.io/en/latest/cluster/cli.html#ray-start for all available options in `rayStartParams`.
      rayStartParams:
        dashboard-host: '0.0.0.0'
      #pod template
      template:
        spec:
          containers:
            - name: ray-head
              image: gcr.io/spark-dev-083/ray/sample-app:autoscaler
              resources:
                limits:
                  cpu: 2
                  memory: 2Gi
                requests:
                  cpu: 2
                  memory: 2Gi
              ports:
                - containerPort: 6379
                  name: gcs-server
                - containerPort: 8265 # Ray dashboard
                  name: dashboard
                - containerPort: 10001
                  name: client
                - containerPort: 8000
                  name: serve
    workerGroupSpecs:
      # the pod replicas in this group typed worker
      - replicas: 1
        minReplicas: 1
        maxReplicas: 50
        # logical group name, for this called small-group, also can be functional
        groupName: small-group
        # The `rayStartParams` are used to configure the `ray start` command.
        # See https://github.com/ray-project/kuberay/blob/master/docs/guidance/rayStartParams.md for the default settings of `rayStartParams` in KubeRay.
        # See https://docs.ray.io/en/latest/cluster/cli.html#ray-start for all available options in `rayStartParams`.
        rayStartParams: {}
        #pod template
        template:
          spec:
            containers:
              - name: ray-worker # must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name', or '123-abc'
                image: gcr.io/spark-dev-083/ray/sample-app:autoscaler
                lifecycle:
                  preStop:
                    exec:
                      command: ["/bin/sh","-c","ray stop"]
                resources:
                  limits:
                    cpu: 3
                    memory: "2Gi"
                  requests:
                    cpu: 3
                    memory: "2Gi"

```

1. Full output of `kubectl describe rayservice rayservice-sample`:

```auto
kubectl describe rayservice rayservice-sample
Name: rayservice-sample
Namespace: rubrik-spark
Labels: <none>
Annotations: <none>
API Version: ray.io/v1
Kind: RayService
Metadata:
  Creation Timestamp: 2024-04-19T21:14:09Z
  Generation: 3
  Managed Fields:
    API Version: ray.io/v1
    Fields Type: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:rayClusterConfig:
          .:
          f:headGroupSpec:
            .:
            f:rayStartParams:
              .:
              f:dashboard-host:
            f:template:
              .:
              f:spec:
                .:
                f:containers:
          f:rayVersion:
          f:workerGroupSpecs:
        f:serveConfigV2:
    Manager: kubectl-client-side-apply
    Operation: Update
    Time: 2024-04-19T21:27:20Z
    API Version: ray.io/v1
    Fields Type: FieldsV1
    fieldsV1:
      f:status:
        .:
        f:activeServiceStatus:
          .:
          f:rayClusterStatus:
            .:
            f:desiredCPU:
            f:desiredGPU:
            f:desiredMemory:
            f:desiredTPU:
            f:desiredWorkerReplicas:
            f:endpoints:
              .:
              f:client:
              f:dashboard:
              f:gcs-server:
              f:metrics:
              f:serve:
            f:head:
              .:
              f:podIP:
              f:serviceIP:
            f:lastUpdateTime:
            f:maxWorkerReplicas:
            f:minWorkerReplicas:
            f:observedGeneration:
        f:observedGeneration:
        f:pendingServiceStatus:
          .:
          f:rayClusterName:
          f:rayClusterStatus:
            .:
            f:desiredCPU:
            f:desiredGPU:
            f:desiredMemory:
            f:desiredTPU:
            f:head:
        f:serviceStatus:
    Manager: kuberay-operator
    Operation: Update
    Subresource: status
    Time: 2024-04-19T21:27:25Z
  Resource Version: 556781297
  UID: 07255e8a-788b-49d8-8a66-04ce7f996caa
Spec:
  Ray Cluster Config:
    Head Group Spec:
      Ray Start Params:
        Dashboard - Host: 0.0.0.0
      Template:
        Spec:
          Containers:
            Image: gcr.io/spark-dev-083/ray/sample-app:autoscaler
            Name: ray-head
            Ports:
              Container Port: 6379
              Name: gcs-server
              Protocol: TCP
              Container Port: 8265
              Name: dashboard
              Protocol: TCP
              Container Port: 10001
              Name: client
              Protocol: TCP
              Container Port: 8000
              Name: serve
              Protocol: TCP
            Resources:
              Limits:
                Cpu: 2
                Memory: 2Gi
              Requests:
                Cpu: 2
                Memory: 2Gi
    Ray Version: 2.11.0
    Worker Group Specs:
      Group Name: small-group
      Max Replicas: 50
      Min Replicas: 1
      Num Of Hosts: 1
      Ray Start Params:
      Replicas: 1
      Template:
        Spec:
          Containers:
            Image: gcr.io/spark-dev-083/ray/sample-app:autoscaler
            Lifecycle:
              Pre Stop:
                Exec:
                  Command:
                    /bin/sh
                    -c
                    ray stop
            Name: ray-worker
            Resources:
              Limits:
                Cpu: 3
                Memory: 2Gi
              Requests:
                Cpu: 3
                Memory: 2Gi
  serveConfigV2: applications:
  - name: math_app
    import_path: conditional_dag.serve_dag
    route_prefix: /calc
    deployments:
      - name: Adder
        user_config:
          increment: 3
        max_ongoing_requests: 2
        autoscaling_config:
          target_ongoing_requests: 1
          metrics_interval_s: 0.2
          min_replicas: 0
          initial_replicas: 0
          max_replicas: 100
          look_back_period_s: 2
          downscale_delay_s: 10
          upscale_delay_s: 0
        graceful_shutdown_timeout_s: 5
        ray_actor_options:
          num_cpus: 0.5
      - name: Multiplier
        user_config:
          factor: 5
        max_ongoing_requests: 2
        autoscaling_config:
          target_ongoing_requests: 1
          metrics_interval_s: 0.2
          min_replicas: 0
          initial_replicas: 0
          max_replicas: 100
          look_back_period_s: 2
          downscale_delay_s: 10
          upscale_delay_s: 0
        graceful_shutdown_timeout_s: 5
        ray_actor_options:
          num_cpus: 1.5
      - name: Router
        max_ongoing_requests: 2
        autoscaling_config:
          target_ongoing_requests: 1
          metrics_interval_s: 0.2
          min_replicas: 1
          max_replicas: 100
          look_back_period_s: 2
          downscale_delay_s: 10
          upscale_delay_s: 0
        graceful_shutdown_timeout_s: 5
        ray_actor_options:
              num_cpus: 1

Status:
  Active Service Status:
    Ray Cluster Status:
      Desired CPU: 5
      Desired GPU: 0
      Desired Memory: 4Gi
      Desired TPU: 0
      Desired Worker Replicas: 1
      Endpoints:
        Client: 10001
        Dashboard: 8265
        Gcs - Server: 6379
        Metrics: 8080
        Serve: 8000
      Head:
        Pod IP: 10.88.22.45
        Service IP: 10.92.12.75
      Last Update Time: 2024-04-19T21:27:21Z
      Max Worker Replicas: 50
      Min Worker Replicas: 1
      Observed Generation: 1
  Observed Generation: 3
  Pending Service Status:
    Ray Cluster Name: rayservice-sample-raycluster-7pwbq
    Ray Cluster Status:
      Desired CPU: 0
      Desired GPU: 0
      Desired Memory: 0
      Desired TPU: 0
      Head:
  Service Status: WaitForServeDeploymentReady
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal WaitForServeDeploymentReady 26m (x129 over 36m) rayservice-controller Fail to create / update Serve applications. If you observe this error consistently, please check "Issue 5: Fail to create / update Serve applications." in https://docs.ray.io/en/master/cluster/kubernetes/troubleshooting/rayservice-troubleshooting.html#kuberay-raysvc-troubleshoot for more details. err: Put "http://rayservice-sample-raycluster-7pwbq-head-svc.rubrik-spark.svc.cluster.local:8265/api/serve/applications/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

```

---

<div class="post-metadata">

**Author:** ![shaikhismail](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/shaikhismail/32/5853_2.png) [@shaikhismail](https://discuss.ray.io/u/shaikhismail)\
**Post date:** [April 20, 2024, 2:26am UTC](https://discuss.ray.io/t/kuberay-sample-rayservice-not-launching-serve-apps/14461/2 "2024-04-20T02:26:55Z")

</div>

1. Kuberay operator logs: `kubectl logs $KUBERAY_OPERATOR_POD -n $YOUR_NAMESPACE | tee operator-log`

```auto
{"level":"info","ts":"2024-04-20T01:11:24.753Z","logger":"setup","msg":"Flag watchNamespace is not set. Watch custom resources in all namespaces."}
{"level":"info","ts":"2024-04-20T01:11:24.753Z","logger":"setup","msg":"Setup manager"}
{"level":"info","ts":"2024-04-20T01:11:24.861Z","logger":"setup","msg":"starting manager"}
{"level":"info","ts":"2024-04-20T01:11:24.861Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"}
{"level":"info","ts":"2024-04-20T01:11:24.861Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":":8080","secure":false}
{"level":"info","ts":"2024-04-20T01:11:24.861Z","msg":"starting server","kind":"health probe","addr":"[::]:8082"}
I0420 01:11:27.650475 1 leaderelection.go:250] attempting to acquire leader lease rubrik-spark/ray-operator-leader...
I0420 01:11:43.072466 1 leaderelection.go:260] successfully acquired lease rubrik-spark/ray-operator-leader
{"level":"info","ts":"2024-04-20T01:11:43.073Z","logger":"controllers.RayCluster","msg":"Starting EventSource","source":"kind source: *v1.RayCluster"}
{"level":"info","ts":"2024-04-20T01:11:43.073Z","logger":"controllers.RayCluster","msg":"Starting EventSource","source":"kind source: *v1.Pod"}
{"level":"info","ts":"2024-04-20T01:11:43.073Z","logger":"controllers.RayCluster","msg":"Starting EventSource","source":"kind source: *v1.Service"}
{"level":"info","ts":"2024-04-20T01:11:43.073Z","logger":"controllers.RayCluster","msg":"Starting Controller"}
{"level":"info","ts":"2024-04-20T01:11:43.073Z","logger":"controllers.RayJob","msg":"Starting EventSource","source":"kind source: *v1.RayJob"}
{"level":"info","ts":"2024-04-20T01:11:43.074Z","logger":"controllers.RayJob","msg":"Starting EventSource","source":"kind source: *v1.RayCluster"}
{"level":"info","ts":"2024-04-20T01:11:43.074Z","logger":"controllers.RayJob","msg":"Starting EventSource","source":"kind source: *v1.Service"}
{"level":"info","ts":"2024-04-20T01:11:43.074Z","logger":"controllers.RayJob","msg":"Starting EventSource","source":"kind source: *v1.Job"}
{"level":"info","ts":"2024-04-20T01:11:43.074Z","logger":"controllers.RayJob","msg":"Starting Controller"}
{"level":"info","ts":"2024-04-20T01:11:43.074Z","logger":"controllers.RayService","msg":"Starting EventSource","source":"kind source: *v1.RayService"}
{"level":"info","ts":"2024-04-20T01:11:43.074Z","logger":"controllers.RayService","msg":"Starting EventSource","source":"kind source: *v1.RayCluster"}
{"level":"info","ts":"2024-04-20T01:11:43.074Z","logger":"controllers.RayService","msg":"Starting EventSource","source":"kind source: *v1.Service"}
{"level":"info","ts":"2024-04-20T01:11:43.074Z","logger":"controllers.RayService","msg":"Starting EventSource","source":"kind source: *v1.Ingress"}
{"level":"info","ts":"2024-04-20T01:11:43.074Z","logger":"controllers.RayService","msg":"Starting Controller"}
{"level":"info","ts":"2024-04-20T01:11:43.975Z","logger":"controllers.RayCluster","msg":"Starting workers","worker count":1}
{"level":"info","ts":"2024-04-20T01:11:43.980Z","logger":"controllers.RayService","msg":"Starting workers","worker count":1}
{"level":"info","ts":"2024-04-20T01:11:44.051Z","logger":"controllers.RayJob","msg":"Starting workers","worker count":1}
{"level":"info","ts":"2024-04-20T01:14:13.968Z","logger":"controllers.RayService","msg":"No active Ray cluster. RayService operator should prepare a new Ray cluster.","RayService":{"name":"rayservice-sample","namespace":"rubrik-spark"},"reconcileID":"a7afdd7c-47d2-4012-be34-18a62df2936f"}
{"level":"info","ts":"2024-04-20T01:14:13.968Z","logger":"controllers.RayService","msg":"Current cluster is unhealthy, prepare to restart.","RayService":{"name":"rayservice-sample","namespace":"rubrik-spark"},"reconcileID":"a7afdd7c-47d2-4012-be34-18a62df2936f","Status":{"activeServiceStatus":{"rayClusterStatus":{"desiredCPU":"0","desiredMemory":"0","desiredGPU":"0","desiredTPU":"0","head":{}}},"pendingServiceStatus":{"rayClusterStatus":{"desiredCPU":"0","desiredMemory":"0","desiredGPU":"0","desiredTPU":"0","head":{}}},"observedGeneration":1}}
{"level":"info","ts":"2024-04-20T01:14:13.986Z","logger":"KubeAPIWarningLogger","msg":"unknown field \"spec.rayClusterConfig.headGroupSpec.template.metadata.creationTimestamp\""}
{"level":"info","ts":"2024-04-20T01:14:13.986Z","logger":"KubeAPIWarningLogger","msg":"unknown field \"spec.rayClusterConfig.workerGroupSpecs[0].template.metadata.creationTimestamp\""}
{"level":"info","ts":"2024-04-20T01:14:14.050Z","logger":"controllers.RayService","msg":"Done reconcileRayCluster update status, enter next loop to create new ray cluster.","RayService":{"name":"rayservice-sample","namespace":"rubrik-spark"},"reconcileID":"a7afdd7c-47d2-4012-be34-18a62df2936f"}
{"level":"info","ts":"2024-04-20T01:14:16.051Z","logger":"controllers.RayService","msg":"Creating a new pending RayCluster instance.","RayService":{"name":"rayservice-sample","namespace":"rubrik-spark"},"reconcileID":"afbab1ea-067b-4d31-8b07-553026007b3a"}
{"level":"info","ts":"2024-04-20T01:14:16.051Z","logger":"controllers.RayService","msg":"createRayClusterInstance","RayService":{"name":"rayservice-sample","namespace":"rubrik-spark"},"reconcileID":"afbab1ea-067b-4d31-8b07-553026007b3a","rayClusterInstanceName":"rayservice-sample-raycluster-9s5d7"}
{"level":"info","ts":"2024-04-20T01:14:16.051Z","logger":"controllers.RayService","msg":"No pending RayCluster, creating RayCluster.","RayService":{"name":"rayservice-sample","namespace":"rubrik-spark"},"reconcileID":"afbab1ea-067b-4d31-8b07-553026007b3a"}
{"level":"info","ts":"2024-04-20T01:14:16.063Z","logger":"KubeAPIWarningLogger","msg":"unknown field \"spec.headGroupSpec.template.metadata.creationTimestamp\""}
{"level":"info","ts":"2024-04-20T01:14:16.063Z","logger":"KubeAPIWarningLogger","msg":"unknown field \"spec.workerGroupSpecs[0].template.metadata.creationTimestamp\""}
{"level":"info","ts":"2024-04-20T01:14:16.064Z","logger":"controllers.RayService","msg":"created rayCluster for rayService","RayService":{"name":"rayservice-sample","namespace":"rubrik-spark"},"reconcileID":"afbab1ea-067b-4d31-8b07-553026007b3a","rayCluster":{"namespace":"rubrik-spark","name":"rayservice-sample-raycluster-9s5d7"}}
{"level":"info","ts":"2024-04-20T01:14:16.064Z","logger":"controllers.RayService","msg":"Check the head Pod status of the pending RayCluster","RayService":{"name":"rayservice-sample","namespace":"rubrik-spark"},"reconcileID":"afbab1ea-067b-4d31-8b07-553026007b3a","RayCluster name":"rayservice-sample-raycluster-9s5d7"}
{"level":"info","ts":"2024-04-20T01:14:16.064Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"07f82f27-a29c-41fc-a2eb-a5ce955d4265"}
{"level":"error","ts":"2024-04-20T01:14:16.064Z","logger":"controllers.RayService","msg":"Failed to check if head Pod is running and ready!","RayService":{"name":"rayservice-sample","namespace":"rubrik-spark"},"reconcileID":"afbab1ea-067b-4d31-8b07-553026007b3a","error":"Found 0 head pods for RayCluster rayservice-sample-raycluster-9s5d7 in the namespace rubrik-spark","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayServiceReconciler).reconcileServe\n\t/home/runner/work/kuberay/kuberay/ray-operator/controllers/ray/rayservice_controller.go:1069\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayServiceReconciler).Reconcile\n\t/home/runner/work/kuberay/kuberay/ray-operator/controllers/ray/rayservice_controller.go:168\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-04-20T01:14:16.064Z","logger":"controllers.RayService","msg":"Fail to reconcileServe.","RayService":{"name":"rayservice-sample","namespace":"rubrik-spark"},"reconcileID":"afbab1ea-067b-4d31-8b07-553026007b3a","error":"Found 0 head pods for RayCluster rayservice-sample-raycluster-9s5d7 in the namespace rubrik-spark","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayServiceReconciler).Reconcile\n\t/home/runner/work/kuberay/kuberay/ray-operator/controllers/ray/rayservice_controller.go:169\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-04-20T01:14:16.065Z","logger":"controllers.RayService","msg":"Check the head Pod status of the pending RayCluster","RayService":{"name":"rayservice-sample","namespace":"rubrik-spark"},"reconcileID":"3354d6f3-9a15-4a87-b56b-9eb7c809f7f1","RayCluster name":"rayservice-sample-raycluster-9s5d7"}
{"level":"error","ts":"2024-04-20T01:14:16.066Z","logger":"controllers.RayService","msg":"Failed to check if head Pod is running and ready!","RayService":{"name":"rayservice-sample","namespace":"rubrik-spark"},"reconcileID":"3354d6f3-9a15-4a87-b56b-9eb7c809f7f1","error":"Found 0 head pods for RayCluster rayservice-sample-raycluster-9s5d7 in the namespace rubrik-spark","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayServiceReconciler).reconcileServe\n\t/home/runner/work/kuberay/kuberay/ray-operator/controllers/ray/rayservice_controller.go:1069\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayServiceReconciler).Reconcile\n\t/home/runner/work/kuberay/kuberay/ray-operator/controllers/ray/rayservice_controller.go:168\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-04-20T01:14:16.066Z","logger":"controllers.RayService","msg":"Fail to reconcileServe.","RayService":{"name":"rayservice-sample","namespace":"rubrik-spark"},"reconcileID":"3354d6f3-9a15-4a87-b56b-9eb7c809f7f1","error":"Found 0 head pods for RayCluster rayservice-sample-raycluster-9s5d7 in the namespace rubrik-spark","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayServiceReconciler).Reconcile\n\t/home/runner/work/kuberay/kuberay/ray-operator/controllers/ray/rayservice_controller.go:169\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-04-20T01:14:16.106Z","logger":"controllers.RayCluster","msg":"Pod Service created successfully","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"07f82f27-a29c-41fc-a2eb-a5ce955d4265","service name":"rayservice-sample-raycluster-9s5d7-head-svc"}
{"level":"info","ts":"2024-04-20T01:14:16.107Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"07f82f27-a29c-41fc-a2eb-a5ce955d4265","Found 0 head Pods; creating a head Pod for the RayCluster.":"rayservice-sample-raycluster-9s5d7"}
{"level":"info","ts":"2024-04-20T01:14:16.107Z","logger":"controllers.RayCluster","msg":"head pod labels","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"07f82f27-a29c-41fc-a2eb-a5ce955d4265","labels":{"app.kubernetes.io/created-by":"kuberay-operator","app.kubernetes.io/name":"kuberay","ray.io/cluster":"rayservice-sample-raycluster-9s5d7","ray.io/group":"headgroup","ray.io/identifier":"rayservice-sample-raycluster-9s5d7-head","ray.io/is-ray-node":"yes","ray.io/node-type":"head"}}
{"level":"info","ts":"2024-04-20T01:14:16.107Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"07f82f27-a29c-41fc-a2eb-a5ce955d4265","nodeType":"head","rayStartParams":{"block":"true","dashboard-agent-listen-port":"52365","dashboard-host":"0.0.0.0","metrics-export-port":"8080"},"Ray container resource":{"limits":{"cpu":"2","memory":"2Gi"},"requests":{"cpu":"2","memory":"2Gi"}}}
{"level":"info","ts":"2024-04-20T01:14:16.107Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"07f82f27-a29c-41fc-a2eb-a5ce955d4265","rayStartCmd":"ray start --head --dashboard-agent-listen-port=52365 --num-cpus=2 --memory=2147483648 --dashboard-host=0.0.0.0 --metrics-export-port=8080 --block "}
{"level":"info","ts":"2024-04-20T01:14:16.107Z","logger":"controllers.RayCluster","msg":"BuildPod","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"07f82f27-a29c-41fc-a2eb-a5ce955d4265","rayNodeType":"head","generatedCmd":"ulimit -n 65536; ray start --head --dashboard-agent-listen-port=52365 --num-cpus=2 --memory=2147483648 --dashboard-host=0.0.0.0 --metrics-export-port=8080 --block "}
{"level":"info","ts":"2024-04-20T01:14:16.107Z","logger":"controllers.RayCluster","msg":"Probes injection feature flag","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"07f82f27-a29c-41fc-a2eb-a5ce955d4265","enabled":true}
{"level":"info","ts":"2024-04-20T01:14:16.107Z","logger":"controllers.RayCluster","msg":"createHeadPod","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"07f82f27-a29c-41fc-a2eb-a5ce955d4265","head pod with name":"rayservice-sample-raycluster-9s5d7-head-"}
{"level":"info","ts":"2024-04-20T01:14:16.150Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"07f82f27-a29c-41fc-a2eb-a5ce955d4265","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":50,"minReplicas":1,"replicas":1}
{"level":"info","ts":"2024-04-20T01:14:16.151Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"07f82f27-a29c-41fc-a2eb-a5ce955d4265","removing the pods in the scaleStrategy of":"small-group"}
{"level":"info","ts":"2024-04-20T01:14:16.151Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"07f82f27-a29c-41fc-a2eb-a5ce955d4265","workerReplicas":1,"runningPods":0,"diff":1}
{"level":"info","ts":"2024-04-20T01:14:16.151Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"07f82f27-a29c-41fc-a2eb-a5ce955d4265","Number workers to add":1,"Worker group":"small-group"}
{"level":"info","ts":"2024-04-20T01:14:16.151Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"07f82f27-a29c-41fc-a2eb-a5ce955d4265","creating worker for group":"small-group","index 0":"in total 1"}
{"level":"info","ts":"2024-04-20T01:14:16.151Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"07f82f27-a29c-41fc-a2eb-a5ce955d4265","nodeType":"worker","rayStartParams":{"address":"rayservice-sample-raycluster-9s5d7-head-svc.rubrik-spark.svc.cluster.local:6379","block":"true","dashboard-agent-listen-port":"52365","metrics-export-port":"8080"},"Ray container resource":{"limits":{"cpu":"3","memory":"2Gi"},"requests":{"cpu":"3","memory":"2Gi"}}}
{"level":"info","ts":"2024-04-20T01:14:16.151Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"07f82f27-a29c-41fc-a2eb-a5ce955d4265","rayStartCmd":"ray start --address=rayservice-sample-raycluster-9s5d7-head-svc.rubrik-spark.svc.cluster.local:6379 --metrics-export-port=8080 --block --dashboard-agent-listen-port=52365 --num-cpus=3 --memory=2147483648 "}
{"level":"info","ts":"2024-04-20T01:14:16.151Z","logger":"controllers.RayCluster","msg":"BuildPod","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"07f82f27-a29c-41fc-a2eb-a5ce955d4265","rayNodeType":"worker","generatedCmd":"ulimit -n 65536; ray start --address=rayservice-sample-raycluster-9s5d7-head-svc.rubrik-spark.svc.cluster.local:6379 --metrics-export-port=8080 --block --dashboard-agent-listen-port=52365 --num-cpus=3 --memory=2147483648 "}
{"level":"info","ts":"2024-04-20T01:14:16.151Z","logger":"controllers.RayCluster","msg":"Probes injection feature flag","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"07f82f27-a29c-41fc-a2eb-a5ce955d4265","enabled":true}
{"level":"info","ts":"2024-04-20T01:14:16.174Z","logger":"controllers.RayCluster","msg":"Created pod","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"07f82f27-a29c-41fc-a2eb-a5ce955d4265","Pod ":"ervice-sample-raycluster-9s5d7-worker-small-group-"}
{"level":"info","ts":"2024-04-20T01:14:16.175Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running; Pod Name: rayservice-sample-raycluster-9s5d7-head-z5st7; Pod Status.Phase: Pending","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"07f82f27-a29c-41fc-a2eb-a5ce955d4265"}
{"level":"info","ts":"2024-04-20T01:14:16.176Z","logger":"controllers.RayCluster","msg":"inconsistentRayClusterStatus","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"07f82f27-a29c-41fc-a2eb-a5ce955d4265","detect inconsistency":"old AvailableWorkerReplicas: 0, new AvailableWorkerReplicas: 0, old DesiredWorkerReplicas: 0, new DesiredWorkerReplicas: 1, old MinWorkerReplicas: 0, new MinWorkerReplicas: 1, old MaxWorkerReplicas: 0, new MaxWorkerReplicas: 50"}
{"level":"info","ts":"2024-04-20T01:14:16.176Z","logger":"controllers.RayCluster","msg":"rayClusterReconcile","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"07f82f27-a29c-41fc-a2eb-a5ce955d4265","Update CR status":"rayservice-sample-raycluster-9s5d7","status":{"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":50,"desiredCPU":"5","desiredMemory":"4Gi","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2024-04-20T01:14:16Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"serviceIP":"10.92.15.52"},"observedGeneration":1}}
{"level":"info","ts":"2024-04-20T01:14:16.251Z","logger":"controllers.RayCluster","msg":"Environment variable RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV is not set, using default value of 300 seconds","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"07f82f27-a29c-41fc-a2eb-a5ce955d4265","cluster name":"rayservice-sample-raycluster-9s5d7"}
{"level":"info","ts":"2024-04-20T01:14:16.251Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"07f82f27-a29c-41fc-a2eb-a5ce955d4265","cluster name":"rayservice-sample-raycluster-9s5d7","seconds":300}
{"level":"info","ts":"2024-04-20T01:14:16.251Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"44413d62-4bfb-4ea0-81cb-492cc75b00fc"}
{"level":"info","ts":"2024-04-20T01:14:16.251Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"44413d62-4bfb-4ea0-81cb-492cc75b00fc","1 head service found":"rayservice-sample-raycluster-9s5d7-head-svc"}
{"level":"info","ts":"2024-04-20T01:14:16.252Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"44413d62-4bfb-4ea0-81cb-492cc75b00fc","Found 1 head Pod":"rayservice-sample-raycluster-9s5d7-head-z5st7","Pod status":"Pending","Pod restart policy":"Always","Ray container terminated status":"nil"}
{"level":"info","ts":"2024-04-20T01:14:16.252Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"44413d62-4bfb-4ea0-81cb-492cc75b00fc","head Pod":"rayservice-sample-raycluster-9s5d7-head-z5st7","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod rayservice-sample-raycluster-9s5d7-head-z5st7. The Pod status is Pending, and the Ray container terminated status is nil."}
{"level":"info","ts":"2024-04-20T01:14:16.252Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"44413d62-4bfb-4ea0-81cb-492cc75b00fc","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":50,"minReplicas":1,"replicas":1}
{"level":"info","ts":"2024-04-20T01:14:16.252Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"44413d62-4bfb-4ea0-81cb-492cc75b00fc","worker Pod":"ervice-sample-raycluster-9s5d7-worker-small-group-5zqcc","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod ervice-sample-raycluster-9s5d7-worker-small-group-5zqcc. The Pod status is Pending, and the Ray container terminated status is nil."}
{"level":"info","ts":"2024-04-20T01:14:16.252Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"44413d62-4bfb-4ea0-81cb-492cc75b00fc","removing the pods in the scaleStrategy of":"small-group"}
{"level":"info","ts":"2024-04-20T01:14:16.252Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"44413d62-4bfb-4ea0-81cb-492cc75b00fc","workerReplicas":1,"runningPods":1,"diff":0}
{"level":"info","ts":"2024-04-20T01:14:16.252Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"44413d62-4bfb-4ea0-81cb-492cc75b00fc","all workers already exist for group":"small-group"}
{"level":"info","ts":"2024-04-20T01:14:16.252Z","logger":"controllers.RayService","msg":"Check the head Pod status of the pending RayCluster","RayService":{"name":"rayservice-sample","namespace":"rubrik-spark"},"reconcileID":"5935e12d-dfe7-4c36-9bf7-56bc5a3858a7","RayCluster name":"rayservice-sample-raycluster-9s5d7"}
{"level":"info","ts":"2024-04-20T01:14:16.253Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running; Pod Name: rayservice-sample-raycluster-9s5d7-head-z5st7; Pod Status.Phase: Pending","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"44413d62-4bfb-4ea0-81cb-492cc75b00fc"}
{"level":"info","ts":"2024-04-20T01:14:16.253Z","logger":"controllers.RayService","msg":"Skipping the update of Serve deployments because the Ray head Pod is not ready.","RayService":{"name":"rayservice-sample","namespace":"rubrik-spark"},"reconcileID":"5935e12d-dfe7-4c36-9bf7-56bc5a3858a7"}
{"level":"info","ts":"2024-04-20T01:14:16.253Z","logger":"controllers.RayService","msg":"Ray Serve applications are not ready to serve requests: checking again in 2ss","RayService":{"name":"rayservice-sample","namespace":"rubrik-spark"},"reconcileID":"5935e12d-dfe7-4c36-9bf7-56bc5a3858a7"}
{"level":"info","ts":"2024-04-20T01:14:16.254Z","logger":"controllers.RayCluster","msg":"inconsistentRayClusterStatus","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"44413d62-4bfb-4ea0-81cb-492cc75b00fc","detect inconsistency":"old AvailableWorkerReplicas: 0, new AvailableWorkerReplicas: 0, old DesiredWorkerReplicas: 0, new DesiredWorkerReplicas: 1, old MinWorkerReplicas: 0, new MinWorkerReplicas: 1, old MaxWorkerReplicas: 0, new MaxWorkerReplicas: 50"}
{"level":"info","ts":"2024-04-20T01:14:16.254Z","logger":"controllers.RayCluster","msg":"rayClusterReconcile","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"44413d62-4bfb-4ea0-81cb-492cc75b00fc","Update CR status":"rayservice-sample-raycluster-9s5d7","status":{"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":50,"desiredCPU":"5","desiredMemory":"4Gi","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2024-04-20T01:14:16Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"serviceIP":"10.92.15.52"},"observedGeneration":1}}
{"level":"info","ts":"2024-04-20T01:14:16.262Z","logger":"controllers.RayCluster","msg":"Got error when updating status","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"44413d62-4bfb-4ea0-81cb-492cc75b00fc","cluster name":"rayservice-sample-raycluster-9s5d7","error":"Operation cannot be fulfilled on rayclusters.ray.io \"rayservice-sample-raycluster-9s5d7\": the object has been modified; please apply your changes to the latest version and try again","RayCluster":{"apiVersion":"ray.io/v1","kind":"RayCluster","namespace":"rubrik-spark","name":"rayservice-sample-raycluster-9s5d7"}}

```

---

<div class="post-metadata">

**Author:** ![shaikhismail](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/shaikhismail/32/5853_2.png) [@shaikhismail](https://discuss.ray.io/u/shaikhismail)\
**Post date:** [April 20, 2024, 2:27am UTC](https://discuss.ray.io/t/kuberay-sample-rayservice-not-launching-serve-apps/14461/3 "2024-04-20T02:27:34Z")

</div>

…continued: kuberay operator logs:

```auto
{"level":"info","ts":"2024-04-20T01:14:16.262Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes reqeueuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"44413d62-4bfb-4ea0-81cb-492cc75b00fc"}
{"level":"error","ts":"2024-04-20T01:14:16.262Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"44413d62-4bfb-4ea0-81cb-492cc75b00fc","error":"Operation cannot be fulfilled on rayclusters.ray.io \"rayservice-sample-raycluster-9s5d7\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-04-20T01:14:16.350Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"4f046e1e-b68f-4bf8-b682-2811b52084a0"}
{"level":"info","ts":"2024-04-20T01:14:16.350Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"4f046e1e-b68f-4bf8-b682-2811b52084a0","1 head service found":"rayservice-sample-raycluster-9s5d7-head-svc"}
{"level":"info","ts":"2024-04-20T01:14:16.351Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"4f046e1e-b68f-4bf8-b682-2811b52084a0","Found 1 head Pod":"rayservice-sample-raycluster-9s5d7-head-z5st7","Pod status":"Pending","Pod restart policy":"Always","Ray container terminated status":"nil"}
{"level":"info","ts":"2024-04-20T01:14:16.351Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"4f046e1e-b68f-4bf8-b682-2811b52084a0","head Pod":"rayservice-sample-raycluster-9s5d7-head-z5st7","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod rayservice-sample-raycluster-9s5d7-head-z5st7. The Pod status is Pending, and the Ray container terminated status is nil."}
{"level":"info","ts":"2024-04-20T01:14:16.351Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"4f046e1e-b68f-4bf8-b682-2811b52084a0","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":50,"minReplicas":1,"replicas":1}
{"level":"info","ts":"2024-04-20T01:14:16.352Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"4f046e1e-b68f-4bf8-b682-2811b52084a0","worker Pod":"ervice-sample-raycluster-9s5d7-worker-small-group-5zqcc","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod ervice-sample-raycluster-9s5d7-worker-small-group-5zqcc. The Pod status is Pending, and the Ray container terminated status is nil."}
{"level":"info","ts":"2024-04-20T01:14:16.352Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"4f046e1e-b68f-4bf8-b682-2811b52084a0","removing the pods in the scaleStrategy of":"small-group"}
{"level":"info","ts":"2024-04-20T01:14:16.352Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"4f046e1e-b68f-4bf8-b682-2811b52084a0","workerReplicas":1,"runningPods":1,"diff":0}
{"level":"info","ts":"2024-04-20T01:14:16.352Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"4f046e1e-b68f-4bf8-b682-2811b52084a0","all workers already exist for group":"small-group"}
{"level":"info","ts":"2024-04-20T01:14:16.353Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running; Pod Name: rayservice-sample-raycluster-9s5d7-head-z5st7; Pod Status.Phase: Pending","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"4f046e1e-b68f-4bf8-b682-2811b52084a0"}
{"level":"info","ts":"2024-04-20T01:14:16.354Z","logger":"controllers.RayCluster","msg":"Environment variable RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV is not set, using default value of 300 seconds","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"4f046e1e-b68f-4bf8-b682-2811b52084a0","cluster name":"rayservice-sample-raycluster-9s5d7"}
{"level":"info","ts":"2024-04-20T01:14:16.354Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"rayservice-sample-raycluster-9s5d7","namespace":"rubrik-spark"},"reconcileID":"4f046e1e-b68f-4bf8-b682-2811b52084a0","cluster name":"rayservice-sample-raycluster-9s5d7","seconds":300}
{"level":"info","ts":"2024-04-20T01:14:18.066Z","logger":"controllers.RayService","msg":"Check the head Pod status of the pending RayCluster","RayService":{"name":"rayservice-sample","namespace":"rubrik-spark"},"reconcileID":"5d86c0a5-fcba-483f-8c12-bcff7a21bbac","RayCluster name":"rayservice-sample-raycluster-9s5d7"}
{"level":"info","ts":"2024-04-20T01:14:18.066Z","logger":"controllers.RayService","msg":"Skipping the update of Serve deployments because the Ray head Pod is not ready.","RayService":{"name":"rayservice-sample","namespace":"rubrik-spark"},"reconcileID":"5d86c0a5-fcba-483f-8c12-bcff7a21bbac"}
{"level":"info","ts":"2024-04-20T01:14:18.066Z","logger":"controllers.RayService","msg":"Ray Serve applications are not ready to serve requests: checking again in 2ss","RayService":{"name":"rayservice-sample","namespace":"rubrik-spark"},"reconcileID":"5d86c0a5-fcba-483f-8c12-bcff7a21bbac"}
{"level":"info","ts":"2024-04-20T01:14:20.068Z","logger":"controllers.RayService","msg":"Check the head Pod status of the pending RayCluster","RayService":{"name":"rayservice-sample","namespace":"rubrik-spark"},"reconcileID":"dcea53ea-5ac7-4f09-af58-1a50fb26906e","RayCluster name":"rayservice-sample-raycluster-9s5d7"}
{"level":"info","ts":"2024-04-20T01:14:20.069Z","logger":"controllers.RayService","msg":"Skipping the update of Serve deployments because the Ray head Pod is not ready.","RayService":{"name":"rayservice-sample","namespace":"rubrik-spark"},"reconcileID":"dcea53ea-5ac7-4f09-af58-1a50fb26906e"}
{"level":"info","ts":"2024-04-20T01:14:20.069Z","logger":"controllers.RayService","msg":"Ray Serve applications are not ready to serve requests: checking again in 2ss","RayService":{"name":"rayservice-sample","namespace":"rubrik-spark"},"reconcileID":"dcea53ea-5ac7-4f09-af58-1a50fb26906e"}

```

---

<div class="post-metadata">

**Author:** ![shaikhismail](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/shaikhismail/32/5853_2.png) [@shaikhismail](https://discuss.ray.io/u/shaikhismail)\
**Post date:** [April 20, 2024, 2:48am UTC](https://discuss.ray.io/t/kuberay-sample-rayservice-not-launching-serve-apps/14461/4 "2024-04-20T02:48:27Z")

</div>

**Update: I deleted the sample app by** `kubectl delete -f ../sample-app/ray-service.sample.yaml` **and re-deployed by** `kubectl apply -f ../sample-app/ray-service.sample.yaml`, **with the following changes to the yaml file** :

```auto
apiVersion: ray.io/v1
kind: RayService
metadata:
  name: rayservice-sample
...
  rayClusterConfig:
    rayVersion: '2.11.0' # should match the Ray version in the image of the containers
# enableInTreeAutoscaling: true
    ######################headGroupSpecs#################################
    # Ray head pod template.
    headGroupSpec:
      ...
              readinessProbe:
                exec:
                  command:
                    - sh
                    - -c
                    - wget -T 2 -q -O- http://localhost:52365/api/local_raylet_healthz | grep
                      success && wget -T 2 -q -O- http://localhost:8265/api/gcs_healthz | grep
                      success
                periodSeconds: 5
                timeoutSeconds: 60
                successThreshold: 1
                failureThreshold: 5
              livenessProbe:
                exec:
                  command:
                    - sh
                    - -c
                    - wget -T 2 -q -O- http://localhost:52365/api/local_raylet_healthz | grep success && wget -T 2 -q -O- http://localhost:8265/api/gcs_healthz | grep success
                periodSeconds: 5
                timeoutSeconds: 60
                successThreshold: 1
                failureThreshold: 5
    workerGroupSpecs:
      ...
                livenessProbe:
                  exec:
                    command:
                      - sh
                      - -c
                      - wget -T 2 -q -O- http://localhost:52365/api/local_raylet_healthz | grep success
                  periodSeconds: 5
                  timeoutSeconds: 60
                  successThreshold: 1
                  failureThreshold: 5

```

**I saw this error in the operator logs:**

```auto
{"level":"info","ts":"2024-04-20T02:43:55.060Z","logger":"controllers.RayService","msg":"Check the head Pod status of the pending RayCluster","RayService":{"name":"rayservice-sample","namespace":"rubrik-spark"},"reconcileID":"1372cf45-186c-4c4c-ae4b-a9efe93b5985","RayCluster name":"rayservice-sample-raycluster-ch9j5"}
{"level":"info","ts":"2024-04-20T02:43:55.061Z","logger":"controllers.RayService","msg":"FetchHeadServiceURL","RayService":{"name":"rayservice-sample","namespace":"rubrik-spark"},"reconcileID":"1372cf45-186c-4c4c-ae4b-a9efe93b5985","head service name":"rayservice-sample-raycluster-ch9j5-head-svc","namespace":"rubrik-spark"}
{"level":"info","ts":"2024-04-20T02:43:55.061Z","logger":"controllers.RayService","msg":"FetchHeadServiceURL","RayService":{"name":"rayservice-sample","namespace":"rubrik-spark"},"reconcileID":"1372cf45-186c-4c4c-ae4b-a9efe93b5985","head service URL":"rayservice-sample-raycluster-ch9j5-head-svc.rubrik-spark.svc.cluster.local:8265","port":"dashboard"}
{"level":"info","ts":"2024-04-20T02:43:55.061Z","logger":"controllers.RayService","msg":"shouldUpdate","RayService":{"name":"rayservice-sample","namespace":"rubrik-spark"},"reconcileID":"1372cf45-186c-4c4c-ae4b-a9efe93b5985","shouldUpdateServe":true,"reason":"Nothing has been cached for cluster rayservice-sample-raycluster-ch9j5 with key rubrik-spark/rayservice-sample/rayservice-sample-raycluster-ch9j5"}
{"level":"info","ts":"2024-04-20T02:43:55.061Z","logger":"controllers.RayService","msg":"updateServeDeployment","RayService":{"name":"rayservice-sample","namespace":"rubrik-spark"},"reconcileID":"1372cf45-186c-4c4c-ae4b-a9efe93b5985","V2 config":"applications:\n - name: math_app\n import_path: conditional_dag.serve_dag\n route_prefix: /calc\n deployments:\n - name: Adder\n user_config:\n increment: 3\n max_ongoing_requests: 2\n autoscaling_config:\n target_ongoing_requests: 1\n metrics_interval_s: 0.2\n min_replicas: 0\n initial_replicas: 0\n max_replicas: 100\n look_back_period_s: 2\n downscale_delay_s: 10\n upscale_delay_s: 0\n graceful_shutdown_timeout_s: 5\n ray_actor_options:\n num_cpus: 0.5\n - name: Multiplier\n user_config:\n factor: 5\n max_ongoing_requests: 2\n autoscaling_config:\n target_ongoing_requests: 1\n metrics_interval_s: 0.2\n min_replicas: 0\n initial_replicas: 0\n max_replicas: 100\n look_back_period_s: 2\n downscale_delay_s: 10\n upscale_delay_s: 0\n graceful_shutdown_timeout_s: 5\n ray_actor_options:\n num_cpus: 1.5\n - name: Router\n max_ongoing_requests: 2\n autoscaling_config:\n target_ongoing_requests: 1\n metrics_interval_s: 0.2\n min_replicas: 1\n max_replicas: 100\n look_back_period_s: 2\n downscale_delay_s: 10\n upscale_delay_s: 0\n graceful_shutdown_timeout_s: 5\n ray_actor_options:\n num_cpus: 1\n"}
{"level":"info","ts":"2024-04-20T02:43:55.062Z","logger":"controllers.RayService","msg":"updateServeDeployment","RayService":{"name":"rayservice-sample","namespace":"rubrik-spark"},"reconcileID":"1372cf45-186c-4c4c-ae4b-a9efe93b5985","MULTI_APP json config":"{\"applications\":[{\"deployments\":[{\"autoscaling_config\":{\"downscale_delay_s\":10,\"initial_replicas\":0,\"look_back_period_s\":2,\"max_replicas\":100,\"metrics_interval_s\":0.2,\"min_replicas\":0,\"target_ongoing_requests\":1,\"upscale_delay_s\":0},\"graceful_shutdown_timeout_s\":5,\"max_ongoing_requests\":2,\"name\":\"Adder\",\"ray_actor_options\":{\"num_cpus\":0.5},\"user_config\":{\"increment\":3}},{\"autoscaling_config\":{\"downscale_delay_s\":10,\"initial_replicas\":0,\"look_back_period_s\":2,\"max_replicas\":100,\"metrics_interval_s\":0.2,\"min_replicas\":0,\"target_ongoing_requests\":1,\"upscale_delay_s\":0},\"graceful_shutdown_timeout_s\":5,\"max_ongoing_requests\":2,\"name\":\"Multiplier\",\"ray_actor_options\":{\"num_cpus\":1.5},\"user_config\":{\"factor\":5}},{\"autoscaling_config\":{\"downscale_delay_s\":10,\"look_back_period_s\":2,\"max_replicas\":100,\"metrics_interval_s\":0.2,\"min_replicas\":1,\"target_ongoing_requests\":1,\"upscale_delay_s\":0},\"graceful_shutdown_timeout_s\":5,\"max_ongoing_requests\":2,\"name\":\"Router\",\"ray_actor_options\":{\"num_cpus\":1}}],\"import_path\":\"conditional_dag.serve_dag\",\"name\":\"math_app\",\"route_prefix\":\"/calc\"}]}"}
{"level":"error","ts":"2024-04-20T02:43:57.077Z","logger":"controllers.RayService","msg":"Fail to reconcileServe.","RayService":{"name":"rayservice-sample","namespace":"rubrik-spark"},"reconcileID":"1372cf45-186c-4c4c-ae4b-a9efe93b5985","error":"Fail to create / update Serve applications. If you observe this error consistently, please check \"Issue 5: Fail to create / update Serve applications.\" in https://docs.ray.io/en/master/cluster/kubernetes/troubleshooting/rayservice-troubleshooting.html#kuberay-raysvc-troubleshoot for more details. err: Put \"http://rayservice-sample-raycluster-ch9j5-head-svc.rubrik-spark.svc.cluster.local:8265/api/serve/applications/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayServiceReconciler).Reconcile\n\t/home/runner/work/kuberay/kuberay/ray-operator/controllers/ray/rayservice_controller.go:169\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}

```

---

<div class="post-metadata">

**Author:** ![shrekris](https://avatars.discourse-cdn.com/v4/letter/s/ce73a5/32.png) [@shrekris](https://discuss.ray.io/u/shrekris)\
**Post date:** [April 22, 2024, 8:50pm UTC](https://discuss.ray.io/t/kuberay-sample-rayservice-not-launching-serve-apps/14461/5 "2024-04-22T20:50:40Z")

</div>

Could you exec into your cluster’s head node and run `serve status`? What’s the output?

---

<div class="post-metadata">

**Author:** ![shaikhismail](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/shaikhismail/32/5853_2.png) [@shaikhismail](https://discuss.ray.io/u/shaikhismail)\
**Post date:** [April 22, 2024, 11:57pm UTC](https://discuss.ray.io/t/kuberay-sample-rayservice-not-launching-serve-apps/14461/6 "2024-04-22T23:57:22Z")

</div>

Btw I’m running this app on GKE, fyi.

```auto
kubectl get pods -l=ray.io/is-ray-node=yes
NAME READY STATUS RESTARTS AGE
ervice-sample-raycluster-ch9j5-worker-small-group-m7b4t 0/1 Init:0/1 0 69m
rayservice-sample-raycluster-ch9j5-head-5m57v 1/1 Running 0 2d17h

kubectl exec -it rayservice-sample-raycluster-ch9j5-head-5m57v -- /bin/bash

(base) ray@rayservice-sample-raycluster-ch9j5-head-5m57v:/app$
(base) ray@rayservice-sample-raycluster-ch9j5-head-5m57v:/app$
(base) ray@rayservice-sample-raycluster-ch9j5-head-5m57v:/app$ ls
__pycache__ conditional_dag.py
(base) ray@rayservice-sample-raycluster-ch9j5-head-5m57v:/app$ serve status
proxies: {}
applications: {}
target_capacity: null
serve status
proxies: {}
applications: {}
target_capacity: null

```

---

<div class="post-metadata">

**Author:** ![Kai-Hsun\_Chen](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kai-hsun_chen/32/3760_2.png) [@Kai-Hsun\_Chen](https://discuss.ray.io/u/Kai-Hsun_Chen)\
**Post date:** [April 23, 2024, 4:02am UTC](https://discuss.ray.io/t/kuberay-sample-rayservice-not-launching-serve-apps/14461/7 "2024-04-23T04:02:37Z")

</div>

I can’t reproduce the issue. I published my image to Quay at `quay.io/kevin85421/ray:test-dag`, and you can try it. I use a local Kind cluster.

```sh
helm install kuberay-operator kuberay/kuberay-operator --version 1.1.0

# Create a RayService with https://gist.github.com/kevin85421/fc292ee8b0e24fd9b422833e208dbf51
kubectl apply -f ray-service.sample-no-runtimeEnv.yaml

# Verify
export HEAD_POD=$(kubectl get pods --selector=ray.io/node-type=head -o custom-columns=POD:metadata.name --no-headers)
kubectl exec -it $HEAD_POD -- bash

# In the head Pod
serve status

```

 ![Screenshot 2024-04-22 at 9.01.15 PM](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/f/fcf903e152dab91d84d5bffa43250908471a5506.jpeg)

 ![Screenshot 2024-04-22 at 9.01.59 PM](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/0/07af179c938abea02b5375e26fa2318982f38552.png)

---

<div class="post-metadata">

**Author:** ![psydok](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/psydok/32/3518_2.png) [@psydok](https://discuss.ray.io/u/psydok)\
**Post date:** [April 23, 2024, 5:38pm UTC](https://discuss.ray.io/t/kuberay-sample-rayservice-not-launching-serve-apps/14461/8 "2024-04-23T17:38:18Z")

</div>

I tested kuberay with version ray 2.10.0-py311 and 2.9.0. I have the same problem. It fails to create rayservice-sample-head-svc and rayservice-sample-serve-svc. But I didn’t notice it right away.  
And I only noticed it because I got them to run once. I tried uninstalling them all and running them again to check for reproducibility. Now there is no way to reproduce a successful launch of rayservice.

I tested versions of kuberay 1.1.0 (doesn’t work with argocd), 1.1.0-rc.1 and 1.1.0-rc.0.

---

<div class="post-metadata">

**Author:** ![psydok](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/psydok/32/3518_2.png) [@psydok](https://discuss.ray.io/u/psydok)\
**Post date:** [April 23, 2024, 5:50pm UTC](https://discuss.ray.io/t/kuberay-sample-rayservice-not-launching-serve-apps/14461/9 "2024-04-23T17:50:05Z")

</div>

I tried to deploy my own applications with runtimeEnv. The `serve start` is not started inside the vorkers. I connected the jar to the cluster in kuberay separately. On this separate jar I started `serve start ...` - so the applications started to be deployed. But still svc with universal names did not get up.

---

<div class="post-metadata">

**Author:** ![shaikhismail](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/shaikhismail/32/5853_2.png) [@shaikhismail](https://discuss.ray.io/u/shaikhismail)\
**Post date:** [April 24, 2024, 11:23pm UTC](https://discuss.ray.io/t/kuberay-sample-rayservice-not-launching-serve-apps/14461/10 "2024-04-24T23:23:04Z")

</div>

@Kai-Hsun_Chen repeated the steps mentioned by you in local Kind cluster:

```auto
$ export HEAD_POD=$(kubectl get pods --selector=ray.io/node-type=head -o custom-columns=POD:metadata.name --no-headers)
$ kubectl exec -it $HEAD_POD -- bash
error: unable to upgrade connection: container not found ("ray-head")

```

---

<div class="post-metadata">

**Author:** ![Kai-Hsun\_Chen](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kai-hsun_chen/32/3760_2.png) [@Kai-Hsun\_Chen](https://discuss.ray.io/u/Kai-Hsun_Chen)\
**Post date:** [April 25, 2024, 5:48am UTC](https://discuss.ray.io/t/kuberay-sample-rayservice-not-launching-serve-apps/14461/11 "2024-04-25T05:48:00Z")

</div>

Solved by [Slack](https://ray-distributed.slack.com/archives/C02GFQ82JPM/p1712267296145549)

---

<div class="post-metadata">

**Author:** ![zenssh](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/zenssh/32/6519_2.png) [@zenssh](https://discuss.ray.io/u/zenssh)\
**Post date:** [September 10, 2024, 6:22am UTC](https://discuss.ray.io/t/kuberay-sample-rayservice-not-launching-serve-apps/14461/12 "2024-09-10T06:22:59Z")

</div>

I’m having the same issue and don’t have access to slack - is it possible to post the fix here?