# Unable to run Sample RayJob on EKS: No Available Workers

**URL:** <https://discuss.ray.io/t/unable-to-run-sample-rayjob-on-eks-no-available-workers/14679>\
**Category:** Kubernetes\
**Created:** [May 13, 2024, 8:40pm UTC](https://discuss.ray.io/t/unable-to-run-sample-rayjob-on-eks-no-available-workers/14679 "2024-05-13T20:40:49Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![JamesTann](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jamestann/32/5629_2.png) [@JamesTann](https://discuss.ray.io/u/JamesTann)\
**Post date:** [May 13, 2024, 8:40pm UTC](https://discuss.ray.io/t/unable-to-run-sample-rayjob-on-eks-no-available-workers/14679/1 "2024-05-13T20:40:49Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- High: It blocks me to complete my task.

**TL;DR:** I am unable to run the sample RayJob (or any RayJob) because Ray starts but never submits the job. I believe this is do to no workers being available but I am not sure how to fix that.

I am trying to follow [this simple RayJob example from the docs](https://docs.ray.io/en/latest/cluster/kubernetes/getting-started/rayjob-quick-start.html#example-run-a-simple-ray-job-with-rayjob).

**Ray Version:** 2.9.0  
**Kuberay Operator Version:** 1.1.0  
**K8s Version:** 1.29  
**Environment:** AWS EKS

What I did:

1. Set up my EKS cluster with 2 node groups, 1 of them has GPUs. I followed [the instructions from the docs.](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/aws-eks-gpu-cluster.html)

```auto
eksctl get nodegroup --cluster test_cluster
CLUSTER NODEGROUP STATUS CREATED MIN SIZE MAX SIZE DESIRED CAPACITY INSTANCE TYPE TYPE
test_cluster test_cpu_nodegroup ACTIVE 2024-01-17T16:32:45Z 0 1 1 m5.xlarge AL2_x86_64 managed
test_cluster test_gpu_nodegroup ACTIVE 2024-01-22T20:10:38Z 0 5 1 g5.2xlarge BOTTLEROCKET_x86_64_NVIDIA managed

```

1. Installed the Kuberay Operator Helm chart, according to the docs Step 2.
2. Applied the sample RayJob according to the docs Step 3:

```auto
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/v1.1.0/ray-operator/config/samples/ray-job.sample.yaml

```

1. Ran the commands in step 4 of the docs:

```auto
kubectl get rayjob
NAME AGE
rayjob-sample 22m

kubectl get raycluster
NAME DESIRED WORKERS AVAILABLE WORKERS STATUS AGE
rayjob-sample-raycluster-fmw7d 1 ready 30m

kubectl get pods
NAME READY STATUS RESTARTS AGE
kuberay-operator-5488cc8c8c-45mt7 1/1 Running 0 74m
rayjob-sample-raycluster-fmw7d-head-2f7sk 1/1 Running 0 31m

```

As you can see, no workers are created to run the actual job despite the cluster starting. Does anyone have any advice or tips on how I can find out why this is the case?

---

<div class="post-metadata">

**Author:** ![JamesTann](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jamestann/32/5629_2.png) [@JamesTann](https://discuss.ray.io/u/JamesTann)\
**Post date:** [May 13, 2024, 8:51pm UTC](https://discuss.ray.io/t/unable-to-run-sample-rayjob-on-eks-no-available-workers/14679/2 "2024-05-13T20:51:33Z")

</div>

Sure enough, I am able to run it locally with Kind, so I am guessing I must be missing something with my EKS configuration.

---

<div class="post-metadata">

**Author:** ![JamesTann](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jamestann/32/5629_2.png) [@JamesTann](https://discuss.ray.io/u/JamesTann)\
**Post date:** [May 13, 2024, 9:03pm UTC](https://discuss.ray.io/t/unable-to-run-sample-rayjob-on-eks-no-available-workers/14679/3 "2024-05-13T21:03:22Z")

</div>

I was able to find the following in the KubeRay Operator logs:

```auto
{
  "level": "error",
  "ts": "2024-05-13T21:01:12.397Z",
  "logger": "controllers.RayJob",
  "msg": "Failed to get job info",
  "RayJob": { "name": "rayjob-sample", "namespace": "default" },
  "reconcileID": "579c7e3e-5232-4432-a961-df3c3538d70b",
  "JobId": "rayjob-sample-n4lr2",
  "error": "Job rayjob-sample-n4lr2 does not exist on the cluster",
  "stacktrace": "github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayJobReconciler).Reconcile\n\t/home/runner/work/kuberay/kuberay/ray-operator/controllers/ray/rayjob_controller.go:223\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"
}

```

---

<div class="post-metadata">

**Author:** ![JamesTann](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jamestann/32/5629_2.png) [@JamesTann](https://discuss.ray.io/u/JamesTann)\
**Post date:** [May 14, 2024, 2:18pm UTC](https://discuss.ray.io/t/unable-to-run-sample-rayjob-on-eks-no-available-workers/14679/4 "2024-05-14T14:18:50Z")

</div>

Downgrading KubeRay Operator from 1.1.0 → 1.0.0 fixed the issue. No idea why.