# Submitted containerized job is stuck in pending mode

**URL:** <https://discuss.ray.io/t/submitted-containerized-job-is-stuck-in-pending-mode/11340>\
**Category:** Ray Core\
**Created:** [July 11, 2023, 8:52am UTC](https://discuss.ray.io/t/submitted-containerized-job-is-stuck-in-pending-mode/11340 "2023-07-11T08:52:38Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![stn73](https://avatars.discourse-cdn.com/v4/letter/s/6f9a4e/32.png) [@stn73](https://discuss.ray.io/u/stn73)\
**Post date:** [July 11, 2023, 8:52am UTC](https://discuss.ray.io/t/submitted-containerized-job-is-stuck-in-pending-mode/11340/1 "2023-07-11T08:52:38Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- High: It blocks me to complete my task.

Hi, I want to use ray to submit containerized jobs to a kubernetes cluster. I’ve tried scheduling non-containerized jobs and it works fine. However, once I submit a containerized job, it is stuck in pending mode forever. The command below submits the job successfully, but is stuck forever.

```auto
ray job submit --address http://localhost:8265 --runtime-env-json='{"container": {"image": "<my-cuda-docker-image>", "worker_path": "/root"}}' -- nvidia-smi

Job submission server address: http://localhost:8265

-------------------------------------------------------
Job 'raysubmit_KKgyZumhXYm1y3ng' submitted successfully
-------------------------------------------------------

Next steps
  Query the logs of the job:
    ray job logs raysubmit_KKgyZumhXYm1y3ng
  Query the status of the job:
    ray job status raysubmit_KKgyZumhXYm1y3ng
  Request the job to be stopped:
    ray job stop raysubmit_KKgyZumhXYm1y3ng

Tailing logs until the job exits (disable with --no-wait)

```

Checking the job status confirms this issue.

```auto
ray job status raysubmit_KKgyZumhXYm1y3ng --address http://localhost:8265

Status for job 'raysubmit_KKgyZumhXYm1y3ng': PENDING
Status message: Job has not started yet. It may be waiting for the runtime environment to be set up.

```

Terminating the submitted job also does not work and basically breaks the ray cluster for me.

```auto
ray job stop raysubmit_KKgyZumhXYm1y3ng --address http://localhost:8265

Job submission server address: http://localhost:8265
Attempting to stop job 'raysubmit_KKgyZumhXYm1y3ng'
Waiting for job 'raysubmit_KKgyZumhXYm1y3ng' to exit (disable with --no-wait):
Job has not exited yet. Status: PENDING
Job has not exited yet. Status: PENDING
Job has not exited yet. Status: PENDING
Job has not exited yet. Status: PENDING
Job has not exited yet. Status: PENDING
Job has not exited yet. Status: PENDING

```

Any experiences with this? Could anyone help me with this issue? 🙂

---

<div class="post-metadata">

**Author:** ![jjyao](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jjyao/32/1799_2.png) [@jjyao](https://discuss.ray.io/u/jjyao)\
**Post date:** [July 17, 2023, 5:04am UTC](https://discuss.ray.io/t/submitted-containerized-job-is-stuck-in-pending-mode/11340/2 "2023-07-17T05:04:28Z")

</div>

Hi @stn73,

Sorry for the late reply. Currently this feature is experimental so it may have some rough edges. What’s the Ray version you are using?

---

<div class="post-metadata">

**Author:** ![stn73](https://avatars.discourse-cdn.com/v4/letter/s/6f9a4e/32.png) [@stn73](https://discuss.ray.io/u/stn73)\
**Post date:** [July 17, 2023, 6:44am UTC](https://discuss.ray.io/t/submitted-containerized-job-is-stuck-in-pending-mode/11340/3 "2023-07-17T06:44:45Z")

</div>

Hi @jjyao , thanks for responding!

I also opened an issue on github with some additional information, since I believe this to be a bug, see [here](https://github.com/ray-project/ray/issues/37293).

I am using version version 2.5.1 of ray.