# Unable to pass Docker image to run to Ray Job Submit

**URL:** <https://discuss.ray.io/t/unable-to-pass-docker-image-to-run-to-ray-job-submit/23299>\
**Category:** Ray Core\
**Created:** [November 2, 2025, 5:45pm UTC](https://discuss.ray.io/t/unable-to-pass-docker-image-to-run-to-ray-job-submit/23299 "2025-11-02T17:45:14Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![neilmatrix](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/neilmatrix/32/7526_2.png) [@neilmatrix](https://discuss.ray.io/u/neilmatrix)\
**Post date:** [November 2, 2025, 5:45pm UTC](https://discuss.ray.io/t/unable-to-pass-docker-image-to-run-to-ray-job-submit/23299/1 "2025-11-02T17:45:14Z")

</div>

I pulled this image - `rayproject/ray:2.51.0.801bd7-extra-py310` from dockerhub with podman (ray prefers podman internally).

On AWS-EC2 then I started a head instance with - `Deep Learning Base AMI with Single CUDA (Ubuntu 22.04)`, `ami-06eff6f62c23006e9 (64-bit (x86))`

On the instance, I have:

```bash
ubuntu@ip-172-31-15-118:~$ python --version
Python 3.10.19
ubuntu@ip-172-31-15-118:~$ python3 --version
Python 3.10.19
ubuntu@ip-172-31-15-118:~$ docker --version
Docker version 28.5.1, build e180ab8
ubuntu@ip-172-31-15-118:~$ podman --version
podman version 3.4.4

python3 -m venv ~/raycli-env
source ~/raycli-env/bin/activate
pip install -U "ray[default]"

(raycli-env) ubuntu@ip-172-31-15-118:~$ ray --version
ray, version 2.51.0

```

I have exported the following environment variables also:

```auto
export RAY_RUNTIME_ENV_DOCKER=1
export RAY_RUNTIME_ENV_PODMAN_EXE=/usr/bin/docker
export PATH=/usr/bin:$PATH

```

And I have started the ray cluster with:

```auto
ray start --head --port=6379 --dashboard-host=0.0.0.0

```

Then I have done:

```bash
mkdir ray-job
cd ray-job

echo 'import ray
ray.init(address="auto")
@ray.remote
def hello():
    return "Hello World from Ray on GPU!"
print(ray.get(hello.remote()))' > hello_ray.py

```

Then I extended the image with the following Dockerfile:

```auto
ARG RAY_UID=1000
ARG RAY_GID=100

FROM rayproject/ray:2.51.0.801bd7-extra-py310

USER root
RUN pip install "ray[default]"

# go back to ray user for running workloads
USER ray
WORKDIR /home/ray

CMD ["python", "hello_ray.py"]

```

And built the image with:

```auto
podman build -t ray-image:latest

```

But when I run, the following command, I don’t get any errors, and the job stays in `pending` forever.

```bash
ray job submit \
  --address="http://127.0.0.1:8265" \
  --runtime-env-json '{"image_uri": "localhost/ray-image:latest"}' \
  -- python hello_ray.py

```

But when I run this with `python-slim` with python versions matched, I am able to successfully submit the job.

I don’t know what I am doing wrong. The python and ray versions on EC2 instance and the podman image are same.  
Please help me. I have been stucked here for so long.

---

<div class="post-metadata">

**Author:** ![RunLLM](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/runllm/32/8249_2.png) [@RunLLM](https://discuss.ray.io/u/RunLLM)\
**Post date:** [November 2, 2025, 5:45pm UTC](https://discuss.ray.io/t/unable-to-pass-docker-image-to-run-to-ray-job-submit/23299/2 "2025-11-02T17:45:50Z")

</div>

Your Ray job is stuck in “pending” because the `image_uri` (container runtime environment) feature requires Podman to be installed and available on the host, and the Ray container must be able to launch nested containers using Podman. The Ray documentation and recent issues confirm that:

- **Podman is required** : Ray’s container runtime environment support is experimental and currently only works with Podman, not Docker, even if you set `RAY_RUNTIME_ENV_PODMAN_EXE=/usr/bin/docker` ([Ray multi-app container guide](https://docs.ray.io/en/latest/serve/advanced-guides/multi-app-container.html#troubleshooting), [Ray GitHub issue #37293](https://github.com/ray-project/ray/issues/37293)).
- **Podman version** : Your Podman version (3.4.4) is quite old; Ray recommends Podman 4.x+ for better compatibility ([Ray multi-app container guide](https://docs.ray.io/en/latest/serve/advanced-guides/multi-app-container.html#troubleshooting)).
- **Privileged container** : If Ray is running inside a container, that container must be started with `--privileged` to allow Podman to launch nested containers ([Ray multi-app container guide](https://docs.ray.io/en/latest/serve/advanced-guides/multi-app-container.html#troubleshooting)).
- **Job stuck in pending** : This is a known symptom when Podman is missing, misconfigured, or not accessible to the Ray process ([Ray GitHub issue #37293](https://github.com/ray-project/ray/issues/37293), [Ray Discuss thread](https://discuss.ray.io/t/submitted-containerized-job-is-stuck-in-pending-mode/11340)).

**Summary:**  
You cannot use Docker as a drop-in replacement for Podman in Ray’s container runtime environment. You must install Podman (preferably v4.x+) on your EC2 instance, ensure it is in the PATH, and, if running Ray inside a container, start that container with `--privileged`. Only then will `image_uri` jobs work as expected.

Would you like more detail on troubleshooting steps or how to set up Podman for Ray?

Sources:

- [Ray multi-app container guide: Troubleshooting](https://docs.ray.io/en/latest/serve/advanced-guides/multi-app-container.html#troubleshooting)
- [Ray GitHub issue #37293](https://github.com/ray-project/ray/issues/37293)
- [Ray Discuss: Submitted containerized job is stuck in pending mode](https://discuss.ray.io/t/submitted-containerized-job-is-stuck-in-pending-mode/11340)

Hint: Mention @RunLLM in the post for followups.