# Runtime env docker image: Check failed: !job\_id\_env.empty() error

**URL:** <https://discuss.ray.io/t/runtime-env-docker-image-check-failed-job-id-env-empty-error/10503>\
**Category:** Ray Core\
**Created:** [May 4, 2023, 6:07am UTC](https://discuss.ray.io/t/runtime-env-docker-image-check-failed-job-id-env-empty-error/10503 "2023-05-04T06:07:10Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![yejingxin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/yejingxin/32/4351_2.png) [@yejingxin](https://discuss.ray.io/u/yejingxin)\
**Post date:** [May 4, 2023, 6:07am UTC](https://discuss.ray.io/t/runtime-env-docker-image-check-failed-job-id-env-empty-error/10503/1 "2023-05-04T06:07:10Z")

</div>

The docker image is created by

```auto
FROM ubuntu:20.04
RUN apt-get update -y
RUN apt-get install -y python3 python3-pip
RUN ln -s /usr/bin/python3 /usr/bin/python
RUN pip3 install --no-cache-dir absl-py ray[default]
RUN pip3 install --no-cache-dir jax[tpu] -f https://storage.googleapis.com/jax-releases/libtpu_releases.html
ENTRYPOINT ["python3" , "--version"]

```

And when I run the job, I have the following error

```auto
{"command_prefix": ["cd", "/tmp/ray/session_2023-05-04_05-54-58_484700_14533/runtime_resources/working_dir_files/_ray_pkg_6068c19fb3b8530f", "&&"], "env_vars": {"RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES": "1", "PYTHONPATH": "/tmp/ray/session_2023-05-04_05-54-58_484700_14533/runtime_resources/working_dir_files/_ray_pkg_6068c19fb3b8530f"}, "py_executable": "docker run -v /tmp/ray:/tmp/ray --network=host --privileged --pid=host --ipc=host --env RAY_RAYLET_PID=15100 -v /home/yejingxin/ray_venv/lib/python3.8/site-packages/ray:/home/yejingxin/ray_venv/lib/python3.8/site-packages/ray --entrypoint python3 gcr.io/cloud-tpu-v2-images/grpc_tpu_worker_v4:yejingxin-debug", "resources_dir": null, "container": {}, "java_jars": []}
[2023-05-04 05:55:10,491 C 15714 15714] core_worker.cc:50: Check failed: !job_id_env.empty() 
***StackTrace Information***
/usr/local/lib/python3.8/dist-packages/ray/_raylet.so(+0xd7dcaa) [0x7f3a4b1f6caa] ray::operator<<()
/usr/local/lib/python3.8/dist-packages/ray/_raylet.so(+0xd7f792) [0x7f3a4b1f8792] ray::SpdLogMessage::Flush()
/usr/local/lib/python3.8/dist-packages/ray/_raylet.so(_ZN3ray6RayLogD1Ev+0x37) [0x7f3a4b1f8aa7] ray::RayLog::~RayLog()
/usr/local/lib/python3.8/dist-packages/ray/_raylet.so(_ZN3ray4core15GetProcessJobIDERKNS0_17CoreWorkerOptionsE+0x10b) [0x7f3a4aafbdfb] ray::core::GetProcessJobID()
/usr/local/lib/python3.8/dist-packages/ray/_raylet.so(_ZN3ray4core10CoreWorkerC1ERKNS0_17CoreWorkerOptionsERKNS_8WorkerIDE+0x8a) [0x7f3a4aafbf0a] ray::core::CoreWorker::CoreWorker()
/usr/local/lib/python3.8/dist-packages/ray/_raylet.so(_ZN3ray4core21CoreWorkerProcessImplC2ERKNS0_17CoreWorkerOptionsE+0x587) [0x7f3a4ab03167] ray::core::CoreWorkerProcessImpl::CoreWorkerProcessImpl()
/usr/local/lib/python3.8/dist-packages/ray/_raylet.so(_ZN3ray4core17CoreWorkerProcess10InitializeERKNS0_17CoreWorkerOptionsE+0xcf) [0x7f3a4ab041bf] ray::core::CoreWorkerProcess::Initialize()
/usr/local/lib/python3.8/dist-packages/ray/_raylet.so(+0x53a895) [0x7f3a4a9b3895] __pyx_pw_3ray_7_raylet_10CoreWorker_1__ cinit__()
/usr/local/lib/python3.8/dist-packages/ray/_raylet.so(+0x53bf33) [0x7f3a4a9b4f33] __pyx_tp_new_3ray_7_raylet_CoreWorker()
ray::IDLE(_PyObject_MakeTpCall+0x183) [0x5f6f43] _PyObject_MakeTpCall
ray::IDLE(_PyEval_EvalFrameDefault+0x5dae) [0x57107e] _PyEval_EvalFrameDefault
ray::IDLE(_PyEval_EvalCodeWithName+0x26a) [0x569cea] _PyEval_EvalCodeWithName
ray::IDLE(_PyFunction_Vectorcall+0x393) [0x5f6a13] _PyFunction_Vectorcall
ray::IDLE(_PyEval_EvalFrameDefault+0x1901) [0x56cbd1] _PyEval_EvalFrameDefault
ray::IDLE(_PyEval_EvalCodeWithName+0x26a) [0x569cea] _PyEval_EvalCodeWithName
ray::IDLE(PyEval_EvalCode+0x27) [0x68e7b7] PyEval_EvalCode
ray::IDLE() [0x680001]
ray::IDLE() [0x68007f]
ray::IDLE() [0x680121]
ray::IDLE(PyRun_SimpleFileExFlags+0x197) [0x680db7] PyRun_SimpleFileExFlags
ray::IDLE(Py_RunMain+0x212) [0x6b8122] Py_RunMain
ray::IDLE(Py_BytesMain+0x2d) [0x6b84ad] Py_BytesMain
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f3a4bda2083]__libc_start_main
ray::IDLE(_start+0x2e) [0x5fb39e] _start

```

---

<div class="post-metadata">

**Author:** ![yejingxin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/yejingxin/32/4351_2.png) [@yejingxin](https://discuss.ray.io/u/yejingxin)\
**Post date:** [May 4, 2023, 6:08am UTC](https://discuss.ray.io/t/runtime-env-docker-image-check-failed-job-id-env-empty-error/10503/2 "2023-05-04T06:08:56Z")

</div>

@GuyangSong Could you help me with the issue?

---

<div class="post-metadata">

**Author:** ![GuyangSong](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/guyangsong/32/181_2.png) [@GuyangSong](https://discuss.ray.io/u/GuyangSong)\
**Post date:** [May 4, 2023, 12:59pm UTC](https://discuss.ray.io/t/runtime-env-docker-image-check-failed-job-id-env-empty-error/10503/3 "2023-05-04T12:59:58Z")

</div>

Can you show me your code?

---

<div class="post-metadata">

**Author:** ![yejingxin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/yejingxin/32/4351_2.png) [@yejingxin](https://discuss.ray.io/u/yejingxin)\
**Post date:** [May 4, 2023, 5:27pm UTC](https://discuss.ray.io/t/runtime-env-docker-image-check-failed-job-id-env-empty-error/10503/4 "2023-05-04T17:27:38Z")

</div>

```auto
client = JobSubmissionClient("http://127.0.0.1:8265")

job_ids = []
for _ in range(1):
  job_id = client.submit_job(
      # Entrypoint shell command to execute
      entrypoint=(
         "python3 -c \"import jax; print('num: ', jax.device_count())\""
      ),
      # Working dir
      runtime_env={
          "working_dir": "/home/yejingxin/docker_ray",
          "pip": [],
          "container":{"image": "gcr.io/cloud-tpu-v2-images/grpc_tpu_worker_v4:yejingxin-debug",
          #"worker_path": "/usr/local/lib/python3.8/dist-packages/ray/_private/workers/default_worker.py"
          "run_options": ["--env PATH=/home/yejingxin/ray2/bin:$PATH", "-v /home/yejingxin/ray2/:/home/yejingxin/ray2/"],
          },
      },
  )
  job_ids.append(job_id)

print(job_ids)

```

I also modified container driver to be docker, since podman does not work well for us

```auto
container_driver = "docker"
        container_command = [
            container_driver,
            "run",
            "-v",
            self._ray_tmp_dir + ":" + self._ray_tmp_dir,
            #"--cgroup-manager=cgroupfs",
            "--privileged",
            "--network=host",
            "--pid=host",
            "--ipc=host",
            #"--env-host",
        ]

```

---

<div class="post-metadata">

**Author:** ![GuyangSong](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/guyangsong/32/181_2.png) [@GuyangSong](https://discuss.ray.io/u/GuyangSong)\
**Post date:** [May 5, 2023, 8:54am UTC](https://discuss.ray.io/t/runtime-env-docker-image-check-failed-job-id-env-empty-error/10503/5 "2023-05-05T08:54:07Z")

</div>

I’m sorry that the container feature has been broken for a long time and I’m not sure you it can work for you now. 🤣  
What’s your ray version?

---

<div class="post-metadata">

**Author:** ![yejingxin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/yejingxin/32/4351_2.png) [@yejingxin](https://discuss.ray.io/u/yejingxin)\
**Post date:** [May 6, 2023, 4:36pm UTC](https://discuss.ray.io/t/runtime-env-docker-image-check-failed-job-id-env-empty-error/10503/6 "2023-05-06T16:36:31Z")

</div>

it is 2.4.0

could you give some instruction to debug the issue?

If there is a potential fix, which part of the code we should look at?

What happened after `docker run image default_worker.py` ?

Docker does not support `--env-host`, is it the root cause that ray cannot get job id info from env vars?

---

<div class="post-metadata">

**Author:** ![GuyangSong](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/guyangsong/32/181_2.png) [@GuyangSong](https://discuss.ray.io/u/GuyangSong)\
**Post date:** [May 8, 2023, 3:33am UTC](https://discuss.ray.io/t/runtime-env-docker-image-check-failed-job-id-env-empty-error/10503/7 "2023-05-08T03:33:27Z")

</div>

> [@yejingxin](#):
>
> Docker does not support `--env-host`, is it the root cause that ray cannot get job id info from env vars?

Yep, It seems an issue about env vars. `RAY_JOB_ID` should be forwarded to the worker process in docker.

---

<div class="post-metadata">

**Author:** ![yejingxin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/yejingxin/32/4351_2.png) [@yejingxin](https://discuss.ray.io/u/yejingxin)\
**Post date:** [May 8, 2023, 3:51am UTC](https://discuss.ray.io/t/runtime-env-docker-image-check-failed-job-id-env-empty-error/10503/8 "2023-05-08T03:51:05Z")

</div>

> [@GuyangSong](#):
>
> Yep, It seems an issue about env vars. `RAY_JOB_ID` should be forwarded to the worker process in docker.

do you know where `RAY_JOB_ID` is set? I am trying to think whether it is possible to save those info into files, and restore from file, instead relying on env vars.

---

<div class="post-metadata">

**Author:** ![GuyangSong](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/guyangsong/32/181_2.png) [@GuyangSong](https://discuss.ray.io/u/GuyangSong)\
**Post date:** [May 8, 2023, 4:10am UTC](https://discuss.ray.io/t/runtime-env-docker-image-check-failed-job-id-env-empty-error/10503/9 "2023-05-08T04:10:05Z")

</div>

You can search it in the ray repo, it is set by raylet.

---

<div class="post-metadata">

**Author:** ![yejingxin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/yejingxin/32/4351_2.png) [@yejingxin](https://discuss.ray.io/u/yejingxin)\
**Post date:** [June 12, 2023, 7:26am UTC](https://discuss.ray.io/t/runtime-env-docker-image-check-failed-job-id-env-empty-error/10503/10 "2023-06-12T07:26:49Z")

</div>

tried to run the example run

```auto
job_id = client.submit_job(
      # Entrypoint shell command to execute
      entrypoint=(
         "python3 -c \"import ray; print('ncsoft: ', ray. __version__ )\""
      ),
      # Working dir
      runtime_env={     
          "container":{"image": "docker.io/anyscale/ray-ml:nightly-py38-cpu", 
          "run_options": ["--tmpfs /tmp:rw", "--env PATH=/home//ray/anaconda3/bin:$PATH", "--cap-drop SYS_ADMIN","--log-level=debug", "-v /home/yejingxin/ray_venv/ray/:/home/yejingxin/ray_venv/ray/"],
          },
      },
  )

```

got the tmp lock file permission error

```auto
103time="2023-06-12T07:20:22Z" level=debug msg="Enabling signal proxying"

104Traceback (most recent call last):

105 File "/home/yejingxin/ray_venv/ray/python/ray/_private/workers/default_worker.py", line 203, in <module>

106 node = ray._private.node.Node(

107 File "/home/ray/anaconda3/lib/python3.8/site-packages/ray/_private/node.py", line 255, in __init__

108 self.metrics_agent_port = self._get_cached_port(

109 File "/home/ray/anaconda3/lib/python3.8/site-packages/ray/_private/node.py", line 851, in _get_cached_port

110 with FileLock(file_path + ".lock"):

111 File "/home/ray/anaconda3/lib/python3.8/site-packages/filelock/_api.py", line 255, in __enter__

112 self.acquire()

113 File "/home/ray/anaconda3/lib/python3.8/site-packages/filelock/_api.py", line 213, in acquire

114 self._acquire()

115 File "/home/ray/anaconda3/lib/python3.8/site-packages/filelock/_unix.py", line 37, in _acquire

116 fd = os.open(self.lock_file, open_flags, self._context.mode)

117PermissionError: [Errno 13] Permission denied: '/tmp/ray/session_2023-06-12_07-20-11_997308_503491/ports_by_node.json.lock'

```