# How does "container" in "runtime\_env" work? **URL:** \ **Category:** Ray Core\ **Created:** [August 8, 2022, 2:30am UTC](https://discuss.ray.io/t/how-does-container-in-runtime-env-work/7108 "2022-08-08T02:30:08Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![jie](https://avatars.discourse-cdn.com/v4/letter/j/3bc359/32.png) [@jie](https://discuss.ray.io/u/jie)\ **Post date:** [August 8, 2022, 2:30am UTC](https://discuss.ray.io/t/how-does-container-in-runtime-env-work/7108/1 "2022-08-08T02:30:08Z")
In this [document](https://docs.ray.io/en/latest/ray-core/handling-dependencies.html#runtime-environments-api-ref) , There has a lot of descriptions about `runtime_env`, **but I don’t known how will it work if I specify some details about `container`.** I submit a job using this command： ```auto ray job submit --address='http://192.168.0.192:8265' --runtime-env-json='{"working_dir":"./","container":{"image": "anyscale/ray-ml:nightly-py38-cpu", "worker_path": "/root/python/ray/workers/default_worker.py", "run_options": ["--cap-drop SYS_ADMIN","--log-level=debug"]}}' -- python ./debug.py ``` This is the debug.py. ```auto import ray ray.init() @ray.remote def f(x): return x * x futures = [f.remote(i) for i in range(2)] print(ray.get(futures)) ``` Surprisingly, I get some feedback, but the process doesn’t seem to stop. I am not getting the result I want. What’s wrong with it? Is there some prework I haven’t done? This is the feedback: ```auto Job submission server address: http://192.168.0.192:8265 2022-08-08 10:08:21,309 INFO dashboard_sdk.py:272 -- Uploading package gcs://_ray_pkg_698a6544fb43c3a9.zip. 2022-08-08 10:08:21,310 INFO packaging.py:479 -- Creating a file package for local directory './'. ------------------------------------------------------- Job 'raysubmit_YVTpnmAzV7ysFmpk' submitted successfully ------------------------------------------------------- Next steps Query the logs of the job: ray job logs raysubmit_YVTpnmAzV7ysFmpk Query the status of the job: ray job status raysubmit_YVTpnmAzV7ysFmpk Request the job to be stopped: ray job stop raysubmit_YVTpnmAzV7ysFmpk Tailing logs until the job exits (disable with --no-wait): ``` ---
**Author:** ![rickyyx](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rickyyx/32/2671_2.png) [@rickyyx](https://discuss.ray.io/u/rickyyx)\ **Post date:** [August 9, 2022, 2:33am UTC](https://discuss.ray.io/t/how-does-container-in-runtime-env-work/7108/2 "2022-08-09T02:33:17Z")
Hey @jie, have your tried `ray job logs raysubmit_YVTpnmAzV7ysFmpk` to see if there’s any output? ```auto but the process doesn’t seem to stop. ``` Does it hang and stuck at the feedback you provided above? If you query the status of the job with `ray job status`, what’s the output there? A few more questions: 1. Which version of ray are you using? 2. Do you have ssh access to the head node? ---
**Author:** ![jie](https://avatars.discourse-cdn.com/v4/letter/j/3bc359/32.png) [@jie](https://discuss.ray.io/u/jie)\ **Post date:** [August 9, 2022, 3:07am UTC](https://discuss.ray.io/t/how-does-container-in-runtime-env-work/7108/3 "2022-08-09T03:07:22Z")
Hi @rickyyx thanks! Yes, it hang and stuck at that feedback. The output like this: ```auto $ ray job logs raysubmit_SpYXzcr6hVkq973i Job submission server address: None 2022-08-09 10:55:08,772 INFO dashboard_sdk.py:129 -- No address provided, defaulting to http://localhost:8265. $ ray job status raysubmit_SpYXzcr6hVkq973i Job submission server address: None 2022-08-09 10:55:36,855 INFO dashboard_sdk.py:129 -- No address provided, defaulting to http://localhost:8265. Status for job 'raysubmit_SpYXzcr6hVkq973i': PENDING Status message: Job has not started yet, likely waiting for the runtime_env to be set up. ``` And my ray version is `3.0.0.dev0`, I have the ssh access to head node. Moreover, I get some messages from `raylet.err` and `runtime_env_setup-01000000.log`: ```auto $ cat raylet.err bash: 第 0 行： exec: podman：未找到 [2022-08-09 10:26:25,244 E 494944 494944] (raylet) worker_pool.cc:500: Some workers of the worker process(506291) have not registered within the timeout. The process is dead, probably it crashed during start. $ cat runtime_env_setup-01000000.log 2022-08-09 10:27:25,248 INFO container.py:47 -- start worker in container with prefix: podman run -v /tmp/ray:/tmp/ray --cgroup-manager=cgroupfs --n etwork=host --pid=host --ipc=host --env-host --env RAY_RAYLET_PID=494944 --cap-drop SYS_ADMIN --log-level=debug --entrypoint python anyscale/ray-ml: nightly-py38-cpu ``` ---
**Author:** ![jjyao](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jjyao/32/1799_2.png) [@jjyao](https://discuss.ray.io/u/jjyao)\ **Post date:** [August 10, 2022, 6:13pm UTC](https://discuss.ray.io/t/how-does-container-in-runtime-env-work/7108/4 "2022-08-10T18:13:59Z")
> [@jie](#): > > ```auto > Some workers of the worker process(506291) have not registered within the timeout. The process is dead, probably it crashed during start. > > ``` Could you try to set `RAY_worker_register_timeout_seconds` env var to a larger time (e.g. 300) and see if it works? Also could you see the worker log to see if it actually crashes? ---
**Author:** ![architkulkarni](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/architkulkarni/32/10_2.png) [@architkulkarni](https://discuss.ray.io/u/architkulkarni)\ **Post date:** [August 10, 2022, 6:28pm UTC](https://discuss.ray.io/t/how-does-container-in-runtime-env-work/7108/5 "2022-08-10T18:28:30Z")
@GuyangSong can you help answer this user’s questions about `container`? ---
**Author:** ![jie](https://avatars.discourse-cdn.com/v4/letter/j/3bc359/32.png) [@jie](https://discuss.ray.io/u/jie)\ **Post date:** [August 11, 2022, 1:46am UTC](https://discuss.ray.io/t/how-does-container-in-runtime-env-work/7108/6 "2022-08-11T01:46:38Z")
Thanks @jjyao! In the past two days, I tried to solve this problem - I installed Podman locally, manually pulled the image before submitting the job, etc. But there is one new issue that is holding me back at the moment: ```auto time="2022-08-11T09:36:46+08:00" level=warning msg="Error validating CNI config file /home/wangjie/.config/cni/net.d/87-podman.conflist: [netplugin failed with no error mess age: fork/exec /opt/cni/bin/bridge: exec format error netplugin failed with no error message: fork/exec /opt/cni/bin/portmap: exec format error netplugin failed with no erro r message: fork/exec /opt/cni/bin/firewall: exec format error netplugin failed with no error message: fork/exec /opt/cni/bin/tuning: exec format error]" Error: executable file `python` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found ``` ---
**Author:** ![GuyangSong](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/guyangsong/32/181_2.png) [@GuyangSong](https://discuss.ray.io/u/GuyangSong)\ **Post date:** [August 11, 2022, 9:41am UTC](https://discuss.ray.io/t/how-does-container-in-runtime-env-work/7108/7 "2022-08-11T09:41:17Z")
I’m sorry to say that the container support is still experimental and has been broken unfortunately in the latest ray versions. We will enhance this part in few months and make a clear document for it. ---
**Author:** ![rahloff](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rahloff/32/4529_2.png) [@rahloff](https://discuss.ray.io/u/rahloff)\ **Post date:** [June 2, 2023, 4:30pm UTC](https://discuss.ray.io/t/how-does-container-in-runtime-env-work/7108/8 "2023-06-02T16:30:39Z")
Hi @GuyangSong, any update on this? I just tried to start a local Ray cluster via `ray.init` with a local image (for automated testing), and ran into the same issue: `Error: executable file `python` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found`