# Cannot setup the runtime env

**URL:** <https://discuss.ray.io/t/cannot-setup-the-runtime-env/5105>\
**Category:** Ray Core\
**Created:** [February 17, 2022, 12:16pm UTC](https://discuss.ray.io/t/cannot-setup-the-runtime-env/5105 "2022-02-17T12:16:25Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![pamparana](https://avatars.discourse-cdn.com/v4/letter/p/df788c/32.png) [@pamparana](https://discuss.ray.io/u/pamparana)\
**Post date:** [February 17, 2022, 12:16pm UTC](https://discuss.ray.io/t/cannot-setup-the-runtime-env/5105/1 "2022-02-17T12:16:25Z")

</div>

So, I am just trying to setup the run time env on the worker nodes and I submit the job using:

```auto
ray job submit --runtime-env-json='{"working_dir": "./", "pip": ["requests==2.26.0", "torch==1.10.2"]}' -- "python test_ray_script.py"

```

And this comes back with:

```auto
-------------------------------------------------------
Job 'raysubmit_7wziNAjmKgDLXtat' submitted successfully
-------------------------------------------------------

```

and then it fails with:

```auto
---------------------------------------
Job 'raysubmit_7wziNAjmKgDLXtat' failed
---------------------------------------

Status message: runtime_env setup failed: The runtime_env failed to be set up.

```

The logs generated are not much helpful either.

When I run it without `"torch==1.10.2"` dependency, it is ok. I can install torch fine by connecting to the pod as:

```auto
kubectl attach test-pod -c test-pod -i -t -n ray
pip install torch==1.10.2

```

---

<div class="post-metadata">

**Author:** ![architkulkarni](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/architkulkarni/32/10_2.png) [@architkulkarni](https://discuss.ray.io/u/architkulkarni)\
**Post date:** [February 17, 2022, 5:33pm UTC](https://discuss.ray.io/t/cannot-setup-the-runtime-env/5105/2 "2022-02-17T17:33:25Z")

</div>

Hi @pamparana, which Ray version are you using? Also, are you able to access the logs on the head node of your cluster? By default they are at `/tmp/ray/session_latest/logs`. There you may find `dashboard_agent.log` or other runtime env setup logs which should contain the traceback from `pip`.

---

<div class="post-metadata">

**Author:** ![pamparana](https://avatars.discourse-cdn.com/v4/letter/p/df788c/32.png) [@pamparana](https://discuss.ray.io/u/pamparana)\
**Post date:** [February 18, 2022, 1:32am UTC](https://discuss.ray.io/t/cannot-setup-the-runtime-env/5105/3 "2022-02-18T01:32:18Z")

</div>

I am on ray 1.10.0. Same as the cloud deployment.

So the error message is not the most helpful. Looking into `raylet.err` on the head node, I have:

```auto
[2022-02-17 17:25:47,224 E 185 185] agent_manager.cc:190: Failed to create runtime env: {"workingDir":"gcs://_ray_pkg_afd927254bbfea17.zip","uris":{"workingDirUri":"gcs://_ray_pkg_afd927254bbfea17.zip","pipUri":"pip://468bfc785de311165ce6b9a0297743fe9c87bdb1"},"extensions":{"_ray_commit":"5ea565317a8104c04ae7892bb9bb41c6d72f12df"},"pipRuntimeEnv":{"config":{"packages":["requests==2.25.1","torch==1.10.2"]}}}, error message: Failed to install pip requirements:
Collecting requests==2.25.1
Using cached requests-2.25.1-py2.py3-none-any.whl (61 kB)
Collecting torch==1.10.2

```

It does not really point to what actually failed. Not sure where else to look. This was the most useful message…

Looking into the worker node, I see this:

```auto
cat raylet.err 
[2022-02-17 04:30:03,988 E 128 128] agent_manager.cc:231: Failed to delete URIs, error message: Local files for URI(s) ['gcs://_ray_pkg_bd5d1281efde2694.zip'] not found.
[2022-02-17 04:30:08,852 E 128 128] agent_manager.cc:190: Failed to create runtime env: {"extensions": {"_ray_commit": "5ea565317a8104c04ae7892bb9bb41c6d72f12df"}, "pipRuntimeEnv": {"config": {"packages": ["requests==2.26.0", "torch==1.10.1"]}}, "uris": {"pipUri": "pip://117f8e5b8c624ffefb6479d46f89922b934ca89e", "workingDirUri": "gcs://_ray_pkg_bd5d1281efde2694.zip"}, "workingDir": "gcs://_ray_pkg_bd5d1281efde2694.zip"}, error message: [Errno 2] No such file or directory: '/tmp/ray/session_2022-02-17_03-41-34_828948_115/runtime_resources/pip/117f8e5b8c624ffefb6479d46f89922b934ca89e'
[2022-02-17 04:30:08,853 E 128 128] worker_pool.cc:623: [Eagerly] Couldn't create a runtime environment for job 03000000.
[2022-02-17 04:30:08,853 E 128 128] agent_manager.cc:190: Failed to create runtime env: {"extensions": {"_ray_commit": "5ea565317a8104c04ae7892bb9bb41c6d72f12df"}, "pipRuntimeEnv": {"config": {"packages": ["requests==2.26.0", "torch==1.10.1"]}}, "uris": {"pipUri": "pip://117f8e5b8c624ffefb6479d46f89922b934ca89e", "workingDirUri": "gcs://_ray_pkg_bd5d1281efde2694.zip"}, "workingDir": "gcs://_ray_pkg_bd5d1281efde2694.zip"}, error message: [Errno 2] No such file or directory: '/tmp/ray/session_2022-02-17_03-41-34_828948_115/runtime_resources/pip/117f8e5b8c624ffefb6479d46f89922b934ca89e'
[2022-02-17 04:30:08,853 E 128 128] worker_pool.cc:623: [Eagerly] Couldn't create a runtime environment for job 03000000.

```

---

<div class="post-metadata">

**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\
**Post date:** [February 19, 2022, 2:15pm UTC](https://discuss.ray.io/t/cannot-setup-the-runtime-env/5105/4 "2022-02-19T14:15:16Z")

</div>

There must be a log file under log directory that contains runtime env specific log `runtime_env_setup-[job_id].log` → I believe it is this

---

<div class="post-metadata">

**Author:** ![pamparana](https://avatars.discourse-cdn.com/v4/letter/p/df788c/32.png) [@pamparana](https://discuss.ray.io/u/pamparana)\
**Post date:** [February 20, 2022, 7:19am UTC](https://discuss.ray.io/t/cannot-setup-the-runtime-env/5105/5 "2022-02-20T07:19:26Z")

</div>

This log file on the head node, unfortunately does not give much information. I see:

```auto
2022-02-18 05:28:53,472 INFO conda_utils.py:198 -- Collecting requests==2.25.1
2022-02-18 05:28:53,490 INFO conda_utils.py:198 -- Using cached requests-2.25.1-py2.py3-none-any.whl (61 kB)
2022-02-18 05:28:53,625 INFO conda_utils.py:198 -- Collecting torch==1.10.2

```

---

<div class="post-metadata">

**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\
**Post date:** [February 21, 2022, 12:57am UTC](https://discuss.ray.io/t/cannot-setup-the-runtime-env/5105/6 "2022-02-21T00:57:01Z")

</div>

Can you also check `dashboard_agent.log`?

---

<div class="post-metadata">

**Author:** ![pamparana](https://avatars.discourse-cdn.com/v4/letter/p/df788c/32.png) [@pamparana](https://discuss.ray.io/u/pamparana)\
**Post date:** [February 21, 2022, 2:56pm UTC](https://discuss.ray.io/t/cannot-setup-the-runtime-env/5105/7 "2022-02-21T14:56:11Z")

</div>

That does not show anything special either.

```auto
2022-02-19 23:34:33,609 INFO agent.py:191 -- Dashboard agent http address: 0.0.0.0:41335
2022-02-19 23:34:33,609 INFO agent.py:199 -- <ResourceRoute [GET] <StaticResource /logs -> PosixPath('/tmp/ray/session_2022-02-19_23-34-29_285231_420/logs')> -> <bound method StaticResource._handle of <StaticResource /logs -> PosixPath('/tmp/ray/session_2022-02-19_23-34-29_285231_420/logs')>>
2022-02-19 23:34:33,609 INFO agent.py:199 -- <ResourceRoute [OPTIONS] <StaticResource /logs -> PosixPath('/tmp/ray/session_2022-02-19_23-34-29_285231_420/logs')> -> <bound method _PreflightHandler._preflight_handler of <aiohttp_cors.cors_config._CorsConfigImpl object at 0x7feb3ed96160>>
2022-02-19 23:34:33,609 INFO agent.py:200 -- Registered 2 routes.

```

---

<div class="post-metadata">

**Author:** ![Manoj\_Kumar\_Dobbali](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/manoj_kumar_dobbali/32/2172_2.png) [@Manoj\_Kumar\_Dobbali](https://discuss.ray.io/u/Manoj_Kumar_Dobbali)\
**Post date:** [February 22, 2022, 5:44pm UTC](https://discuss.ray.io/t/cannot-setup-the-runtime-env/5105/8 "2022-02-22T17:44:28Z")

</div>

I am facing similar issue, looks like torch installation is failing because installation command for torch on linux machine is as follows,

`pip3 install torch==1.10.2+cpu torchvision==0.11.3+cpu torchaudio==0.10.2+cpu -f https://download.pytorch.org/whl/cpu/torch_stable.html` . Source : [Start Locally | PyTorch](https://pytorch.org/get-started/locally/)

I am not sure how to add this line to requirements.txt or runtime\_env

---

<div class="post-metadata">

**Author:** ![architkulkarni](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/architkulkarni/32/10_2.png) [@architkulkarni](https://discuss.ray.io/u/architkulkarni)\
**Post date:** [February 22, 2022, 6:07pm UTC](https://discuss.ray.io/t/cannot-setup-the-runtime-env/5105/9 "2022-02-22T18:07:56Z")

</div>

Ah thanks for finding this! It’s too bad that `pip` doesn’t show an error message in this case (it seems to hang at ` Collecting torch==1.10.2` according to the traceback posted earlier.)

> I am not sure how to add this line to requirements.txt or runtime\_env

[python - How to format requirements.txt when package source is from specific websites? - Stack Overflow](https://stackoverflow.com/a/61742742/14651138) has a detailed answer; you can just include it as a line in requirements.txt.

Note that we unfortunately had a regression for including such options in `requirements.txt` in Ray 1.10 ([[Bug] Python: --extra-index-url does not work in requirements file anymore · Issue #22056 · ray-project/ray · GitHub](https://github.com/ray-project/ray/issues/22056)), so you would need to use Ray 1.9 or earlier or use Ray 1.11.0rc0 (`pip install ray[default]==1.11.0rc0`) for this to work, or use a nightly build that contains the bugfix. Alternatively, you can set the `-f` `--find-links` flag using an environment variable: [Configuration - pip documentation v23.3.1](https://pip.pypa.io/en/stable/topics/configuration/#environment-variables). But note that this environment variable should be set on the cluster before Ray is started (e.g. with `ray start`), otherwise the runtime env pip process won’t see it.

---

<div class="post-metadata">

**Author:** ![Manoj\_Kumar\_Dobbali](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/manoj_kumar_dobbali/32/2172_2.png) [@Manoj\_Kumar\_Dobbali](https://discuss.ray.io/u/Manoj_Kumar_Dobbali)\
**Post date:** [February 22, 2022, 7:29pm UTC](https://discuss.ray.io/t/cannot-setup-the-runtime-env/5105/10 "2022-02-22T19:29:56Z")

</div>

Thanks @architkulkarni . Adding to the alternatives, installing pytorch with conda and rest of the requirements using pip seems to work for me

```auto
runtime_env = (
            { "conda": {"dependencies": ["pytorch", {"pip": [list_of_reqs or path_to_reqs.txt file]}]},
             ...
            ..

```