# Ray submit fails on EKS, runtime\_env creation error

**URL:** <https://discuss.ray.io/t/ray-submit-fails-on-eks-runtime-env-creation-error/4701>\
**Category:** Kubernetes\
**Created:** [January 13, 2022, 10:57pm UTC](https://discuss.ray.io/t/ray-submit-fails-on-eks-runtime-env-creation-error/4701 "2022-01-13T22:57:35Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![lihost](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/lihost/32/1896_2.png) [@lihost](https://discuss.ray.io/u/lihost)\
**Post date:** [January 13, 2022, 10:57pm UTC](https://discuss.ray.io/t/ray-submit-fails-on-eks-runtime-env-creation-error/4701/1 "2022-01-13T22:57:35Z")

</div>

Hi Team,

I have deployed Ray on EKS using helm charts. I am also specifying pip `runtime_env` during `init`.

This is how I am initializing.

```auto
ray.init(address="ray://example-cluster-ray-head:10001", namespace="ray", runtime_env={"pip": ["ray[default]", "ray[serve]", "flair==0.9", "nltk==3.6.2"]})

```

  
  
  
Here is the complete traceback.

```auto
❯ ray submit example-full.yaml FlairModel.py
2022-01-13 14:23:54,613	INFO util.py:282 -- setting max workers for head node type to 0
Loaded cached provider configuration
If you experience issues with the cloud provider, try re-running the command with --no-config-cache.
2022-01-13 14:24:06,024	INFO util.py:282 -- setting max workers for head node type to 0
2022-01-13 14:24:06,148	INFO command_runner.py:172 -- NodeUpdater: example-cluster-ray-head-type-s59z8: Running kubectl -n ray exec -it example-cluster-ray-head-type-s59z8 -- bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (python ~/FlairModel.py)'
Traceback (most recent call last):
  File "/home/ray/FlairModel.py", line 2, in <module>
    ray.init(address="ray://example-cluster-ray-head:10001", namespace="ray", runtime_env={"pip": ["ray[default]", "ray[serve]", "flair==0.9", "nltk==3.6.2"]})
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
    return func(*args, **kwargs)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 775, in init
    return builder.connect()
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/client_builder.py", line 155, in connect
    ray_init_kwargs=self._remote_init_kwargs)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/util/client_connect.py", line 42, in connect
    ray_init_kwargs=ray_init_kwargs)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/util/client/ __init__.py", line 228, in connect
    conn = self.get_context().connect(*args, **kw_args)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/util/client/ __init__.py", line 88, in connect
    self.client_worker._server_init(job_config, ray_init_kwargs)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/util/client/worker.py", line 698, in _server_init
    f"Initialization failure from server:\n{response.msg}")
ConnectionAbortedError: Initialization failure from server:
Traceback (most recent call last):
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/util/client/server/proxier.py", line 624, in Datapath
    client_id, job_config):
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/util/client/server/proxier.py", line 281, in start_specific_server
    specific_server=specific_server,
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/util/client/server/proxier.py", line 234, in _create_runtime_env
    "Failed to create runtime_env for Ray client "
RuntimeError: Failed to create runtime_env for Ray client server: [Errno 2] No such file or directory: '/tmp/ray/session_2022-01-12_16-25-08_622146_119/runtime_resources/conda/6aa502b772617ee2bec7c93351970b6ed85aa479'

command terminated with exit code 1
2022-01-13 14:28:45,460	ERROR command_runner.py:182 -- NodeUpdater: example-cluster-ray-head-type-s59z8: Command failed:

  kubectl -n ray exec -it example-cluster-ray-head-type-s59z8 --'bash --login -c -i '"'"'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (python ~/FlairModel.py)'"'"''

```

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![lihost](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/lihost/32/1896_2.png) [@lihost](https://discuss.ray.io/u/lihost)\
**Post date:** [January 15, 2022, 12:29am UTC](https://discuss.ray.io/t/ray-submit-fails-on-eks-runtime-env-creation-error/4701/2 "2022-01-15T00:29:19Z")

</div>

Based on errors it says `ConnectionAbortedError: Initialization failure from server` but this is failing at exactly the line where I have mentioned `runtime_env`.

I logged in to head-node to install these libraries and I could not install because of unavailability of network or proxy.

I am not sure if my understanding is correct or there is any other issue.

---

<div class="post-metadata">

**Author:** ![ckw017](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/ckw017/32/1683_2.png) [@ckw017](https://discuss.ray.io/u/ckw017)\
**Post date:** [January 19, 2022, 12:48am UTC](https://discuss.ray.io/t/ray-submit-fails-on-eks-runtime-env-creation-error/4701/3 "2022-01-19T00:48:21Z")

</div>

Hi, are you saying that the head node itself doesn’t have network access? If that’s the case, the runtime env might not be able to download the necessary dependencies which is causing the error you’re seeing.