# Failed to lease worker from node **URL:** \ **Category:** Ray Core\ **Created:** [February 21, 2022, 3:53pm UTC](https://discuss.ray.io/t/failed-to-lease-worker-from-node/5140 "2022-02-21T15:53:50Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![pamparana](https://avatars.discourse-cdn.com/v4/letter/p/df788c/32.png) [@pamparana](https://discuss.ray.io/u/pamparana)\ **Post date:** [February 21, 2022, 3:53pm UTC](https://discuss.ray.io/t/failed-to-lease-worker-from-node/5140/1 "2022-02-21T15:53:50Z")
I have been struggling for days with ray trying to basically import pytorch and close to giving up and finding another solution. Basically my submission is as follows: ```auto ray job submit --runtime-env-json='{"working_dir": "./", "pip": ["requests==2.25.1", "torch==1.10.2", "torchvision==0.11.3"]}' -- "python test_ray_script.py" Job submission server address: http://127.0.0.1:8265 2022-02-21 16:39:29,898 INFO sdk.py:189 -- Uploading package gcs://_ray_pkg_cc1f90b5cd6565c7.zip. 2022-02-21 16:39:29,900 INFO packaging.py:353 -- Creating a file package for local directory './'. ``` It usually comes back with: ```auto ------------------------------------------------------- Job 'raysubmit_a1GgQCA5jvemK1FR' submitted successfully ------------------------------------------------------- Next steps Query the logs of the job: ray job logs raysubmit_a1GgQCA5jvemK1FR Query the status of the job: ray job status raysubmit_a1GgQCA5jvemK1FR Request the job to be stopped: ray job stop raysubmit_a1GgQCA5jvemK1FR Tailing logs until the job exits (disable with --no-wait): --------------------------------------- Job 'raysubmit_a1GgQCA5jvemK1FR' failed --------------------------------------- ``` Logs have been pretty unhelpful so far. Looking at `gcs_server.err`, I see something like: ```auto 2022-02-21 07:25:20,785 E 161 161] gcs_actor_scheduler.cc:316: Failed to lease worker from node cac9e236ed0dac867f7a77d4f9b953e20fbdf42b535be63dbc5466ef for actor 3f78dc477a4de988d752876302000000(Counter. __init__ ) as the runtime environment setup failed, job id = 02000000 [2022-02-21 07:29:22,805 E 161 161] gcs_actor_scheduler.cc:316: Failed to lease worker from node cac9e236ed0dac867f7a77d4f9b953e20fbdf42b535be63dbc5466ef for actor 2f36444a140ac223e8c1d28a01000000(JobSupervisor. __init__ ) as the runtime environment setup failed, job id = 01000000 [2022-02-21 07:40:03,152 E 161 161] gcs_actor_scheduler.cc:316: Failed to lease worker from node cac9e236ed0dac867f7a77d4f9b953e20fbdf42b535be63dbc5466ef for actor 2ea9d20139d71f06e51fe3bb01000000(JobSupervisor. __init__ ) as the runtime environment setup failed, job id = 01000000 ``` Could this be something or is it because of the previous error? Looking at the log of job id=010000000 ```auto 2022-02-21 07:39:53,109 INFO conda_utils.py:198 -- Using cached requests-2.25.1-py2.py3-none-any.whl (61 kB) 2022-02-21 07:39:53,185 INFO conda_utils.py:198 -- Collecting torch==1.10.2 2022-02-21 07:40:02,147 INFO conda_utils.py:198 -- ``` Looking at raylet.err: ```auto Using cached requests-2.25.1-py2.py3-none-any.whl (61 kB) Collecting torch [2022-02-21 07:40:03,152 E 218 218] agent_manager.cc:190: Failed to create runtime env: {"workingDir":"gcs://_ray_pkg_cc1f90b5cd6565c7.zip","uris":{"workingDirUri":"gcs://_ray_pkg_cc1f90b5cd6565c7.zip","pipUri":"pip://23ada401a940d4ca08aab4cb77421f32396eeb8b"},"extensions":{"_ray_commit":"5ea565317a8104c04ae7892bb9bb41c6d72f12df"},"pipRuntimeEnv":{"config":{"packages":["requests==2.25.1","torch==1.10.2","torchvision==0.11.3"]}}}, error message: Failed to install pip requirements: Collecting requests==2.25.1 Using cached requests-2.25.1-py2.py3-none-any.whl (61 kB) Collecting torch==1.10.2 ``` Again, says cannot install package but not really what went wrong. I have spent 3 days on this without making much progress and at this point close to trying some other tool. ---
**Author:** ![yic](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/yic/32/437_2.png) [@yic](https://discuss.ray.io/u/yic)\ **Post date:** [February 22, 2022, 6:47pm UTC](https://discuss.ray.io/t/failed-to-lease-worker-from-node/5140/2 "2022-02-22T18:47:51Z")
From the log, it seems that runtime env can’t be set up here. It seems like the pip failed to be installed. Could you check the logs of `ray_client_server.[err|out]`? cc @architkulkarni ---
**Author:** ![yic](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/yic/32/437_2.png) [@yic](https://discuss.ray.io/u/yic)\ **Post date:** [February 22, 2022, 6:53pm UTC](https://discuss.ray.io/t/failed-to-lease-worker-from-node/5140/3 "2022-02-22T18:53:03Z")
I come across this post [Cannot setup the runtime env - #9 by architkulkarni](https://discuss.ray.io/t/cannot-setup-the-runtime-env/5105/9) It seems like the same issue. Could you follow the instruction there? ---
**Author:** ![architkulkarni](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/architkulkarni/32/10_2.png) [@architkulkarni](https://discuss.ray.io/u/architkulkarni)\ **Post date:** [February 22, 2022, 6:53pm UTC](https://discuss.ray.io/t/failed-to-lease-worker-from-node/5140/4 "2022-02-22T18:53:15Z")
Is this the same issue as [Cannot setup the runtime env - #9 by architkulkarni](https://discuss.ray.io/t/cannot-setup-the-runtime-env/5105/9)? Could you check the latest response there? It shows that installing `pytorch` requires an extra pip option. Another idea, which is more of a guess: it looks like the `pip install` is failing in the middle. Usually the logs say `Killed` when this happens, I’m not sure why it didn’t show up in the logs though. Maybe try increasing the memory on the node? [python - pip install - killed - Stack Overflow](https://stackoverflow.com/questions/30550235/pip-install-killed) ---
**Author:** ![architkulkarni](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/architkulkarni/32/10_2.png) [@architkulkarni](https://discuss.ray.io/u/architkulkarni)\ **Post date:** [February 22, 2022, 6:56pm UTC](https://discuss.ray.io/t/failed-to-lease-worker-from-node/5140/5 "2022-02-22T18:56:19Z")
As another option, if you are okay with manually preinstalling pytorch on the cluster in advance instead of dynamically installing it per-job using `runtime_env`, you can do that as instead, and `import pytorch` will still work in your Ray script. ---
**Author:** ![Manoj\_Kumar\_Dobbali](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/manoj_kumar_dobbali/32/2172_2.png) [@Manoj\_Kumar\_Dobbali](https://discuss.ray.io/u/Manoj_Kumar_Dobbali)\ **Post date:** [February 22, 2022, 7:33pm UTC](https://discuss.ray.io/t/failed-to-lease-worker-from-node/5140/6 "2022-02-22T19:33:33Z")
@pamparana Adding to the alternatives, installing pytorch with conda and rest of the requirements using pip seems to work for me ```auto runtime_env = ( { "conda": {"dependencies": ["pytorch", "torchvision", {"pip": [list_of_reqs or path_to_reqs.txt file]}]}, ... .. ``` ---
**Author:** ![Manoj\_Kumar\_Dobbali](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/manoj_kumar_dobbali/32/2172_2.png) [@Manoj\_Kumar\_Dobbali](https://discuss.ray.io/u/Manoj_Kumar_Dobbali)\ **Post date:** [February 22, 2022, 7:39pm UTC](https://discuss.ray.io/t/failed-to-lease-worker-from-node/5140/7 "2022-02-22T19:39:04Z")
@architkulkarni By installing on cluster in advance, Do you mean adding it to Ray Docker image? I have built a custom docker image on top of Ray image as following ```auto ARG BASE_IMAGE=rayproject/ray:1.10.0 FROM $BASE_IMAGE RUN pip install --no-cache-dir torch==1.10.2+cpu -f https://download.pytorch.org/whl/torch_stable.html RUN pip install pytorch-lightning==1.5.0 ``` and removed torch installation from requirements.txt so that runtime env will be setup without error but I still got `ModuleNotFoundError`. It looked like the Ray client code is compiled before using the Docker image? ---
**Author:** ![architkulkarni](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/architkulkarni/32/10_2.png) [@architkulkarni](https://discuss.ray.io/u/architkulkarni)\ **Post date:** [February 22, 2022, 7:57pm UTC](https://discuss.ray.io/t/failed-to-lease-worker-from-node/5140/8 "2022-02-22T19:57:12Z")
Thanks for adding the alternative! Hmm, building it into the Docker image should definitely work. The Ray code is just Python, so there’s no compilation that happens before or after the Docker image. Does `import pytorch` work on the cluster as expected (without using Ray)? ---
**Author:** ![architkulkarni](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/architkulkarni/32/10_2.png) [@architkulkarni](https://discuss.ray.io/u/architkulkarni)\ **Post date:** [February 22, 2022, 7:59pm UTC](https://discuss.ray.io/t/failed-to-lease-worker-from-node/5140/9 "2022-02-22T19:59:43Z")
(To add more detail, Ray just calls `pip install`, and Ray is already running in the Docker container, so the new packages would just be installed in the container at runtime but the existing packages should still be importable (see this PR which added this functionality [[runtime\_env] Make pip installs incremental by edoakes · Pull Request #20341 · ray-project/ray · GitHub](https://github.com/ray-project/ray/pull/20341)) ---
**Author:** ![Manoj\_Kumar\_Dobbali](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/manoj_kumar_dobbali/32/2172_2.png) [@Manoj\_Kumar\_Dobbali](https://discuss.ray.io/u/Manoj_Kumar_Dobbali)\ **Post date:** [February 23, 2022, 6:17am UTC](https://discuss.ray.io/t/failed-to-lease-worker-from-node/5140/10 "2022-02-23T06:17:56Z")
Thanks @architkulkarni . Yeah. Docker is doing as expected. My deployment is on k8s and I missed updating operator image to the updated Docker image. once it is changed, it is working as expected.