# Dependency management with ray for remote task execution

**URL:** <https://discuss.ray.io/t/dependency-management-with-ray-for-remote-task-execution/1479>\
**Category:** Kubernetes\
**Created:** [March 30, 2021, 11:36pm UTC](https://discuss.ray.io/t/dependency-management-with-ray-for-remote-task-execution/1479 "2021-03-30T23:36:44Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![asm582](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/asm582/32/669_2.png) [@asm582](https://discuss.ray.io/u/asm582)\
**Post date:** [March 30, 2021, 11:36pm UTC](https://discuss.ray.io/t/dependency-management-with-ray-for-remote-task-execution/1479/1 "2021-03-30T23:36:44Z")

</div>

Hello, I have a question regarding dependency management: if a user spawns a task on the ray remote cluster, how are the binaries required to run the task copied from the local laptop to the remote ray cluster? is there any documentation from a dependency management perspective for remote tasks?

---

<div class="post-metadata">

**Author:** ![yic](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/yic/32/437_2.png) [@yic](https://discuss.ray.io/u/yic)\
**Post date:** [March 31, 2021, 6:22pm UTC](https://discuss.ray.io/t/dependency-management-with-ray-for-remote-task-execution/1479/2 "2021-03-31T18:22:59Z")

</div>

Right now, all ray nodes should be homogeneous, which means that you need them running in the same environment.

For script functions, it’ll be picked to run in remote node, but for other dependency it’s not done automatically, which means you need to copy it there by yourself.

We are putting an effort to let user specify a runtime environment for each job so this can be done automatically. It’s not ready yet, but should be ready to use within April. You can check it later if you are interested in it.

---

<div class="post-metadata">

**Author:** ![asm582](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/asm582/32/669_2.png) [@asm582](https://discuss.ray.io/u/asm582)\
**Post date:** [March 31, 2021, 8:35pm UTC](https://discuss.ray.io/t/dependency-management-with-ray-for-remote-task-execution/1479/3 "2021-03-31T20:35:48Z")

</div>

Thank you @yic, April is not too far! I am looking forward to trying out this feature.

---

<div class="post-metadata">

**Author:** ![asm582](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/asm582/32/669_2.png) [@asm582](https://discuss.ray.io/u/asm582)\
**Post date:** [April 29, 2021, 6:22pm UTC](https://discuss.ray.io/t/dependency-management-with-ray-for-remote-task-execution/1479/4 "2021-04-29T18:22:54Z")

</div>

Hello @yic , can you please share if dependency management with conda is supported on ray?

Thanks

---

<div class="post-metadata">

**Author:** ![yic](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/yic/32/437_2.png) [@yic](https://discuss.ray.io/u/yic)\
**Post date:** [May 18, 2021, 9:06pm UTC](https://discuss.ray.io/t/dependency-management-with-ray-for-remote-task-execution/1479/5 "2021-05-18T21:06:43Z")

</div>

Hi @asm582 sorry for the delay!  
The project got delayed. It’ll come with the 1.4.0 release which is planed for the end of this month (or early next month).

Thanks  
Yi Cheng.

---

<div class="post-metadata">

**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\
**Post date:** [June 8, 2021, 6:41pm UTC](https://discuss.ray.io/t/dependency-management-with-ray-for-remote-task-execution/1479/6 "2021-06-08T18:41:48Z")

</div>

This thread seems like a good place to chat about runtime envs and their relationship to Ray deployments on k8s – lots of questions from k8s users on dependency management recently.  
cc @yic @eoakes @architkulkarni 😃

---

<div class="post-metadata">

**Author:** ![asm582](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/asm582/32/669_2.png) [@asm582](https://discuss.ray.io/u/asm582)\
**Post date:** [June 9, 2021, 10:03pm UTC](https://discuss.ray.io/t/dependency-management-with-ray-for-remote-task-execution/1479/7 "2021-06-09T22:03:27Z")

</div>

Hello, is this feature available in ray 1.4?

---

<div class="post-metadata">

**Author:** ![yic](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/yic/32/437_2.png) [@yic](https://discuss.ray.io/u/yic)\
**Post date:** [June 11, 2021, 5:40am UTC](https://discuss.ray.io/t/dependency-management-with-ray-for-remote-task-execution/1479/8 "2021-06-11T05:40:16Z")

</div>

Hi @asm582 we just released 1.4.0, please check the document here [Advanced Usage — Ray v2.0.0.dev0](https://docs.ray.io/en/master/advanced.html#runtime-environments-experimental)

Note, this is still an experimental feature, and welcome any feedback. cc @architkulkarni

---

<div class="post-metadata">

**Author:** ![asm582](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/asm582/32/669_2.png) [@asm582](https://discuss.ray.io/u/asm582)\
**Post date:** [July 20, 2021, 3:39pm UTC](https://discuss.ray.io/t/dependency-management-with-ray-for-remote-task-execution/1479/9 "2021-07-20T15:39:57Z")

</div>

Hello,

Question with respect to dependency management of ray operator and ray nodes, do we have some sort of compatibility check if operator image is compatible with ray nodes when users build images from ray source?

Thanks

---

<div class="post-metadata">

**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\
**Post date:** [July 22, 2021, 2:48am UTC](https://discuss.ray.io/t/dependency-management-with-ray-for-remote-task-execution/1479/10 "2021-07-22T02:48:40Z")

</div>

There’s no explicit check — but as of Ray 1.2.0 the interfaces are designed for cross-compatibility of scaling functionality with core ray functionality.

You’d have to modify very specific code to break compatibility — I think the relevant protobufs are labelled with “do not change” warnings.
