# K8s - submitting jobs with python deps

**URL:** <https://discuss.ray.io/t/k8s-submitting-jobs-with-python-deps/2442>\
**Category:** Ray Clusters\
**Created:** [June 7, 2021, 9:40pm UTC](https://discuss.ray.io/t/k8s-submitting-jobs-with-python-deps/2442 "2021-06-07T21:40:00Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![blu](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/blu/32/1099_2.png) [@blu](https://discuss.ray.io/u/blu)\
**Post date:** [June 7, 2021, 9:40pm UTC](https://discuss.ray.io/t/k8s-submitting-jobs-with-python-deps/2442/1 "2021-06-07T21:40:00Z")

</div>

Trying to find examples of submitting a k8s job that runs a custom ray program using multiprocessing. Getting ModuleNotFoundError on ‘mycustompackage’ due to package not on worker nodes. Seeing mentions of file mounts but examples would be appreciated. Also how do we solve for if 'mycustompackage` is updated in future?

---

<div class="post-metadata">

**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\
**Post date:** [June 8, 2021, 6:38pm UTC](https://discuss.ray.io/t/k8s-submitting-jobs-with-python-deps/2442/2 "2021-06-08T18:38:19Z")

</div>

Hi! For now, the best officially available solution is to build your dependencies into the image you’re using.  
In the near future, the preferred way to do this will be via a feature called `runtime environments`  
An experimental version of this functionality exists in Ray 1.4.0 but is not yet documented.  
I recommend bugging the maintainers of runtime envs on this thread: [Dependency management with ray for remote task execution - #3 by asm582](https://discuss.ray.io/t/dependency-management-with-ray-for-remote-task-execution/1479/3)

---

<div class="post-metadata">

**Author:** ![blu](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/blu/32/1099_2.png) [@blu](https://discuss.ray.io/u/blu)\
**Post date:** [June 8, 2021, 8:23pm UTC](https://discuss.ray.io/t/k8s-submitting-jobs-with-python-deps/2442/3 "2021-06-08T20:23:06Z")

</div>

Thanks Dmitri!

We have the dependency in the image for the k8s job but how do we get that propagated to worker nodes. Or do you mean we add the dependencies to the ray image we use in the [cluster](https://github.com/ray-project/ray/blob/master/deploy/charts/ray/templates/raycluster.yaml#L45)?

---

<div class="post-metadata">

**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\
**Post date:** [June 8, 2021, 9:17pm UTC](https://discuss.ray.io/t/k8s-submitting-jobs-with-python-deps/2442/4 "2021-06-08T21:17:03Z")

</div>

yes, exactly – until runtime envs are available, client and server ends for Ray client must have matching dependencies installed, so you’d need to add the dependencies to the image used in the cluster