# Pythonpath on a kubernetes cluster

**URL:** <https://discuss.ray.io/t/pythonpath-on-a-kubernetes-cluster/2190>\
**Category:** Kubernetes\
**Created:** [May 18, 2021, 2:05pm UTC](https://discuss.ray.io/t/pythonpath-on-a-kubernetes-cluster/2190 "2021-05-18T14:05:50Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![delioda79](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/delioda79/32/312_2.png) [@delioda79](https://discuss.ray.io/u/delioda79)\
**Post date:** [May 18, 2021, 2:05pm UTC](https://discuss.ray.io/t/pythonpath-on-a-kubernetes-cluster/2190/1 "2021-05-18T14:05:50Z")

</div>

I’m having the same issue I see here:

> <https://github.com/ray-project/ray/issues/3116>
>
> \<!--
> General questions should be asked on the mailing list ray-dev@googlegroups….com.
> 
> Before submitting an issue, please fill out the following form.
> \--\>
> 
> \### System information
> \- \*\*OS Platform and Distribution (e.g., Linux Ubuntu 16.04)\*\*: Ubuntu 16.04
> \- \*\*Ray installed from (source or binary)\*\*: pip
> \- \*\*Ray version\*\*: 0.5.3
> \- \*\*Python version\*\*: 3.5.2
> \- \*\*Exact command to reproduce\*\*: python3 hello.py
> 
> \<!--
> You can obtain the Ray version with
> 
> python -c "import ray; print(ray.\_\_version\_\_)"
> \--\>
> 
> \### Describe the problem
> I have 2 AWS EC2 instances running with one as head and one as worker. Set up was done using the instructions for Manual cluster setup. I have two python files. One of them imports a function from the other and calls that function inside a \`remote\` function. Running it gives \`Exception: This function was not imported properly.\`. The set up runs fine if the second file is not imported or if there is only the head node in the cluster.
> 
> \### Source code / logs
> 
> Two files:
> \`\`\`
> \# hello.py
> 
> import ray
> import sys
> 
> ray.init(redis\_address="172.31.2.108:6379")
> 
> import time
> from testimport import sleep
> 
> @ray.remote
> def f():
> time.sleep(0.01)
> sleep(0.01)
> return "python version: %s, ip: %s" % (sys.version\_info, ray.services.get\_node\_ip\_address())
> 
> \# Get a list of the IP addresses of the nodes that have joined the cluster.
> print(set(ray.get(\[f.remote() for \_ in range(100)\])))
> \`\`\`
> \`\`\`
> \# testimport.py
> import time
> 
> def sleep(n):
> time.sleep(n)
> \`\`\`
> Command run:
> \`python3 hello.py\`
> 
> Output:
> 
> \`\`\`
> Traceback (most recent call last):
> File "/home/ubuntu/.local/lib/python3.5/site-packages/ray/import\_thread.py", line 131, in fetch\_and\_register\_remote\_function
> function = pickle.loads(serialized\_function)
> AttributeError: Can't get attribute 'sleep' on \<module 'testimport' from '/home/ubuntu/workspace/raytest/testimport.py'\>
> 
> Remote function \_\_main\_\_.f failed with:
> 
> Traceback (most recent call last):
> File "/home/ubuntu/.local/lib/python3.5/site-packages/ray/import\_thread.py", line 123, in f
> raise Exception("This function was not imported properly.")
> Exception: This function was not imported properly.
> 
> Suppressing duplicate error message.
> Suppressing duplicate error message.
> Suppressing duplicate error message.
> .
> .
> .
> Suppressing duplicate error message.
> Suppressing duplicate error message.
> Suppressing duplicate error message.
> Traceback (most recent call last):
> File "hello.py", line 16, in \<module\>
> print(set(ray.get(\[f.remote() for \_ in range(100)\])))
> File "/home/ubuntu/.local/lib/python3.5/site-packages/ray/worker.py", line 2514, in get
> raise RayGetError(object\_ids\[i\], value)
> ray.worker.RayGetError: Could not get objectid ObjectID(46fe73ad1bbfdc7c6293a0d80e570329a911ecf9). It was created by remote function \_\_main\_\_.f which failed with:
> 
> Remote function \_\_main\_\_.f failed with:
> 
> Traceback (most recent call last):
> File "/home/ubuntu/.local/lib/python3.5/site-packages/ray/import\_thread.py", line 123, in f
> raise Exception("This function was not imported properly.")
> Exception: This function was not imported properly.
> \`\`\`
> 
> 
> Commenting out \`sleep(0.01)\` from \`hello.py\` gives:
> \`{"python version: sys.version\_info(major=3, minor=5, micro=2, releaselevel='final', serial=0), ip: 172.31.2.108", "python version: sys.version\_info(major=3, minor=5, micro=2, releaselevel='final', serial=0), ip: 172.31.0.97"}\`
> So the set up is working for a single file with no imports.
> 
> Removing the worker node from the cluster gives:
> \`{"python version: sys.version\_info(major=3, minor=5, micro=2, releaselevel='final', serial=0), ip: 172.31.2.108"}\`
> So the import works on a single node cluster.
> 
> Seems like the import of the second file does not work when running on a cluster.

But I cannot understand how to make it work on a k8s cluster. I mean spinning up the cluster some pods are created, and I have some other external pod just connecting to the cluster…now how can I install the packages I need in all these machines?

---

<div class="post-metadata">

**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\
**Post date:** [May 18, 2021, 3:24pm UTC](https://discuss.ray.io/t/pythonpath-on-a-kubernetes-cluster/2190/2 "2021-05-18T15:24:15Z")

</div>

I’d recommend using a Dockerfile to build images with the packages installed, using the ray images as the base.

---

<div class="post-metadata">

**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\
**Post date:** [May 18, 2021, 3:29pm UTC](https://discuss.ray.io/t/pythonpath-on-a-kubernetes-cluster/2190/3 "2021-05-18T15:29:44Z")

</div>

There’s ongoing work for Ray to internally sync packages and environments among nodes, but it’s not quite GA yet  
@architkulkarni or @eoakes or yi c. (can’t find his @ handle) might know more about the status of that project

---

<div class="post-metadata">

**Author:** ![delioda79](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/delioda79/32/312_2.png) [@delioda79](https://discuss.ray.io/u/delioda79)\
**Post date:** [May 18, 2021, 7:00pm UTC](https://discuss.ray.io/t/pythonpath-on-a-kubernetes-cluster/2190/4 "2021-05-18T19:00:58Z")

</div>

Oh OK seems good like that, did not thin it, thanks!