Using py_modules URI to set up dependencies per task failed for pypng package

Ray Core
1

Hi, I appreciate your help with the following issue:
We’re using runtime_env to set up dependencies via py_modules per task, with a URI to a remote zip file. We’re doing so for all our required packages and its 3rd party dependencies.

we encountered the following issue, while trying to pass 3rd party packages such as ‘pypng’ via URI:

ModuleNotFoundError: No module named ‘png’

Installation of ‘pypng’ generates 3 folders (‘pycache’, ‘bin’, and ‘pypng-0.0.21.dist-info’) and one single file png.py.

While passing directory that contains png.py file via URI (as a zip), we see that the directory was successfully unpacked to /tmp/ray/session_latest/runtime_resources and contains the file as expected:

From the container:
root@ray-algo-plugin-worker1-7d4f5658b4-t6hns:/home/wf/App# cd /tmp/ray/session_2023-11-07_12-08-10_478243_7/runtime_resources/py_modules_files/s3_dependencies_png_py
root@ray-algo-plugin-worker1-7d4f5658b4-t6hns:/tmp/ray/session_2023-11-07_12-08-10_478243_7/runtime_resources/py_modules_files/s3_dependencies_png_py# ls
png.py
root@ray-algo-plugin-worker1-7d4f5658b4-t6hns:/tmp/ray/session_2023-11-07_12-08-10_478243_7/runtime_resources/py_modules_files/s3_dependencies_png_py#

But still we got import error:

ModuleNotFoundError: No module named ‘png’

  1. We cannot use ‘pip’ for passing 3rd parties packages.
  2. We’re looking for a solution per actor/task (and not per job)
2

Can you print PYTHONPATH in the code to see whether the module is in the path?