Using ray in when imported as module and run as module

Ray Core
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

Hello guys,

I have a problem using ray in the context of using it as a module.

I have the following code (main.py), which is a simplified version of the actual file, but it has the same structure and error messages:

import ray
import importlib
from pathlib import Path
import sys
import os

try:
     from parallel.rayParallel import mainWrapper
except ImportError:
     from rayParallel import mainWrapper


if __name__ == "__main__":
     if not ray.is_initialized():
         ray.init()
     path = Path("path/to/python_script")
    
     sys.path.insert(0, f"{path.parent}{os.sep}")
     func = importlib.import_module(path.stem)
     run_func = func.mainWrapper
     run_func()

It initializes ray and than interanalizes my simulation script as a module and than runs the “mainWrapper” function which actually does the computation and parallelization with ray.

The thing is, when I run this script as a normal python script (python3 main.py) it works just fine.

But when I want to run it as a module (which is my ultimate goal) with
python3 -m parallel (the test module is named parallel, which has a init.py and the above main.py) function, I get the following error:

RuntimeError: The remote function failed to import on the worker. This may be because needed library dependencies are not installed in the worker environment:

ray::parallel() (pid=96716, ip=134.61.193.179)
ModuleNotFoundError: No module named 'rayParallel'

I already tried debugging, searched the internet, but did not find any solution.

Thank in advance :slight_smile:

2

Could you make sure rayParallel can be found from the sys.path?

3

Yes, the file rayParallal.py is included in the sys.path, see line (sys.path.insert etc…)

4

I also checked the sys.path, and the folder where rayParallel.py (or later the module rayParallel) is included is contained in the systems path.

5

Does anyone have a solution?
I still haven’t solved this problem.

6

anyone?
I have been working on this problem for a week now, and I cannot continue without solving it.

7

@jjyao I wonder if making this part of runtime env in the ray.init() would help?

8

I fixed it, but it was quite an endeavour.

1 Like