Ray submit - refer to a script in the file mount


So I have a local module that’s structured as below :

* "my_ml_module"
 * __init__.py
 * scripts/
   * my_scripts.py
   * __init__.py
 * routines/
   * __init__.py
   * useful_routines.py

In my_scripts.py, it may have import statements such as from my_ml_module.routines.useful_routines import abc.

I have a EC2 autoscaling scaler where I specified my_ml_module as file_mounts, so the cluster nodes do have the my_ml_module under directory /.

Now my question is how do I submit the my_scripts.py to the cluster, and have the cluster refer to the script under the mounted location so that PYTHONPATH works out?

Currently when I do ray submit my_ml_module/scripts/my_scripts.py, I think the new copy of the script is getting synced to the os.path.join(“~”, os.path.basename(script)) as specified in API and Package Reference — Ray v2.0.0.dev0 and I am getting ModuleNotFoundError: No module named my_ml_module.routines.useful_routines.


I think ray exec may come in handy here.
You can try ray attach to ssh into the head node and experiment with commands to see what works.
Then use the command you figure out that with ray exec in your workflow.

1 Like