Runtime_env on driver node

Ray Core
1
  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

I used runtime_env to setup the dependencies on ray cluster, but it seems on driver side, I can’t import the installed modules. Is it expected that the new dependencies are only accessible in remote node?

My initial guess was that the dependency installation took a long time, therefore the driver side doesn’t have all the dependencies ready, and any remote task doesn’t run into issues because ray may have validation for each remote task’s runtime_env.

In order to ensure all dependencies are ready, I launched a remote task with the runtime_env, and wait for the task to finish before importing the module on driver side, however, the driver side still can’t import the module.

2

Yes, runtime environments only install the dependencies on the Ray cluster nodes, and the environment is only accessible by those actors and tasks.

By “driver side” do you mean a client machine that’s not part of the Ray cluster? If you want to run a driver script on the Ray cluster itself, you can try using the Ray Jobs API and specifying the runtime_env using that API. Ray Jobs Overview — Ray 2.2.0 The driver script (entrypoint) will then run inside the specified runtime_env.

3

By driver side, I mean the head node, where I run the driver script and ray.init. Yes, it’s within the ray cluster. But looks like I can’t import modules on there.

4

Ah, the runtime env is only active for tasks and actors. It uses virtualenv and modifies PYTHONPATH for the worker processes; the environment isn’t installed “globally” on the node.

If you want to import things outside of tasks and actors, you can install the environment manually, or use the Ray Jobs API and specify the runtime_env there.

1 Like