How can I pass all(most?) ENV parameters of the driver to remote workers?

1

Hello!

I’ve been trying to find a way to pass most if not all ENV parameters set on the driver down to each remote tasks. I can pass them as remote parameter and re-set them once it gets to the remote(), but is there more elegant / ray native way of passing those ENVs?

Thank you!

2

Could you use runtime environments to do something like ray.init(runtime_env={"env_vars": os.environ})? This would read all the environment variables on the driver at the time ray.init is called and set those environment variables in each subsequent remote task.

https://docs.ray.io/en/latest/ray-core/handling-dependencies.html#runtime-environments

3

@architkulkarni Thank you for the pointer!!!

I will try what you suggested, but I also noticed this text

env_vars (Dict[str, str]): Environment variables to set.
Environment variables already set on the cluster will still be visible to the Ray workers; so there is no need to include os.environ or similar in the env_vars field.

I don’t believe this is actually true. I have a few ENV parameters in .env that’s picked up vscode debugger and my main application is successfully using, but the ray workers submitted from it don’t seem to have those parameters set. I am using ray 2.1.0… so I am not sure why that’s not working. Maybe bug?

4

Ah, I think I’m not familiar with .env and ENV parameters, what’s that?

That text is just referring to the environment variables in the process which starts Ray; for example, the environment variables you see when running env in the terminal. If you then start ray with ray start --head and print os.environ inside a Ray task (without using runtime_env), you should see the same environment variables.

5

@architkulkarni That makes sense! Thank you for clarifying it!