How can I tell if I'm on a worker or the driver? Is there something like `multiprocessing.parent_process()`?

Ray Core
1

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

Is there a way of telling, from within a Python module, whether I’m on a ray worker or the driver? For example, something like the builtin multiprocessing.parent_process().

2

Can you tell me the use case a little bit? (I imagine that normally it’s straightforward to figure out what’s driver?) I don’t think there’s an official API, but there is a way to do this by combining some existing APIs.

3

One way to do this is to get ray.get_runtime_context().task_id. It will raise an error if the current process is not a task worker.

More about runtime_context here: ray.runtime_context — Ray 1.12.0

4

Thanks @ericl , I didn’t know about that—very useful. I ended up setting a custom environment variable using the runtime_env argument to ray.remote, but your solution is more elegant.

@sangcho, the use case is the following: I have a configuration object that the user can modify at will, and I want any changes to be visible to workers. My approach is to write the configuration to disk anytime it’s changed, and then load that file on the workers when my library is imported (I’m not worried about potential race conditions here, but of course this is not ideal). But the catch is, I also want to load the user’s configuration on import—so I need to load from different locations depending on whether I’m on the driver or not.

If you know of a better pattern for this kind of global-variable sharing, I’m all ears!