Is it possible to use Ray in a subprocess created with multiprocessing.Process?

Ray Core
1

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

Hi,

I wonder if it is possible to use Ray (for example, call ray.get(), or create an Actor) from a subprocess created with multiprocessing.Process of the driver script.
I tested it myself, and the code got stuck when creating the Actor.
Environment: Python 3.8, Ray 2.0.0.

import ray
import multiprocessing

ray.init()

@ray.remote
class Foo:
    def __init__(self, a):
        self.a = a
    def p(self):
        return self.a


def create_and_run(a):
    print("before creating")
    foo = Foo.remote(a)
    print("after creating")
    result = ray.get(foo.p.remote())
    print(result)

p = multiprocessing.Process(target=create_and_run, args=(1,))
p.start()
p.join()

Output:

root@iZ2zeinq87v01lgdaj1xdiZ:/home/hanyu/ray_tests# python multiprocess.py
2022-10-20 02:35:25,176 INFO worker.py:1333 -- Connecting to existing Ray cluster at address: 172.20.5.216:6379...
2022-10-20 02:35:25,182 INFO worker.py:1509 -- Connected to Ray cluster. View the dashboard at http://127.0.0.1:8265
before creating

If I create the Actor in the main process, and call ray.get() in the subprocess (as shown below), the code got stuck at the ray.get().

import ray
import multiprocessing

ray.init()

@ray.remote
class Foo:
    def __init__(self, a):
        self.a = a
    def p(self):
        return self.a

def run(foo):
    print("before result")
    result = foo.p.remote()
    print("before get")
    print(ray.get(result))

foo = Foo.remote(1)
p = multiprocessing.Process(target=run, args=(foo,))
p.start()
p.join()

Output:

root@iZ2zeinq87v01lgdaj1xdiZ:/home/hanyu/ray_tests# python multiprocess.py
2022-10-20 02:40:06,906 INFO worker.py:1333 -- Connecting to existing Ray cluster at address: 172.20.5.216:6379...
2022-10-20 02:40:06,912 INFO worker.py:1509 -- Connected to Ray cluster. View the dashboard at http://127.0.0.1:8265
before result
before get

Thank you!

2

We don’t recommend using Ray this way. Ray doesn’t work well with fork because 1. Ray processes are supposed to be managed by Ray systems, but if you fork the process, they may not be managed by Ray properly. 2. Ray uses gRPC internally, and it doesn’t play well with forking (there’s a way to make gRPC play well with fork though). This usage pattern is also not tested and behaviors are undefined. What’s the usecase btw?

3

I see. Thanks for the prompt reply! I was using some strange legacy code that led me to such a weird situation. Never mind :rofl: