GIl and race conditions with Ray multiprocessing

1

I am parallelizing functions that read the same 3D numpy array (a series of images) using Ray multiprocessing Pool().

I share the 3D array images with ray.put() as follows:

images_ref = ray.put(images)
myfunc_partial = functools.partial(myfunc, obj_ref=images_ref, arg2=blah2)

with Pool(processes=ncpus) as pool:
   pool.map(myfunc_partial, params_list)  #params_list has at least 10 elements

Insise myfunc_partial, images is processed after calling images = ray.get(obj_ref)

First of all, is that the proper way to share my 3D Numpy array across the 10 processes so that there are no copies of the array?

If so, is the GIL stepping in by preventing the mutliple processes to read the shared-memory array at the same time?

2

GIL is only applied within the same process. Ray’s Shared memory objects are read-only, so it doesn’t require any explicit lock. So all the processes can concurrently access the buffer without any synchornization

1 Like