I am parallelizing functions that read the same 3D numpy array (a series of images) using Ray multiprocessing
I share the 3D array
ray.put() as follows:
images_ref = ray.put(images) myfunc_partial = functools.partial(myfunc, obj_ref=images_ref, arg2=blah2) with Pool(processes=ncpus) as pool: pool.map(myfunc_partial, params_list) #params_list has at least 10 elements
images is processed after calling
images = ray.get(obj_ref)
First of all, is that the proper way to share my 3D Numpy array across the 10 processes so that there are no copies of the array?
If so, is the GIL stepping in by preventing the mutliple processes to read the shared-memory array at the same time?