I have an expensive initialization that I want each worker to perform before it starts processing tasks. The resulting object should then be accessible to all tasks on that worker. With Actors, it is quite straightforward, I initialize it in the constructor. However, I am not clear on how to do it…

cc @eoakes do you know how we can achieve this? This seems to be a common use case of our multi processing pool?

2 things come to mind In general, if you want global state, you can wrap it in an actor, then ray.get() it inside your parallelized function. The caveat is that this incurs deserialization overhead (which can be large if your object is a large, non-array-like object). Use Actor Pool, which is …

I am indeed using the ActorPool now, which works well, although I need to specify in advance how many actors I will have. I was under the impression that the ray multiprocessing.Pool grows/shrinks automatically with the number of tasks and the current cluster size (ie, if I supply many tasks, it wil…

oh i see, i think the multiprocessing pool defaults to creating one actor per cpu in the cluster, i’m not aware of any fancy tricks there. You’re right that ActorPool doesn’t have a way of adding actors to an existing pool right now, but it should be pretty easy to add (as long as someone is willin…

@Alex is using the ActorPool still the recommended approach for situations where you want a pool of processes and you have expensive initialisation? I see it’s deprecated now.

@Yoav regarding your questions of expensive initialization, if it is about process warming / code loading, Ray should already handle that given we do some caching / re-use of workers Otherwise have you considered using the object store / ray.put & get at the beginning of the task?

Worker initializer in ray.util.multiprocessing.Pool

Ray Core

ClarenceNg November 15, 2022, 9:14am 7

As noted in Deprecation of ray.utils.ActorPool - #9 by ClarenceNg the actor pool is no longer deprecated

Topic		Replies	Views
Ray Worker Initialiser Ray Clusters	0	385	June 13, 2023
Creating pool using cluster started with runtime_env Ray Core	2	65	April 8, 2026
[Core] How to make sure an actor is initialized? Ray Core	5	1396	February 17, 2023
Deprecation of ray.utils.ActorPool Ray Core	8	1070	November 15, 2022
Run writing file jobs with the prespecified number of workers(?) Ray Core	6	499	December 22, 2021

Worker initializer in ray.util.multiprocessing.Pool

Related topics