# Worker initializer in ray.util.multiprocessing.Pool

**URL:** https://discuss.ray.io/t/worker-initializer-in-ray-util-multiprocessing-pool/896
**Category:** Ray Core
**Created:** [February 15, 2021, 8:40pm UTC](https://discuss.ray.io/t/worker-initializer-in-ray-util-multiprocessing-pool/896 "2021-02-15T20:40:59Z")
**Posts on this page:** 8
**Page:** 1

<div class="post-metadata">

### Author: ![Yoav](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/yoav/32/456_2.png) [@Yoav](https://discuss.ray.io/u/Yoav)
#### Post date: [February 15, 2021, 8:40pm UTC](https://discuss.ray.io/t/worker-initializer-in-ray-util-multiprocessing-pool/896/1 "2021-02-15T20:40:59Z")

</div>

I have an expensive initialization that I want each worker to perform before it starts processing tasks. The resulting object should then be accessible to all tasks on that worker.  
With Actors, it is quite straightforward, I initialize it in the constructor.

However, I am not clear on how to do it with tasks, and how to do it with the `ray.util.multiprocessing.Pool`.

Specifically, the initializer in the Pool API is not returning any value. The pattern I know from the python Pool is to assign to a global variable, which is then accessible also from the worker task. However, this does not seem to work with the ray Pool. What is the intended usage pattern for initializers in the ray multiprocessing Pool?

---

<div class="post-metadata">

### Author: ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)
#### Post date: [February 15, 2021, 10:38pm UTC](https://discuss.ray.io/t/worker-initializer-in-ray-util-multiprocessing-pool/896/2 "2021-02-15T22:38:04Z")

</div>

cc @eoakes do you know how we can achieve this? This seems to be a common use case of our multi processing pool?

---

<div class="post-metadata">

### Author: ![Alex](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/alex/32/341_2.png) [@Alex](https://discuss.ray.io/u/Alex)
#### Post date: [February 15, 2021, 10:42pm UTC](https://discuss.ray.io/t/worker-initializer-in-ray-util-multiprocessing-pool/896/3 "2021-02-15T22:42:00Z")

</div>

2 things come to mind

1. In general, if you want global state, you can wrap it in an actor, then `ray.get()` it inside your parallelized function. The caveat is that this incurs deserialization overhead (which can be large if your object is a large, non-array-like object).

2. Use Actor Pool, which is built for this exact case. You could even wrap it and call it from your pool map function if you wanted.

---

<div class="post-metadata">

### Author: ![Yoav](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/yoav/32/456_2.png) [@Yoav](https://discuss.ray.io/u/Yoav)
#### Post date: [February 15, 2021, 11:32pm UTC](https://discuss.ray.io/t/worker-initializer-in-ray-util-multiprocessing-pool/896/4 "2021-02-15T23:32:00Z")

</div>

I am indeed using the ActorPool now, which works well, although I need to specify in advance how many actors I will have. I was under the impression that the ray multiprocessing.Pool grows/shrinks automatically with the number of tasks and the current cluster size (ie, if I supply many tasks, it will create more actors and autoscale up). Or is it just my wishful thinking?

---

<div class="post-metadata">

### Author: ![Alex](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/alex/32/341_2.png) [@Alex](https://discuss.ray.io/u/Alex)
#### Post date: [February 16, 2021, 1:10am UTC](https://discuss.ray.io/t/worker-initializer-in-ray-util-multiprocessing-pool/896/5 "2021-02-16T01:10:44Z")

</div>

oh i see, i think the multiprocessing pool defaults to creating one actor per cpu in the cluster, i’m not aware of any fancy tricks there.

You’re right that ActorPool doesn’t have a way of adding actors to an existing pool right now, but it should be pretty easy to add (as long as someone is willing to implement it). Do you mind filing a github feature request?

---

<div class="post-metadata">

### Author: ![Andrea\_Pisoni](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/andrea_pisoni/32/1827_2.png) [@Andrea\_Pisoni](https://discuss.ray.io/u/Andrea_Pisoni)
#### Post date: [November 7, 2022, 3:30pm UTC](https://discuss.ray.io/t/worker-initializer-in-ray-util-multiprocessing-pool/896/6 "2022-11-07T15:30:03Z")

</div>

@Alex is using the ActorPool still the recommended approach for situations where you want a pool of processes and you have expensive initialisation? I see it’s deprecated now.

---

<div class="post-metadata">

### Author: ![ClarenceNg](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/clarenceng/32/3108_2.png) [@ClarenceNg](https://discuss.ray.io/u/ClarenceNg)
#### Post date: [November 15, 2022, 9:14am UTC](https://discuss.ray.io/t/worker-initializer-in-ray-util-multiprocessing-pool/896/7 "2022-11-15T09:14:27Z")

</div>

As noted in [Deprecation of ray.utils.ActorPool - #9 by ClarenceNg](https://discuss.ray.io/t/deprecation-of-ray-utils-actorpool/7648/9) the actor pool is no longer deprecated

---

<div class="post-metadata">

### Author: ![ClarenceNg](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/clarenceng/32/3108_2.png) [@ClarenceNg](https://discuss.ray.io/u/ClarenceNg)
#### Post date: [November 15, 2022, 9:27am UTC](https://discuss.ray.io/t/worker-initializer-in-ray-util-multiprocessing-pool/896/8 "2022-11-15T09:27:11Z")

</div>

@Yoav

regarding your questions of expensive initialization, if it is about process warming / code loading, Ray should already handle that given we do some caching / re-use of workers

Otherwise have you considered using the object store / ray.put & get at the beginning of the task?
