# Allreduce primitive?

**URL:** <https://discuss.ray.io/t/allreduce-primitive/1134>\
**Category:** Ray Core\
**Created:** [March 5, 2021, 5:26am UTC](https://discuss.ray.io/t/allreduce-primitive/1134 "2021-03-05T05:26:21Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![KristianHolsheimer](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kristianholsheimer/32/582_2.png) [@KristianHolsheimer](https://discuss.ray.io/u/KristianHolsheimer)\
**Post date:** [March 5, 2021, 5:26am UTC](https://discuss.ray.io/t/allreduce-primitive/1134/1 "2021-03-05T05:26:21Z")

</div>

Hi there,

I was wondering if there is a way that I can use allreduce in core ray? The way I picture this is something like:

```python
# on a remote worker:
ray.allreduce(x, operation='sum', axis_name='foo', axis_size=16)
x # now holds the sum of all x's across all 16 workers

```

I imagine that ray would block on the remote workers until it has collected `axis_size=16` calls to allreduce along `axis_name='foo'`. It would then do the allreduce operation and then lift the block.

Is something like this possible? If not, do you think it’s a good idea to add this? I would love to have this kind of functionality.

---

<div class="post-metadata">

**Author:** ![rliaw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rliaw/32/24_2.png) [@rliaw](https://discuss.ray.io/u/rliaw)\
**Post date:** [March 5, 2021, 5:28am UTC](https://discuss.ray.io/t/allreduce-primitive/1134/2 "2021-03-05T05:28:58Z")

</div>

Yeah!

This is now supported with `ray.util.collective`. It’s still a work-in-progress, but you should definitely try it out!

> **[ray-project/ray](https://github.com/ray-project/ray/tree/master/python/ray/util/collective)**
>
> master/python/ray/util/collective

cc @zhisbug

---

<div class="post-metadata">

**Author:** ![KristianHolsheimer](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kristianholsheimer/32/582_2.png) [@KristianHolsheimer](https://discuss.ray.io/u/KristianHolsheimer)\
**Post date:** [March 5, 2021, 5:36am UTC](https://discuss.ray.io/t/allreduce-primitive/1134/3 "2021-03-05T05:36:53Z")

</div>

Awesome, thanks!

I also like the design… define a group and then referencing it by name.