# Multi-agent Env with different reward functions for different agents?

**URL:** <https://discuss.ray.io/t/multi-agent-env-with-different-reward-functions-for-different-agents/3502>\
**Category:** RLlib\
**Created:** [September 10, 2021, 11:56pm UTC](https://discuss.ray.io/t/multi-agent-env-with-different-reward-functions-for-different-agents/3502 "2021-09-10T23:56:31Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![lucas\_spangher](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/lucas_spangher/32/938_2.png) [@lucas\_spangher](https://discuss.ray.io/u/lucas_spangher)\
**Post date:** [September 10, 2021, 11:56pm UTC](https://discuss.ray.io/t/multi-agent-env-with-different-reward-functions-for-different-agents/3502/1 "2021-09-10T23:56:31Z")

</div>

Hi,

I’m interested in creating a multi-agent env with two agents in copies of the same (custom) environment. I’m interested in implementing a different reward for the two agents. Is there an easy way to make this happen?

Also, would this make sense with a centralized critic, or would it mess up the value function for the critic?

---

<div class="post-metadata">

**Author:** ![rsv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rsv/32/703_2.png) [@rsv](https://discuss.ray.io/u/rsv)\
**Post date:** [September 12, 2021, 7:03am UTC](https://discuss.ray.io/t/multi-agent-env-with-different-reward-functions-for-different-agents/3502/2 "2021-09-12T07:03:38Z")

</div>

In the multi-agent environments API rewards are dict mapping agent names to their rewards as well as observations.

```auto
> print(rewards)
{"car_1": 3, "car_2": -1, "traffic_light_1": 0}

```

You can calculate rewards in your env and return it for the agent concerned

---

<div class="post-metadata">

**Author:** ![mannyv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mannyv/32/606_2.png) [@mannyv](https://discuss.ray.io/u/mannyv)\
**Post date:** [September 12, 2021, 11:35am UTC](https://discuss.ray.io/t/multi-agent-env-with-different-reward-functions-for-different-agents/3502/3 "2021-09-12T11:35:27Z")

</div>

@lucas_spangher,

When I use a centralized critic in these cases I include an agent index variable that indicates which agent the reward is from. If the number agents is small I use a one-hot encoding and if they are \>4 I use a binary encoding.

00000100 ← 6th agent id one-hot encoding  
0110 ← 6th agent id binary encoding

Good luck.

* * *

> **[Manny is helping people do cool things with RL](https://www.buymeacoffee.com/mannyv)**
>
> Hey, so glad we got to connect. If you found my suggestions useful feel free to buy me a coffee!

---

<div class="post-metadata">

**Author:** ![lucas\_spangher](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/lucas_spangher/32/938_2.png) [@lucas\_spangher](https://discuss.ray.io/u/lucas_spangher)\
**Post date:** [September 14, 2021, 12:04am UTC](https://discuss.ray.io/t/multi-agent-env-with-different-reward-functions-for-different-agents/3502/4 "2021-09-14T00:04:54Z")

</div>

Thank you both, I really appreciate!

@rsv , just a note, in the example you listed, this dict just returns ints for rewards. Would I return a dict in the reward function in order for this to be the output of a function, i.e. dynamic?

---

<div class="post-metadata">

**Author:** ![mannyv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mannyv/32/606_2.png) [@mannyv](https://discuss.ray.io/u/mannyv)\
**Post date:** [September 14, 2021, 12:25am UTC](https://discuss.ray.io/t/multi-agent-env-with-different-reward-functions-for-different-agents/3502/5 "2021-09-14T00:25:13Z")

</div>

@lucas_spangher

The link below shows a simple multi-agent env example. You are going to have a step method in your environment that will return 4 dicts. One for new\_obs, reward, agentandenv\_done, extra\_info respectively.

> <https://github.com/ray-project/ray/blob/master/rllib/examples/env/multi_agent.py>

---

<div class="post-metadata">

**Author:** ![lucas\_spangher](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/lucas_spangher/32/938_2.png) [@lucas\_spangher](https://discuss.ray.io/u/lucas_spangher)\
**Post date:** [September 14, 2021, 12:38am UTC](https://discuss.ray.io/t/multi-agent-env-with-different-reward-functions-for-different-agents/3502/6 "2021-09-14T00:38:05Z")

</div>

I appreciate it, thank you. Thanks for pointing me in the right direction.

---

<div class="post-metadata">

**Author:** ![rsv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rsv/32/703_2.png) [@rsv](https://discuss.ray.io/u/rsv)\
**Post date:** [September 14, 2021, 7:03am UTC](https://discuss.ray.io/t/multi-agent-env-with-different-reward-functions-for-different-agents/3502/7 "2021-09-14T07:03:39Z")

</div>

Your environment returns reward dict in multiagent case. Rllib doesn’t calculate reward, so you can make it by function in the environment and it may be int or float