# Mean reward per agent in MARL
**URL:** https://discuss.ray.io/t/mean-reward-per-agent-in-marl/8917
**Category:** RLlib
**Created:** [January 11, 2023, 9:26am UTC](https://discuss.ray.io/t/mean-reward-per-agent-in-marl/8917 "2023-01-11T09:26:09Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@Username1](https://discuss.ray.io/u/Username1)
#### Post date: [January 11, 2023, 9:26am UTC](https://discuss.ray.io/t/mean-reward-per-agent-in-marl/8917/1 "2023-01-11T09:26:09Z")
**How severe does this issue affect your experience of using Ray?**
- High: It blocks me to complete my task.
Hi, as said [on this post](https://discuss.ray.io/t/meaning-of-episode-reward-mean/3839/8) " in a multi-agent RL configuration, the reported **episode\_reward\_mean** in json\_object is the sum of the episode\_reward\_mean obtained by RL each agent."
1. How can I report the reward per agent?
2. How can I see it on Tensorboard as well?
Thanks!
---
### Author:  [@Lars\_Simon\_Zehnder](https://discuss.ray.io/u/Lars_Simon_Zehnder)
#### Post date: [January 11, 2023, 12:50pm UTC](https://discuss.ray.io/t/mean-reward-per-agent-in-marl/8917/2 "2023-01-11T12:50:55Z")
Hi @Username1, custom metrics have been discusses here a couple of times. We usually refer to the very good example [here](https://github.com/ray-project/ray/blob/master/rllib/examples/custom_metrics_and_callbacks.py).
In there you find a callback names `on_episode_end()`. This is the callback you want to need as the episode has ended and mean rewards could be properly computed. The `episode` object in the arguments contains all the data you look for. You have to loop over the agents in the `rewards` therein and then add your mean rewards for each agent to the `episode`’ s `custom_metrics` attribute as shown in [this line](https://github.com/ray-project/ray/blob/8e375d081eed978d9a9a7e440b7ae43d6f64d78b/rllib/examples/custom_metrics_and_callbacks.py#L98). The `hist_data` attribute will create for you histogram and distributions in TensorBoard whereas the `custom_metrics` create scalars.
---
### Author:  [@Username1](https://discuss.ray.io/u/Username1)
#### Post date: [January 11, 2023, 4:16pm UTC](https://discuss.ray.io/t/mean-reward-per-agent-in-marl/8917/3 "2023-01-11T16:16:04Z")
Thank you very much @Lars_Simon_Zehnder for your reply.
Where should I see the custom metrics? Are they supposed to be printed to the console after training? I can’t see them printed or displayed on TB. Should I set ‘verbose = 3’?
This is my code:
```auto
class MyCallback(Callback):
def on_episode_end(self, worker: RolloutWorker, base_env: BaseEnv,
policies: Dict[str, Policy], episode: MultiAgentEpisode,
**kwargs):
episode.custom_metrics['agents_lst'] = episode.agent_rewards.keys()
episode.custom_metrics['mean_return_per_agent'] = list(episode.agent_rewards.keys())
# Graphs of Hist over time.
episode.custom_metrics["return_hist"] = episode.hist_data["mean_return_per_agent"]
```
---
### Author:  [@Lars\_Simon\_Zehnder](https://discuss.ray.io/u/Lars_Simon_Zehnder)
#### Post date: [January 11, 2023, 4:29pm UTC](https://discuss.ray.io/t/mean-reward-per-agent-in-marl/8917/4 "2023-01-11T16:29:22Z")
@Username1 , this won’t work, as `custom_metrics["my_metric"]` needs a scalar to work, so you need to create a custom metric for each of your agents. You then want to define a summarization for the rewards a single agent collected.
Take a look into the example - it shows you with a simple example, how to do it.
---
### Author:  [@Username1](https://discuss.ray.io/u/Username1)
#### Post date: [January 11, 2023, 4:51pm UTC](https://discuss.ray.io/t/mean-reward-per-agent-in-marl/8917/5 "2023-01-11T16:51:40Z")
Thank you very much @Lars_Simon_Zehnder
First, I am not sure, but I guess I should change the example from `episode: Episode` to `episode: MultiAgentEpisode` since my env is multi-agent.
Then, I’ve tried something very simple, but I can’t find the metrics printed out in the console:
```auto
class MyCallback(Callback):
def on_episode_end(self, worker: RolloutWorker, base_env: BaseEnv,
policies: Dict[str, Policy], episode: Episode,
**kwargs):
episode.custom_metrics['agents_lst'] = 1
episode.custom_metrics['mean_return_per_agent'] = 2
```
Do you have any multi-agent example? or what I am missing to make it work? Thanks!
---
### Author:  [@Lars\_Simon\_Zehnder](https://discuss.ray.io/u/Lars_Simon_Zehnder)
#### Post date: [January 11, 2023, 5:49pm UTC](https://discuss.ray.io/t/mean-reward-per-agent-in-marl/8917/6 "2023-01-11T17:49:29Z")
@Username1, you do not need to use the `MultiAgentEpisode` specifically, as the callback here is simply inherited and that one uses the `Episode` which is also the base class for `MultiAgentEpisode`. Don’t worry the episode that is passed in at runtime is a `MultiAgentEpisode`.
To your problem, have you followed the workflow in the example? Did you add the callback to your `config`s `callbacks`? Do you see it in TensorBoard?
---
### Author:  [@Username1](https://discuss.ray.io/u/Username1)
#### Post date: [January 12, 2023, 9:30am UTC](https://discuss.ray.io/t/mean-reward-per-agent-in-marl/8917/7 "2023-01-12T09:30:50Z")
Hello @Lars_Simon_Zehnder this is my training module. I can’t see the custom metrics on the console nor on Tensorboard.
I also don’t know how to get the rewards per agent. I guess `episode.agent_rewards` brings back a dictionary with the rewards per agent or how to get the info I want. Thanks
```auto
class MyCallback(Callback):
def on_episode_end(self, worker: RolloutWorker, base_env: BaseEnv,
policies: Dict[str, Policy], episode: Episode,
**kwargs):
episode.custom_metrics['agents_lst'] = 1
episode.custom_metrics['mean_return_per_agent'] = 2
```
```auto
def setup_and_train():
# config dict.. etc
train_steps = 1
experiment_name = 'my_env'
tuner = tune.Tuner("PPO", param_space=config,
run_config=air.RunConfig(
name = experiment_name,
stop={"timesteps_total": train_steps},
checkpoint_config=air.CheckpointConfig(checkpoint_frequency=50, checkpoint_at_end=True),
callbacks= [MyCallback()] #here
)
)
results = tuner.fit()
```
---
### Author:  [@Lars\_Simon\_Zehnder](https://discuss.ray.io/u/Lars_Simon_Zehnder)
#### Post date: [January 12, 2023, 12:45pm UTC](https://discuss.ray.io/t/mean-reward-per-agent-in-marl/8917/8 "2023-01-12T12:45:10Z")
@Username1 , you are almost there. The callbacks in this case are RLlib callbacks and not Tune callbacks. So you have to add them to your `config`:
```auto
config.callbacks(MyCallback)
tuner = tune.Tune("PPO", ....)
```
---
### Author:  [@Username1](https://discuss.ray.io/u/Username1)
#### Post date: [January 12, 2023, 1:16pm UTC](https://discuss.ray.io/t/mean-reward-per-agent-in-marl/8917/9 "2023-01-12T13:16:32Z")
Thank you very much @Lars_Simon_Zehnder for your time. So the entry on the tune dictionary `callbacks= [MyCallback()]` has to be removed right?
Now, when I add `config.callbacks(MyCallback)` to the PPO config like this:
```auto
#RLLIB Configs
N_CPUS = 4
learning_rate = 1e-3
config = PPOConfig()\
.training(lr=learning_rate,num_sgd_iter=10, train_batch_size = 4000)\
.framework("torch")\
.rollouts(num_rollout_workers=1, observation_filter="MeanStdFilter")\
.resources(num_gpus=0,num_cpus_per_worker=1)\
.evaluation(evaluation_interval=100,evaluation_duration = 5, evaluation_duration_unit='episodes',
evaluation_config= {"explore": False})\
.environment(env = env_name, env_config={
"num_workers": N_CPUS - 1,
"disable_env_checking":True} )
#RLLIB callbacks
config.callbacks(MyCallback)
```
I get the following error:
```auto
(PPO pid=4329) File "/opt/anaconda3/lib/python3.8/site-packages/ray/rllib/evaluation/rollout_worker.py", line 650, in __init__
(PPO pid=4329) self.callbacks.on_sub_environment_created(
(PPO pid=4329) AttributeError: 'MyCallback' object has no attribute 'on_sub_environment_created'
```
---
### Author:  [@sven1977](https://discuss.ray.io/u/sven1977)
#### Post date: [January 12, 2023, 3:23pm UTC](https://discuss.ray.io/t/mean-reward-per-agent-in-marl/8917/10 "2023-01-12T15:23:02Z")
> [@Username1](#):
>
> `MyCallback`
Hey @Username1 , thanks for raising this issue. Not sure what’s the issue exactly, but it seems like you are subclassing your custom MyCallback from an older DefaultCallbacks class? The current master one has this method here. As a hack, you might just want to add it as-is to your MyCallback class:
```auto
@OverrideToImplementCustomLogic
def on_sub_environment_created(
self,
*,
worker: "RolloutWorker",
sub_environment: EnvType,
env_context: EnvContext,
env_index: Optional[int] = None,
**kwargs,
) -> None:
pass
```
---
### Author:  [@sven1977](https://discuss.ray.io/u/sven1977)
#### Post date: [January 12, 2023, 3:24pm UTC](https://discuss.ray.io/t/mean-reward-per-agent-in-marl/8917/11 "2023-01-12T15:24:25Z")
Oh, I see, you are subclassing from a `Callback` class (maybe tune callbacks?). Could you subclass from
the `ray.rllib.algorithms.callbacks::DefaultCallbacks` class?
---
### Author:  [@Username1](https://discuss.ray.io/u/Username1)
#### Post date: [January 12, 2023, 4:22pm UTC](https://discuss.ray.io/t/mean-reward-per-agent-in-marl/8917/12 "2023-01-12T16:22:17Z")
Thank you very much @sven1977 for your time and your response.
Thanks!