# How can I train a multi-objective model in rllib?

**URL:** <https://discuss.ray.io/t/how-can-i-train-a-multi-objective-model-in-rllib/4148>\
**Category:** RLlib\
**Created:** [November 16, 2021, 11:57am UTC](https://discuss.ray.io/t/how-can-i-train-a-multi-objective-model-in-rllib/4148 "2021-11-16T11:57:34Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![saeid93](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/saeid93/32/411_2.png) [@saeid93](https://discuss.ray.io/u/saeid93)\
**Post date:** [November 16, 2021, 11:57am UTC](https://discuss.ray.io/t/how-can-i-train-a-multi-objective-model-in-rllib/4148/1 "2021-11-16T11:57:34Z")

</div>

Hello,  
I have a model which has rewards for two different objectives, I want to know if there is a way to train a policy in rllib that could achieve a trade-off between these two objectives (finding pareto front maybe)? Is there any multi-objective rl algorithm implementation in the rllib? If not, how can I add a multiobjective algorithm suitable for my environment to the rllib?

---

<div class="post-metadata">

**Author:** ![sven1977](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sven1977/32/53_2.png) [@sven1977](https://discuss.ray.io/u/sven1977)\
**Post date:** [November 16, 2021, 2:22pm UTC](https://discuss.ray.io/t/how-can-i-train-a-multi-objective-model-in-rllib/4148/2 "2021-11-16T14:22:12Z")

</div>

Hey @saeid93 , great question. There is no algorithm in RLlib that was designed especially for that purpose (multi-reward envs). You could add a post-processing step to merge the rewards into one by providing a custom callback and overriding the `on_postprocess_trajectory` method. In there, you can change the incoming batch’s “rewards” key.

See here for an example:

> <https://github.com/ray-project/ray/blob/master/rllib/examples/custom_metrics_and_callbacks.py#L86>

---

<div class="post-metadata">

**Author:** ![saeid93](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/saeid93/32/411_2.png) [@saeid93](https://discuss.ray.io/u/saeid93)\
**Post date:** [November 17, 2021, 12:33pm UTC](https://discuss.ray.io/t/how-can-i-train-a-multi-objective-model-in-rllib/4148/3 "2021-11-17T12:33:13Z")

</div>

Thanks, @sven1977 . That’s a nice way of doing it! Could you please confirm that in your solution you mean ‘summing up’ both rewards by ‘merge the reward into one’? That’s a nice solution but currently, I am doing the same thing (summing up) the rewards in my environment and passing that to ray. Then I pass each reward separately in my info dict separately as a custom metric to the monitoring callback for viewing them in tensorboard. Does your solution make any difference to what I am doing right now since as far as I understand both of them are summing up both objectives rewards?  
Summing up the rewards works to some extent but the problem with that is the rewards come from different rages and even normalizing them will not result in a smooth curve. If I want to move beyond summing up is there any particular RL solution that you suggest for integrating (implementing) into rllib?

---

<div class="post-metadata">

**Author:** ![mannyv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mannyv/32/606_2.png) [@mannyv](https://discuss.ray.io/u/mannyv)\
**Post date:** [November 17, 2021, 1:42pm UTC](https://discuss.ray.io/t/how-can-i-train-a-multi-objective-model-in-rllib/4148/4 "2021-11-17T13:42:01Z")

</div>

Hi @saeid93,

Have a look at this related post.

> [@Multi-objective RL](https://discuss.ray.io/t/multi-objective-rl/4055):
>
> Is there a support for training multi-objective based PPO and DQN agent in RLlib? Can I refer to some tuned example if any such thing is feasible?