# How to deploy a trained Ray RLlib PPO policy/model in multi-agent-case?
**URL:** https://discuss.ray.io/t/how-to-deploy-a-trained-ray-rllib-ppo-policy-model-in-multi-agent-case/4072
**Category:** RLlib
**Created:** [November 9, 2021, 2:14pm UTC](https://discuss.ray.io/t/how-to-deploy-a-trained-ray-rllib-ppo-policy-model-in-multi-agent-case/4072 "2021-11-09T14:14:39Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@klausk55](https://discuss.ray.io/u/klausk55)
#### Post date: [November 9, 2021, 2:14pm UTC](https://discuss.ray.io/t/how-to-deploy-a-trained-ray-rllib-ppo-policy-model-in-multi-agent-case/4072/1 "2021-11-09T14:14:39Z")
Hello,
How can I deploy a trained Ray RLlib PPO policy/model in multi-agent-case and using an RNN-based policy?
I guess the first step is to load/restore the PPO Trainer (i.e. `trainer.restore(checkpoint)`).
Then there are the functions `trainer.compute_single_action` and `trainer.compute_actions`. The latter seems to compute actions for a batch of observations under one specific policy.
What I want is to compute a single action for one of the agents using its RNN-based policy.
Do I have to use `trainer.compute_single_action` and pass observation, RNN-state and policy ID to it?
---
### Author:  [@amogkam](https://discuss.ray.io/u/amogkam)
#### Post date: [November 10, 2021, 1:02am UTC](https://discuss.ray.io/t/how-to-deploy-a-trained-ray-rllib-ppo-policy-model-in-multi-agent-case/4072/2 "2021-11-10T01:02:51Z")
@gjoliver any ideas here?
---
### Author:  [@mannyv](https://discuss.ray.io/u/mannyv)
#### Post date: [November 10, 2021, 1:17pm UTC](https://discuss.ray.io/t/how-to-deploy-a-trained-ray-rllib-ppo-policy-model-in-multi-agent-case/4072/3 "2021-11-10T13:17:42Z")
Hi @klausk55,
Have a look at this documentation: [https://docs.ray.io/en/latest/rllib-training.html?highlight=compute\_action#computing-actions](https://docs.ray.io/en/latest/rllib-training.html?highlight=compute_action#computing-actions)
In the multiagent case the obs should be a dictionary with the agent(s) you want to compute the actions for.
Also don’t forget you need to chain the output state of one call as the input state of the next call to compute actions for that same agent.
## Here is an example from serve: [https://docs.ray.io/en/latest/serve/tutorials/rllib.html](https://docs.ray.io/en/latest/serve/tutorials/rllib.html)
> **[Manny is helping people do cool things with RL](https://buymeacoffee.com/mannyv)**
>
> Hey, so glad we got to connect. If you found my suggestions useful feel free to buy me a coffee!
---
### Author:  [@klausk55](https://discuss.ray.io/u/klausk55)
#### Post date: [November 10, 2021, 3:59pm UTC](https://discuss.ray.io/t/how-to-deploy-a-trained-ray-rllib-ppo-policy-model-in-multi-agent-case/4072/4 "2021-11-10T15:59:24Z")
Thanks @mannyv!
> [@mannyv](#):
>
> Hi @klausk55,
>
> Have a look at this documentation: [Getting Started with RLlib — Ray 2.8.0](https://docs.ray.io/en/latest/rllib-training.html?highlight=compute_action#computing-actions)
>
> In the multiagent case the obs should be a dictionary with the agent(s) you want to compute the actions for.
I guess in the multi-agent case where the obs is a MultiAgentDict, the invoking method should be `compute_actionS` since it accepts a dict as an obs.
> [@](#):
>
> Also don’t forget you need to chain the output state of one call as the input state of the next call to compute actions for that same agent.
Here you mean the internal state in case of an RNN-based policy, right? If so, what would you say is an approriate initial state for the first call to compute an action? Simply zero arrays?
> [@](#):
>
> ## Here is an example from serve: [Serving RLlib Models — Ray 2.8.0](https://docs.ray.io/en/latest/serve/tutorials/rllib.html)
>
> [https://buymeacoffee.com/mannyv](https://buymeacoffee.com/mannyv)
Yes, that’s a great example for an online serving use case! You’ve already made me aware of this in a previous post. I appreciate your help, thanks!
---
### Author:  [@mannyv](https://discuss.ray.io/u/mannyv)
#### Post date: [November 10, 2021, 4:33pm UTC](https://discuss.ray.io/t/how-to-deploy-a-trained-ray-rllib-ppo-policy-model-in-multi-agent-case/4072/5 "2021-11-10T16:33:56Z")
@klausk55,
compute\_actions should also accept a dictionary observation. You can use either.
The trainer has a `get_initial_state` method you can use.
---
### Author:  [@klausk55](https://discuss.ray.io/u/klausk55)
#### Post date: [November 10, 2021, 4:40pm UTC](https://discuss.ray.io/t/how-to-deploy-a-trained-ray-rllib-ppo-policy-model-in-multi-agent-case/4072/6 "2021-11-10T16:40:41Z")
What do you think @mannyv? Could it look like this?
```auto
state = {}
done = False
obs_dict = env.reset()
while not done:
action_dict = {}
for agent_id, obs in obs_dict.items():
if state.get(agent_id) is None:
state[agent_id] = trainer.get_policy(
policy_id="policy_{}".format(agent_id)).get_initial_state()
action_dict[agent_id], state[agent_id], _ = trainer.compute_single_action(
observation=obs, state=state[agent_id], policy_id="policy_{}".format(agent_id))
obs_dict, reward, done, info = env.step(action_dict)
```