# How to train multiple policies in one environment?

**URL:** <https://discuss.ray.io/t/how-to-train-multiple-policies-in-one-environment/8889>\
**Category:** RLlib\
**Created:** [January 9, 2023, 9:12am UTC](https://discuss.ray.io/t/how-to-train-multiple-policies-in-one-environment/8889 "2023-01-09T09:12:40Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![coco](https://avatars.discourse-cdn.com/v4/letter/c/8797f3/32.png) [@coco](https://discuss.ray.io/u/coco)\
**Post date:** [January 9, 2023, 9:12am UTC](https://discuss.ray.io/t/how-to-train-multiple-policies-in-one-environment/8889/1 "2023-01-09T09:12:41Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- High: It blocks me to complete my task.

 ![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/d/d23f9ccefab737236af8e777ca84e67179e4c5b2.jpeg)  
 ![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/4/4704edbe16838b6d50ebb138d82146df66cf552d.jpeg)

I am modifying my code by referring to this example.  
This example shows how to bring in pre-trained policies and train other policies.

However, my problem is to checkpoint only one policy that controls nine agents and then restore it, and train another policy that controls the one agent in 10-agents environment that 9 agent’s policy is fixed.  
Since my source code is made to have multiple agents follow one policy,  
config[‘multiagent’] ['policy\_mapping\_fn '] was like  
" policy\_mapping\_fn = lambda x : original policy "  
By the way, I wanted to make one of these agents follow a policy other than the original policy.  
so,

![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/4/4de61ae4065f5363ae4ec3dd07855d636948c308.png)

The code was written as shown in the example above.  
And then, it’s execution result seemed that ‘one policy’ was assigned to ‘every agent’ except ‘agent0’. (one policy per one agent)  
Can you tell me how to modify the code to have ‘multiple agents(exept ‘agent0’)’ refer to ‘only one policy’ ?  
thank you. Have a nice day.

---

<div class="post-metadata">

**Author:** ![arturn](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/arturn/32/2096_2.png) [@arturn](https://discuss.ray.io/u/arturn)\
**Post date:** [January 11, 2023, 4:30pm UTC](https://discuss.ray.io/t/how-to-train-multiple-policies-in-one-environment/8889/2 "2023-01-11T16:30:55Z")

</div>

Hi @coco  
I’m sorry but I don’t understand the problem. The policy mapping function you posted already maps only one agent to one policy while all other agents will be mapped to another policy.  
How is that different from the following?

> Can you tell me how to modify the code to have ‘multiple agents(exept ‘agent0’)’ refer to ‘only one policy’ ?

Also, you can post code here as “Preformatted Text” so you don’t have to make screenshots and others can properly copy your code.

---

<div class="post-metadata">

**Author:** ![arturn](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/arturn/32/2096_2.png) [@arturn](https://discuss.ray.io/u/arturn)\
**Post date:** [January 11, 2023, 4:34pm UTC](https://discuss.ray.io/t/how-to-train-multiple-policies-in-one-environment/8889/3 "2023-01-11T16:34:33Z")

</div>

Also, this is a duplicate of [How can I train multiple 'trainer' in same environment?(or embed trained trainer in environment?) - #3 by coco](https://discuss.ray.io/t/how-can-i-train-multiple-trainer-in-same-environment-or-embed-trained-trainer-in-environment/8700/3) to which you still have to answer.  
Please don’t open a new thread if you have opened another one for the same problem. 🙂

---

<div class="post-metadata">

**Author:** ![coco](https://avatars.discourse-cdn.com/v4/letter/c/8797f3/32.png) [@coco](https://discuss.ray.io/u/coco)\
**Post date:** [January 12, 2023, 2:20am UTC](https://discuss.ray.io/t/how-to-train-multiple-policies-in-one-environment/8889/4 "2023-01-12T02:20:30Z")

</div>

Sorry, I’ll be more careful when I open the thread. 😅