I asked this question on Ray’s Slack channel and now I am transcribing the Thread here so it’s available to everyone. Thanks to @sven1977 and @rliaw for answering my question. Bruno Brandao: Does anyone know how can I pre-train a model with Behaviour Cloning and then load it into PPO for further …

Hi @Ryan_Spangler , could you be a little more specific about what exactly you do not get to work? There is an example here about how to use a pretrained Policy to copy weights over. This is exactly how you would do this after you trained your Policy with BC and want to start PPO with these parameter…

Thanks @BrunoBSM for copying and sharing this here! :slight_smile:

Hello @BrunoBSM , thanks for sharing, it helped me a lot. I was able to do this transfer learning between BC and PPO. I know that because get a good reward when evaluating the PPO agent. However, when I start training the PPO agent it behaves likes it is starting to learn from zero, it has bad episo…

Hi @Anas_BELFADIL , can you provide some more information as of how you’re starting the PPO training? I ask because I’ve been having trouble using these weights inside Tune. Although today I found a solution by mathpluscode that might be the answer to this: It’s the las comment by him When I start …

I just start naively with ppo_agent.train(). Does this re-intialize the weights and discards the weights we set before?

No worries, ppo_agent.train() itself does not reset the weights. Only creating the trainer (first step: trainer = ppo.PPOTrainer(config=...)) does, but if you load the weights (trainer.set_weights() or trainer.get_policy().set_weights()) after that and then start train()ing, it’ll be fine and the p…

@sven1977 , I think without the past data, probably the restored weights gonna be forgotten after a few iterations (due to catastrophic forgetting).

@felipeeeantunes , yes that could very well be the case, that starting to learn with new data (from a previous BC run), e.g. via PPO, could quickly lead to degrading performance, meaning worse than what the BC run achieved. But this may have to be tried from case to case.

In my experience, train()ing after trainer.set_weights() doesn’t continue training, it somehow loses the pre-training. But using this solution in the last comment here last comment as pointed by @BrunoBSM , solves the problem. Here is a notebook Colab with recreation of the problem and the solution, …

Hi I had the same issues but have fixed it by just saving and loading a new checkpoint bcweights = bcloning.get_weights() agent = ppo.PPOTrainer(config) agent.set_weights(bcweights) checkpoint_path = agent.save() print(checkpoint_path) # may want to add ray.shutdown and ray.init here agent = pp…

How to pretrain a model with behavior cloning

RLlib

Finebouche December 5, 2023, 3:28pm 16

HI @Ryan_Spangler , did you managed to make it work using @Lars_Simon_Zehnder example ? I am struggling to recover weight

Topic		Replies	Views
Transfer Learning for Multi-Agent env. with RLlib RLlib	4	872	September 21, 2022
Loading pre-trained single-agent policy weights for multi-agent training RLlib	2	957	June 11, 2021
Loading pre-trained BC policy weight for tunning with hyper-parameter optimization Checkpointing, Restoring	1	77	August 28, 2024
Hybrid Offline learning and PPO? Offline RL	4	1099	April 17, 2025
Behavioural Cloning Algo RLlib	6	1808	May 24, 2021

How to pretrain a model with behavior cloning

Related topics