I want to do some complicated training using RLlib and I’m not sure how.
I have an environment for two agents, and I want to train the first agent while I’m forcing the policy of the second agent to be a hard-coded policy that I write. I want to run that training for 10 steps. Then I want to continue training both agents normally for 10 more steps. That means that in the second training, the first agent is starting out with the policy that I trained in the first training phase, while the second agent is starting with a blank policy.
I think yes, it’s possible; and here is a sketch of how using callbacks.
Consider having two policies. Let’s call them {‘main’, ‘hard-coded’}.
make main the only trainable policy.
using the on_train_result() hook of your callback you can check a criteria (e.g. self.iteration % 10 == 0) and do what ever you want with the algorithm and policy states. You can switch out the policy states, and let the new policy start from scratch. You can add a new policy. You can update the policy_mapping fn, etc. I recommend taking a look at self_play_with_open_spiel.py to get ideas on how this can be done.
