Policy weights overwritten in self-play

RLlib
14

Hi @mannyv and @sven1977 ,

I want to help me clarify something on weight syncing on this example.

As I understand when I set weights for a policy I have to do the syncing manually. In this topic, https://discuss.ray.io/t/board-game-self-play-ppo/1425/13 Sven had proposed:

trainer.set_weights({"shared_policy_2": self.men[sel]})  #loading weights 

weights = ray.put(trainer.workers.local_worker().save())
trainer.workers.foreach_worker(
lambda w: w.restore(ray.get(weights))
            )

or

trainer.set_weights({"shared_policy_2": self.men[sel]})  #loading weights 

local_weights = trainer.workers.local_worker().get_weights()
trainer.workers.foreach_worker(lambda worker: worker.set_weights(local_weights))

and Manny here said he had an error with the first method and proposed:

trainer.set_weights({"shared_policy_2": self.men[sel]})

weights = ray.put(trainer.get_weights("shared_policy_2"))
trainer.workers.foreach_worker(
lambda w: w.set_weights(ray.get(weights))
            )

I want to ask if all these method are equivalent (meaning that they sync the weights).
Especially for Manny’s method since it does:

weights = ray.put(trainer.get_weights("shared_policy_2"))

does this sync the weights only for “shared_policy_2” or overwrites the weights of “shared_policy_1” with the ones of “shared_policy_2”, since it is applied on all the workers? Because it seems to me like that, but I am not sure how it works.
Also, all three methods run on my pc without errors.

Thanks,
George