Population-Based Training Replays Do Not Execute Callbacks

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

My Ray.tune hyperarameter study incorporates Population_Based Training and includes a callback to create custom metrics. The tuning process is successful, and the pbt_policy—.txt files are created as expected, in particular with a callbacks entry:

“callbacks”: “<class ‘train_DW.CB_on_episode_end’>”`“callbacks”: “<class ‘train_DW.CB_on_episode_end’>”.

Note the reference to the callback class is a string.

When I then attempt replay using the policy file using PopulationBasedTrainingReplay, I receive the following error:

ValueError: callbacks must be a callable method that returns a subclass of DefaultCallbacks, got <class ‘train_DW.CB_on_episode_end’>

If I remove the callbacks entry from the policy file, the replay succeeds, but of course without including the custom metrics.

I explored trying to register the callback, but there does not seem to be an registry option for this use case.

I would appreciate any guidance on if and how to solve this issue, as I can’t imagine that use of callbacks would be prohibited from PBT replays.

Thanks very much.

David Wilt