Persisting values across callbacks

RLlib
#1

I’d like to know if I can add an attribute to my DefaultCallbacks and have it persist across multiple on_episode_end callbacks.

What i want is a metric that will average a value over the last 100 episodes. However, there are two things I need in this metric:

  1. if the model completed less than 100 episodes, I still want to sum them up and divide by 100.
  2. At some points in training, I need to “reset” this list of 100 episodes as if the model were starting again.

So, my thoughts were “let’s make a custom_metric.” If I add a collections.deque to my DefaultCallbacks with maxlen=100 and append to it at every on_episode_end, I can check it at the on_train_result callback (where I need it) and add it to the results dictionary. However, I do not know exactly if, when RAY parallelizes into multiple workers, there will be multiple instances of this callback object. Will I be able to trust this value? the on_episode_end callback is called in a “centralized” way?

#2

Hey @BrunoBSM , custom callback objects are not centralized, meaning each worker (num_workers) will have its own instance of your provided class and will thus do its own on_episode_end call.

However, you can add episode.hist_data (build lists with values per-timestep) and episode.custom_metrics (put per-episode metrics here).

In the on_train_results method (only called on your “learner”, meaning centralized), you should then see these metrics in the results argument:

image
image1658×1214 354 KB

This is from our example script in ray.rllib.examples.custom_metrics_and_callbacks.py