Logging metrics/observations on every step

RLlib
1

tldr: has anyone found a nice way to log metrics on every environment step (i.e. more than the default min/avg/max across 400 steps)?

Long version: I need to compute a moving window of sample variances on my observation space which requires that I log every observation (or have access to the last 1000 or so at any given time). Currently I have a custom impl of DefaultCallback that incrementally writes rows to a csv on episode end, then gzips at the end of my training run to save space, but this feels like a pretty lame hack. I tried to add a dataframe as an instance variable to my callback class to append to and write every couple thousand steps, but I think with whatever ray magic that goes on causes the underlying numpy array to have the writeable : FALSE flag, and I can’t modify it. I’ve also noticed that any data put in episode.hist_data ends up getting returned when calling .train() on my Trainable, so I could put it all there and deal with saving it outside the Callback class, but I don’t want to pollute my tensorboard with so many histograms. Is there any alternative I’m missing?

2

Hey @aphoh , did you try using a rolling list as instance property instead of your dataframe? I don’t have an explanation for why your dataframe is not writeable. With the list, you would always (at any env step) have access to the last n values or so.

3

Thanks @sven1977 that worked. I don’t know how the Callback object is handled with ray but I agree it’s a little confusing why that writable flag would get flipped. For whatever reason just using a dictionary of python lists works. Thanks!

1 Like
4

Awesome, thanks for sharing this result @aphoh . Making callbacks and custom metrics more intuitive in the future is definitely on our todo list.