Hi!
TLDR: How do I format the file name of evaluation output files to include the current training step? Target format for single episodes similar to: output-{TRAINSTEP}_worker{ID}_{BATCH}.json
Full story:
I’m rather new to the ray architecture and I’m surprised how easy it is to execute complex RL models. Thanks to everybody who is contributing! Still I am having trouble finding my way through the architecture.
I am training a Multi-Agent model using PPO in a custom environment. To assess the performance of my model during training time, I want to evaluate my model periodically. Fortunately, I can parameterize my Trainer as follows to save my evaluation episodes:
...
"evaluation_interval": int(5),
"evaluation_num_episodes": 1,
"evaluation_config": {
"explore": False,
"output_compress_columns": [],
"output": "eval_results/",
},
...
Now, every 5 iterations, the evaluation worker will gather a single episode and dump it into a json file, which gets split into parts if a size threshold is reached.
Since I would like to use these evaluation episodes to judge how the training process is progressing, I would like to match the episode to the training step. Until now, I am not sure how to retrieve the training step from the data inside the json file. Furthermore, I think it would be more elegant to split the episode files completely: Each worker gather its trajectories and creates files similar to the previous structure, maybe following a filename format like output-{TRAINSTEP}-{WALLTIME}_worker{ID}_{BATCH}.json. I assume the splitting can be achieved by setting "output_max_file_size": 0. The renaming, however, remains a mystery to me.
I hope there is someone who can help me with this. Thanks!