[RLlib] Make it easier to play trained policies

[drozzy]

I’m surprised no one else complains about this… but it is near impossible to try out a trained policy (or at least I can’t find ANY docs about this).

Basically training is easy, but stepping through the environment and feeding it an action selected by policy is not documented anywhere:

Here is an issue I filed about this:

Load, play, rollout a trained/saved policy (without cli rollout.py)

[RickLan]

Hi @drozzy I feel your pain. I came from stable_baselines too. I just wrote a runnable script to try out a trained policy below. It’s for multi-agent but can be easily modified for single agent. I think the docs has an example for single agent, but I couldn’t remember where atm. Cheers,

TF error when restoring from checkpoint, multi-agent

import ray import ray.rllib.agents.ppo as ppo from ray.tune.logger import pretty_print from ray.rllib.examples.env.random_env import RandomMultiAgentEnv num_agents = 2 config = ppo.DEFAULT_CONFIG.copy() config["num_workers"] = 1 config["env_config"] = { "num_agents" : num_agents, } env = RandomMultiAgentEnv(config["env_config"]) config["multiagent"] = { "policies" : { # (policy_cls, obs_space, act_space, config) "{}".format(x): (None, env.observation_space, env.action_space, {}) for …

Edit: there is definitely a higher learning curve for RLlib than stable_baselines, imho. For my research work, I wish I had started with RLlib than stable_baselines.

Edit 2: the single agent version is here: Getting Started with RLlib — Ray 3.0.0.dev0

[sven1977]

Hey @drozzy , great point, and thanks for all your help on this @stefanbschneider and @RickLan .
Yes, we should document this better.

For the LSTM and attention cases, you can also take a look at these example script, where these env loops are described in the comments:

ray.rllib.examples.attention_net.py and ray.rllib.examples.cartpole_lstm.py.