Testing RNNs after Training

RLlib
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

I’m training and testing a custom RNN, but I receive the following error message:
File “/home/.conda/envs/ray/lib/python3.9/site-packages/ray/rllib/utils/threading.py”, line 24, in wrapper
return func(self, *a, **k)
File “/home/.conda/envs/ray/lib/python3.9/site-packages/ray/rllib/policy/torch_policy_v2.py”, line 1291, in _compute_action_helper
dist_inputs, state_out = self.model(input_dict, state_batches, seq_lens)
File “/home/.conda/envs/ray/lib/python3.9/site-packages/ray/rllib/models/modelv2.py”, line 259, in call
res = self.forward(restored, state or , seq_lens)
File “/home/.conda/envs/ray/lib/python3.9/site-packages/ray/rllib/models/torch/recurrent_net.py”, line 92, in forward
inputs = add_time_dimension(
File “/home/.conda/envs/ray/lib/python3.9/site-packages/ray/rllib/policy/rnn_sequencing.py”, line 240, in add_time_dimension
new_batch_size = seq_lens.shape[0]
AttributeError: ‘NoneType’ object has no attribute ‘shape’

This is my code, I think I need to add states to it such as here (line 106), but I don’t really know what states or how to get to them or why I’m adding them.

Thank you for your help!

tuner = ray.tune.Tuner(
    "PPO",
    run_config=ray.air.RunConfig(
        storage_path="./results_rllib",
        stop=MaximumIterationStopper(max_iter=100),
        checkpoint_config=ray.air.CheckpointConfig(checkpoint_at_end=True),
    ),
    param_space=config,
)

result_grid = tuner.fit()
best_result = result_grid.get_best_result(metric="episode_reward_mean", mode="max")

env = gym.make("MyEnv-v0", disable_env_checker=True)
obs, info = env.reset()
i = 0
for i in range(20000):
    action = {}
    for agent_id in obs.keys():
        agent_obs = obs[agent_id]
        action[agent_id] = ppo.compute_single_action(agent_obs, policy_id="shared_policy")
    obs, reward, terminated, truncated, info = env.step(action)
    i=+1