Assert seq_lens is not None -> PPOTrainer

RLlib
1

Hi all,

I am trying wrap the default FCNET with an LSTM, but even setting the “use_LSTM”: True, results in the following error:

Traceback (most recent call last):
  File "policy_server.py", line 198, in <module>
    trainer = PPOTrainer(config=DEFAULT_CONFIG, env=RandomEnv)
  File "C:\Users\ashyk\AppData\Local\Programs\Python\Python38\lib\site-packages\ray\rllib\agents\trainer_template.py", line 123, in __init__
    Trainer.__init__(self, config, env, logger_creator)
  File "C:\Users\ashyk\AppData\Local\Programs\Python\Python38\lib\site-packages\ray\rllib\agents\trainer.py", line 548, in __init__
    super().__init__(config, logger_creator)
  File "C:\Users\ashyk\AppData\Local\Programs\Python\Python38\lib\site-packages\ray\tune\trainable.py", line 98, in __init__
    self.setup(copy.deepcopy(self.config))
  File "C:\Users\ashyk\AppData\Local\Programs\Python\Python38\lib\site-packages\ray\rllib\agents\trainer.py", line 709, in setup
    self._init(self.config, self.env_creator)
  File "C:\Users\ashyk\AppData\Local\Programs\Python\Python38\lib\site-packages\ray\rllib\agents\trainer_template.py", line 150, in _init
    self.workers = self._make_workers(
  File "C:\Users\ashyk\AppData\Local\Programs\Python\Python38\lib\site-packages\ray\rllib\agents\trainer.py", line 791, in _make_workers
    return WorkerSet(
  File "C:\Users\ashyk\AppData\Local\Programs\Python\Python38\lib\site-packages\ray\rllib\evaluation\worker_set.py", line 92, in __init__
    self._local_worker = self._make_worker(
  File "C:\Users\ashyk\AppData\Local\Programs\Python\Python38\lib\site-packages\ray\rllib\evaluation\worker_set.py", line 367, in _make_worker
    worker = cls(
  File "C:\Users\ashyk\AppData\Local\Programs\Python\Python38\lib\site-packages\ray\rllib\evaluation\rollout_worker.py", line 534, in __init__
    self._build_policy_map(policy_dict, policy_config)
  File "C:\Users\ashyk\AppData\Local\Programs\Python\Python38\lib\site-packages\ray\rllib\evaluation\rollout_worker.py", line 1193, in _build_policy_map
    policy_map[name] = cls(obs_space, act_space, merged_conf)
  File "C:\Users\ashyk\AppData\Local\Programs\Python\Python38\lib\site-packages\ray\rllib\policy\tf_policy_template.py", line 237, in __init__
    DynamicTFPolicy.__init__(
  File "C:\Users\ashyk\AppData\Local\Programs\Python\Python38\lib\site-packages\ray\rllib\policy\dynamic_tf_policy.py", line 351, in __init__
    self._initialize_loss_from_dummy_batch(
  File "C:\Users\ashyk\AppData\Local\Programs\Python\Python38\lib\site-packages\ray\rllib\policy\dynamic_tf_policy.py", line 467, in _initialize_loss_from_dummy_batch
    self._optimizer = self.optimizer()
  File "C:\Users\ashyk\AppData\Local\Programs\Python\Python38\lib\site-packages\ray\rllib\policy\tf_policy_template.py", line 280, in optimizer
    optimizers = self.exploration.get_exploration_optimizer(
  File "C:\Users\ashyk\AppData\Local\Programs\Python\Python38\lib\site-packages\ray\rllib\utils\exploration\curiosity.py", line 221, in get_exploration_optimizer
    self._postprocess_helper_tf(
  File "C:\Users\ashyk\AppData\Local\Programs\Python\Python38\lib\site-packages\ray\rllib\utils\exploration\curiosity.py", line 267, in _postprocess_helper_tf
    phis, _ = self.model._curiosity_feature_net({
  File "C:\Users\ashyk\AppData\Local\Programs\Python\Python38\lib\site-packages\ray\rllib\models\modelv2.py", line 234, in __call__
    res = self.forward(restored, state or [], seq_lens)
  File "C:\Users\ashyk\AppData\Local\Programs\Python\Python38\lib\site-packages\ray\rllib\models\tf\recurrent_net.py", line 196, in forward
    assert seq_lens is not None
AssertionError

I am using the latest Windows wheel with python 3.8. Any idea what it could be?

2

Try disabling the curiosity module. I think it should work without it.

3

You are correct that this worked, however, I was under the impression that the two should be able to work together/seperately? Is this a bug or intended interaction?

4

@Denys_Ashikhin,

Here is what I think is happening.

The curiosity module takes an optional feature_net_config. As you can see below if you do not provide one, it will use a copy of the one that generated the policy model. Since you are using the auto_lstm feature that will create a curiosity module with an lstm in it. Try passing a feature_net_config with model[“use_lstm”] = False.

@sven1977 @gjoliver Perhaps there should be a warning, error, or auto-disabling of the use_lstm feature for this case in the curiosity exploration module.

1 Like
5

Thanks a bunch, that seems to get it running - now to see if it will actually learn hahah :sweat_smile: