QMixagent.compute_single_action error

RLlib Configure Algorithm, Training, Evaluation, Scaling
1 
ray_results/DangerUAV2_v1/QMIX/QMIX_DangerUAV2_v1_83686_00000_0_2022-10-05_13-49-20/checkpoint_001500/checkpoint-1500
2022-10-13 13:00:31,396	INFO trainable.py:669 -- Restored on 172.28.0.2 from checkpoint: ray_results/DangerUAV2_v1/QMIX/QMIX_DangerUAV2_v1_83686_00000_0_2022-10-05_13-49-20/checkpoint_001500
2022-10-13 13:00:31,397	INFO trainable.py:677 -- Current state after restoring: {'_iteration': 1500, '_timesteps_total': None, '_time_total': 18878.440465450287, '_episodes_total': 73197}
Traceback (most recent call last):
  File "rllib_test2_danger_v1.py", line 193, in <module>
    action,state_out,_=QMixagent.compute_single_action(obs['group_1'], state)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/algorithms/algorithm.py", line 1148, in compute_single_action
    episode=episode,
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/policy/policy.py", line 331, in compute_single_action
    timestep=timestep,
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/algorithms/qmix/qmix_policy.py", line 317, in compute_actions_from_input_dict
    explore=explore,
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/utils/exploration/epsilon_greedy.py", line 101, in get_exploration_action
    action_distribution, explore, timestep
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/utils/exploration/epsilon_greedy.py", line 188, in _get_torch_exploration_action
    epsilon = self.epsilon_schedule(self.last_timestep)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/utils/schedules/schedule.py", line 46, in __call__
    return self.value(t)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/utils/schedules/schedule.py", line 42, in value
    return self._value(t)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/utils/schedules/piecewise_schedule.py", line 62, in _value
    if l_t <= t < r_t:
TypeError: '<=' not supported between instances of 'int' and 'NoneType'

Any idea?

2

Seems like you are compute_actions_from_input_dict before a timestep has been made. This should also work in theory, but might be a bug on our side. Can you tell us more about when this happens or even provide a repro script?