Ray 1.6.0 Impala multiagent, PolicyID 'default_policy' not found in this PolicyMap

Hi
I am trying to train an RL agent for GFootball using ImpalaTrainer from ray 1.6.0 in which I am facing this issue when using multiagent config for setting two Impala agents.

(pid=7732)   File "/opt/conda/lib/python3.7/site-packages/ray/rllib/agents/trainer_template.py", line 136, in __init__
(pid=7732)     Trainer.__init__(self, config, env, logger_creator)
(pid=7732)   File "/opt/conda/lib/python3.7/site-packages/ray/rllib/agents/trainer.py", line 592, in __init__
(pid=7732)     super().__init__(config, logger_creator)
(pid=7732)   File "/opt/conda/lib/python3.7/site-packages/ray/tune/trainable.py", line 103, in __init__
(pid=7732)     self.setup(copy.deepcopy(self.config))
(pid=7732)   File "/opt/conda/lib/python3.7/site-packages/ray/rllib/agents/trainer_template.py", line 146, in setup
(pid=7732)     super().setup(config)
(pid=7732)   File "/opt/conda/lib/python3.7/site-packages/ray/rllib/agents/trainer.py", line 739, in setup
(pid=7732)     self._init(self.config, self.env_creator)
(pid=7732)   File "/opt/conda/lib/python3.7/site-packages/ray/rllib/agents/trainer_template.py", line 177, in _init
(pid=7732)     self.train_exec_impl = execution_plan(self.workers, config)
(pid=7732)   File "/opt/conda/lib/python3.7/site-packages/ray/rllib/agents/impala/impala.py", line 298, in execution_plan
(pid=7732)     learner_thread = make_learner_thread(workers.local_worker(), config)
(pid=7732)   File "/opt/conda/lib/python3.7/site-packages/ray/rllib/agents/impala/impala.py", line 177, in make_learner_thread
(pid=7732)     learner_queue_timeout=config["learner_queue_timeout"])
(pid=7732)   File "/opt/conda/lib/python3.7/site-packages/ray/rllib/execution/multi_gpu_learner_thread.py", line 66, in __init__
(pid=7732)     self.policy = self.local_worker.policy_map[DEFAULT_POLICY_ID]
(pid=7732)   File "/opt/conda/lib/python3.7/site-packages/ray/rllib/policy/policy_map.py", line 152, in __getitem__
(pid=7732)     raise KeyError(f"PolicyID '{item}' not found in this PolicyMap!")
(pid=7732) KeyError: "PolicyID 'default_policy' not found in this PolicyMap!"

Hi @Avatara_Ten,

Can you share your config and if possible a reproduction script?