RLlib MARL examples are not working

james116blue · December 12, 2022, 12:53pm

How severe does this issue affect your experience of using Ray?

High: It blocks me to complete my task.

I struggle to find working examples on PettingZooEnv.
For example, this one

ray-project/ray/blob/ray-2.1.0/rllib/examples/rock_paper_scissors_multiagent.py

"""A simple multi-agent env with two agents playing rock paper scissors.

This demonstrates running the following policies in competition:
    (1) heuristic policy of repeating the same move
    (2) heuristic policy of beating the last opponent move
    (3) LSTM/feedforward PG policies
    (4) LSTM policy with custom entropy loss
"""

import argparse
import os
from pettingzoo.classic import rps_v2
import random

import ray
from ray import air, tune
from ray.rllib.algorithms.pg import (
    PG,
    PGTF2Policy,
    PGTF1Policy,

This file has been truncated. show original

gives me following error:

2022-12-12 15:50:54,850	ERROR trial_runner.py:993 -- Trial PG_RockPaperScissors_9c2e7_00000: Error processing event.
ray.tune.error._TuneNoNextExecutorEventError: Traceback (most recent call last):
  File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/tune/execution/ray_trial_executor.py", line 1050, in get_next_executor_event
    future_result = ray.get(ready_future)
  File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
    return func(*args, **kwargs)
  File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/_private/worker.py", line 2291, in get
    raise value
ray.exceptions.RayActorError: The actor died because of an error raised in its creation task, ray::PG.__init__() (pid=32718, ip=192.168.31.157, repr=PG)
  File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/algorithms/algorithm.py", line 414, in __init__
    super().__init__(config=config, logger_creator=logger_creator, **kwargs)
  File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/tune/trainable/trainable.py", line 161, in __init__
    self.setup(copy.deepcopy(self.config))
  File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/algorithms/algorithm.py", line 524, in setup
    self.workers = WorkerSet(
  File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/evaluation/worker_set.py", line 185, in __init__
    self._local_worker = self._make_worker(
  File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/evaluation/worker_set.py", line 892, in _make_worker
    worker = cls(
  File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/evaluation/rollout_worker.py", line 567, in __init__
    self.policy_dict = _determine_spaces_for_multi_agent_dict(
  File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/evaluation/rollout_worker.py", line 2121, in _determine_spaces_for_multi_agent_dict
    raise ValueError(
ValueError: `observation_space` not provided in PolicySpec for default_policy and env does not have an observation space OR no spaces received from other workers' env(s) OR no `observation_space` specified in config!

2022-12-12 15:50:54,855	ERROR ray_trial_executor.py:111 -- An exception occurred when trying to stop the Ray actor:Traceback (most recent call last):
  File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/tune/execution/ray_trial_executor.py", line 102, in _post_stop_cleanup
    ray.get(future, timeout=timeout)
  File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
    return func(*args, **kwargs)
  File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/_private/worker.py", line 2291, in get
    raise value
ray.exceptions.RayActorError: The actor died because of an error raised in its creation task, ray::PG.__init__() (pid=32718, ip=192.168.31.157, repr=PG)
  File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/algorithms/algorithm.py", line 414, in __init__
    super().__init__(config=config, logger_creator=logger_creator, **kwargs)
  File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/tune/trainable/trainable.py", line 161, in __init__
    self.setup(copy.deepcopy(self.config))
  File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/algorithms/algorithm.py", line 524, in setup
    self.workers = WorkerSet(
  File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/evaluation/worker_set.py", line 185, in __init__
    self._local_worker = self._make_worker(
  File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/evaluation/worker_set.py", line 892, in _make_worker
    worker = cls(
  File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/evaluation/rollout_worker.py", line 567, in __init__
    self.policy_dict = _determine_spaces_for_multi_agent_dict(
  File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/evaluation/rollout_worker.py", line 2121, in _determine_spaces_for_multi_agent_dict
    raise ValueError(
ValueError: `observation_space` not provided in PolicySpec for default_policy and env does not have an observation space OR no spaces received from other workers' env(s) OR no `observation_space` specified in config!

(PG pid=32718) 2022-12-12 15:50:54,841	INFO algorithm.py:457 -- Current log_level is WARN. For more information, set 'log_level': 'INFO' / 'DEBUG' or use the -v and -vv flags.
(PG pid=32718) 2022-12-12 15:50:54,844	WARNING env.py:51 -- Skipping env checking for this experiment
(PG pid=32718) 2022-12-12 15:50:54,846	ERROR worker.py:763 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::PG.__init__() (pid=32718, ip=192.168.31.157, repr=PG)
(PG pid=32718)   File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/algorithms/algorithm.py", line 414, in __init__
(PG pid=32718)     super().__init__(config=config, logger_creator=logger_creator, **kwargs)
(PG pid=32718)   File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/tune/trainable/trainable.py", line 161, in __init__
(PG pid=32718)     self.setup(copy.deepcopy(self.config))
(PG pid=32718)   File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/algorithms/algorithm.py", line 524, in setup
(PG pid=32718)     self.workers = WorkerSet(
(PG pid=32718)   File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/evaluation/worker_set.py", line 185, in __init__
(PG pid=32718)     self._local_worker = self._make_worker(
(PG pid=32718)   File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/evaluation/worker_set.py", line 892, in _make_worker
(PG pid=32718)     worker = cls(
(PG pid=32718)   File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/evaluation/rollout_worker.py", line 567, in __init__
(PG pid=32718)     self.policy_dict = _determine_spaces_for_multi_agent_dict(
(PG pid=32718)   File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/evaluation/rollout_worker.py", line 2121, in _determine_spaces_for_multi_agent_dict
(PG pid=32718)     raise ValueError(
(PG pid=32718) ValueError: `observation_space` not provided in PolicySpec for default_policy and env does not have an observation space OR no spaces received from other workers' env(s) OR no `observation_space` specified in config!
2022-12-12 15:50:54,957	ERROR tune.py:773 -- Trials did not complete: [PG_RockPaperScissors_9c2e7_00000]
2022-12-12 15:50:54,958	INFO tune.py:777 -- Total run time: 2.82 seconds (2.71 seconds for the tuning loop).
2022-12-12 15:50:54,968	INFO algorithm.py:457 -- Current log_level is WARN. For more information, set 'log_level': 'INFO' / 'DEBUG' or use the -v and -vv flags.
2022-12-12 15:50:54,972	WARNING env.py:51 -- Skipping env checking for this experiment
Traceback (most recent call last):
  File "/home/student/projects/rllib/examples/rock_paper_scissors_multiagent_2.1.0.py", line 206, in <module>
    run_heuristic_vs_learned(args, use_lstm=False)
  File "/home/student/projects/rllib/examples/rock_paper_scissors_multiagent_2.1.0.py", line 130, in run_heuristic_vs_learned
    algo = cls(config=config)
  File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/algorithms/algorithm.py", line 414, in __init__
    super().__init__(config=config, logger_creator=logger_creator, **kwargs)
  File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/tune/trainable/trainable.py", line 161, in __init__
    self.setup(copy.deepcopy(self.config))
  File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/algorithms/algorithm.py", line 524, in setup
    self.workers = WorkerSet(
  File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/evaluation/worker_set.py", line 185, in __init__
    self._local_worker = self._make_worker(
  File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/evaluation/worker_set.py", line 892, in _make_worker
    worker = cls(
  File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/evaluation/rollout_worker.py", line 567, in __init__
    self.policy_dict = _determine_spaces_for_multi_agent_dict(
  File "/home/student/anaconda3/envs/rllib/lib/python3.8/site-packages/ray/rllib/evaluation/rollout_worker.py", line 2121, in _determine_spaces_for_multi_agent_dict
    raise ValueError(
ValueError: `observation_space` not provided in PolicySpec for always_same and env does not have an observation space OR no spaces received from other workers' env(s) OR no `observation_space` specified in config!

Actually, there is the same error for other MARL examples.

Single agent algorithms work perfectly.

arturn · December 15, 2022, 8:45am

Hi @james116blue ,

This happens if you use the “wrong” pettingzoo version.
As per our requirements.txt files, we currently support pettingzoo==1.15.0.

george_sk · December 15, 2022, 3:06pm

Hi @arturn ,

since I have the same problem with waterworld_v4 (has corrected some bugs compared to the current supported version) that is later than pettingzoo 1.15.0, do you know if the newer version will be supported in the future or if there are any tricks to use them like using older RLlib versons etc?

Thanks,
George

james116blue · December 15, 2022, 7:12pm

It works.

Thank you for advice

arturn · December 16, 2022, 8:46am

Hi @george_sk ,

I’m can’t give an ETA, but it is on our radar.
We are currently catching up with the gym API first.

Cheers

george_sk · December 16, 2022, 11:58am

Thanks for the reply @arturn . The thing is I am working or a research progress with waterworld and I want to use the fixed v4 version and all my code is with RLlib, so I can’t change now. So if minor changes are needed (imports etc) pm me or let me know in order to maybe try and make it run, especially for versions before ray 2.0.0 that I am more familiar. Otherwise I will declare it on github to be in the queue for fixing.

arturn · December 16, 2022, 1:07pm

Yes, please file an enhancement issue there so people can thumbs up it

Topic		Replies	Views
Rllib multiagent examples are not working RLlib	2	447	January 5, 2023
MultiAgent training Issues RLlib	1	543	April 9, 2024
Failure in running the example code from Ray document RLlib	2	1042	September 22, 2022
Rllid with pettingzoo environment faild RLlib	0	266	July 24, 2022
PettingZoo can not use with rllib RLlib	3	980	August 11, 2022

RLlib MARL examples are not working

Related topics