Does RLlib algorithm support both discrete and continuous action spaces simultaneously?

Jay · February 16, 2023, 1:59am

kourosh · February 16, 2023, 2:38am

Hey @Jay , what do you mean by simultaneously? RLlib supports both spaces, i.e. gym.spaces.Box and gym.spaces.Discrete. It also supports more complex spaces such as multi-discrete space.

Jay · February 16, 2023, 3:39am

Hi @kourosh

Thank you for the reply! I am currently working the MLagent Dodgeball environment. The agents have both continuous and discrete action space. I tried to implement it as such.

However, I am still facing difficulties running the script.

Thanks for the help!

kourosh · February 16, 2023, 4:06am

Yes, here is an example:

github.com

ray-project/ray/blob/master/rllib/examples/nested_action_spaces.py

import argparse
from gymnasium.spaces import Dict, Tuple, Box, Discrete
import os

import ray
from ray import air, tune
from ray.tune.registry import register_env
from ray.rllib.examples.env.nested_space_repeat_after_me_env import (
    NestedSpaceRepeatAfterMeEnv,
)
from ray.rllib.utils.test_utils import check_learning_achieved
from ray.tune.registry import get_trainable_cls

parser = argparse.ArgumentParser()
parser.add_argument(
    "--run", type=str, default="PPO", help="The RLlib-registered algorithm to use."
)
parser.add_argument(
    "--framework",
    choices=["tf", "tf2", "torch"],

This file has been truncated. show original

Jay · February 16, 2023, 4:19am

Hi @kourosh

Can I ask for a MARL example and is it possible to tell me if my implementation is faulty in any way? Really appreciate the help!

policies = {
“DodgeballAgent”: PolicySpec(
observation_space=TupleSpace(
[
Box(float(“-inf”), float(“inf”), (3,8)),
Box(float(“-inf”), float(“inf”), (738,)),
Box(float(“-inf”), float(“inf”), (252,)),
Box(float(“-inf”), float(“inf”), (36,)),
Box(float(“-inf”), float(“inf”), (378,)),
Box(float(“-inf”), float(“inf”), (20,))
]
),
action_space=TupleSpace([
Box(-1.0, 1.0, (3,), dtype = np.float32),
MultiDiscrete([2,2])
]
)),
}

config = (
    PPOConfig()
    .environment(
        "unity3d",
        env_config={
            "file_name": None,
            "episode_horizon": None,
        },
        disable_env_checking = True
    )
    .framework("torch")
    # For running in editor, force to use just one Worker (we only have
    # one Unity running)!
    .rollouts(
        num_rollout_workers=0,
        rollout_fragment_length=200,
    )
    .training(
        lr=0.0003,
        lambda_=0.95,
        gamma=0.99,
        sgd_minibatch_size=256,
        train_batch_size=4000,
        num_sgd_iter=20,
        clip_param=0.2,
        model={"fcnet_hiddens": [512, 512]},
    )
    .multi_agent(policies=policies, 
                 policy_mapping_fn=lambda agent_id, *args, **kwargs: "DodgeballAgent",)
    # Use GPUs iff `RLLIB_NUM_GPUS` env var set to > 0.
    .resources(num_gpus=1)
)

kourosh · February 16, 2023, 11:57pm

Is it possible for you to share a clean and concise repro script, so that I can run it on my end to test?

Jay · February 17, 2023, 1:49am

github.com

lohdaijiu/RLlib-implementation/blob/main/Dodgeball_Training.py

"""
Example of running an RLlib Trainer against a locally running Unity3D editor
instance (available as Unity3DEnv inside RLlib).
For a distributed cloud setup example with Unity,
see `examples/serving/unity3d_[server|client].py`

To run this script against a local Unity3D engine:
1) Install Unity3D and `pip install mlagents`.

2) Open the Unity3D Editor and load an example scene from the following
   ml-agents pip package location:
   `.../ml-agents/Project/Assets/ML-Agents/Examples/`
   This script supports the `3DBall`, `3DBallHard`, `SoccerStrikersVsGoalie`,
    `Tennis`, and `Walker` examples.
   Specify the game you chose on your command line via e.g. `--env 3DBall`.
   Feel free to add more supported examples here.

3) Then run this script (you will have to press Play in your Unity editor
   at some point to start the game and the learning process):
$ python unity3d_env_local.py --env 3DBall --stop-reward [..]

This file has been truncated. show original

This is the code that I tried to run. Do note that the error only occurs after I try to run the Unity Dodgeball environment to train the agents.

This is a screenshot of the error for reference.

kourosh · February 22, 2023, 4:46pm

conversation is continued here: Is mixed action spaces supported?

Topic		Replies	Views
Is mixed action spaces supported? RLlib	10	1322	February 23, 2023
RLlib and gym.space RLlib	4	656	November 14, 2021
Example running RL on tuple space RLlib	7	481	February 4, 2022
Action space Discrete is not supported for DQN RLlib	0	36	September 28, 2024
Is any multi discrete action example for PPO or other algorithms? RLlib	9	4109	January 29, 2023

Does RLlib algorithm support both discrete and continuous action spaces simultaneously?

Related topics