# Does RLlib algorithm support both discrete and continuous action spaces simultaneously?

**URL:** <https://discuss.ray.io/t/does-rllib-algorithm-support-both-discrete-and-continuous-action-spaces-simultaneously/9395>\
**Category:** RLlib\
**Created:** [February 16, 2023, 1:59am UTC](https://discuss.ray.io/t/does-rllib-algorithm-support-both-discrete-and-continuous-action-spaces-simultaneously/9395 "2023-02-16T01:59:23Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jay](https://avatars.discourse-cdn.com/v4/letter/j/ed8c4c/32.png) [@Jay](https://discuss.ray.io/u/Jay)\
**Post date:** [February 16, 2023, 1:59am UTC](https://discuss.ray.io/t/does-rllib-algorithm-support-both-discrete-and-continuous-action-spaces-simultaneously/9395/1 "2023-02-16T01:59:23Z")

</div>

Does RLlib algorithm support both discrete and continuous action spaces simultaneously?

---

<div class="post-metadata">

**Author:** ![kourosh](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kourosh/32/2130_2.png) [@kourosh](https://discuss.ray.io/u/kourosh)\
**Post date:** [February 16, 2023, 2:38am UTC](https://discuss.ray.io/t/does-rllib-algorithm-support-both-discrete-and-continuous-action-spaces-simultaneously/9395/2 "2023-02-16T02:38:57Z")

</div>

Hey @Jay , what do you mean by simultaneously? RLlib supports both spaces, i.e. gym.spaces.Box and gym.spaces.Discrete. It also supports more complex spaces such as [multi-discrete space](https://discuss.ray.io/t/is-any-multi-discrete-action-example-for-ppo-or-other-algorithms/4693/6).

---

<div class="post-metadata">

**Author:** ![Jay](https://avatars.discourse-cdn.com/v4/letter/j/ed8c4c/32.png) [@Jay](https://discuss.ray.io/u/Jay)\
**Post date:** [February 16, 2023, 3:39am UTC](https://discuss.ray.io/t/does-rllib-algorithm-support-both-discrete-and-continuous-action-spaces-simultaneously/9395/3 "2023-02-16T03:39:21Z")

</div>

Hi @kourosh

Thank you for the reply! I am currently working the MLagent Dodgeball environment. The agents have both continuous and discrete action space. I tried to implement it as such.  
 ![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/5/506e99d3220c8e623175b7600e247a5b936fc804.png)  
However, I am still facing difficulties running the script.

Thanks for the help!

---

<div class="post-metadata">

**Author:** ![kourosh](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kourosh/32/2130_2.png) [@kourosh](https://discuss.ray.io/u/kourosh)\
**Post date:** [February 16, 2023, 4:06am UTC](https://discuss.ray.io/t/does-rllib-algorithm-support-both-discrete-and-continuous-action-spaces-simultaneously/9395/4 "2023-02-16T04:06:10Z")

</div>

Yes, here is an example:

> <https://github.com/ray-project/ray/blob/master/rllib/examples/nested_action_spaces.py>

---

<div class="post-metadata">

**Author:** ![Jay](https://avatars.discourse-cdn.com/v4/letter/j/ed8c4c/32.png) [@Jay](https://discuss.ray.io/u/Jay)\
**Post date:** [February 16, 2023, 4:19am UTC](https://discuss.ray.io/t/does-rllib-algorithm-support-both-discrete-and-continuous-action-spaces-simultaneously/9395/5 "2023-02-16T04:19:10Z")

</div>

Hi @kourosh

Can I ask for a MARL example and is it possible to tell me if my implementation is faulty in any way? Really appreciate the help!

policies = {  
“DodgeballAgent”: PolicySpec(  
observation\_space=TupleSpace(  
[  
Box(float(“-inf”), float(“inf”), (3,8)),  
Box(float(“-inf”), float(“inf”), (738,)),  
Box(float(“-inf”), float(“inf”), (252,)),  
Box(float(“-inf”), float(“inf”), (36,)),  
Box(float(“-inf”), float(“inf”), (378,)),  
Box(float(“-inf”), float(“inf”), (20,))  
]  
),  
action\_space=TupleSpace([  
Box(-1.0, 1.0, (3,), dtype = np.float32),  
MultiDiscrete([2,2])  
]  
)),  
}

```
config = (
    PPOConfig()
    .environment(
        "unity3d",
        env_config={
            "file_name": None,
            "episode_horizon": None,
        },
        disable_env_checking = True
    )
    .framework("torch")
    # For running in editor, force to use just one Worker (we only have
    # one Unity running)!
    .rollouts(
        num_rollout_workers=0,
        rollout_fragment_length=200,
    )
    .training(
        lr=0.0003,
        lambda_=0.95,
        gamma=0.99,
        sgd_minibatch_size=256,
        train_batch_size=4000,
        num_sgd_iter=20,
        clip_param=0.2,
        model={"fcnet_hiddens": [512, 512]},
    )
    .multi_agent(policies=policies, 
                 policy_mapping_fn=lambda agent_id, *args, **kwargs: "DodgeballAgent",)
    # Use GPUs iff `RLLIB_NUM_GPUS` env var set to > 0.
    .resources(num_gpus=1)
)

```

---

<div class="post-metadata">

**Author:** ![kourosh](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kourosh/32/2130_2.png) [@kourosh](https://discuss.ray.io/u/kourosh)\
**Post date:** [February 16, 2023, 11:57pm UTC](https://discuss.ray.io/t/does-rllib-algorithm-support-both-discrete-and-continuous-action-spaces-simultaneously/9395/6 "2023-02-16T23:57:58Z")

</div>

Is it possible for you to share a clean and concise repro script, so that I can run it on my end to test?

---

<div class="post-metadata">

**Author:** ![Jay](https://avatars.discourse-cdn.com/v4/letter/j/ed8c4c/32.png) [@Jay](https://discuss.ray.io/u/Jay)\
**Post date:** [February 17, 2023, 1:49am UTC](https://discuss.ray.io/t/does-rllib-algorithm-support-both-discrete-and-continuous-action-spaces-simultaneously/9395/7 "2023-02-17T01:49:26Z")

</div>

> <https://github.com/lohdaijiu/RLlib-implementation/blob/main/Dodgeball_Training.py>

This is the code that I tried to run. Do note that the error only occurs after I try to run the Unity Dodgeball environment to train the agents.

 ![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/5/5ee25808d9a02f56440a3e640fa34dfccc21ae31.png)  
This is a screenshot of the error for reference.

---

<div class="post-metadata">

**Author:** ![kourosh](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kourosh/32/2130_2.png) [@kourosh](https://discuss.ray.io/u/kourosh)\
**Post date:** [February 22, 2023, 4:46pm UTC](https://discuss.ray.io/t/does-rllib-algorithm-support-both-discrete-and-continuous-action-spaces-simultaneously/9395/8 "2023-02-22T16:46:56Z")

</div>

conversation is continued here: [Is mixed action spaces supported?](https://discuss.ray.io/t/is-mixed-action-spaces-supported/9452)