# MultiAgent Setup

**URL:** <https://discuss.ray.io/t/multiagent-setup/23572>\
**Category:** Uncategorized\
**Created:** [June 27, 2026, 10:17am UTC](https://discuss.ray.io/t/multiagent-setup/23572 "2026-06-27T10:17:58Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tory](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/tory/32/8527_2.png) [@Tory](https://discuss.ray.io/u/Tory)\
**Post date:** [June 27, 2026, 10:17am UTC](https://discuss.ray.io/t/multiagent-setup/23572/1 "2026-06-27T10:17:58Z")

</div>

MultiAgent setup involving multiple agents.

---

<div class="post-metadata">

**Author:** ![RunLLM](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/runllm/32/8249_2.png) [@RunLLM](https://discuss.ray.io/u/RunLLM)\
**Post date:** [June 27, 2026, 10:18am UTC](https://discuss.ray.io/t/multiagent-setup/23572/2 "2026-06-27T10:18:15Z")

</div>

Yes, hierarchical RL using a single `MultiAgentEpisodeReplayBuffer` is possible. The buffer is designed to store episodes from multi-agent environments, including hierarchical setups, by keeping track of experiences for each agent and policy separately, and can handle complex agent-policy mappings as in hierarchical RL scenarios. See [MultiAgentEpisodeReplayBuffer source](https://github.com/ray-project/ray/blob/master/rllib/utils/replay_buffers/multi_agent_episode_buffer.py) and [official docs](https://github.com/ray-project/ray/blob/master/doc/source/rllib/package_ref/replay-buffers.rst).

Would you like more detail on how to configure or use this buffer in a hierarchical RL setup?

Sources:

- [MultiAgentEpisodeReplayBuffer source](https://github.com/ray-project/ray/blob/master/rllib/utils/replay_buffers/multi_agent_episode_buffer.py)
- [official docs](https://github.com/ray-project/ray/blob/master/doc/source/rllib/package_ref/replay-buffers.rst)

Hint: Mention @Herald in the post for followups.

---

<div class="post-metadata">

**Author:** ![RunLLM](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/runllm/32/8249_2.png) [@RunLLM](https://discuss.ray.io/u/RunLLM)\
**Post date:** [June 27, 2026, 10:26am UTC](https://discuss.ray.io/t/multiagent-setup/23572/4 "2026-06-27T10:26:11Z")

</div>

You can implement this by designing your environment’s `step()` method to only return observations for agents expected to act at each timestep. For example, at t1, return obs for both Manager and Workers; at t2–t4, only for Workers; at t5, again for Manager and Workers, and so on. This pattern is supported by RLlib’s multi-agent API and is compatible with a single `MultiAgentEpisodeReplayBuffer` as used in MASAC. See [this discussion](https://discuss.ray.io/t/multi-agent-setting-different-step-sizes-for-agents-and-how-actions-are-passed/5906) and [test\_multi\_agent\_episode.py](https://github.com/ray-project/ray/blob/master/rllib/env/tests/test_multi_agent_episode.py) for reference.

Here’s a minimal dummy environment step logic:

```python
from ray.rllib.env.multi_agent_env import MultiAgentEnv

class HierarchicalEnv(MultiAgentEnv):
    def __init__ (self):
        self.t = 0
        self.manager_steps = {0, 4, 8} # t1, t5, t9 (0-based)
        self.worker_steps = set(range(13)) # t1-t13 (0-based)
        self.agents = ["manager", "worker_1", "worker_2"]

    def reset(self):
        self.t = 0
        return {"manager": 0, "worker_1": 0, "worker_2": 0}

    def step(self, action_dict):
        obs, rew, done, info = {}, {}, {}, {}
        self.t += 1
        # Manager acts only at specified steps
        if self.t in self.manager_steps:
            obs["manager"] = self.t
            rew["manager"] = 0
            done["manager"] = False
            info["manager"] = {}
        # Workers act at every step
        if self.t in self.worker_steps:
            for wid in ["worker_1", "worker_2"]:
                obs[wid] = self.t
                rew[wid] = 0
                done[wid] = False
                info[wid] = {}
        done[" __all__"] = self.t >= 12 # End after t13
        return obs, rew, done, info

```

This structure ensures only the relevant agents act at each timestep, matching your hierarchical RL timing.

Sources:

- [multi-agent step timing discussion](https://discuss.ray.io/t/multi-agent-setting-different-step-sizes-for-agents-and-how-actions-are-passed/5906)
- [test\_multi\_agent\_episode.py](https://github.com/ray-project/ray/blob/master/rllib/env/tests/test_multi_agent_episode.py)

Hint: Mention @Herald in the post for followups.

---

<div class="post-metadata">

**Author:** ![RunLLM](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/runllm/32/8249_2.png) [@RunLLM](https://discuss.ray.io/u/RunLLM)\
**Post date:** [June 27, 2026, 11:20am UTC](https://discuss.ray.io/t/multiagent-setup/23572/6 "2026-06-27T11:20:34Z")

</div>

Certainly! To use `terminated` and `truncated` (as required by RLlib’s Gymnasium-style API), return both per-agent dicts and always include the `" __all__"` key. Here’s the modified dummy environment step:

```python
from ray.rllib.env.multi_agent_env import MultiAgentEnv

class HierarchicalEnv(MultiAgentEnv):
    def __init__ (self):
        self.t = 0
        self.manager_steps = {0, 4, 8} # t1, t5, t9 (0-based)
        self.worker_steps = set(range(13)) # t1-t13 (0-based)
        self.agents = ["manager", "worker_1", "worker_2"]

    def reset(self, *, seed=None, options=None):
        self.t = 0
        return {"manager": 0, "worker_1": 0, "worker_2": 0}, {}

    def step(self, action_dict):
        obs, rew, terminated, truncated, info = {}, {}, {}, {}, {}
        self.t += 1
        # Manager acts only at specified steps
        if self.t in self.manager_steps:
            obs["manager"] = self.t
            rew["manager"] = 0
            terminated["manager"] = False
            truncated["manager"] = False
            info["manager"] = {}
        # Workers act at every step
        if self.t in self.worker_steps:
            for wid in ["worker_1", "worker_2"]:
                obs[wid] = self.t
                rew[wid] = 0
                terminated[wid] = False
                truncated[wid] = False
                info[wid] = {}
        # End after t13 (t=12, 0-based)
        terminated[" __all__"] = self.t >= 12
        truncated[" __all__"] = False
        return obs, rew, terminated, truncated, info

```

This follows the required API: always return both `terminated` and `truncated` dicts with `" __all__"` keys, as shown in [test\_multi\_agent\_env.py](https://github.com/ray-project/ray/blob/master/rllib/env/tests/test_multi_agent_env.py).

Sources:

- [test\_multi\_agent\_env.py](https://github.com/ray-project/ray/blob/master/rllib/env/tests/test_multi_agent_env.py)

Hint: Mention @Herald in the post for followups.

---

<div class="post-metadata">

**Author:** ![RunLLM](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/runllm/32/8249_2.png) [@RunLLM](https://discuss.ray.io/u/RunLLM)\
**Post date:** [June 27, 2026, 11:44am UTC](https://discuss.ray.io/t/multiagent-setup/23572/8 "2026-06-27T11:44:36Z")

</div>

This error means that when calling `concat_episode`, the current episode’s timestep (`self.t`) does not match the next episode’s starting timestep (`other.t_started`). This usually happens if episodes are not split or continued at the correct boundaries, or if your environment does not provide agent actions/observations in the expected sequence. See [single\_agent\_episode.py source](https://github.com/ray-project/ray/blob/master/rllib/env/single_agent_episode.py) and [related discussion](https://discuss.ray.io/t/self-t-other-t-started-training-error/23021).

Would you like more detail on how to debug or fix this in your environment?

Sources:

- [single\_agent\_episode.py source](https://github.com/ray-project/ray/blob/master/rllib/env/single_agent_episode.py)
- [related discussion](https://discuss.ray.io/t/self-t-other-t-started-training-error/23021)

Hint: Mention @Herald in the post for followups.

---

<div class="post-metadata">

**Author:** ![RunLLM](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/runllm/32/8249_2.png) [@RunLLM](https://discuss.ray.io/u/RunLLM)\
**Post date:** [June 27, 2026, 11:46am UTC](https://discuss.ray.io/t/multiagent-setup/23572/10 "2026-06-27T11:46:41Z")

</div>

This error typically occurs when the episode chunks for an agent are not aligned correctly—meaning the end timestep of one chunk (`self.t`) does not match the start timestep of the next chunk (`other.t_started`). This can happen if your environment skips agent actions/observations at certain timesteps or if the episode splitting logic is inconsistent. According to [this Ray issue](https://github.com/ray-project/ray/pull/59895), you must ensure that for each agent, the episode chunks are concatenated only when `self.t == other.t_started`.

To fix this, check that your environment always provides agent data in a way that the timesteps line up across episode boundaries. If your agents act at different frequencies (as in hierarchical RL), make sure that the episode splitting and concatenation logic accounts for “hanging” actions or observations, and that the `MultiAgentEpisode` and `SingleAgentEpisode` objects are constructed with consistent `t` and `t_started` values for each agent. See the robust test and fix approach in [PR #59895](https://github.com/ray-project/ray/pull/59895).

Sources:

- [PR #59895](https://github.com/ray-project/ray/pull/59895)

Hint: Mention @Herald in the post for followups.

---

<div class="post-metadata">

**Author:** ![RunLLM](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/runllm/32/8249_2.png) [@RunLLM](https://discuss.ray.io/u/RunLLM)\
**Post date:** [June 29, 2026, 11:26am UTC](https://discuss.ray.io/t/multiagent-setup/23572/13 "2026-06-29T11:26:43Z")

</div>

You do not need to manually accumulate Worker rewards; RLlib automatically accumulates rewards for each agent between their actions, including in hierarchical RL. Any rewards given to an agent when it is not acting are stored and summed until the agent acts again, at which point the total is attributed to the last action. See [multi-agent reward handling discussion](https://discuss.ray.io/t/multiagentenv-delayed-rewards/22587) and [multi-agent episode code](https://github.com/ray-project/ray/blob/master/rllib/env/multi_agent_episode.py).

Would you like more detail on how this works internally?

Sources:

- [multi-agent reward handling discussion](https://discuss.ray.io/t/multiagentenv-delayed-rewards/22587)
- [multi-agent episode code](https://github.com/ray-project/ray/blob/master/rllib/env/multi_agent_episode.py)

Hint: Mention @Herald in the post for followups.

---

<div class="post-metadata">

**Author:** ![RunLLM](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/runllm/32/8249_2.png) [@RunLLM](https://discuss.ray.io/u/RunLLM)\
**Post date:** [July 9, 2026, 6:24am UTC](https://discuss.ray.io/t/multiagent-setup/23572/15 "2026-07-09T06:24:59Z")

</div>

You can read the source files for RLlib’s MASAC (Multi-Agent Soft Actor-Critic) by exploring the RLlib repository, specifically the SAC algorithm files and multi-agent utilities. The main SAC implementation is in [`rllib/algorithms/sac/sac.py`](https://github.com/ray-project/ray/blob/master/rllib/algorithms/sac/sac.py), and multi-agent support is handled by replay buffers like [`multi_agent_episode_buffer.py`](https://github.com/ray-project/ray/blob/master/rllib/utils/replay_buffers/multi_agent_episode_buffer.py). You may also want to look at example scripts such as [`multi_agent_pendulum_sac.py`](https://github.com/ray-project/ray/blob/master/rllib/examples/algorithms/sac/multi_agent_pendulum_sac.py).

Would you like a step-by-step guide on navigating these files?

Sources:

- [multi\_agent\_pendulum\_sac.py](https://github.com/ray-project/ray/blob/master/rllib/examples/algorithms/sac/multi_agent_pendulum_sac.py)
- [`rllib/algorithms/sac/sac.py`](https://github.com/ray-project/ray/blob/master/rllib/algorithms/sac/sac.py)
- [`multi_agent_episode_buffer.py`](https://github.com/ray-project/ray/blob/master/rllib/utils/replay_buffers/multi_agent_episode_buffer.py)

Hint: Mention @Herald in the post for followups.

---

<div class="post-metadata">

**Author:** ![RunLLM](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/runllm/32/8249_2.png) [@RunLLM](https://discuss.ray.io/u/RunLLM)\
**Post date:** [July 31, 2026, 11:09am UTC](https://discuss.ray.io/t/multiagent-setup/23572/17 "2026-07-31T11:09:45Z")

</div>

It could be possible that RLlib’s MASAC (Multi-Agent Soft Actor-Critic) is based on the standard SAC algorithm, extended for multi-agent settings. Each agent has its own actor (policy network) and twin critics (Q-networks), and the entropy temperature (alpha) is learned to balance exploration and exploitation. The actor outputs actions, the critics estimate Q-values, and the temperature controls entropy regularization. See [SACConfig and code](https://github.com/ray-project/ray/blob/master/rllib/algorithms/sac/sac.py) and [default SAC RLModule](https://github.com/ray-project/ray/blob/master/rllib/algorithms/sac/default_sac_rl_module.py).

To extend or subclass actors, critics, or temperature modules, you might subclass the relevant RLModule or model classes. For example, you can subclass `DefaultSACRLModule` for custom actor/critic logic, or register a custom model for policy or Q-networks via `policy_model_config` and `q_model_config` in `SACConfig`. See [default\_sac\_rl\_module.py](https://github.com/ray-project/ray/blob/master/rllib/algorithms/sac/default_sac_rl_module.py) and [custom model discussion](https://discuss.ray.io/t/custome-the-sactorchmodel/20956).

Would you like a step-by-step example of subclassing one of these components?

Sources:

- [SACConfig and code](https://github.com/ray-project/ray/blob/master/rllib/algorithms/sac/sac.py)
- [default SAC RLModule](https://github.com/ray-project/ray/blob/master/rllib/algorithms/sac/default_sac_rl_module.py)
- [custom model discussion](https://discuss.ray.io/t/custome-the-sactorchmodel/20956)

Hint: Mention @Herald in the post for followups.

---

<div class="post-metadata">

**Author:** ![RunLLM](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/runllm/32/8249_2.png) [@RunLLM](https://discuss.ray.io/u/RunLLM)\
**Post date:** [July 31, 2026, 11:13am UTC](https://discuss.ray.io/t/multiagent-setup/23572/19 "2026-07-31T11:13:23Z")

</div>

It seems likely that RLlib’s MASAC (Multi-Agent Soft Actor-Critic) follows the standard SAC structure, extended for multiple agents. Each agent maintains its own actor, twin critics, and entropy temperature. The high-level pseudocode is:

```auto
Initialize replay buffer (MultiAgentEpisodeReplayBuffer)
For each agent:
    Initialize actor network (policy)
    Initialize two critic networks (Q1, Q2)
    Initialize target critic networks
    Initialize entropy temperature (alpha)

For each episode:
    Reset environment
    For each timestep:
        For each agent:
            Select action using actor (with exploration)
        Step environment with all actions
        Store (obs, actions, rewards, next_obs, terminated, truncated) for all agents in buffer

        If enough samples in buffer:
            For each agent:
                Sample batch from buffer
                Update critics using Bellman backup
                Update actor using policy gradient and entropy regularization
                Adjust temperature alpha to match target entropy
                Soft update target critics

Repeat until training complete

```

To extend, subclass the actor/critic/temperature modules as described in [default\_sac\_rl\_module.py](https://github.com/ray-project/ray/blob/master/rllib/algorithms/sac/default_sac_rl_module.py) and configure via `SACConfig`.

Sources:

- [https://github.com/ray-project/ray/blob/master/rllib/examples/algorithms/sac/multi\_agent\_pendulum\_sac.py](https://github.com/ray-project/ray/blob/master/rllib/examples/algorithms/sac/multi_agent_pendulum_sac.py)

Hint: Mention @Herald in the post for followups.

---

<div class="post-metadata">

**Author:** ![clearonyx718](https://avatars.discourse-cdn.com/v4/letter/c/858c86/32.png) [@clearonyx718](https://discuss.ray.io/u/clearonyx718)\
**Post date:** [November 3, 2026, 11:07pm UTC](https://discuss.ray.io/t/multiagent-setup/23572/20 "2026-11-03T23:07:18Z")

</div>

This post was flagged by the community and is temporarily hidden.

---

<div class="post-metadata">

**Author:** ![mildgasket933](https://avatars.discourse-cdn.com/v4/letter/m/2acd7d/32.png) [@mildgasket933](https://discuss.ray.io/u/mildgasket933)\
**Post date:** [November 12, 2026, 2:57am UTC](https://discuss.ray.io/t/multiagent-setup/23572/21 "2026-11-12T02:57:23Z")

</div>

This post was flagged by the community and is temporarily hidden.

---

<div class="post-metadata">

**Author:** ![oddcanyon200](https://avatars.discourse-cdn.com/v4/letter/o/5daacb/32.png) [@oddcanyon200](https://discuss.ray.io/u/oddcanyon200)\
**Post date:** [November 12, 2026, 3:51am UTC](https://discuss.ray.io/t/multiagent-setup/23572/22 "2026-11-12T03:51:58Z")

</div>

This post was flagged by the community and is temporarily hidden.

---

<div class="post-metadata">

**Author:** ![sharponyx602](https://avatars.discourse-cdn.com/v4/letter/s/c37758/32.png) [@sharponyx602](https://discuss.ray.io/u/sharponyx602)\
**Post date:** [November 12, 2026, 4:49am UTC](https://discuss.ray.io/t/multiagent-setup/23572/23 "2026-11-12T04:49:15Z")

</div>

This post was flagged by the community and is temporarily hidden.

---

<div class="post-metadata">

**Author:** ![sharpquartz113](https://avatars.discourse-cdn.com/v4/letter/s/7feea3/32.png) [@sharpquartz113](https://discuss.ray.io/u/sharpquartz113)\
**Post date:** [November 12, 2026, 5:15am UTC](https://discuss.ray.io/t/multiagent-setup/23572/24 "2026-11-12T05:15:46Z")

</div>

This post was flagged by the community and is temporarily hidden.

---

<div class="post-metadata">

**Author:** ![swiftcircuit327](https://avatars.discourse-cdn.com/v4/letter/s/848f3c/32.png) [@swiftcircuit327](https://discuss.ray.io/u/swiftcircuit327)\
**Post date:** [November 12, 2026, 2:25pm UTC](https://discuss.ray.io/t/multiagent-setup/23572/25 "2026-11-12T14:25:42Z")

</div>

This post was flagged by the community and is temporarily hidden.

---

<div class="post-metadata">

**Author:** ![solidcircuit692](https://avatars.discourse-cdn.com/v4/letter/s/a587f6/32.png) [@solidcircuit692](https://discuss.ray.io/u/solidcircuit692)\
**Post date:** [November 12, 2026, 2:44pm UTC](https://discuss.ray.io/t/multiagent-setup/23572/26 "2026-11-12T14:44:52Z")

</div>

This post was flagged by the community and is temporarily hidden.

---

<div class="post-metadata">

**Author:** ![vividcanyon233](https://avatars.discourse-cdn.com/v4/letter/v/eada6e/32.png) [@vividcanyon233](https://discuss.ray.io/u/vividcanyon233)\
**Post date:** [November 12, 2026, 4:15pm UTC](https://discuss.ray.io/t/multiagent-setup/23572/27 "2026-11-12T16:15:15Z")

</div>

This post was flagged by the community and is temporarily hidden.
