RLlib: How to use saved SingleAgentEpisodes for SAC?

RLlib
1

I’m sorry my code is too large, I can’t reduce it to a reproducible code.

How can I use a saved data that is generated from expert for SAC?

The data is saved similar to Working with offline data — Ray 2.49.2’s
”””If your expert data is given in columnar format and you need to train on full expert trajectories you can follow the code in the following example to convert your own data into RLlib’s SingleAgentEpisode objects:””” example.

I use ray.data to save SingleAgentEpisodes to a path. But I don’t know how to use this data for SAC.

When I use SACConfig().offline_data(input=”path_to_single_agent_episodes”), it just runs without using the offline data. My expectation was for it to add it to its replay buffer.

2

To use expert data saved as RLlib SingleAgentEpisode objects for SAC, you must ensure that the SAC algorithm is configured to read data in RLlib’s episode format. By default, SAC expects tabular (columnar) data, not SingleAgentEpisode objects. You need to set the config option input_read_episodes=True in .offline_data() so RLlib knows to load episodes, not tabular rows. Without this, SAC will ignore your episode data and not add it to the replay buffer, as you observed.

Example:

from ray.rllib.algorithms.sac import SACConfig

config = (
    SACConfig()
    .offline_data(
        input_="path_to_single_agent_episodes",
        input_read_episodes=True,
    )
)

This tells RLlib to read your saved SingleAgentEpisode data and use it for offline training with SAC. For more, see the relevant section in the RLlib offline data docs.

Would you like a step-by-step breakdown or more details on how to structure your data or config?

Sources:

Hint: Mention @RunLLM in the post for followups.