What to expect from RLlib Connectors Alpha State

RLlib
1

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

Hello everyone,

I have been exploring the RLlib connectors (Connectors (Alpha) — Ray 2.5.0) for our team’s migration from Stable Baselines3 (SB3) to Ray. While the release notes mention that all algorithms are supported since version 2.3.0, the connectors themselves are still marked as alpha. Our specific requirement involves a complex connector that transforms observations using a Graph Neural Network (GNN) before feeding them to the policy.

We are hesitant to build our solution on alpha features and would like to gather more information about their current state and implications. It would be greatly appreciated if you could provide insights on the following:

  1. What can we expect from the alpha state of RLlib connectors?
  2. Are there any known limitations or risks associated with using alpha features?
  3. Are there alternative approaches or methods we can consider to achieve similar functionality without relying on alpha features or connectors?

We value the stability and reliability of our system, and we want to ensure we make informed decisions regarding our migration. Any advice, guidance, or suggestions you can offer would be of great help to our team.

Thank you for your time and assistance.

2

Hello @Nicolas_Gehring

  1. The connectors are marked as alpha because there are several TODOs left before making it generally available across all use-cases that RLlib supports today. If your use-cases has nothing to do with the remaining items then you can use connectors.

image
image1179×1300 192 KB

We have intentionally not made progress on these items because we have been working on redefining some of the existing internal abstractions to reduce the complexity of the library. So by the time that that is done, some of these todo items may be replaced by new concepts, or may become irrelevant.

  1. For connectors not so much, we have paying customers that are using these “so called” alpha features and we have not updated any of the requirements of that feature for them for almost a year. Having said that, the use-cases that we have been exposed to may not be representative of what you might do with connectors. But since connectors are stateful python code, they should be very flexible in terms of the use-cases they can support.

  2. There is no alternative to replace connectors. Connectors are preprocessing components that will be check-pointed withe the algorithm so that you can get durable policies even if your environment changes a little bit (for example you add a new action head to your env). If you do not need durability and the checkpointing of preprocessors you can use environment wrappers and apply your custom transformations on the environment side.

Please Note that connectors are already enabled by default unless we have explicitly disabled it for one of those todo reasons.