Usage of num_aggregator_actors_per_learner with PPO

RLlib
1

Hi,

I am confused about the documentation and usage of num_aggregator_actors_per_learner in the new API using PPO.
In the docstring it says “Must be at least 1” while the default is set to 0.

If I set it to 1, I get the following error:

ValueError: If episodes provided for training, Learner (<ray.train._internal.worker_group._WrappedExecutable object at 0x7fb75c49d1b0>) must have a LearnerConnector pipeline (but pipeline is None)!), taking actor 0 out of service.

Am I supposed to just keep the default?
If I would like to set a higher value am I supposed to provide a custom LearnerConnector pipeline?

1. Severity of the issue: (select one)
Low: Annoying but doesn’t hinder my work.

2. Environment:

  • Ray version: 2.44.1
  • Python version: 3.10.15
  • OS: Ubuntu 22.04
  • Cloud/Infrastructure: AKS
  • Other libs/tools (if relevant):

3. What happened vs. what you expected:

  • Expected: PPO can make use of aggregator actors by default.
  • Actual: Fails with an error telling me to provide a LearnerConnector.

Gihub Issue on this topic: [RLlib] Documentation of num_aggregator_actors_per_learner · Issue #51874 · ray-project/ray · GitHub