I am new to RLlib, and I just got some lab test scripts in the old API, and I would like to learn the use of RLlib by moving them into the new API. Right now, the first thing that confused me is about learners and resources config for multi-GPU training cases.
Here, I would like to have 8 parallel workers, each to sample a single environment, and training could utilise 2 GPUs. I think in the old API, the above should be set by “rollouts” and “resources” in the config. And in the new API, this is set in “env_runners” and “learners”, and no need to specify in the resources anymore. I am not sure about that the num_gpus option in the resources with New API is still meaningful in this case. Below are my sample setup scripts. Would anyone can clarify this to me? thanks very much.
# config old API
config = (
PPOConfig()
.environment("Pendulum-v1")
.framework("torch")
.training(
gamma=0.99,
lr=3e-4,
model={
"fcnet_hiddens": [128, 128, 64],
"fcnet_activation": "relu",
"vf_share_layers": True}
)
.rollouts(
num_rollout_workers=8,
num_envs_per_worker=1
)
.resources(
num_gpus=2,
num_cpus_per_worker=1
)
)
# config new API
config = (
PPOConfig()
.api_stack(
enable_rl_module_and_learner=True,
enable_env_runner_and_connector_v2=True
)
.framework("torch")
.environment("Pendulum-v1")
.rl_module(
model_config_dict={
"fcnet_hiddens": [128, 128, 64],
"fcnet_activation": "relu",
"vf_share_layers": True}
)
.training(
gamma=0.99,
lr=3e-4
)
.env_runners(
num_env_runners=8,
num_envs_per_env_runner=1,
num_cpus_per_env_runner=1,
num_gpus_per_env_runner=0
)
.learners(
num_learners=2,
num_gpus_per_learner=1
)
.resources(
num_cpus_for_main_process=8,
num_gpus=2 # Would this param still needed in new API?
)
)