Low steps per second after migrating from stablebaselines3

How severe does this issue affect your experience of using Ray?

  • None: Just asking a question out of curiosity
  • Low: It annoys or frustrates me for a moment.
  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.
  • High: It blocks me to complete my task.

High. I’m unable to proceed with my hyperparameter tuning due to a low steps per second.

I have computer with 1 gpu and 1 cpu with 8 logical processors (aka 8 cpus) and used to train my models using stablebaselines3 and would be able to train as fast as 1000 steps per second using the synchronous DummyVecEnv with 16 workers. I just recently migrated to ray and rllib recently and am having trouble reaching higher than 100 steps per second. I have read the documentation over and over and attempted to have multiple workers, multiple envs, fractional gpus, local trainer and/or local worker. Please let me know how I can speed up the training of my models, I heard ray and rllib are optimized for speed so I’m hoping it’s just a config error on my part. Thank for helping.

Note: I have been training using PPO and the default model

Pinging people in the RLlib team

cc: @gjoliver @arturn @avnishn @kourosh

Could you share a little bit about your setup? i.e env, config, etc? Also how expensive is your env.step() calls?