Num workers speedup?

RLlib
1
  • Low: It annoys or frustrates me for a moment.

I’ve been running some experiments with a simple multiagent game varying the number of workers using PG algorithm. Here are some tensorboard results with 0 workers, 1 worker, and 7 workers (my laptop only has 8 cpus). The checkpoint frequency is 50 and the number of episodes is 2000 for each run.

Screen Shot 2022-04-28 at 6.56.13 PM
Screen Shot 2022-04-28 at 6.56.13 PM1754×1106 166 KB

The larger the number of workers, the faster the run. It’s not a perfect geometric scale, but 8x workers for 2-3x gain is still good, and like I said this is a very simple environment.

It gets a little bit interesting when I look at how how long it takes for a certain score to be reached. I wish I could share the interactive tensorboard, but I’ll just show a table here, where data is number of seconds for each run (rows) to achieve a score (columns). (I collected the data using the un-smoothed curves.)

+ ------------- + --- + --- + --- + --- +
| workers/score | 200 | 300 | 400 | 450 |
+ ------------- + --- + --- + --- + --- +
| 0 workers     | 3   | 4.5 | 7.5 | 10  |
| 1 worker      | 4   | 6   | 8   | 13  |
| 7 workers     | 1   | 2   | 5.5 | 18  |
+ ------------- + --- + --- + --- + --- +

So in every case, 0 workers hits the score markers faster than 1 worker. 7 workers is faster than 0 workers at first, but at the upper levels of the score, it seriously slows down, taking almost 2x longer to reach 450. This table suggests that there may not really be much gain in increasing the number of workers, which doesn’t seem to make sense. What am I missing?

2

Hi @rusu24edward ,

A couple of things for you to determine what causes this:

  1. Have a look at the reward per sampled experience, rather than wall clock time.
    This will tell you if the algorithm itself is influenced by the number of workers. Some algorithms may be influenced heavily by this - for example DDPPO.
  2. Are these rewards from only three independent runs? If you compare settings, always remember to fix seeds!
  3. For an even better picture, not only fix seeds, but fix them to different values.

Let me know if you get any more insight or consider sharing your results with tensorboard dev so we can have a better look at them!