In the IMPALA trainer when the learner queue is full (I am using an expensive model so the GPU training is slow) the algorithm seems to move training to the CPU, making the training unbearably slow. The desired behavior would be that the CPUs go idle if the GPU is catching up. This effect can be pos…

EDIT - the below fix is not a fix. Setting learner queue size to 0 just makes it infinite size, which makes the learner learn from way off policy data and consumes more and more memory until an out of memory error occurs. For anyone else experiencing extremely slow training when the learner queue f…

Interesting, I’m not sure I understand why RLlib would move training to the CPU (I don’t think that’ll happen as the trained policy is created once on the local worker and thus only placed once on the GPU (or CPU if no GPU is available)). Are you on tf or torch? Is there an easy setup that would r…

Thanks for the reply - I am using torch and would rather not switch as I am using a custom model written in pytorch and in other parts of the project a custom trainer also written for pytorch. I suspect what you said will work as far as reproducing - I have tried a bunch of different methods to circ…

Ok, I would like to debug this. Do you think this is reproducible with e.g. CartPole and a very simple model on the GPU by making the action-compute pass much faster than the batched train pass (by inserting an artificial sleep for example)? I guess one would have to set the queue to something large…

Yes, I think this should work - In a custom model assuming you do not have batched inference set up you can add a sleep that is only activated when the batch size is bigger than one. The queue should not have to be too big (since I believe it is in number of rollout fragments) if the sleep is set at…

I also think I’m running into this bug and would like to know if there’s any fix, or workarounds. Other things I have noticed: the “sample_throughput” variable skyrockets and i get values that are something like 600k samples per second. Graphs on tensorflow show the point at which training become…

[image] Bam4d: the “sample_throughput” variable skyrockets and i get values that are something like 600k samples per second. crossref to Very slow gradient descent on remote workers

[RLlib] Ray trains extremely slow when learner queue is full

RLlib

smorad May 3, 2021, 12:59pm 8

crossref to Very slow gradient descent on remote workers

Topic		Replies	Views
Very slow gradient descent on remote workers RLlib	14	2592	June 8, 2021
Impala does not respect GPU allocation RLlib	4	667	February 26, 2021
Impala Bugs and some other observations RLlib	9	1190	April 27, 2023
Impala seems inefficient (slow), how to properly initialize? RLlib	1	305	September 22, 2022
IMPALA agent not working RLlib	1	357	January 9, 2023

[RLlib] Ray trains extremely slow when learner queue is full

Related topics