Very slow gradient descent on remote workers

@sven1977 this is the same issue as in this discussion: [RLlib] Ray trains extremely slow when learner queue is full