RLlib uses synchronous_parallel_sample when collecting data from multiple workers.
sample_batches = ray.get([worker.sample.remote() for worker in worker_set.remote_workers()])
This function means that each worker samples a round of episode respectively and then samples the next round. Will this synchronous method slow down the sampling rate of multiple workers? For example, one worker collects an episode quickly while another worker collects it slowly. Are there any problems if you rewrite this method to be asynchronous? Will it affect training?
Hi @zhangzhang,
I think the idea of the synchronous sampling algorithms is that is that you have a loop where you collect some fixed number of samples, do a training updates, samples new samples, update, … .
One thing that may not be obvious is that when you call .remote() that starts off asynchronous computation. The ray get will wait till all of them return but they are all running at the same time. It is not like it collects the first remote call before it starts the second.
There are asynchronous versions of many algorithms already implemented if that is what you are looking for. You can find them here: Algorithms — Ray 2.2.0
Did you have something else in mind with your question?
If one rollout samples to an episode too slowly, will it affect the sampling speed of other rollouts
yes that is correct @zhangzhang. However we typically sample fragments of a rollout to mitigate this, meaning that from each environment, we might only sample 4 timesteps before returning from a call to synchronous parallel sample.