I’ve been reading through the source code of ray train / ray data, so I can get a clearer understanding of what is happening.
One thing I noticed is that–by default–the trainers set the randomize_block_order
parameter to True
.
From looking at what this does; my assumption is that this is useful if you are running multiple separate training runs on the same dataset.
But, presumably, if you are running a single training run (with, for example, 10 GPUs) – then this parameter will have no effect, since each block should roughly only be fetched once? (Since the dataset will be split amongst each worker).
Is my intuition correct?