Hi,
Is there a way that ray can internally split the given data set into n batches as per the number of worker processes available so that each can process a range of given data set?
I have come across ds.split(n=n, locality_hints=workers) but this ds has to be stored locally. Instead, I am looking for a solution where ray can internally split the data and provide those ranges to each worker process.
Thanks