Using Ray Plasma Effectively

[SumanthDatta]

Hi ,

We are using RaySGD dataset for our training ,parallel iterator reads data from our persistent store.

During every training epoch , iterator is fetching data again from persistent store. Ray Plasma is filled up to some extent.Instead of reading from persistent store often, iterator has to read most of the data from a plasma store. How to effectively achieve that.

[sangcho]

cc @rliaw Can you answer his question?

[rliaw]

@SumanthDatta

During every training epoch , iterator is fetching data again from persistent store. Ray Plasma is filled up to some extent.Instead of reading from persistent store often, iterator has to read most of the data from a plasma store. How to effectively achieve that.

Maybe you could try using the Dataset API? The MLDataset API cc @Kai_Huang could also be useful here.

https://docs.ray.io/en/master/raysgd/raysgd_dataset.html

[SumanthDatta]

@rliaw we are already using raysgd dataset API.

[Kai_Huang]

Thanks @rliaw I’ll look at it