I have been using raysgd for parallel distribution of fetching data using sgd dataset API and training using TFtrainer and Torch trainer.
Data fetching happens from different datasources. Trying to evaluate the performance of reading , using Python Cprofiler , any good approach to evaluate the performance , what metrics to look for while evaluating.
@rliaw What’s the best way to profile datasets APIs?
Hey @SumanthDatta , great question and sorry for the delay, which was caused by the question being “uncategorized”. It helps if you set a category (e.g. “Tune”) when you post a new question. That way, we’ll find it more easily and can assign the right person to answer it.
@rliaw , could someone from the SGD team answer this? Thanks 