Inference pipeline examples

Ray Libraries (Data, Train, Tune, Serve) Ray Serve
1

Hi community

I am doing research about autoscaling of ML inference pipelines and I’m planning to use ray pipeline as my backend system. Currently I am looking for examples, repos or datasets of real-world examples of ML inference pipelines, I was wondering if anyone could provide me real-world examples of inference pipelines being used in production or are used as benchmards? e.g. for microservices GitHub - clowee/MicroserviceDataset: Microservice Dependency Graph Dataset there is this dataset of microservices graphs with the links to the repos but I wasn’t able to find something similar for the ml inference graphs. There is this work that is based on some Google dataset training pipelines [2103.16007] Production Machine Learning Pipelines: Empirical Analysis and Optimization Opportunities but unfortunately it is not open source. whether if you know some examples of real-world repos or a metadata dataset of the graphs both will be super helpful to me.

2

Hi @saeid93 , we’re also actively looking for good patterns to showcase ongoing serve pipeline development work and benchmarks, here’re some candidates based on our user calls and understanding of the industry:

  1. CV/NLP pipeline for complex product content understanding stack. It’s common to have a set of identical images being sent to orders of 10s models, where each is responsible for a given task (segmentation, classification, labeling …) and final prediction is made base on outputs of all combined.

  2. Some combination of the NN recommendation models GitHub - facebookresearch/dlrm: An implementation of a deep learning recommendation model (DLRM)

I think 1) is much more actionable to get stated with public dataset from ImageNet / NLP with pre-trained models with various architecture and size.

1 Like
3

Thank you, that was very helpful, could you please introduce (if any) more real-world multip step systems like the DLRM one that you mentioned?

4

real-world multip step systems like the DLRM

It’s commonly adopted in industry in companies at certain scale, I’m not sure if it’s easy to find public patterns. One such example is multi-stage ranking, where for a given user / product context, rank the top N candidates out of M in the pool, M >>> N.

It’s typically done via 3~4 funnels, using ads as example:
A:O(10000) → B: O(500) → C: O(50) → 10 candidates

At each stage from A to C, model complexity significantly increases with much higher weight on final conversion.

Some datasets of recsys can be found in:

1 Like