Torch Ensemble serving

Ray Serve
1

I’m wondering if anyone has used ray to server models in an ensemble for inference?
Serving ML Models — Ray v1.8.0 references ensembles and pages like Model Ensembling - YOLOv5 Documentation

Looking for any pointers/gists/examples…thx

2

Hi @puntime_error , this is a feature we discussed in serve and actively planning on it as well. You can see a public doc for it: [Public] [Serve] Pipeline Proposal Draft - Google Docs , feel free to leave your comments and feedback !

3

Check this out Pipeline API (Experimental) — Ray v2.0.0.dev0

4

This link seems to be broken, any updates in this regard?

5

Hi @Alex_Al-Saffar , check this out: Model Composition — Ray 2.3.0

1 Like