Sequence/Tensor Parallelism with Ray Serve

1

Are there any examples/demos on how to do this for inference. Got a big model which needs sequence parallelism and looking to split the workload 8x on a node.

Thanks

2

vLLM on Ray Serve will give you Tensor Parallelism baked in and probably your best bet. Guide coming soon!

3

here’s an example of setting up vllm with Ray Serve - Serve a Large Language Model with vLLM — Ray 3.0.0.dev0