# Model Parallelism in Ray **URL:** \ **Category:** Ray Train\ **Created:** [January 7, 2022, 6:57pm UTC](https://discuss.ray.io/t/model-parallelism-in-ray/4617 "2022-01-07T18:57:24Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Vipul\_Gupta](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/vipul_gupta/32/1806_2.png) [@Vipul\_Gupta](https://discuss.ray.io/u/Vipul_Gupta)\ **Post date:** [January 7, 2022, 6:57pm UTC](https://discuss.ray.io/t/model-parallelism-in-ray/4617/1 "2022-01-07T18:57:25Z")
Hi folks, it seems like Ray Train focuses on distributed training with data parallelism. I am wondering if there is a use case with model parallelism. In our specific use-case, we are training large-scale embeddings, and these typically require model parallelism due to a large embedding matrix that cannot fit in the memory of one machine. ---
**Author:** ![rliaw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rliaw/32/24_2.png) [@rliaw](https://discuss.ray.io/u/rliaw)\ **Post date:** [January 7, 2022, 7:09pm UTC](https://discuss.ray.io/t/model-parallelism-in-ray/4617/2 "2022-01-07T19:09:35Z")
That’s a great use case. You mentioned offline that you were looking at Pytorch Biggraph in particular? ---
**Author:** ![Vipul\_Gupta](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/vipul_gupta/32/1806_2.png) [@Vipul\_Gupta](https://discuss.ray.io/u/Vipul_Gupta)\ **Post date:** [January 8, 2022, 6:28am UTC](https://discuss.ray.io/t/model-parallelism-in-ray/4617/3 "2022-01-08T06:28:11Z")
Thanks, Richard, for the quick response. Yes, in particular, we are looking to train node embeddings on large graphs, and use Pytorch-BigGraph as a framework for training over a Ray cluster. ---
**Author:** ![rliaw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rliaw/32/24_2.png) [@rliaw](https://discuss.ray.io/u/rliaw)\ **Post date:** [January 11, 2022, 3:29am UTC](https://discuss.ray.io/t/model-parallelism-in-ray/4617/4 "2022-01-11T03:29:04Z")
Officially, we don’t have any pre-existing examples. However, it should work fine (given that Ray Train just constructs the process group for you). We would be happy to help guide you through the implementation, if you have any particular questions. ---
**Author:** ![rliaw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rliaw/32/24_2.png) [@rliaw](https://discuss.ray.io/u/rliaw)\ **Post date:** [January 11, 2022, 3:29am UTC](https://discuss.ray.io/t/model-parallelism-in-ray/4617/5 "2022-01-11T03:29:23Z")
(and also subsequently highlight your use case as a successful example down the road!) ---
**Author:** ![Vipul\_Gupta](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/vipul_gupta/32/1806_2.png) [@Vipul\_Gupta](https://discuss.ray.io/u/Vipul_Gupta)\ **Post date:** [January 13, 2022, 7:16pm UTC](https://discuss.ray.io/t/model-parallelism-in-ray/4617/6 "2022-01-13T19:16:40Z")
Of course! Would be happy to contribute to the ray ecosystem in any which way. The fact that Ray Train just constructs the process group makes sense. However, the examples provided in the [documentation](https://docs.ray.io/en/latest/train/train.html) give an impression that the API is limited and works mostly for the data parallel case. After decoding PyTorch-BigGraph, I think I can train with model parallelism on Ray if only I can replace init\_process\_group from torch.distributed by an equivalent ray function. Is there a similar API in Ray? Thanks! ---
**Author:** ![rliaw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rliaw/32/24_2.png) [@rliaw](https://discuss.ray.io/u/rliaw)\ **Post date:** [January 15, 2022, 5:13am UTC](https://discuss.ray.io/t/model-parallelism-in-ray/4617/7 "2022-01-15T05:13:56Z")
Can you post an example of what you want to do? ---
**Author:** ![jamjambles](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jamjambles/32/5252_2.png) [@jamjambles](https://discuss.ray.io/u/jamjambles)\ **Post date:** [October 28, 2023, 3:23am UTC](https://discuss.ray.io/t/model-parallelism-in-ray/4617/8 "2023-10-28T03:23:09Z")
Hi all, Coming across a similar use case where we want to use Ray Train to split a large model across multiple GPUs rather than replicate (data parallel). For example I have a cluster with A10 GPUs (24GB) but the model requires ~50GB of GPU memory. Can I use Ray Train APIs to partition the model across multiple GPUs? Are there any examples? Thanks! ---
**Author:** ![justinvyu](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/justinvyu/32/4309_2.png) [@justinvyu](https://discuss.ray.io/u/justinvyu)\ **Post date:** [November 2, 2023, 9:27pm UTC](https://discuss.ray.io/t/model-parallelism-in-ray/4617/9 "2023-11-02T21:27:26Z")
Hey @jamjambles, The easiest way would be to use deepspeed with a Zero-1/2/3 sharding strategy using Ray Train. See this user guide for more details: [Get Started with DeepSpeed — Ray 2.7.1](https://docs.ray.io/en/latest/train/deepspeed.html) Let me know if you’re able to set that up. I’m also happy to move to a slack conversation to discuss further! ---
**Author:** ![Magic\_Liu\_Liu](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/magic_liu_liu/32/5358_2.png) [@Magic\_Liu\_Liu](https://discuss.ray.io/u/Magic_Liu_Liu)\ **Post date:** [November 18, 2023, 3:49pm UTC](https://discuss.ray.io/t/model-parallelism-in-ray/4617/10 "2023-11-18T15:49:52Z")
@rliaw can you help me, i just wanna use ray to serve llama2-70b in 2 a10-4gpus vms. ---
**Author:** ![psnilesh](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/psnilesh/32/8424_2.png) [@psnilesh](https://discuss.ray.io/u/psnilesh)\ **Post date:** [January 9, 2026, 4:07am UTC](https://discuss.ray.io/t/model-parallelism-in-ray/4617/11 "2026-01-09T04:07:10Z")
@justinvyu Could you point me to an example using Megatron?