# Init device mesh in pytorch distributed
**URL:** https://discuss.ray.io/t/init-device-mesh-in-pytorch-distributed/22371
**Category:** Ray Train
**Created:** [April 24, 2025, 11:28pm UTC](https://discuss.ray.io/t/init-device-mesh-in-pytorch-distributed/22371 "2025-04-24T23:28:03Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@navmarri1](https://discuss.ray.io/u/navmarri1)
#### Post date: [April 24, 2025, 11:28pm UTC](https://discuss.ray.io/t/init-device-mesh-in-pytorch-distributed/22371/1 "2025-04-24T23:28:03Z")
**1. Severity of the issue: (select one)**
None: I’m just curious or want clarification.
Low: Annoying but doesn’t hinder my work.
Medium: Significantly affects my productivity but can find a workaround.
High: Completely blocks me.
**2. Environment:**
- Ray version: 2.4
- Python version: 3.10
- OS: ubuntu
- Cloud/Infrastructure:
- Other libs/tools (if relevant):
**3. What happened vs. what you expected:**
- Expected:
I am planning to torch [device mesh](https://pytorch.org/tutorials/recipes/distributed_device_mesh.html) when using ray train. However, ray train initializes `dist.init_process_group()` as default ([ref](https://github.com/ray-project/ray/blob/7bf8c41a8faf90683eb60310312a14faf1ab11be/python/ray/train/torch/config.py#L115-L121)). This is needed since fsdp2 requires a device mesh to customize sharding strategies. Is there a workaround for this?
---
### Author:  [@rliaw](https://discuss.ray.io/u/rliaw)
#### Post date: [April 25, 2025, 5:22pm UTC](https://discuss.ray.io/t/init-device-mesh-in-pytorch-distributed/22371/2 "2025-04-25T17:22:00Z")
hi, this is a cool use case. Can you share about about what you want to do with your sharding strategies?
---
### Author:  [@navmarri1](https://discuss.ray.io/u/navmarri1)
#### Post date: [April 26, 2025, 6:08pm UTC](https://discuss.ray.io/t/init-device-mesh-in-pytorch-distributed/22371/3 "2025-04-26T18:08:48Z")
FSDP2 expects device\_mesh to get the device placement and infer sharding strategy. at the moment I’m looking at hybrid sharding. However, device\_mesh doesn’t seem to be an option in ray train. It doesn’t seem to be a blocker since `init_device_mesh` already [checks](https://github.com/pytorch/pytorch/blob/9480ed4cd328e27e5ea74d45e3a9363da51263ce/torch/distributed/device_mesh.py#L455-L457) if `dist.is_initialized()` and doesn’t throw error for duplicate initialization.
It would be great to provide user choice to choose between `init_device_mesh` vs `init_process_group`