# Will the Ray DAG/Workflows choose the best transport for data between two node? **URL:** \ **Category:** Ray Core\ **Created:** [October 23, 2024, 7:31pm UTC](https://discuss.ray.io/t/will-the-ray-dag-workflows-choose-the-best-transport-for-data-between-two-node/16065 "2024-10-23T19:31:26Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![MoFHeka](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mofheka/32/6461_2.png) [@MoFHeka](https://discuss.ray.io/u/MoFHeka)\ **Post date:** [October 23, 2024, 7:31pm UTC](https://discuss.ray.io/t/will-the-ray-dag-workflows-choose-the-best-transport-for-data-between-two-node/16065/1 "2024-10-23T19:31:27Z")
Just like what UCX can do. For example NCCL between GPUs, shared memory/IPC between two CPU workers if they are in one machine and RDMA/TCP directly without object storage if they are in different machine. I know an additional hint could use NCCL, but it only support torch. How about Jax/Tensorflow? What’s more, the transmission usually N producer to M consumer. How can I make this N to M pipe with DAG or WorkFlow? ---
**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\ **Post date:** [October 24, 2024, 4:21am UTC](https://discuss.ray.io/t/will-the-ray-dag-workflows-choose-the-best-transport-for-data-between-two-node/16065/2 "2024-10-24T04:21:32Z")
Default Ray APIs don’t support it. We are building a new API called ray compiled graph which is going to support something similar. But it is starting from - GPU to GPU using nccl when user annotate it - shared memory for local - will use higher performance transport multi node soon > **[\[PUBLIC\] Ray Compiled Graphs developer guide](https://docs.google.com/document/d/1xDs-yIBFLXuqUDLou-_KELERkzWrmGWri6p_xG18SH4/edit#heading=h.kzrrbhon06yz)** > > Ray Compiled Graphs (previously “accelerated DAGs/aDAGs”) Alpha (Developer Preview) Ray 2.32 ships the Alpha (Developer Preview) for a new Ray Core feature called Ray Compiled Graphs Compiled Graphs give you a classic Ray Core-like API but with:... ---
**Author:** ![MoFHeka](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mofheka/32/6461_2.png) [@MoFHeka](https://discuss.ray.io/u/MoFHeka)\ **Post date:** [October 24, 2024, 2:29pm UTC](https://discuss.ray.io/t/will-the-ray-dag-workflows-choose-the-best-transport-for-data-between-two-node/16065/3 "2024-10-24T14:29:00Z")
> [@sangcho](#): > > multi node Thank you for your reply. May I ask if multi-node transport DAG has a design draft? ---
**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\ **Post date:** [October 24, 2024, 6:08pm UTC](https://discuss.ray.io/t/will-the-ray-dag-workflows-choose-the-best-transport-for-data-between-two-node/16065/4 "2024-10-24T18:08:35Z")
It will be posted to the compiled graph channel. But the simple idea is to just use gloo under the hood ---
**Author:** ![ruisearch42](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/ruisearch42/32/6012_2.png) [@ruisearch42](https://discuss.ray.io/u/ruisearch42)\ **Post date:** [October 24, 2024, 6:57pm UTC](https://discuss.ray.io/t/will-the-ray-dag-workflows-choose-the-best-transport-for-data-between-two-node/16065/5 "2024-10-24T18:57:24Z")
For Compiled Graphs multi-node communication, so far it supports NCCL transport (TCP) and shared-memory+RPC based transport, depending on the type hint passed in. As mentioned, we are working on an optimized mechanism. ---
**Author:** ![MoFHeka](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mofheka/32/6461_2.png) [@MoFHeka](https://discuss.ray.io/u/MoFHeka)\ **Post date:** [October 25, 2024, 3:53am UTC](https://discuss.ray.io/t/will-the-ray-dag-workflows-choose-the-best-transport-for-data-between-two-node/16065/6 "2024-10-25T03:53:14Z")
Gloo is a collective communication library. It’s synchronization mechanism may cause some unnecessary overhead between different tasks. ---
**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\ **Post date:** [October 26, 2024, 4:35am UTC](https://discuss.ray.io/t/will-the-ray-dag-workflows-choose-the-best-transport-for-data-between-two-node/16065/7 "2024-10-26T04:35:50Z")
we can use async apis to overlap compute/comm. we are already doing something similar with nccl (which is also collective). Also the design is not final. We are open to hear if you have any recommendation on active message style high performance communication library that’s easy to support! ---
**Author:** ![MoFHeka](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mofheka/32/6461_2.png) [@MoFHeka](https://discuss.ray.io/u/MoFHeka)\ **Post date:** [October 27, 2024, 7:35am UTC](https://discuss.ray.io/t/will-the-ray-dag-workflows-choose-the-best-transport-for-data-between-two-node/16065/8 "2024-10-27T07:35:34Z")
Actually we’re doing something similar, in order to avoid doing the same job, maybe we can get involved in this in some way? In addition to LLM, we focus more on the application on the recommender system. Are there any early open versions? Why choose Gloo? And what’s the key difference between compiled graph and Alpa(A MPMD Jax with Ray ) by the way, here is the comparison between UCX and Gloo: [https://www.researchgate.net/publication/367280846\_Supercharging\_Distributed\_Computing\_Environments\_For\_High\_Performance\_Data\_Engineering/figures?lo=1](https://www.researchgate.net/publication/367280846_Supercharging_Distributed_Computing_Environments_For_High_Performance_Data_Engineering/figures?lo=1) ---
**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\ **Post date:** [October 31, 2024, 3:40pm UTC](https://discuss.ray.io/t/will-the-ray-dag-workflows-choose-the-best-transport-for-data-between-two-node/16065/9 "2024-10-31T15:40:40Z")
I think Alpa can be implemented using compiled graphs. so it is more like lower level abstraction to express task graph that has GPU to GPU communication (and more optimized). ---
**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\ **Post date:** [October 31, 2024, 3:42pm UTC](https://discuss.ray.io/t/will-the-ray-dag-workflows-choose-the-best-transport-for-data-between-two-node/16065/10 "2024-10-31T15:42:12Z")
> Actually we’re doing something similar, in order to avoid doing the same job, maybe we can get involved in this in some way? In addition to LLM, we focus more on the application on the recommender system. btw, I may miss some context. what are you guys working on? ---
**Author:** ![MoFHeka](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mofheka/32/6461_2.png) [@MoFHeka](https://discuss.ray.io/u/MoFHeka)\ **Post date:** [November 1, 2024, 10:45am UTC](https://discuss.ray.io/t/will-the-ray-dag-workflows-choose-the-best-transport-for-data-between-two-node/16065/11 "2024-11-01T10:45:15Z")
Make something like a Pathway with Ray. A MPMD framework which implement is one controller assign different SPMD meshes(For example, Jax, Megatron). ---
**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\ **Post date:** [November 4, 2024, 4:48pm UTC](https://discuss.ray.io/t/will-the-ray-dag-workflows-choose-the-best-transport-for-data-between-two-node/16065/12 "2024-11-04T16:48:47Z")
on interseting. maybe we can definitely talk in person? the motivation of this feature is very similar.