How is large data copied between two nodes?

Ray Core
1

I understand that when data that is saved in one node’s plasma store is required in another, the raylet will copy it from one node to the other.
I would like to understand the protocol that is used to copy the data.
Is it pure tcp? does it use grpc? Does it copy it in chunks? How is the chunk size defined?
Where in the source code can I find this?

Is it possible that in the future this can be exchanged by the user? for example being able to use RDMA?

2

Hi. Ray uses the gRPC.
For high performance communication, for instance using RDMA, its is possible to use libraries like the Ray Collective Communication Library that wrappers libraries like Gloo and NCCL, that can use RDMA/Infiniband

1 Like