# How to divide data freely to worker? **URL:** \ **Category:** Ray Train\ **Created:** [November 3, 2022, 11:48am UTC](https://discuss.ray.io/t/how-to-divide-data-freely-to-worker/8127 "2022-11-03T11:48:58Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Jack\_Chen](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jack_chen/32/2028_2.png) [@Jack\_Chen](https://discuss.ray.io/u/Jack_Chen)\ **Post date:** [November 3, 2022, 11:48am UTC](https://discuss.ray.io/t/how-to-divide-data-freely-to-worker/8127/1 "2022-11-03T11:48:58Z")
**How severe does this issue affect your experience of using Ray?** - High: It blocks me to complete my task. How to divide data freely to worker? Now there are two models. In distributed training, data transfer is required. How to ensure that the data obtained by workers of the two models are consistent ---
**Author:** ![xwjiang2010](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/xwjiang2010/32/1476_2.png) [@xwjiang2010](https://discuss.ray.io/u/xwjiang2010)\ **Post date:** [November 3, 2022, 4:18pm UTC](https://discuss.ray.io/t/how-to-divide-data-freely-to-worker/8127/2 "2022-11-03T16:18:44Z")
Are you referring to data parallel training? If you are using torch, you can take a look at [ray/torch\_trainer.py at master · ray-project/ray · GitHub](https://github.com/ray-project/ray/blob/master/python/ray/train/torch/torch_trainer.py). Similarly we have examples for other frameworks. ---
**Author:** ![Jack\_Chen](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jack_chen/32/2028_2.png) [@Jack\_Chen](https://discuss.ray.io/u/Jack_Chen)\ **Post date:** [November 4, 2022, 2:34am UTC](https://discuss.ray.io/t/how-to-divide-data-freely-to-worker/8127/3 "2022-11-04T02:34:49Z")
can I specify a data block to a specified woker? ---
**Author:** ![xwjiang2010](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/xwjiang2010/32/1476_2.png) [@xwjiang2010](https://discuss.ray.io/u/xwjiang2010)\ **Post date:** [November 4, 2022, 5:18pm UTC](https://discuss.ray.io/t/how-to-divide-data-freely-to-worker/8127/4 "2022-11-04T17:18:17Z")
Are you planning to use Ray Dataset? There is indeed logic in Ray Dataset that decides which dataset block goes to which worker for distributed training. And if the remaining blocks cannot be divided evenly among workers, some logic makes sure to further divide up some blocks into smaller ones and assign them evenly to workers. However, usually this kind of logic is not that important for end users to know about and we abstract that part away from end user. From their perspective, they have a Ray Dataset and all workers can get their shards of data evenly. Do you mind telling us a bit more about your use case and why you would care about such details? ---
**Author:** ![Jack\_Chen](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jack_chen/32/2028_2.png) [@Jack\_Chen](https://discuss.ray.io/u/Jack_Chen)\ **Post date:** [November 5, 2022, 2:45am UTC](https://discuss.ray.io/t/how-to-divide-data-freely-to-worker/8127/5 "2022-11-05T02:45:53Z")
I want to use Ray Train for vertical federated learning, and provider and promoter workers need to accept consistent data. I have another question, can I use TorchTrainer for the model that implements gradient update manually? ---
**Author:** ![xwjiang2010](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/xwjiang2010/32/1476_2.png) [@xwjiang2010](https://discuss.ray.io/u/xwjiang2010)\ **Post date:** [November 7, 2022, 7:38pm UTC](https://discuss.ray.io/t/how-to-divide-data-freely-to-worker/8127/6 "2022-11-07T19:38:56Z")
what do you mean by “consistent data”? My understanding is that the participants in VFL don’t need to share the sample space but rather feature space is split. Could you speak more concretely about what is the concern around data consistency? Maybe something around how do you plan to map provider/promoter worker to ray actor and the data requirement. A schematic would be helpful. As for gradient update, ray train doesn’t do any of the gradient/weight syncing, it’s all offered by torch. Again, I would be curious to learn why you would want to do that manually. ---
**Author:** ![jovany-wang](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jovany-wang/32/364_2.png) [@jovany-wang](https://discuss.ray.io/u/jovany-wang)\ **Post date:** [March 11, 2023, 5:14am UTC](https://discuss.ray.io/t/how-to-divide-data-freely-to-worker/8127/7 "2023-03-11T05:14:00Z")
Hi @Jack_Chen , I can understand your pain points on federated learning scenario, like splitting your models to different parties. We’re working on supporting federated learning on the top of Ray, in [RayFed](https://github.com/ray-project/rayfed) repo. But unfortunately, RayFed hasn’t ported RayTrain APIs. If you have requirements on it or you’re interested in it, please let me know. Thanks Qing ---
**Author:** ![Jules\_Damji](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jules_damji/32/4058_2.png) [@Jules\_Damji](https://discuss.ray.io/u/Jules_Damji)\ **Post date:** [March 14, 2023, 5:56pm UTC](https://discuss.ray.io/t/how-to-divide-data-freely-to-worker/8127/8 "2023-03-14T17:56:21Z")
@Jack_Chen Do you still have questions that have not been resolved. As @xwjiang2010 has suggested the capabilities of Ray Data and Ray Train: both are higher level abstractions so you don’t have to do all bits manually. ---
**Author:** ![121onto](https://avatars.discourse-cdn.com/v4/letter/1/a183cd/32.png) [@121onto](https://discuss.ray.io/u/121onto)\ **Post date:** [April 11, 2024, 3:13am UTC](https://discuss.ray.io/t/how-to-divide-data-freely-to-worker/8127/9 "2024-04-11T03:13:10Z")
I would like to have control over how data is sent to workers. For example, if I want to train an autoregressive sequence model (not an LLM), it would be very memory inefficient to create a dataset where each row represents a record for training.