I am Instantiate the Hugging Face Dataset directly in the train_loop_per_worker but I am not clear if it automatically distributed data among workers (DDP/DP) or each worker work on the full copy of the dataset.
https://docs.ray.io/en/latest/train/user-guides/data-loading-preprocessing.html
Could anyone explain what happens in the background here? My end goal is to enable DDP on single-node/multi-gpu…