How severe does this issue affect your experience of using Ray? High: It blocks me to complete my task. Hi Team, The Ray Train job fails after the completion of all the epochs (Huggingface trainable). In the training arguments, if load_best_model_at_end: true is passed, the below error occurs,…

TorchTrainer: Collective operation timeout: WorkNCCL

saivivek15 July 17, 2023, 4:43pm 2

@kai @xwjiang2010, could you please suggest any workaround for this issue?

AIR, TorchTrainer, DDP and NCCL Timeout

Topic		Replies	Views
Errors when test TorchTrainer with the "getting started" code Ray Train	1	489	October 1, 2021
TorchTrainer hangs when only 1 worker raises error Ray Libraries (Data, Train, Tune, Serve)	15	894	November 2, 2022
Segfault in torchtrainer for num_workers > 0 in dataloader Ray Libraries (Data, Train, Tune, Serve)	1	455	April 9, 2023
Get distributed process group timeout when using torch trainer + FullSyncIterDatapipe Ray Train	5	630	December 20, 2022
Ray Streaming is timing out while trying to get next window Ray Libraries (Data, Train, Tune, Serve)	0	109	November 8, 2023