How severe does this issue affect your experience of using Ray? High: It blocks me to complete my task. Hi Team, The Ray Train job fails after the completion of all the epochs (Huggingface trainable). In the training arguments, if load_best_model_at_end: true is passed, the below error occurs,…

TorchTrainer: Collective operation timeout: WorkNCCL

saivivek15 July 17, 2023, 4:43pm 2

@kai @xwjiang2010, could you please suggest any workaround for this issue?

AIR, TorchTrainer, DDP and NCCL Timeout

Topic		Replies	Views
TorchTrainer hangs when only 1 worker raises error	15	1088	November 2, 2022
AIR, TorchTrainer, DDP and NCCL Timeout	2	1304	August 17, 2023
Ray Train code works locally, not in SageMaker PyTorch job Ray Train	15	1161	January 12, 2022
Any suggestions on how to debug the distributed torch trainer Dashboard, Monitoring & Debugging	7	901	June 9, 2021
Ray Train RuntimeError: unable to write to file </torch_1602_2842463136> Ray Train	3	1145	January 7, 2022