How severe does this issue affect your experience of using Ray? High: It blocks me to complete my task. Hi Team, The Ray Train job fails after the completion of all the epochs (Huggingface trainable). In the training arguments, if load_best_model_at_end: true is passed, the below error occurs,…

Hi @saivivek15 , it looks like this issue has been discussed e.g. here before. Can you try two things (ideally separately). One, can you set export NCCL_P2P_LEVEL=NVL second, it looks like you’re using HDFS. One user in the thread reported issues with HDFS sync taking too long on rank 0. Can you …

TorchTrainer: Collective operation timeout: WorkNCCL

Uncategorized

saivivek15 July 17, 2023, 4:43pm 2

@kai @xwjiang2010, could you please suggest any workaround for this issue?

AIR, TorchTrainer, DDP and NCCL Timeout

Topic		Replies	Views
TorchTrainer hangs when only 1 worker raises error Uncategorized	15	1233	November 2, 2022
AIR, TorchTrainer, DDP and NCCL Timeout Uncategorized	2	1386	August 17, 2023
Ray Train code works locally, not in SageMaker PyTorch job Ray Train	15	1332	January 12, 2022
Any suggestions on how to debug the distributed torch trainer Dashboard, Monitoring & Debugging	7	994	June 9, 2021
Ray Train RuntimeError: unable to write to file </torch_1602_2842463136> Ray Train	3	1210	January 7, 2022

TorchTrainer: Collective operation timeout: WorkNCCL

Related topics