TorchTrainer Timed out waiting 1800000 ms for send operation to complete

2

You can set this in the TorchTrainer like so:

TorchTrainer(torch_config=TorchConfig(timeout_s=...))

However, if your application times out after 30 min (1800 sec) that usually indicates there’s an unwanted hang in the program that could be worth investigating.