When evaluate the model in valid_loop, I get different metrics(such as loss, accuracy, and so on) from different ranks in the ray clusters. Do we have a simple solution to do like pytorch all_reduce work, so that I can get the average loss in Rank0. I tried @ray.remote, but I meet 2 questions: I…

Can you set Tuner(tune_config=tune.TuneConfig(reuse_actors=False))?

Do I understand correctly that you would like to simply average the results from all ranks? Unfortunately we do not have a built-in way to do that just now, but it is on our backlog. You should be able to use pytorch communication primitives directly in the training function, however, as that funct…

The solution you said works, Thank you very much, Thanks for your wonderful product!

@Yard1 Thanks for your help, I want to ask another question, I can use the all_reduce in TorchTrainer, it works well . but it didn’t work when I use ray.tune, the error log: dist.all_reduce(total, dist.ReduceOp.SUM, async_op=False) File "/home/zetlin/miniconda3/lib/python3.8/site-packages/to…

That’s because Train automatically starts the PyTorch distributed backend, while Tune doesn’t. If you want to use PyTorch distributed methods, you should use them with Train. What is that you are trying to accomplish here? Are you trying to communicate between Tune trials?

My situation is like this: One trial may need 2 GPUs， in the valid_loop I need to calulate the average of the metrics from 2 GPUs. So I want to know do you have some solutions of all_reduce in tune?

I will try it, yesterday I was stucked by another problem. Sometimes the trail didn’t realease the GPU memory, I find ray.tune.utils.wait_for_gpu in the documentation, but this function didn’t realease the GPU memory, right? Do you have suggestions how to realease GPU memory in the code? I tried…

Can you post the code for the training function you are running in Tune?

1. Training Code def train_func_per_worker(args): os.environ['TORCH_DISTRIBUTED_DEBUG'] = "DETAIL" global best_ade args = dict_to_namespace(args) log_writer = LogWriter("./") worker_info = f"Rank{session.get_world_rank()}" if not args.use_ray_tune else f"Trial{session.get_trial…

pytorch all_reduce works well in TorchTrainer, thanks a lot.

How can I synchronization metrics in `ray.train` valid_loop

Uncategorized

Yard1 December 6, 2022, 4:47pm 7

Was this solved in How to make all use of the GPU memory in Ray.tune - #3 by Yard1?

Topic		Replies	Views
How to make all use of the GPU memory in Ray.tune Uncategorized	6	1466	December 6, 2022
Ray Train hangs for long time Ray Train	11	1999	July 20, 2022
Ray using so much memory I cannot even start the tuning Ray Tune	5	2600	April 24, 2023
Tune.run() with docker is not using gpu Ray Tune	7	2249	May 31, 2022
Ray Tune process hangs - thread synchronization issue? Ray Tune	16	949	February 10, 2023

How can I synchronization metrics in `ray.train` valid_loop

Related topics