# How to use fraction GPU in \`ray.tune.Tuner\`? **URL:** \ **Category:** Ray Train\ **Created:** [August 20, 2023, 12:52am UTC](https://discuss.ray.io/t/how-to-use-fraction-gpu-in-ray-tune-tuner/11863 "2023-08-20T00:52:00Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Xinchengzelin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/xinchengzelin/32/3352_2.png) [@Xinchengzelin](https://discuss.ray.io/u/Xinchengzelin)\ **Post date:** [August 20, 2023, 12:52am UTC](https://discuss.ray.io/t/how-to-use-fraction-gpu-in-ray-tune-tuner/11863/1 "2023-08-20T00:52:00Z")
I use `ray.tune.Tuner` where the trainable is a `TorchTrainer`(the basic code structure as follows). I have 1 GPU, I want to run 2 tune trials at one GPU at the same time, so only 0.5GPU for one trial. I only find `tune.with_resources` to set fraction GPU in the docs , but when the trainable is a TorchTrainer, which don’t works. ```auto tuner = tune.Tuner( trainer, # TorchTrainer tune_config=tune.TuneConfig( metric="best_fde", mode="min", scheduler=scheduler, num_samples=num_samples, reuse_actors=False ), run_config=ray.air.RunConfig( name = tuner_dir_name, # dir name progress_reporter=tune.CLIReporter(max_report_frequency=600), ), param_space={"train_loop_config": config, # "scaling_config": ray.air.config.ScalingConfig( # num_workers=2, # resources_per_worker={ # "CPU": 4, # "GPU":0.5 # } # ), } ) results = tuner.fit() ``` I also test to set fraction GPU in the TorchTrainer, which also didn’t work as expected. ```auto trainer = TorchTrainer( train_loop_per_worker=train_func_per_worker, train_loop_config={ "args": args, }, scaling_config=ScalingConfig( num_workers=2, # The number of workers (Ray actors) to launch use_gpu=args.use_gpu, resources_per_worker={"GPU":0.5}, ), run_config=ray.air.RunConfig( progress_reporter=ray.tune.CLIReporter(max_report_frequency=600), ), ) ``` So I don’t know how to appropriatelly set the resource parameters. Please help me. ---
**Author:** ![matthewdeng](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/matthewdeng/32/1446_2.png) [@matthewdeng](https://discuss.ray.io/u/matthewdeng)\ **Post date:** [August 20, 2023, 4:04pm UTC](https://discuss.ray.io/t/how-to-use-fraction-gpu-in-ray-tune-tuner/11863/2 "2023-08-20T16:04:35Z")
Hey! What is the total amount of resources you want to schedule for a single `TorchTrainer`? Based on your current `ScalingConfig`, each Trainer/Trial will request a total of ```auto trainer_resources + num_workers * resources_per_worker ``` where ```auto trainer_resources=1 CPU num_workers=2 resources_per_worker=0.5 GPU ``` So in this case, you’ll be requesting a total of 1 CPU and 1 GPU - does that match the output you’re seeing in the console output? ---
**Author:** ![Xinchengzelin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/xinchengzelin/32/3352_2.png) [@Xinchengzelin](https://discuss.ray.io/u/Xinchengzelin)\ **Post date:** [August 21, 2023, 12:58am UTC](https://discuss.ray.io/t/how-to-use-fraction-gpu-in-ray-tune-tuner/11863/3 "2023-08-21T00:58:27Z")
This will have another error `train.torch.prepare_model` when as follows: ```auto 2023-08-21 09:01:33,024 ERROR trial_runner.py:993 -- Trial TorchTrainer_32216_00001: Error processing event. ray.exceptions.RayTaskError(RuntimeError): ray::_Inner.train() (pid=1641548, ip=192.168.1.128, repr=TorchTrainer) File "/home/xxx/miniconda3/lib/python3.8/site-packages/ray/tune/trainable/trainable.py", line 355, in train raise skipped from exception_cause(skipped) File "/home/xxx/miniconda3/lib/python3.8/site-packages/ray/train/_internal/utils.py", line 54, in check_for_failure ray.get(object_ref) ray.exceptions.RayTaskError(RuntimeError): ray::RayTrainWorker._RayTrainWorker__execute() (pid=1641581, ip=192.168.1.128, repr=) File "/home/xxx/miniconda3/lib/python3.8/site-packages/ray/train/_internal/worker_group.py", line 31, in __execute raise skipped from exception_cause(skipped) File "/home/xxx/miniconda3/lib/python3.8/site-packages/ray/train/_internal/utils.py", line 129, in discard_return_wrapper train_func(*args, **kwargs) File "multipathpp_train.py", line 442, in train_func_per_worker model = train.torch.prepare_model(model,parallel_strategy_kwargs={"find_unused_parameters":True}) File "/home/xxx/miniconda3/lib/python3.8/site-packages/ray/train/torch/train_loop_utils.py", line 120, in prepare_model return get_accelerator(_TorchAccelerator).prepare_model( File "/home/xxx/miniconda3/lib/python3.8/site-packages/ray/train/torch/train_loop_utils.py", line 365, in prepare_model model = DataParallel(model, **parallel_strategy_kwargs) File "/home/xxx/miniconda3/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 578, in __init__ dist._verify_model_across_ranks(self.process_group, parameters) RuntimeError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:957, invalid usage, NCCL version 21.0.3 ncclInvalidUsage: This usually reflects invalid usage of NCCL library (such as too many async ops, too many collectives at once, mixing streams in a group, etc). Result for TorchTrainer_32216_00001: date: 2023-08-21_09-01-20 experiment_id: 010ad717f5004de69f94685a292af065 hostname: xxx node_ip: 192.168.1.128 pid: 1641548 timestamp: 1692579680 trial_id: '32216_00001' ``` But I don’t know how to solve it. environment: ```auto ubuntu 20.04 pytorch: 1.10.2+cu113 ray: 2.1.0 ``` ---
**Author:** ![matthewdeng](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/matthewdeng/32/1446_2.png) [@matthewdeng](https://discuss.ray.io/u/matthewdeng)\ **Post date:** [August 21, 2023, 3:42am UTC](https://discuss.ray.io/t/how-to-use-fraction-gpu-in-ray-tune-tuner/11863/4 "2023-08-21T03:42:42Z")
Hmm I think this is because you end up scheduling both workers on the same GPU. Is there a reason you are trying to run `DistributedDataParallel` on fractional GPUs here? ---
**Author:** ![Xinchengzelin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/xinchengzelin/32/3352_2.png) [@Xinchengzelin](https://discuss.ray.io/u/Xinchengzelin)\ **Post date:** [August 21, 2023, 7:03am UTC](https://discuss.ray.io/t/how-to-use-fraction-gpu-in-ray-tune-tuner/11863/5 "2023-08-21T07:03:15Z")
The situation is that: the model is sensitive to the batch\_size, so I want to use a small batch\_size, so I can run more trials at the same GPU at the same time. if for this situation, I shouldn’t use `train.torch.prepare_model`? However, if I don’t use `train.torch.prepare_model`, it as if one trial is running 2 times at the same time. Besides, if I want to run 4 trials in 2 GPUs at the same time, what shoulde I do? ---
**Author:** ![rliaw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rliaw/32/24_2.png) [@rliaw](https://discuss.ray.io/u/rliaw)\ **Post date:** [August 23, 2023, 9:28pm UTC](https://discuss.ray.io/t/how-to-use-fraction-gpu-in-ray-tune-tuner/11863/6 "2023-08-23T21:28:04Z")
If you want to run 4 trials on a total of 2GPUs, all at the same time – you should do: ```auto tune.Tuner( tune.with_resources(train_func, resources={"gpu": 0.5}), tune_config={..}, ...) ``` Without TorchTrainer. Tuner can be used by itself for models that don’t need distributed training. ---
**Author:** ![Xinchengzelin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/xinchengzelin/32/3352_2.png) [@Xinchengzelin](https://discuss.ray.io/u/Xinchengzelin)\ **Post date:** [August 24, 2023, 1:03am UTC](https://discuss.ray.io/t/how-to-use-fraction-gpu-in-ray-tune-tuner/11863/7 "2023-08-24T01:03:09Z")
Thank you @rliaw I find another solution to do this with TorchTrainer, I need to set `torch_backend = 'gloo'`, which can run several trails on one GPU at the same time. However, it cost more time. when I run 4 trials on 2 GPUs, every batch cost 2s for each trial; every batch just cost 0.35s when run 1 trials on each GPU. As my code used TorchTrainer in Tuner, so I can reuse most of the code to fulfill TorchTrainer training and Tuner training. If I use the `tune.with_resources`, could the time issues be solved? That is to say, I couldn’t run 4 trials on a total of 2GPUs with TorchTrainer efficiently?