# How do I set GPU affinity of workers **URL:** \ **Category:** RLlib\ **Created:** [April 16, 2021, 9:08am UTC](https://discuss.ray.io/t/how-do-i-set-gpu-affinity-of-workers/1755 "2021-04-16T09:08:03Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![Bam4d](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/bam4d/32/373_2.png) [@Bam4d](https://discuss.ray.io/u/Bam4d)\ **Post date:** [April 16, 2021, 9:08am UTC](https://discuss.ray.io/t/how-do-i-set-gpu-affinity-of-workers/1755/1 "2021-04-16T09:08:03Z")
Lets say I am using IMPALA with several workers and I have 4 GPUs. I want to be able to say GPUs 0,1,2 should be shared across the workers evenly so inference/rendering is accellerated by them, But i want GPU 3 to be dedicated to training loop. How can I do this? ---
**Author:** ![sven1977](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sven1977/32/53_2.png) [@sven1977](https://discuss.ray.io/u/sven1977)\ **Post date:** [April 16, 2021, 9:25am UTC](https://discuss.ray.io/t/how-do-i-set-gpu-affinity-of-workers/1755/2 "2021-04-16T09:25:30Z")
This should probably work out-of-the-box in the latest master. I would probably set `num_gpus_per_worker=[3/num_workers]` and our new PlacementGroups support should take care of the rest. If you need more details on how we do this now in RLlib (and IMPALA), you can you look at IMPALA’s `DefaultResourceRequest` override (in `rllib/agents/impala.py::OverrideDefaultResourceRequest`). In that method, we define the placement group bundles depending on how many workers we have. The first bundle is always the learner one (“driver”), then the worker bundles (which we will attempt to PACK (on the same node if possible), then the eval workers (you may not have any)). NOTE: There have been recent bug fixes for the placement groups, so you may have to use the latest master for this all to work. NOTE2: There is also a partial\_gpu RLlib example script PR (not merged yet) here: [[RLlib] Partial GPU example. by sven1977 · Pull Request #15334 · ray-project/ray · GitHub](https://github.com/ray-project/ray/pull/15334) ---
**Author:** ![Bam4d](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/bam4d/32/373_2.png) [@Bam4d](https://discuss.ray.io/u/Bam4d)\ **Post date:** [April 18, 2021, 9:18am UTC](https://discuss.ray.io/t/how-do-i-set-gpu-affinity-of-workers/1755/3 "2021-04-18T09:18:15Z")
On the latest master this just fails with an error out-of-the-box: ```auto (pid=13371) 2021-04-18 10:16:52,677 ERROR worker.py:395 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::RolloutWorker. __init__ () (pid=13371, ip=192.168.1.105) (pid=13371) File "python/ray/_raylet.pyx", line 505, in ray._raylet.execute_task (pid=13371) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task.function_executor (pid=13371) File "/home/bam4d/anaconda3/envs/griddly/lib/python3.8/site-packages/ray/_private/function_manager.py", line 566, in actor_method_executor (pid=13371) return method(__ray_actor, *args, **kwargs) (pid=13371) File "/home/bam4d/anaconda3/envs/griddly/lib/python3.8/site-packages/ray/rllib/evaluation/rollout_worker.py", line 516, in __init__ (pid=13371) self.policy_map, self.preprocessors = self._build_policy_map( (pid=13371) File "/home/bam4d/anaconda3/envs/griddly/lib/python3.8/site-packages/ray/rllib/evaluation/rollout_worker.py", line 1158, in _build_policy_map (pid=13371) policy_map[name] = cls(obs_space, act_space, merged_conf) (pid=13371) File "/home/bam4d/anaconda3/envs/griddly/lib/python3.8/site-packages/ray/rllib/policy/policy_template.py", line 243, in __init__ (pid=13371) self.parent_cls. __init__ ( (pid=13371) File "/home/bam4d/anaconda3/envs/griddly/lib/python3.8/site-packages/ray/rllib/policy/torch_policy.py", line 145, in __init__ (pid=13371) self.model_gpu_towers = nn.parallel.replicate.replicate( (pid=13371) AttributeError: 'function' object has no attribute 'replicate' ``` I will do some more investigating ---
**Author:** ![Bam4d](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/bam4d/32/373_2.png) [@Bam4d](https://discuss.ray.io/u/Bam4d)\ **Post date:** [April 18, 2021, 9:20am UTC](https://discuss.ray.io/t/how-do-i-set-gpu-affinity-of-workers/1755/4 "2021-04-18T09:20:14Z")
Also my 2 GPUS are different so it matters which one gets used for driver/workers. Is there a way I can specify this? ---
**Author:** ![Bam4d](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/bam4d/32/373_2.png) [@Bam4d](https://discuss.ray.io/u/Bam4d)\ **Post date:** [April 18, 2021, 9:30am UTC](https://discuss.ray.io/t/how-do-i-set-gpu-affinity-of-workers/1755/5 "2021-04-18T09:30:25Z")
If i fix the `replicate.replicate()` line then I run into another error: ```auto File "/home/bam4d/anaconda3/envs/griddly/lib/python3.8/site-packages/ray/rllib/policy/torch_policy.py", line 145, in __init__ self.model_gpu_towers = nn.parallel.replicate( File "/home/bam4d/anaconda3/envs/griddly/lib/python3.8/site-packages/torch/nn/parallel/replicate.py", line 91, in replicate param_copies = _broadcast_coalesced_reshape(params, devices, detach) File "/home/bam4d/anaconda3/envs/griddly/lib/python3.8/site-packages/torch/nn/parallel/replicate.py", line 71, in _broadcast_coalesced_reshape tensor_copies = Broadcast.apply(devices, *tensors) File "/home/bam4d/anaconda3/envs/griddly/lib/python3.8/site-packages/torch/nn/parallel/_functions.py", line 14, in forward assert all(i.device.type != 'cpu' for i in inputs), ( AssertionError: Broadcast function not implemented for CPU tensors ``` ---
**Author:** ![Bam4d](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/bam4d/32/373_2.png) [@Bam4d](https://discuss.ray.io/u/Bam4d)\ **Post date:** [April 18, 2021, 9:32am UTC](https://discuss.ray.io/t/how-do-i-set-gpu-affinity-of-workers/1755/6 "2021-04-18T09:32:13Z")
FYI there are the settings im using: I have 1x2080ti and 1x980ti in my PC ```auto --num-cpus=8 --num-gpus=2 --num-workers=2 --num-gpus-per-worker=0.5 --num-envs-per-worker=1 ``` ---
**Author:** ![Bam4d](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/bam4d/32/373_2.png) [@Bam4d](https://discuss.ray.io/u/Bam4d)\ **Post date:** [April 18, 2021, 4:32pm UTC](https://discuss.ray.io/t/how-do-i-set-gpu-affinity-of-workers/1755/8 "2021-04-18T16:32:01Z")
@sven1977 I think the multi-GPU code thats currently in master is causing several issues for me: 1. Its creating multiple copies of the model for each worker. (at least 2) so if i have 2 workers it creates 4 copies of the model on the GPU (or CPU) which is extremely detrimental as every model I use is now using 2x the Memory. [[RLlib] Multi-GPU support for Torch algorithms. (#14709) · ray-project/ray@cecfc3b · GitHub](https://github.com/ray-project/ray/commit/cecfc3b43b7a1a7ec4398645556ef64c21704ea8#diff-396cded251382c39ae2a136ee01acb8b085561c1735826666ecae0b43160ad34R133) 2. The `parallel.replicate` does not work because the model is not on the GPU when its replicated, so it runs into the error above. In addition to this… if I fix this by moving the model.to(device) then im creating at least 2 models on the GPU, which causes the GPU to run out of memory. ---
**Author:** ![Bam4d](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/bam4d/32/373_2.png) [@Bam4d](https://discuss.ray.io/u/Bam4d)\ **Post date:** [April 18, 2021, 4:57pm UTC](https://discuss.ray.io/t/how-do-i-set-gpu-affinity-of-workers/1755/9 "2021-04-18T16:57:34Z")
Removing the multi-GPU code and ensuring a single model is placed on the GPU solves my issues. (obviously not a proper fix). ```auto # # Create multi-GPU model towers, if necessary. # # - The central main model will be stored under self.model, residing on # # self.device. # # - Each GPU will have a copy of that model under # # self.model_gpu_towers, matching the devices in self.devices. # # - Parallelization is done by splitting the train batch and passing # # it through the model copies in parallel, then averaging over the # # resulting gradients, applying these averages on the main model and # # updating all towers' weights from the main model. # # - In case of just one device (1 (fake) GPU or 1 CPU), no # # parallelization will be done. # if config["_fake_gpus"] or config["num_gpus"] == 0 or \ # not torch.cuda.is_available(): # logger.info( # "TorchPolicy running on {}.".format("{} fake-GPUs".format( # config["num_gpus"]) if config["_fake_gpus"] else "CPU")) # self.device = torch.device("cpu") # self.model = model.to(self.device) # self.devices = [ # self.device for _ in range(config["num_gpus"] or 1) # ] # self.model_gpu_towers = [ # model if config["num_gpus"] == 0 else copy.deepcopy(model) # for i in range(config["num_gpus"] or 1) # ] # else: # logger.info("TorchPolicy running on {} GPU(s).".format( # config["num_gpus"])) # self.device = torch.device("cuda") # self.model = model.to(self.device) # print(ray.get_gpu_ids()) # self.devices = [ # torch.device("cuda:{}".format(id_)) # for i, id_ in enumerate(ray.get_gpu_ids()) # if i < config["num_gpus"] # ] # self.model_gpu_towers = nn.parallel.replicate( # model, [ # id_ for i, id_ in enumerate(ray.get_gpu_ids()) # if i < config["num_gpus"] # ]) #print(f'{torch.cuda.device_count()}') self.device = torch.device('cuda:0') self.devices = [self.device] self.model = model.to(self.device) self.model_gpu_towers = [self.model] ``` ---
**Author:** ![sven1977](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sven1977/32/53_2.png) [@sven1977](https://discuss.ray.io/u/sven1977)\ **Post date:** [April 20, 2021, 9:21am UTC](https://discuss.ray.io/t/how-do-i-set-gpu-affinity-of-workers/1755/10 "2021-04-20T09:21:04Z")
Hmm, interesting. It should not create more than 1 model per worker with your settings, only 2 model-towers on the learner (but not on the workers). ---
**Author:** ![Bam4d](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/bam4d/32/373_2.png) [@Bam4d](https://discuss.ray.io/u/Bam4d)\ **Post date:** [April 20, 2021, 9:42am UTC](https://discuss.ray.io/t/how-do-i-set-gpu-affinity-of-workers/1755/11 "2021-04-20T09:42:50Z")
So for me, the `model.to(self.device)` adds one model to the GPU and then the `nn.parallel.replicate` adds another. so if i only have 1 GPU per worker, it actually loads two models. ---
**Author:** ![sven1977](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sven1977/32/53_2.png) [@sven1977](https://discuss.ray.io/u/sven1977)\ **Post date:** [April 20, 2021, 10:03am UTC](https://discuss.ray.io/t/how-do-i-set-gpu-affinity-of-workers/1755/13 "2021-04-20T10:03:57Z")
Yeah, I see, it should not do any multi-GPU copying/tower generation if num\_gpus \<= 1. Makes sense. I’ll fix this. Even for num\_gpus=2, we could probably nix the “main” model and just use one of the towers. ---
**Author:** ![sven1977](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sven1977/32/53_2.png) [@sven1977](https://discuss.ray.io/u/sven1977)\ **Post date:** [April 20, 2021, 5:44pm UTC](https://discuss.ray.io/t/how-do-i-set-gpu-affinity-of-workers/1755/14 "2021-04-20T17:44:17Z")
Hey @Bam4d , Here is a preliminary PR with major fixes for torch multi-GPU. Unfortunately, I’m still not able to get num\_gpus\_per\_worker\>0 to work. a strange torch error comes up: “Invalid device ordinal”. Any ideas? I’m running IMPALA on 4 GPUs against Atari: rllib/tuned\_examples/impala/atari-impala-multi-gpu.yaml > > > Torch multi-GPU bug fixes: > \- Adds IMPALA torch multi-GPU tuned example (Atari benchmark). > \- TODO: num\_gpus\_per\_worker\>0 currently causes a torch error > > See this discussion here as well: > https://discuss.ray.io/t/how-do-i-set-gpu-affinity-of-workers/1755/10 > > > \## Why are these changes needed? > > > \## Related issue number > > > \## Checks > > \- \[\] I've run \`scripts/format.sh\` to lint the changes in this PR. > \- \[\] I've included any doc changes needed for https://docs.ray.io/en/master/. > \- \[\] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ > \- Testing Strategy > - \[\] Unit tests > - \[\] Release tests > - \[\] This PR is not tested :( GPU utilization looks also good now: ```auto +-----------------------------------------------------------------------------+ | NVIDIA-SMI 450.51.05 Driver Version: 450.51.05 CUDA Version: 11.0 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Tesla K80 On | 00000000:00:17.0 Off | 0 | | N/A 72C P0 114W / 149W | 2702MiB / 11441MiB | 70% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ | 1 Tesla K80 On | 00000000:00:18.0 Off | 0 | | N/A 56C P0 123W / 149W | 2612MiB / 11441MiB | 53% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ | 2 Tesla K80 On | 00000000:00:19.0 Off | 0 | | N/A 72C P0 119W / 149W | 2612MiB / 11441MiB | 100% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ | 3 Tesla K80 On | 00000000:00:1A.0 Off | 0 | | N/A 56C P0 126W / 149W | 2612MiB / 11441MiB | 98% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ ``` ---
**Author:** ![Bam4d](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/bam4d/32/373_2.png) [@Bam4d](https://discuss.ray.io/u/Bam4d)\ **Post date:** [April 20, 2021, 6:19pm UTC](https://discuss.ray.io/t/how-do-i-set-gpu-affinity-of-workers/1755/15 "2021-04-20T18:19:04Z")
So the invalid device ordinal might be because the ray\_gpu\_id is not consistent with cuda.available\_devices. I cam across this problem before with Griddly, If there is only 1 device available to cuda, it will always re-order the device to cuda:0. so if ray\_gpu\_id is 1 and CUDA\_AVAILABLE\_DEVICES = 1 then cuda:0 will point to the single gpu and cuda:1 will throw the device ordinal error. I’m guessing you are passing CUDA\_AVAILABLE\_DEVICES to worker processes? This might be the cause. ---
**Author:** ![Bam4d](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/bam4d/32/373_2.png) [@Bam4d](https://discuss.ray.io/u/Bam4d)\ **Post date:** [April 20, 2021, 6:27pm UTC](https://discuss.ray.io/t/how-do-i-set-gpu-affinity-of-workers/1755/16 "2021-04-20T18:27:31Z")
This line here I think is the problem, you are using the ray\_gpu\_id, but cuda will set these to 0 indexed. I think if you set ‘\_id’ to ‘i’ it will work for this case, not sure about other cases, might be a bit more complicated. [here](https://github.com/ray-project/ray/pull/15421/files#diff-396cded251382c39ae2a136ee01acb8b085561c1735826666ecae0b43160ad34R150) ---
**Author:** ![sven1977](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sven1977/32/53_2.png) [@sven1977](https://discuss.ray.io/u/sven1977)\ **Post date:** [April 21, 2021, 1:42pm UTC](https://discuss.ray.io/t/how-do-i-set-gpu-affinity-of-workers/1755/17 "2021-04-21T13:42:43Z")
Cool, yeah, that could be that torch always starts enumerating from 0. For the learner, this is fine (it gets the first n gpu IDs), but for the workers, this would then be a problem. Let me try to fix this. … Thanks so much for your help with this btw. I did confirm yesterday that e.g. IMPALA runs (and learns) 4x as fast with 4 GPUs as opposed to w/ 1 GPU with this PR already. Just have to figure out the worker GPUs now … ---
**Author:** ![sven1977](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sven1977/32/53_2.png) [@sven1977](https://discuss.ray.io/u/sven1977)\ **Post date:** [April 21, 2021, 2:54pm UTC](https://discuss.ray.io/t/how-do-i-set-gpu-affinity-of-workers/1755/19 "2021-04-21T14:54:59Z")
Awesome, that was it! @Bam4d , could you do me a favor and try it now on the above PR? I just tested with the following config and I do see a 4x speedup (throughput and learning) for 4 GPUs over single GPU. ```auto # Runs on a p2.8xlarge single head node machine. # Should reach ~400 reward in about 1h and after 15-20M ts. atari-impala: env: BreakoutNoFrameskip-v4 run: IMPALA config: framework: torch rollout_fragment_length: 50 train_batch_size: 4000 num_gpus: 4 num_workers: 2 # 31 on a p2.8xlarge machine to get above results num_gpus_per_worker: 0.5 # or 0 to get above results num_envs_per_worker: 5 clip_rewards: True lr_schedule: [ [0, 0.0005], [20000000, 0.000000000001], ] ``` ---
**Author:** ![Bam4d](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/bam4d/32/373_2.png) [@Bam4d](https://discuss.ray.io/u/Bam4d)\ **Post date:** [April 22, 2021, 11:34am UTC](https://discuss.ray.io/t/how-do-i-set-gpu-affinity-of-workers/1755/21 "2021-04-22T11:34:02Z")
yeah its working for me with num\_gpus\_per\_env=0 which is great. Multi-agent RTS training with inference on GPU speaks for itself: ![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/1X/fa39b291595dcd9945cbbbe562f710ebcabd82d8.png) This is Amazing!!! ---
**Author:** ![sven1977](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sven1977/32/53_2.png) [@sven1977](https://discuss.ray.io/u/sven1977)\ **Post date:** [April 23, 2021, 10:05am UTC](https://discuss.ray.io/t/how-do-i-set-gpu-affinity-of-workers/1755/22 "2021-04-23T10:05:16Z")
Awesome!! Thanks for sharing this @Bam4d !!