# How to stop Ray from managing CUDA\_VISIBLE\_DEVICES?

**URL:** <https://discuss.ray.io/t/how-to-stop-ray-from-managing-cuda-visible-devices/8767>\
**Category:** Ray Core\
**Created:** [December 23, 2022, 12:21pm UTC](https://discuss.ray.io/t/how-to-stop-ray-from-managing-cuda-visible-devices/8767 "2022-12-23T12:21:59Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Architect](https://avatars.discourse-cdn.com/v4/letter/a/d26b3c/32.png) [@Architect](https://discuss.ray.io/u/Architect)\
**Post date:** [December 23, 2022, 12:21pm UTC](https://discuss.ray.io/t/how-to-stop-ray-from-managing-cuda-visible-devices/8767/1 "2022-12-23T12:21:59Z")

</div>

I’m using Ray on a cluster with a variable number of GPUs per node. I want to run a task on each node, and let it consume all the GPUs on that node. I have defined a custom `node` resource to make sure tasks are not ran in parallel on the same node. However, if I don’t set `num_gpus` for the remote function, Ray sets `CUDA_VISIBLE_DEVICES` to an empty string. So I’m forced to proved some `num_gpus` value, which leaves some nodes underutilized.  
Could someone with any of these questions:

- Can I specify a flexible number of GPUs?  
or
- How to stop Ray from editing CUDA\_VISIBLE\_DEVICES?

Thank you for your help!

---

<div class="post-metadata">

**Author:** ![cade](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/cade/32/3837_2.png) [@cade](https://discuss.ray.io/u/cade)\
**Post date:** [March 24, 2023, 5:59pm UTC](https://discuss.ray.io/t/how-to-stop-ray-from-managing-cuda-visible-devices/8767/2 "2023-03-24T17:59:17Z")

</div>

Interesting case of resource requests on non-homogonous cluster cc @jjyao

@Architect not sure what you ended up going with but one workaround is to unset `CUDA_VISIBLE_DEVICES` at the beginning of task execution.

```auto
import os

@ray.remote
def task():
    del os.environ['CUDA_VISIBLE_DEVICES']
    # remaining code that uses all available GPUs on the node

```

---

<div class="post-metadata">

**Author:** ![Yard1](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/yard1/32/508_2.png) [@Yard1](https://discuss.ray.io/u/Yard1)\
**Post date:** [March 24, 2023, 6:13pm UTC](https://discuss.ray.io/t/how-to-stop-ray-from-managing-cuda-visible-devices/8767/3 "2023-03-24T18:13:59Z")

</div>

I think we have the `RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES` env var for that (set it to a non-null value in both the driver code and the runtime env)

---

<div class="post-metadata">

**Author:** ![Jules\_Damji](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jules_damji/32/4058_2.png) [@Jules\_Damji](https://discuss.ray.io/u/Jules_Damji)\
**Post date:** [March 24, 2023, 11:46pm UTC](https://discuss.ray.io/t/how-to-stop-ray-from-managing-cuda-visible-devices/8767/4 "2023-03-24T23:46:44Z")

</div>

Thanks @Yard1 and @cade for your suggestions. @Architect Let us know if either of the suggestions worked for you to disable cuda visibility of devices.

---

<div class="post-metadata">

**Author:** ![Architect](https://avatars.discourse-cdn.com/v4/letter/a/d26b3c/32.png) [@Architect](https://discuss.ray.io/u/Architect)\
**Post date:** [April 13, 2023, 7:17pm UTC](https://discuss.ray.io/t/how-to-stop-ray-from-managing-cuda-visible-devices/8767/5 "2023-04-13T19:17:38Z")

</div>

Hi all, thanks for your help. Indeed I ended up manually managing `CUDA_VISIBLE_DEVICES`. Did not know about `RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES`, this could have helped.