# CUDA error: all CUDA-capable devices are busy or unavailable

**URL:** <https://discuss.ray.io/t/cuda-error-all-cuda-capable-devices-are-busy-or-unavailable/1113>\
**Category:** Ray Tune\
**Created:** [March 3, 2021, 10:47pm UTC](https://discuss.ray.io/t/cuda-error-all-cuda-capable-devices-are-busy-or-unavailable/1113 "2021-03-03T22:47:59Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![import-antigravity](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/import-antigravity/32/562_2.png) [@import-antigravity](https://discuss.ray.io/u/import-antigravity)\
**Post date:** [March 3, 2021, 10:47pm UTC](https://discuss.ray.io/t/cuda-error-all-cuda-capable-devices-are-busy-or-unavailable/1113/1 "2021-03-03T22:47:59Z")

</div>

I have some Pytorch code incorporating Ray Tune which runs fine on my laptop (on CPU), but when I try to run it on my computing cluster I get the following error:

```auto
2021-03-03 17:24:33,331	ERROR function_manager.py:498 -- Failed to load actor class ImplicitFunc.
(pid=6410) Traceback (most recent call last):
(pid=6410) File "/.../lib/python3.8/site-packages/ray/function_manager.py", line 496, in _load_actor_class_from_gcs
(pid=6410) actor_class = pickle.loads(pickled_class)
(pid=6410) File "/.../lib/python3.8/site-packages/torch/storage.py", line 141, in _load_from_bytes
(pid=6410) return torch.load(io.BytesIO(b))
(pid=6410) File "/.../lib/python3.8/site-packages/torch/serialization.py", line 595, in load
(pid=6410) return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
(pid=6410) File "/.../lib/python3.8/site-packages/torch/serialization.py", line 774, in _legacy_load
(pid=6410) result = unpickler.load()
(pid=6410) File "/.../lib/python3.8/site-packages/torch/serialization.py", line 730, in persistent_load
(pid=6410) deserialized_objects[root_key] = restore_location(obj, location)
(pid=6410) File "/.../lib/python3.8/site-packages/torch/serialization.py", line 175, in default_restore_location
(pid=6410) result = fn(storage, location)
(pid=6410) File "/.../lib/python3.8/site-packages/torch/serialization.py", line 155, in _cuda_deserialize
(pid=6410) return storage_type(obj.size())
(pid=6410) File "/.../lib/python3.8/site-packages/torch/cuda/ __init__.py", line 462, in _lazy_new
(pid=6410) return super(_CudaBase, cls). __new__ (cls, *args, **kwargs)
(pid=6410) RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable

```

I’m running the following versions:

ray 1.2.0  
pytorch 1.7.1  
cuda 10.1

I’d submit an issue on GitHub but I’m not sure how to write code which would reproduce the problem.

Additional info: I ran this code on a GPU compute node with 2 2080 Tis

---

<div class="post-metadata">

**Author:** ![import-antigravity](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/import-antigravity/32/562_2.png) [@import-antigravity](https://discuss.ray.io/u/import-antigravity)\
**Post date:** [March 3, 2021, 11:19pm UTC](https://discuss.ray.io/t/cuda-error-all-cuda-capable-devices-are-busy-or-unavailable/1113/2 "2021-03-03T23:19:28Z")

</div>

Update: It looks like the problem was that I had models outside of my training function which I was trying to reference, which I guess isn’t thread safe or something. Is there a way to, for example, have one pretrained model you want to reference from multiple trials? Like maybe if you were doing transfer learning or something?

---

<div class="post-metadata">

**Author:** ![rliaw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rliaw/32/24_2.png) [@rliaw](https://discuss.ray.io/u/rliaw)\
**Post date:** [March 4, 2021, 1:44am UTC](https://discuss.ray.io/t/cuda-error-all-cuda-capable-devices-are-busy-or-unavailable/1113/3 "2021-03-04T01:44:46Z")

</div>

Yeah, you could hypothetically do something like `ref = ray.put(model)` then in your training function, do `model = ray.get(ref)`.

---

<div class="post-metadata">

**Author:** ![import-antigravity](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/import-antigravity/32/562_2.png) [@import-antigravity](https://discuss.ray.io/u/import-antigravity)\
**Post date:** [March 4, 2021, 4:05am UTC](https://discuss.ray.io/t/cuda-error-all-cuda-capable-devices-are-busy-or-unavailable/1113/4 "2021-03-04T04:05:46Z")

</div>

At the moment it seems like there’s no memory bottleneck in storing multiple copies of the models, but I’ll keep this in mind in case it becomes an issue!

---

<div class="post-metadata">

**Author:** ![Nitin\_Pasumarthy](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/nitin_pasumarthy/32/277_2.png) [@Nitin\_Pasumarthy](https://discuss.ray.io/u/Nitin_Pasumarthy)\
**Post date:** [February 11, 2022, 7:03am UTC](https://discuss.ray.io/t/cuda-error-all-cuda-capable-devices-are-busy-or-unavailable/1113/5 "2022-02-11T07:03:52Z")

</div>

I’m facing a similar problem now. Shared my Trainable [here](https://discuss.ray.io/t/status-all-cuda-capable-devices-are-busy-or-unavailable/5012)

@import-antigravity , what do you mean by `I had models outside my training function`? And how did you find those problematic definitions?