# \[Ray Core\] RuntimeError: No CUDA GPUs are available **URL:** \ **Category:** Ray Core\ **Created:** [September 29, 2022, 2:28pm UTC](https://discuss.ray.io/t/ray-core-runtimeerror-no-cuda-gpus-are-available/7718 "2022-09-29T14:28:40Z")\ **Posts on this page:** 1\ **Showing post:** 3
**Author:** ![ravi](https://avatars.discourse-cdn.com/v4/letter/r/58956e/32.png) [@ravi](https://discuss.ray.io/u/ravi)\ **Post date:** [September 30, 2022, 5:14am UTC](https://discuss.ray.io/t/ray-core-runtimeerror-no-cuda-gpus-are-available/7718/3 "2022-09-30T05:14:49Z")
Thanks @rickyyx for quick response. I appreciate it. I am sorry for not adding `(num_gpus=1)` to my actor. Thanks for the sharp observation. Now, I have added it and also added `ray.cluster_resources() ` as suggested. Please see below: ```auto import ray import torch ray.init() ray.cluster_resources() @ray.remote(num_gpus=1) class Counter(object): def __init__ (self): self.tensor = torch.ones((1, 3)) self.device = "cuda:0" def move_and_increment(self): self.tensor.to(self.device) self.tensor += 1 def print(self): return self.tensor print(f"torch.cuda.is_available(): {torch.cuda.is_available()}") counters = [Counter.remote() for i in range(2)] [c.move_and_increment.remote() for c in counters] futures = [c.print.remote() for c in counters] print(ray.get(futures)) ray.shutdown() ``` Unfortunately, the above code does not return. Therefore I have to press CTRL+C after 2-3 minutes to stop it. Below is the information, printed in terminal: ```auto $ python ray_test.py 2022-09-30 13:57:55,626 INFO worker.py:1509 -- Started a local Ray instance. View the dashboard at 127.0.0.1:8265 torch.cuda.is_available(): True (scheduler +16s) Tip: use `ray status` to view detailed cluster status. To disable these messages, set RAY_SCHEDULER_EVENTS=0. (scheduler +16s) Warning: The following resource request cannot be scheduled right now: {'GPU': 1.0, 'CPU': 1.0}. This is likely due to all cluster resources being claimed by actors. Consider creating fewer actors or adding more nodes to this Ray cluster. (scheduler +51s) Warning: The following resource request cannot be scheduled right now: {'GPU': 1.0, 'CPU': 1.0}. This is likely due to all cluster resources being claimed by actors. Consider creating fewer actors or adding more nodes to this Ray cluster. (scheduler +1m26s) Warning: The following resource request cannot be scheduled right now: {'CPU': 1.0, 'GPU': 1.0}. This is likely due to all cluster resources being claimed by actors. Consider creating fewer actors or adding more nodes to this Ray cluster. ^CTraceback (most recent call last): File "/home/ravi/learning_ray/ray_test.py", line 26, in print(ray.get(futures)) File "/home/ravi/anaconda/envs/ray/lib/python3.9/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper return func(*args, **kwargs) File "/home/ravi/anaconda/envs/ray/lib/python3.9/site-packages/ray/_private/worker.py", line 2269, in get values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout) File "/home/ravi/anaconda/envs/ray/lib/python3.9/site-packages/ray/_private/worker.py", line 669, in get_objects data_metadata_pairs = self.core_worker.get_objects( File "python/ray/_raylet.pyx", line 1211, in ray._raylet.CoreWorker.get_objects File "python/ray/_raylet.pyx", line 173, in ray._raylet.check_status KeyboardInterrupt ^CError in atexit._run_exitfuncs: Traceback (most recent call last): File "/home/ravi/anaconda/envs/ray/lib/python3.9/site-packages/ray/_private/node.py", line 1143, in _kill_process_type self._kill_process_impl( File "/home/ravi/anaconda/envs/ray/lib/python3.9/site-packages/ray/_private/node.py", line 1199, in _kill_process_impl process.wait(timeout_seconds) File "/home/ravi/anaconda/envs/ray/lib/python3.9/subprocess.py", line 1189, in wait return self._wait(timeout=timeout) File "/home/ravi/anaconda/envs/ray/lib/python3.9/subprocess.py", line 1911, in _wait time.sleep(delay) KeyboardInterrupt ``` During the above execution, I noticed that my simplest actor is consuming 1089MiB GPU memory. Isn’t it too much for such a simple actor? Please see the output of `nvidia-smi` below: ```auto $ nvidia-smi Fri Sep 30 13:58:45 2022 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 455.23.05 Driver Version: 455.23.05 CUDA Version: 11.1 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 GeForce RTX 208... On | 00000000:01:00.0 Off | N/A | | N/A 47C P8 8W / N/A | 1513MiB / 7982MiB | 2% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 1529 G /usr/lib/xorg/Xorg 45MiB | | 0 N/A N/A 2812 G /usr/lib/xorg/Xorg 161MiB | | 0 N/A N/A 2987 G /usr/bin/gnome-shell 56MiB | | 0 N/A N/A 6112 G ...252301518872410907,131072 144MiB | | 0 N/A N/A 35286 C ray::Counter 1089MiB | +-----------------------------------------------------------------------------+ ``` Finally, below is the output of `ray status`: ```auto $ ray status ======== Autoscaler status: 2022-09-30 14:02:37.908976 ======== Node status --------------------------------------------------------------- Healthy: 1 node_64c97f1b08cc9131d1f160698cae490c845ebb2e1ef0cdfb98e086b4 Pending: (no pending nodes) Recent failures: (no failures) Resources --------------------------------------------------------------- Usage: 1.0/16.0 CPU 1.0/1.0 GPU 0.0/1.0 accelerator_type:RTX 0.00/15.066 GiB memory 0.00/7.533 GiB object_store_memory Demands: {'CPU': 1.0, 'GPU': 1.0}: 1+ pending tasks/actors ``` Please note that I am using Ray v2.0.0. Thank you very much. --- _[View the full topic](https://discuss.ray.io/t/ray-core-runtimeerror-no-cuda-gpus-are-available/7718)._