# Using checkpoint causes GPU failure and error during training process

**URL:** <https://discuss.ray.io/t/using-checkpoint-causes-gpu-failure-and-error-during-training-process/22851>\
**Category:** Checkpointing, Restoring\
**Created:** [July 16, 2025, 3:06am UTC](https://discuss.ray.io/t/using-checkpoint-causes-gpu-failure-and-error-during-training-process/22851 "2025-07-16T03:06:54Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![ZanhaPeng](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/zanhapeng/32/7699_2.png) [@ZanhaPeng](https://discuss.ray.io/u/ZanhaPeng)\
**Post date:** [July 16, 2025, 3:06am UTC](https://discuss.ray.io/t/using-checkpoint-causes-gpu-failure-and-error-during-training-process/22851/1 "2025-07-16T03:06:54Z")

</div>

tuner = tune.Tuner(  
‘PPO’,  
param\_space=ppo\_config,  
run\_config=tune.RunConfig(  
storage\_path=storage\_dir,  
name=‘p0’,  
stop={‘training\_iteration’: 2000},  
verbose=3,  
checkpoint\_config=tune.CheckpointConfig(  
checkpoint\_at\_end=True,  
checkpoint\_frequency=5,  
),  
),  
)  
results = tuner.fit()

error：  
Trial PPO\_multi\_env\_08ebf\_00000 finished iteration 5 at 2025-07-16 10:55:01. Total running time: 1min 39s  
╭──────────────────────────────────────────────────╮  
│ Trial PPO\_multi\_env\_08ebf\_00000 result │  
├──────────────────────────────────────────────────┤  
│ env\_runners/episode\_len\_mean 150 │  
│ env\_runners/episode\_return\_mean -172 │  
│ num\_env\_steps\_sampled\_lifetime 20480 │  
╰──────────────────────────────────────────────────╯  
2025-07-16 10:55:01,180 ERROR tune\_controller.py:1331 – Trial task failed for trial PPO\_multi\_env\_08ebf\_00000  
Traceback (most recent call last):  
File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/ray/air/execution/\_internal/event\_manager.py”, line 110, in resolve\_future  
result = ray.get(future)  
File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/ray/\_private/auto\_init\_hook.py”, line 22, in auto\_init\_wrapper  
return fn(\*args, \*\*kwargs)  
File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/ray/\_private/client\_mode\_hook.py”, line 104, in wrapper  
return func(\*args, \*\*kwargs)  
File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/ray/\_private/worker.py”, line 2849, in get  
values, debugger\_breakpoint = worker.get\_objects(object\_refs, timeout=timeout)  
File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/ray/\_private/worker.py”, line 937, in get\_objects  
raise value.as\_instanceof\_cause()  
ray.exceptions.RayTaskError(RaySystemError): ray::PPO.save() (pid=26518, ip=10.68.4.39, actor\_id=4c6af5385d92900280559f5e01000000, repr=PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True))  
File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/ray/tune/trainable/trainable.py”, line 486, in save  
checkpoint\_dict\_or\_path = self.save\_checkpoint(checkpoint\_dir)  
File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/ray/rllib/algorithms/algorithm.py”, line 2690, in save\_checkpoint  
self.save\_to\_path(  
File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/ray/rllib/utils/checkpoints.py”, line 300, in save\_to\_path  
comp\_state = self.get\_state(components=comp\_name)[comp\_name]  
File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/ray/rllib/algorithms/algorithm.py”, line 2834, in get\_state  
state[COMPONENT\_LEARNER\_GROUP] = self.learner\_group.get\_state(  
File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/ray/rllib/core/learner/learner\_group.py”, line 521, in get\_state  
state[COMPONENT\_LEARNER] = self.\_get\_results(results)[0]  
File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/ray/rllib/core/learner/learner\_group.py”, line 672, in \_get\_results  
raise result\_or\_error  
File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/ray/rllib/utils/actor\_manager.py”, line 861, in \_fetch\_result  
result = ray.get(ready)  
ray.exceptions.RaySystemError: System error: Attempting to deserialize object on a CUDA device but torch.cuda.is\_available() is False. If you are running on a CPU-only machine, please use torch.load with map\_location=torch.device(‘cpu’) to map your storages to the CPU.  
traceback: Traceback (most recent call last):  
File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/torch/storage.py”, line 533, in \_load\_from\_bytes  
return torch.load(io.BytesIO(b), weights\_only=False)  
File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/torch/serialization.py”, line 1495, in load  
return \_legacy\_load(  
File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/torch/serialization.py”, line 1754, in \_legacy\_load  
result = unpickler.load()  
File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/torch/serialization.py”, line 1682, in persistent\_load  
obj = restore\_location(obj, location)  
File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/torch/serialization.py”, line 693, in default\_restore\_location  
result = fn(storage, location)  
File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/torch/serialization.py”, line 631, in \_deserialize  
device = \_validate\_device(location, backend\_name)  
File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/torch/serialization.py”, line 600, in \_validate\_device  
raise RuntimeError(  
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is\_available() is False. If you are running on a CPU-only machine, please use torch.load with map\_location=torch.device(‘cpu’) to map your storages to the CPU.

Trial PPO\_multi\_env\_08ebf\_00000 errored after 5 iterations at 2025-07-16 10:55:01. Total running time: 1min 39s  
Error file: /tmp/ray/session\_2025-07-16\_10-53-20\_879632\_24234/artifacts/2025-07-16\_10-53-21/p0/driver\_artifacts/PPO\_multi\_env\_08ebf\_00000\_0\_2025-07-16\_10-53-21/error.txt  
2025-07-16 10:55:01,202 INFO tune.py:1009 – Wrote the latest version of all result files and experiment state to ‘/home/zanhao/Project\_shuttle/results/p0’ in 0.0206s.

Trial status: 1 ERROR  
Current time: 2025-07-16 10:55:01. Total running time: 1min 39s  
Logical resource usage: 13.0/32 CPUs, 0.99/1 GPUs (0.0/1.0 accelerator\_type:G)  
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮  
│ Trial name status iter total time (s) …lls\_per\_iteration …\_sampled\_lifetime │  
├────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤  
│ PPO\_multi\_env\_08ebf\_00000 ERROR 5 94.4173 1 20480 │  
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Number of errored trials: 1  
╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮  
│ Trial name # failures error file │  
├──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤  
│ PPO\_multi\_env\_08ebf\_00000 1 /tmp/ray/session\_2025-07-16\_10-53-20\_879632\_24234/artifacts/2025-07-16\_10-53-21/p0/driver\_artifacts/PPO\_multi\_env\_08ebf\_00000\_0\_2025-07-16\_10-53-21/error.txt │  
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) Attempting to deserialize object on a CUDA device but torch.cuda.is\_available() is False. If you are running on a CPU-only machine, please use torch.load with map\_location=torch.device(‘cpu’) to map your storages to the CPU.  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) Traceback (most recent call last):  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/ray/\_private/serialization.py”, line 458, in deserialize\_objects  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) obj = self.\_deserialize\_object(data, metadata, object\_ref)  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/ray/\_private/serialization.py”, line 315, in \_deserialize\_object  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) return self.\_deserialize\_msgpack\_data(data, metadata\_fields)  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/ray/\_private/serialization.py”, line 270, in \_deserialize\_msgpack\_data  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) python\_objects = self.\_deserialize\_pickle5\_data(pickle5\_data)  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/ray/\_private/serialization.py”, line 258, in \_deserialize\_pickle5\_data  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) obj = pickle.loads(in\_band, buffers=buffers)  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/torch/storage.py”, line 533, in \_load\_from\_bytes  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) return torch.load(io.BytesIO(b), weights\_only=False)  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/torch/serialization.py”, line 1495, in load  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) return \_legacy\_load(  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/torch/serialization.py”, line 1754, in \_legacy\_load  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) result = unpickler.load()  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/torch/serialization.py”, line 1682, in persistent\_load  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) obj = restore\_location(obj, location)  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/torch/serialization.py”, line 693, in default\_restore\_location  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) result = fn(storage, location)  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/torch/serialization.py”, line 631, in \_deserialize  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) device = \_validate\_device(location, backend\_name)  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/torch/serialization.py”, line 600, in \_validate\_device  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) raise RuntimeError(  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is\_available() is False. If you are running on a CPU-only machine, please use torch.load with map\_location=torch.device(‘cpu’) to map your storages to the CPU.  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) 2025-07-16 10:55:01,178 ERROR actor\_manager.py:873 – Ray error (System error: Attempting to deserialize object on a CUDA device but torch.cuda.is\_available() is False. If you are running on a CPU-only machine, please use torch.load with map\_location=torch.device(‘cpu’) to map your storages to the CPU.  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) traceback: Traceback (most recent call last):  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/ray/\_private/serialization.py”, line 458, in deserialize\_objects  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) obj = self.\_deserialize\_object(data, metadata, object\_ref)  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/ray/\_private/serialization.py”, line 315, in \_deserialize\_object  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) return self.\_deserialize\_msgpack\_data(data, metadata\_fields)  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/ray/\_private/serialization.py”, line 270, in \_deserialize\_msgpack\_data  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) python\_objects = self.\_deserialize\_pickle5\_data(pickle5\_data)  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/ray/\_private/serialization.py”, line 258, in \_deserialize\_pickle5\_data  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) obj = pickle.loads(in\_band, buffers=buffers)  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/torch/storage.py”, line 533, in \_load\_from\_bytes  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) return torch.load(io.BytesIO(b), weights\_only=False)  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/torch/serialization.py”, line 1495, in load  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) return \_legacy\_load(  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/torch/serialization.py”, line 1754, in \_legacy\_load  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) result = unpickler.load()  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/torch/serialization.py”, line 1682, in persistent\_load  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) obj = restore\_location(obj, location)  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/torch/serialization.py”, line 693, in default\_restore\_location  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) result = fn(storage, location)  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/torch/serialization.py”, line 631, in \_deserialize  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) device = \_validate\_device(location, backend\_name)  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) File “/home/zanhao/anaconda3/envs/torch/lib/python3.9/site-packages/torch/serialization.py”, line 600, in \_validate\_device  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) raise RuntimeError(  
(PPO(env=multi\_env; env-runners=4; learners=1; multi-agent=True) pid=26518) RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is\_available() is False. If you are running on a CPU-only machine, please use torch.load with map\_location=torch.device(‘cpu’) to map your storages to the CPU.

---

<div class="post-metadata">

**Author:** ![christina](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/christina/32/7542_2.png) [@christina](https://discuss.ray.io/u/christina)\
**Post date:** [July 16, 2025, 5:06pm UTC](https://discuss.ray.io/t/using-checkpoint-causes-gpu-failure-and-error-during-training-process/22851/2 "2025-07-16T17:06:17Z")

</div>

> [@ZanhaPeng](#):
>
> Attempting to deserialize object on a CUDA device but torch.cuda.is\_available() is False. If you are running on a CPU-only machine, please use torch.load with map\_location=torch.device(‘cpu’) to map your storages to the CPU

It seems like CUDA is not available on your machine, likely because your machine doesn’t have a GPU. Do you know if your computer has a GPU or is it CPU only?

---

<div class="post-metadata">

**Author:** ![ZanhaPeng](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/zanhapeng/32/7699_2.png) [@ZanhaPeng](https://discuss.ray.io/u/ZanhaPeng)\
**Post date:** [July 19, 2025, 6:45am UTC](https://discuss.ray.io/t/using-checkpoint-causes-gpu-failure-and-error-during-training-process/22851/3 "2025-07-19T06:45:06Z")

</div>

yes，Of course, my computer has a GPU, and the GPU runs normally during training. However, when saving checkpoints, the GPU crashes

---

<div class="post-metadata">

**Author:** ![MCW\_Lad](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mcw_lad/32/8009_2.png) [@MCW\_Lad](https://discuss.ray.io/u/MCW_Lad)\
**Post date:** [July 19, 2025, 7:36am UTC](https://discuss.ray.io/t/using-checkpoint-causes-gpu-failure-and-error-during-training-process/22851/4 "2025-07-19T07:36:31Z")

</div>

The error getting thrown indicates that torch.cuda.is\_available() is returning false. Try running that method on a Python console on machine you’re using to deserialize the model, and see if it works in isolation - that might provide more information on what’s going wrong.

The issue, as best I can tell, isn’t within Ray or RLlib - it might be that something’s wrong with your PyTorch install.

---

<div class="post-metadata">

**Author:** ![ZanhaPeng](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/zanhapeng/32/7699_2.png) [@ZanhaPeng](https://discuss.ray.io/u/ZanhaPeng)\
**Post date:** [July 22, 2025, 7:00am UTC](https://discuss.ray.io/t/using-checkpoint-causes-gpu-failure-and-error-during-training-process/22851/5 "2025-07-22T07:00:23Z")

</div>

But I downloaded my torch from the official website, and the versions of cuda and other libraries matched it.

---

<div class="post-metadata">

**Author:** ![MCW\_Lad](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mcw_lad/32/8009_2.png) [@MCW\_Lad](https://discuss.ray.io/u/MCW_Lad)\
**Post date:** [July 22, 2025, 9:08am UTC](https://discuss.ray.io/t/using-checkpoint-causes-gpu-failure-and-error-during-training-process/22851/6 "2025-07-22T09:08:04Z")

</div>

If you run torch.cuda.is\_available() in isolation, outside of another script, what does it output?

---

<div class="post-metadata">

**Author:** ![ZanhaPeng](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/zanhapeng/32/7699_2.png) [@ZanhaPeng](https://discuss.ray.io/u/ZanhaPeng)\
**Post date:** [July 23, 2025, 6:08am UTC](https://discuss.ray.io/t/using-checkpoint-causes-gpu-failure-and-error-during-training-process/22851/7 "2025-07-23T06:08:49Z")

</div>

Of course, it will return true

---

<div class="post-metadata">

**Author:** ![MCW\_Lad](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mcw_lad/32/8009_2.png) [@MCW\_Lad](https://discuss.ray.io/u/MCW_Lad)\
**Post date:** [July 24, 2025, 7:28pm UTC](https://discuss.ray.io/t/using-checkpoint-causes-gpu-failure-and-error-during-training-process/22851/8 "2025-07-24T19:28:10Z")

</div>

That is interesting. If you have the full script/repo on hand, I can try pulling it onto a different machine and see if I get the same error. That should at least narrow things down.

---

<div class="post-metadata">

**Author:** ![Theo\_Fan](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/theo_fan/32/8105_2.png) [@Theo\_Fan](https://discuss.ray.io/u/Theo_Fan)\
**Post date:** [July 25, 2025, 1:26am UTC](https://discuss.ray.io/t/using-checkpoint-causes-gpu-failure-and-error-during-training-process/22851/9 "2025-07-25T01:26:42Z")

</div>

Hi, @MCW_Lad

I encountered the **same issue** , maybe you can reproduce it with the following code:

```py
from ray import train, tune
from ray.rllib.algorithms.ppo import PPOConfig

config = (
    PPOConfig()
    .environment("Pendulum-v1")
    
    .training(
        lr=tune.grid_search([0.001, 0.0001]),
    )
    .env_runners(
        num_env_runners=2,
        batch_mode="complete_episodes"
    )
    .learners(
        num_learners=1,
        num_gpus_per_learner=1, # gpu config
    )
    
)

tuner = tune.Tuner(
    config.algo_class,
    param_space=config,
    run_config=train.RunConfig(
        stop={
        	"training_iteration": 5,
        },
        checkpoint_config=tune.CheckpointConfig(
            checkpoint_at_end=True, # Problem
        ),
    ),
)

results = tuner.fit()

```

And I get the following error:

```txt
ERROR actor_manager.py:873 -- Ray error (System error: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. 
If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

```

---

<div class="post-metadata">

**Author:** ![MCW\_Lad](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mcw_lad/32/8009_2.png) [@MCW\_Lad](https://discuss.ray.io/u/MCW_Lad)\
**Post date:** [July 26, 2025, 10:29am UTC](https://discuss.ray.io/t/using-checkpoint-causes-gpu-failure-and-error-during-training-process/22851/10 "2025-07-26T10:29:29Z")

</div>

Just ran that code on my own machine. It seems to work perfectly fine, making me think that something with your CUDA setup is the problem. Full outputs uploaded to pastebin for reference, [here](https://pastebin.com/ugv5c9Lw).

---

<div class="post-metadata">

**Author:** ![Theo\_Fan](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/theo_fan/32/8105_2.png) [@Theo\_Fan](https://discuss.ray.io/u/Theo_Fan)\
**Post date:** [July 31, 2025, 1:29am UTC](https://discuss.ray.io/t/using-checkpoint-causes-gpu-failure-and-error-during-training-process/22851/11 "2025-07-31T01:29:20Z")

</div>

Your output is precisely what I had hoped to see, I ran the above test code again, and the complete output in [here](https://pastebin.com/k2ySk3MR),

After that, I tested the CUDA installation in my environment manually,

```auto
(rl) ~ % nvidia-smi
Thu Jul 31 09:05:51 2025
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01 Driver Version: 535.183.01 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 4090 Off | 00000000:C1:00.0 Off | Off |
| 32% 30C P8 15W / 450W | 14MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 1 NVIDIA GeForce RTX 4090 Off | 00000000:E1:00.0 Off | Off |
| 31% 29C P8 18W / 450W | 14MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 4520 G /usr/lib/xorg/Xorg 4MiB |
| 1 N/A N/A 4520 G /usr/lib/xorg/Xorg 4MiB |
+---------------------------------------------------------------------------------------+
(rl) ~ %
(rl) ~ % nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0
(rl) ~ % python -c "import torch; print(torch. __version__ ); print(torch.version.cuda); print(torch.cuda.is_available())"
2.7.1+cu118
11.8
True

```

and… well, things look a bit strange. 😵‍💫