# Attempting to deserialize object on a CUDA device... error on 2 GPU machine

**URL:** <https://discuss.ray.io/t/attempting-to-deserialize-object-on-a-cuda-device-error-on-2-gpu-machine/1553>\
**Category:** Ray Tune\
**Created:** [April 5, 2021, 1:50pm UTC](https://discuss.ray.io/t/attempting-to-deserialize-object-on-a-cuda-device-error-on-2-gpu-machine/1553 "2021-04-05T13:50:36Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Many98](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/many98/32/493_2.png) [@Many98](https://discuss.ray.io/u/Many98)\
**Post date:** [April 5, 2021, 1:50pm UTC](https://discuss.ray.io/t/attempting-to-deserialize-object-on-a-cuda-device-error-on-2-gpu-machine/1553/1 "2021-04-05T13:50:36Z")

</div>

**I use TorchTrainer.as\_trainable() to tune my neural net on 2 GPU machine.**  
**I checked Cuda usage with nvidia-smi on first trial and it says I use together about 6 GB out of 26 GB but torch.cuda.is\_available() is obviously False. After first trial I get following error:**

Traceback (most recent call last):  
File “/…/anaconda3/envs/ox/lib/python3.8/site-packages/ray/tune/trial\_runner.py”, line 726, in \_process\_trial  
result = self.trial\_executor.fetch\_result(trial)  
File “/…/anaconda3/envs/ox/lib/python3.8/site-packages/ray/tune/ray\_trial\_executor.py”, line 489, in fetch\_result  
result = ray.get(trial\_future[0], timeout=DEFAULT\_GET\_TIMEOUT)  
File “/…/anaconda3/envs/ox/lib/python3.8/site-packages/ray/worker.py”, line 1452, in get  
raise value.as\_instanceof\_cause()  
ray.exceptions.RayTaskError(RuntimeError): e[36mray::TorchTrainable.train()e[39m (pid=163211, ip=…)  
File “python/ray/\_raylet.pyx”, line 482, in ray.\_raylet.execute\_task  
File “python/ray/\_raylet.pyx”, line 436, in ray.\_raylet.execute\_task.function\_executor  
File “/…/anaconda3/envs/ox/lib/python3.8/site-packages/ray/tune/trainable.py”, line 336, in train  
result = self.step()  
File “/…/anaconda3/envs/ox/lib/python3.8/site-packages/ray/util/sgd/torch/torch\_trainer.py”, line 657, in step  
output = override\_tune\_step(  
File “/…/temp/src/SentiNet.py”, line 941, in step  
validation\_stats = trainer.validate(reduce\_results=False)[0]  
File “/…/anaconda3/envs/ox/lib/python3.8/site-packages/ray/util/sgd/torch/torch\_trainer.py”, line 498, in validate  
worker\_stats = self.worker\_group.validate(  
File “/…/anaconda3/envs/ox/lib/python3.8/site-packages/ray/util/sgd/torch/worker\_group.py”, line 340, in validate  
return ray.get(remote\_worker\_stats)  
File “/…/anaconda3/envs/ox/lib/python3.8/site-packages/ray/serialization.py”, line 308, in deserialize\_objects  
self.\_deserialize\_object(data, metadata, object\_ref))  
File “/…/anaconda3/envs/ox/lib/python3.8/site-packages/ray/serialization.py”, line 247, in \_deserialize\_object  
return self.\_deserialize\_msgpack\_data(data, metadata)  
File “/…/anaconda3/envs/ox/lib/python3.8/site-packages/ray/serialization.py”, line 226, in \_deserialize\_msgpack\_data  
python\_objects = self.\_deserialize\_pickle5\_data(pickle5\_data)  
File “/…/anaconda3/envs/ox/lib/python3.8/site-packages/ray/serialization.py”, line 216, in \_deserialize\_pickle5\_data  
obj = pickle.loads(in\_band)  
File “/…/anaconda3/envs/ox/lib/python3.8/site-packages/torch/storage.py”, line 141, in \_load\_from\_bytes  
return torch.load(io.BytesIO(b))  
File “/…/anaconda3/envs/ox/lib/python3.8/site-packages/torch/serialization.py”, line 595, in load  
return \_legacy\_load(opened\_file, map\_location, pickle\_module, \*\*pickle\_load\_args)  
File “/…/anaconda3/envs/ox/lib/python3.8/site-packages/torch/serialization.py”, line 774, in \_legacy\_load  
result = unpickler.load()  
File “/…/anaconda3/envs/ox/lib/python3.8/site-packages/torch/serialization.py”, line 730, in persistent\_load  
deserialized\_objects[root\_key] = restore\_location(obj, location)  
File “/…/anaconda3/envs/ox/lib/python3.8/site-packages/torch/serialization.py”, line 175, in default\_restore\_location  
result = fn(storage, location)  
File “/…/anaconda3/envs/ox/lib/python3.8/site-packages/torch/serialization.py”, line 151, in \_cuda\_deserialize  
device = validate\_cuda\_device(location)  
File “/…/anaconda3/envs/ox/lib/python3.8/site-packages/torch/serialization.py”, line 135, in validate\_cuda\_device  
raise RuntimeError('Attempting to deserialize object on a CUDA ’  
RuntimeError: **Attempting to deserialize object on a CUDA device but torch.cuda.is\_available() is False. If you are running on a CPU-only machine, please use torch.load with map\_location=torch.device(‘cpu’) to map your storages to the CPU.**

**I guess that ray or torch preallocate all Cuda memory but I have no clue how to avoid it.**  
**I appreciate any help**  
**Here is my code I use:**

```auto
def step(trainer, info: dict):
        """
        Define a custom training loop for tune.
         This is needed because we want to manually update our scheduler.
         """
        train_stats = trainer.train()
        validation_stats = trainer.validate(reduce_results=False)[0]
        # Manually update our scheduler with the given metric.
        # trainer.update_scheduler(metric=validation_stats["val_loss"])
        all_stats = ray.tune.utils.merge_dicts(train_stats, validation_stats[0])
        return all_stats

    TorchTrainable = TorchTrainer.as_trainable(
        override_tune_step=step,
        training_operator_cls=sn_operator_cls,
        num_workers=num_workers,
        num_cpus_per_worker=4,
        use_gpu=use_gpu,
        use_tqdm=True,
        scheduler_step_freq="epoch",
        config=operator_config
    )

    analysis = ray.tune.run(
        TorchTrainable,
        num_samples=3,
        config=tune_config,
        stop={"epoch": 4},
        verbose=1,
        )

    return analysis.get_best_config(metric="val_loss", mode="min")

```

---

<div class="post-metadata">

**Author:** ![eoakes](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/eoakes/32/7825_2.png) [@eoakes](https://discuss.ray.io/u/eoakes)\
**Post date:** [April 5, 2021, 4:15pm UTC](https://discuss.ray.io/t/attempting-to-deserialize-object-on-a-cuda-device-error-on-2-gpu-machine/1553/2 "2021-04-05T16:15:54Z")

</div>

@rliaw @kai any pointers here?

---

<div class="post-metadata">

**Author:** ![kai](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kai/32/3380_2.png) [@kai](https://discuss.ray.io/u/kai)\
**Post date:** [April 6, 2021, 3:35pm UTC](https://discuss.ray.io/t/attempting-to-deserialize-object-on-a-cuda-device-error-on-2-gpu-machine/1553/3 "2021-04-06T15:35:38Z")

</div>

Which verison of Ray are you using? There were some problems recently on the latest master concerning GPU allocation, which should be fixed by this PR: [[core] Fix placement group GPU assignment bug by wuisawesome · Pull Request #15049 · ray-project/ray · GitHub](https://github.com/ray-project/ray/pull/15049)

So if you re-run this using the latest master, does the problem still come up?

---

<div class="post-metadata">

**Author:** ![Many98](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/many98/32/493_2.png) [@Many98](https://discuss.ray.io/u/Many98)\
**Post date:** [April 6, 2021, 4:38pm UTC](https://discuss.ray.io/t/attempting-to-deserialize-object-on-a-cuda-device-error-on-2-gpu-machine/1553/4 "2021-04-06T16:38:04Z")

</div>

I have installed latest wheel build [https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-2.0.0.dev0-cp38-cp38-manylinux2014\_x86\_64.whl](https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-2.0.0.dev0-cp38-cp38-manylinux2014_x86_64.whl) but the error message remains same.