# Ray tune + deepspeed integration **URL:** \ **Category:** Ray Tune\ **Created:** [February 21, 2025, 12:01pm UTC](https://discuss.ray.io/t/ray-tune-deepspeed-integration/21872 "2025-02-21T12:01:24Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![klyakh98](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/klyakh98/32/6487_2.png) [@klyakh98](https://discuss.ray.io/u/klyakh98)\ **Post date:** [February 21, 2025, 12:01pm UTC](https://discuss.ray.io/t/ray-tune-deepspeed-integration/21872/1 "2025-02-21T12:01:24Z")
I am trying to tune my LLM with HuggingFace Trainer.hyperparameter\_search() together with the Ray backend. The model I am using has 13B params, so it is impossible to run trials without sharing the optimizer across multiple GPUs (8 GPUs per trial in my case, max concurrent trials = 1). I tried to add Deepspeed ZeRO-2 config to Trainer, but without success - it always crashes with `module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cpu` error. I guess that this error originates from how Ray starts trials. I also tried to start my tune with `deepspeed/accelerate [args] tune_script.py` instead of `python tune_script.py`. Is it possible to tune hyperparams with HF + Ray + Deepspeed? Or maybe other ways like using FSDP… ---
**Author:** ![christina](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/christina/32/7542_2.png) [@christina](https://discuss.ray.io/u/christina)\ **Post date:** [February 21, 2025, 9:57pm UTC](https://discuss.ray.io/t/ray-tune-deepspeed-integration/21872/2 "2025-02-21T21:57:51Z")
Hi there! Welcome to the Ray community. Can you tell me more about your CUDA setup or the lines of code where it’s erroring out on? It seems like the module is expecting a CUDA because that’s what the `device_id[0]` has it on, but instead it seems like it’s trying to use the `cpu`. What GPU are you running this on?