# Extract GPU statistics per process in Ray Tune

**URL:** <https://discuss.ray.io/t/extract-gpu-statistics-per-process-in-ray-tune/559>\
**Category:** Ray Tune\
**Created:** [January 24, 2021, 12:56am UTC](https://discuss.ray.io/t/extract-gpu-statistics-per-process-in-ray-tune/559 "2021-01-24T00:56:31Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![kovjxkjvklz](https://avatars.discourse-cdn.com/v4/letter/k/ac8455/32.png) [@kovjxkjvklz](https://discuss.ray.io/u/kovjxkjvklz)\
**Post date:** [January 24, 2021, 12:56am UTC](https://discuss.ray.io/t/extract-gpu-statistics-per-process-in-ray-tune/559/1 "2021-01-24T00:56:31Z")

</div>

Hi, I was wondering how I am able to extract GPU statistics per process in Ray Tune train function.  
Right now I’m able to use the nvsmi.get\_gpu\_processes(), matching the current pid with the particular gpu\_process id, and extract used\_memory from the GPU\_process. But I’m not sure how to do the same thing for extracting GPU utilization. Here the assumption can be that no two processes co-locate in the same GPU.  
Also, in ray dashboard, I’m able to monitor the GPU Util, are there any ways to extract that statistics?

---

<div class="post-metadata">

**Author:** ![rliaw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rliaw/32/24_2.png) [@rliaw](https://discuss.ray.io/u/rliaw)\
**Post date:** [January 24, 2021, 8:09am UTC](https://discuss.ray.io/t/extract-gpu-statistics-per-process-in-ray-tune/559/2 "2021-01-24T08:09:33Z")

</div>

I think you will see them in your logs if you do:

```auto
tune.run(config={"log_sys_usage": True})
```

---

<div class="post-metadata">

**Author:** ![kovjxkjvklz](https://avatars.discourse-cdn.com/v4/letter/k/ac8455/32.png) [@kovjxkjvklz](https://discuss.ray.io/u/kovjxkjvklz)\
**Post date:** [January 25, 2021, 8:44am UTC](https://discuss.ray.io/t/extract-gpu-statistics-per-process-in-ray-tune/559/3 "2021-01-25T08:44:29Z")

</div>

For that single configuration/process, the util reported would contain all GPUs’ util percent in that machine, including some other GPUs that this process does not use. How to just get the gpu util that a particular process is using?

---

<div class="post-metadata">

**Author:** ![rliaw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rliaw/32/24_2.png) [@rliaw](https://discuss.ray.io/u/rliaw)\
**Post date:** [January 26, 2021, 1:24am UTC](https://discuss.ray.io/t/extract-gpu-statistics-per-process-in-ray-tune/559/4 "2021-01-26T01:24:30Z")

</div>

Maybe you can instead access the particular GPU via `ray.get_gpu_ids()` in your training code and use GPUtil to get utilization stats for the GPU.