# Map Batches not using all CPUs

**URL:** <https://discuss.ray.io/t/map-batches-not-using-all-cpus/12586>\
**Category:** Ray Data\
**Created:** [October 25, 2023, 12:37pm UTC](https://discuss.ray.io/t/map-batches-not-using-all-cpus/12586 "2023-10-25T12:37:56Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![localh](https://avatars.discourse-cdn.com/v4/letter/l/87869e/32.png) [@localh](https://discuss.ray.io/u/localh)\
**Post date:** [October 25, 2023, 12:37pm UTC](https://discuss.ray.io/t/map-batches-not-using-all-cpus/12586/1 "2023-10-25T12:37:56Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- Low: It annoys or frustrates me for a moment.

I have a TB-sized tabular data set that is going through these preprocessing functions:

```auto
    preprocessor = Chain(Concatenator(include=cat_cols, dtype=np.int32, output_column_name=['x_cat']),
                        Concatenator(exclude=["y", "x_cat"], dtype=np.float32, output_column_name=['x_cont'])
                        )

```

My cluster is a single node with 40 CPUs, 672 GB of RAM and 8x GPUs.

I cannot get Ray to use all the CPUs when processing the data. Fiddling with the object store memory, I observe the following use of CPUs:

```auto
# 10 gb -- 1 / 40 cpu
# default object store memory -- 9 / 40 cpu
# 610 gb eq to shm -- 27 / 40 cpu

```

How can I get Ray to use all CPUs while preprocessing data?

---

<div class="post-metadata">

**Author:** ![raulchen](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/raulchen/32/330_2.png) [@raulchen](https://discuss.ray.io/u/raulchen)\
**Post date:** [October 26, 2023, 12:28am UTC](https://discuss.ray.io/t/map-batches-not-using-all-cpus/12586/2 "2023-10-26T00:28:06Z")

</div>

By default, each map task uses 1 logical CPU. If you find your CPUs are not saturated, it may indicate that the actual task isn’t really using 1 CPU. Decreasing the CPU resource request will help, as there will be more tasks running concurrently.  
To override this parameter, you can create your own Preprocessor subclass and override the `_get_transform_config` method. Simply make it return something like `{"num_cpus": 0.5}`