# Unexplained memory usage with cloudpickle+obj store

**URL:** <https://discuss.ray.io/t/unexplained-memory-usage-with-cloudpickle-obj-store/6877>\
**Category:** Ray Core\
**Created:** [July 19, 2022, 6:02pm UTC](https://discuss.ray.io/t/unexplained-memory-usage-with-cloudpickle-obj-store/6877 "2022-07-19T18:02:13Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![cade](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/cade/32/3837_2.png) [@cade](https://discuss.ray.io/u/cade)\
**Post date:** [July 19, 2022, 6:02pm UTC](https://discuss.ray.io/t/unexplained-memory-usage-with-cloudpickle-obj-store/6877/1 "2022-07-19T18:02:13Z")

</div>

Hi Core team,

In the process of answering a [SO question](https://stackoverflow.com/questions/72970940/python-ray-passing-non-trivial-objects-to-workers-causes-memory-overflow), I found that Ray has unexplained memory usage when passing a 10MB payload (in the object store) and a Python object instance which references another module. See the following measurements:

```auto
+--------------+-----------------+-----------+---------+------------------+
| with_payload | with_config_obj | num_tasks | used_mb | used_mb_per_task |
+--------------+-----------------+-----------+---------+------------------+
| True | True | 1 | 28.47 | 28.47 |
| True | True | 8 | 209.51 | 26.19 |
| True | True | 16 | 419.36 | 26.21 |
| False | True | 1 | 18.27 | 18.27 |
| False | True | 8 | 130.23 | 16.28 |
| False | True | 16 | 256.55 | 16.03 |
| True | False | 1 | 3.01 | 3.01 |
| True | False | 8 | 14.65 | 1.83 |
| True | False | 16 | 29.07 | 1.82 |
| False | False | 1 | 0.52 | 0.52 |
| False | False | 8 | 0.52 | 0.07 |
| False | False | 16 | 2.82 | 0.18 |
+--------------+-----------------+-----------+---------+------------------+

```

Why is `used_mb_per_task` ~10MB higher when both `with_payload` and `with_config_obj` are True than when `with_payload` is False and `with_config_obj` is True?

### Details

#### Memory measurement

I use `used` from [`psutil.virtual_memory()`](https://psutil.readthedocs.io/en/latest/#psutil.virtual_memory). The same pattern exists for `available` and `free`, so I don’t think I’m double-counting the shared memory usage.

#### Config object/payload

The config object and payload are `ray.put()`'d after they are defined in this way:

```python
class DummyObject:
    def do_something(self):
        print(nltk. __version__ )

def create_data(target_size_mb):
    bytes_per_mb = 1 << 20
    payload = 'a' * target_size_mb * bytes_per_mb
    return payload

@ray.remote
def dummy_fun(config, payload):
    return type(config), type(payload)

```

The full code is [here](https://github.com/cadedaniel/cadelabs/blob/master/stackoverflow/72970940/script.py). You can generate the table above by running the driver.py script. I obtained those results with Ray 1.13 on a c5.4xlarge instance on AWS (16CPU).

---

<div class="post-metadata">

**Author:** ![cade](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/cade/32/3837_2.png) [@cade](https://discuss.ray.io/u/cade)\
**Post date:** [July 19, 2022, 6:07pm UTC](https://discuss.ray.io/t/unexplained-memory-usage-with-cloudpickle-obj-store/6877/2 "2022-07-19T18:07:00Z")

</div>

As another datapoint, here is the same table when the payload size is 100MB.

```auto
+--------------+-----------------+-----------+---------+------------------+
| with_payload | with_config_obj | num_tasks | used_mb | used_mb_per_task |
+--------------+-----------------+-----------+---------+------------------+
| True | True | 1 | 117.09 | 117.09 |
| True | True | 8 | 933.07 | 116.63 |
| True | True | 16 | 1862.18 | 116.39 |
| False | True | 1 | 16.9 | 16.9 |
| False | True | 8 | 129.67 | 16.21 |
| False | True | 16 | 255.3 | 15.96 |
| True | False | 1 | 2.48 | 2.48 |
| True | False | 8 | 14.35 | 1.79 |
| True | False | 16 | 28.56 | 1.78 |
| False | False | 1 | 0.65 | 0.65 |
| False | False | 8 | 1.6 | 0.2 |
| False | False | 16 | 0.87 | 0.05 |
+--------------+-----------------+-----------+---------+------------------+

```

Looks like when the object instance is passed to the task, the payload is copied unnecessarily.

---

<div class="post-metadata">

**Author:** ![Chen\_Shen](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/chen_shen/32/1486_2.png) [@Chen\_Shen](https://discuss.ray.io/u/Chen_Shen)\
**Post date:** [July 22, 2022, 9:10am UTC](https://discuss.ray.io/t/unexplained-memory-usage-with-cloudpickle-obj-store/6877/3 "2022-07-22T09:10:12Z")

</div>

Interesting. I think this is most likely due to how the task arguments are serialized: when we call the remote function we need to serialize the object which may incur a copy.

---

<div class="post-metadata">

**Author:** ![cade](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/cade/32/3837_2.png) [@cade](https://discuss.ray.io/u/cade)\
**Post date:** [July 22, 2022, 4:50pm UTC](https://discuss.ray.io/t/unexplained-memory-usage-with-cloudpickle-obj-store/6877/4 "2022-07-22T16:50:27Z")

</div>

I’ll create a bug report in GitHub when I have a chance – thanks for validating my thinking!