Unexplained memory usage with cloudpickle+obj store

Hi Core team,

In the process of answering a SO question, I found that Ray has unexplained memory usage when passing a 10MB payload (in the object store) and a Python object instance which references another module. See the following measurements:

+--------------+-----------------+-----------+---------+------------------+
| with_payload | with_config_obj | num_tasks | used_mb | used_mb_per_task |
+--------------+-----------------+-----------+---------+------------------+
|     True     |       True      |     1     |  28.47  |      28.47       |
|     True     |       True      |     8     |  209.51 |      26.19       |
|     True     |       True      |     16    |  419.36 |      26.21       |
|    False     |       True      |     1     |  18.27  |      18.27       |
|    False     |       True      |     8     |  130.23 |      16.28       |
|    False     |       True      |     16    |  256.55 |      16.03       |
|     True     |      False      |     1     |   3.01  |       3.01       |
|     True     |      False      |     8     |  14.65  |       1.83       |
|     True     |      False      |     16    |  29.07  |       1.82       |
|    False     |      False      |     1     |   0.52  |       0.52       |
|    False     |      False      |     8     |   0.52  |       0.07       |
|    False     |      False      |     16    |   2.82  |       0.18       |
+--------------+-----------------+-----------+---------+------------------+

Why is used_mb_per_task ~10MB higher when both with_payload and with_config_obj are True than when with_payload is False and with_config_obj is True?

Details

Memory measurement

I use used from psutil.virtual_memory(). The same pattern exists for available and free, so I don’t think I’m double-counting the shared memory usage.

Config object/payload

The config object and payload are ray.put()'d after they are defined in this way:

class DummyObject:
    def do_something(self):
        print(nltk.__version__)

def create_data(target_size_mb):
    bytes_per_mb = 1 << 20
    payload = 'a' * target_size_mb * bytes_per_mb
    return payload

@ray.remote
def dummy_fun(config, payload):
    return type(config), type(payload)

The full code is here. You can generate the table above by running the driver.py script. I obtained those results with Ray 1.13 on a c5.4xlarge instance on AWS (16CPU).

As another datapoint, here is the same table when the payload size is 100MB.

+--------------+-----------------+-----------+---------+------------------+
| with_payload | with_config_obj | num_tasks | used_mb | used_mb_per_task |
+--------------+-----------------+-----------+---------+------------------+
|     True     |       True      |     1     |  117.09 |      117.09      |
|     True     |       True      |     8     |  933.07 |      116.63      |
|     True     |       True      |     16    | 1862.18 |      116.39      |
|    False     |       True      |     1     |   16.9  |       16.9       |
|    False     |       True      |     8     |  129.67 |      16.21       |
|    False     |       True      |     16    |  255.3  |      15.96       |
|     True     |      False      |     1     |   2.48  |       2.48       |
|     True     |      False      |     8     |  14.35  |       1.79       |
|     True     |      False      |     16    |  28.56  |       1.78       |
|    False     |      False      |     1     |   0.65  |       0.65       |
|    False     |      False      |     8     |   1.6   |       0.2        |
|    False     |      False      |     16    |   0.87  |       0.05       |
+--------------+-----------------+-----------+---------+------------------+

Looks like when the object instance is passed to the task, the payload is copied unnecessarily.

Interesting. I think this is most likely due to how the task arguments are serialized: when we call the remote function we need to serialize the object which may incur a copy.

1 Like

I’ll create a bug report in GitHub when I have a chance – thanks for validating my thinking!

1 Like