Driver memory increasing indefinitely when returning a numpy array

I have an actor that simply returns a numpy array in its remote method as follows:

import ray
import numpy as np
import time
from memory_profiler import profile

@ray.remote
class ImgActor:
    def __init__(self):
        #init a 500 mb array
        self.img = np.random.randint(0, 255, (50000, 10000)).astype(np.uint8)

    def run(self):
        return self.img

@profile
def actor_loop():
    ray.init(object_store_memory=2 * 1 << 30)

    actor = ImgActor.remote()
    run_duration = 15 * 60
    start = time.time()
    while time.time() - start < run_duration:
        actor.run.remote()

if __name__ == "__main__":
    actor_loop()

When I run this, the memory consumed by the main process/driver keeps increasing linearly. However the memory consumed by the actor process is fairly constant. I am not sure I understand why as I am not storing any explicit references to any objects, so I would assume they get evicted after each invocation to actor.run.remote(). Also even if an increasing amount of objects are being stored in the object store, why should that increase the memory of the driver process? Most likely the memory consumption is due to ObjectsRef being created in loop. But,if I am not holding on to those references (i.e no local reference) shouldn’t they be garbage collected? Probably I am doing something really silly and would love to get any clarification on this.

1 Like