I have an actor that simply returns a numpy array in its remote method as follows:
import ray import numpy as np import time from memory_profiler import profile @ray.remote class ImgActor: def __init__(self): #init a 500 mb array self.img = np.random.randint(0, 255, (50000, 10000)).astype(np.uint8) def run(self): return self.img @profile def actor_loop(): ray.init(object_store_memory=2 * 1 << 30) actor = ImgActor.remote() run_duration = 15 * 60 start = time.time() while time.time() - start < run_duration: actor.run.remote() if __name__ == "__main__": actor_loop()
When I run this, the memory consumed by the main process/driver keeps increasing linearly. However the memory consumed by the actor process is fairly constant. I am not sure I understand why as I am not storing any explicit references to any objects, so I would assume they get evicted after each invocation to
actor.run.remote(). Also even if an increasing amount of objects are being stored in the object store, why should that increase the memory of the driver process? Most likely the memory consumption is due to ObjectsRef being created in loop. But,if I am not holding on to those references (i.e no local reference) shouldn’t they be garbage collected? Probably I am doing something really silly and would love to get any clarification on this.