Currently our pipeline which uses ray with modin spills A LOT of data, and I am talking of multiple terabytes in just a few hours. This is expected, but the files never get deleted. Even if the ray nodes are being shut down.
I would expect the temp files to be temporary and be deleted if not longer needed.
Additionally when deleting manually using trash and empty trash, the files get stuck in trash/expunged. Never actually being deleted. I have to manually delete them with the terminal.
Am I missing something? do I need to call some sort of cleanup?
Thank you!
Can you share your pipeline/code?
Hi there, sorry for the late reply
Its a bit hard to share that cause its a lot and its propriatary. but i can give you more details
And I think it makes sense that it cant cleanup, because we kinda kill the process ray is residing in, so it might not have the time to clean up after itself.
I do call ray.shutdown before doing the killing, but it returnes immediatly, and i dont think it cleans tmp in that time.
Can I somehow wait for that to happen?
Or should .shutdown even clean tmp? If not, how can I initiate that?
more details: I run mutliple docker containers, in those docker containers we have flask apps that host a processing pool (with just 1 worker) in this pool a process is spawned that does some processing using ray. after the process is done, we shutdown ray and immediatly after kill the process in the pool (or even the whole pool).
I can manually purge the tmp directory, but I feel there must be a better way.