Hi!
I just saw the handling of object spilling in Ray, but when I tried ‘ray.put()’ to write 4GB of single object on a node with 2GB of shared memory, I found that it didn’t spill the object (no content in the folder ‘/tmp/spill’ ), but it worked fine.
Execute the ‘ray memory’ command
We have a mechanism called “fallback allocation” in this scenario. We use Swap memory if it doesn’t fit to plasma memory nor spilleable (there are some scenarios this can happen).
cc @ericl is it an intended behavior we use fallback allocation for objects bigger than plasma memory limit?
Yes, the object store will fallback to allocating filesystem-backed pages if insufficient /dev/shm memory is available. This can happen if the object is too large, or if shared memory is too fragmented.
These objects can still be spilled later on as soon as they aren’t mmaped any more (they’re treated similar to any other in-memory plasma object). We try to minimize the number of filesystem-backed objects, since their performance can be much worse due to filesystem page writeback.