Has anyone messed with putting a large (100 million edges, 2 million nodes) igraph in the shared memory object store? I tried that and the RAM got used up in a heartbeat. I read somewhere that only certain objects in shared memory are supported and therefore non-supported ones are still copied? I suppose if an alternative data structure is compatible that serves as a map/dictionary that would be fine in my case. I’m just not sure what those compatible types are - again I’m just trying to avoid the copies because the graph is too large for that, and I have too many threads that need to read that same graph.
Putting object into object store requires 1 copy. So from your program → shared memory.
After that, accessing this object via ray.get is zero-copy, only for objects that support pickle 5 out of band serialization support. I believe Pyarrow or numpy data structure supports this. Any class that uses them under the hood (e.g., pandas) will also be supported. So it depends on if igraph supports the zero copy serialization PEP 574 – Pickle protocol 5 with out-of-band data | peps.python.org