Can Ray handle no space being left on a device?

Ray Clusters
1

How severe does this issue affect your experience of using Ray?

  • Low: It contributes to significant difficulty to complete my task, but I can work around it.
    • (I need to know the answer to this question)

I’m receiving:

  File "python/ray/_raylet.pyx", line 908, in ray._raylet.spill_objects_handler
  File "python/ray/_raylet.pyx", line 911, in ray._raylet.spill_objects_handler
  File "/home/ray/anaconda3/lib/python3.8/site-packages/ray/_private/external_storage.py", line 657, in spill_objects
    return _external_storage.spill_objects(object_refs, owner_addresses)
  File "/home/ray/anaconda3/lib/python3.8/site-packages/ray/_private/external_storage.py", line 302, in spill_objects
    return self._write_multiple_objects(f, object_refs, owner_addresses, url)
  File "/home/ray/anaconda3/lib/python3.8/site-packages/ray/_private/external_storage.py", line 149, in _write_multiple_objects
    written_bytes = f.write(payload)
OSError: [Errno 28] No space left on device
An unexpected internal error occurred while the IO worker was spilling objects: [Errno 28] No space left on device

This makes sense. My dataset is somewhat large and it might be the case that some workers run out of disk space when dataset.count() is parallelized. Does Ray handle this case elegantly, or will this break things?

2

cc @Clark_Zinzow can you address this question?