An unexpected internal error occurred while the IO worker was spilling objects

I’'m running a cluster on my laptop with ray start --head and launching my program with

with ray.init(address='auto', ignore_reinit_error=True):
    ...

When object store is full, object spilling fails with

FileNotFoundError: [Errno 2] No such file or directory: '/tmp/ray/session_2023-05-07_09-24-33_669967_36323/ray_spilled_objects/37539b8a4799451eb12c9c4e96bdcc16-multi-27'
An unexpected internal error occurred while the IO worker was spilling objects: [Errno 2] No such file or directory: '/tmp/ray/session_2023-05-07_09-24-33_669967_36323/ray_spilled_objects/37539b8a4799451eb12c9c4e96bdcc16-multi-27'

Is there any additional configuration I need to set? How do I make spilling work properly?

Hi, can you give us more details about the platform you are running on? Which operating system and version of Ray are you using?

Mac M2, OS: Ventura 13.3.1, Ray 2.4.0

By default there shouldn’t be any configuration you need to set, although if you want, you can set the object_spilling_config during init [Object Spilling — Ray 2.5.0].

How long were you running this cluster for? Can you check if perhaps /tmp/ray/session_2023-05-07_09-24-33_669967_36323/ was deleted after creation?