An unexpected internal error occurred while the IO worker was spilling objects

Ray Core
1

I’'m running a cluster on my laptop with ray start --head and launching my program with

with ray.init(address='auto', ignore_reinit_error=True):
    ...

When object store is full, object spilling fails with

FileNotFoundError: [Errno 2] No such file or directory: '/tmp/ray/session_2023-05-07_09-24-33_669967_36323/ray_spilled_objects/37539b8a4799451eb12c9c4e96bdcc16-multi-27'
An unexpected internal error occurred while the IO worker was spilling objects: [Errno 2] No such file or directory: '/tmp/ray/session_2023-05-07_09-24-33_669967_36323/ray_spilled_objects/37539b8a4799451eb12c9c4e96bdcc16-multi-27'

Is there any additional configuration I need to set? How do I make spilling work properly?

2

Hi, can you give us more details about the platform you are running on? Which operating system and version of Ray are you using?

3

Mac M2, OS: Ventura 13.3.1, Ray 2.4.0

4

By default there shouldn’t be any configuration you need to set, although if you want, you can set the object_spilling_config during init [Object Spilling — Ray 2.5.0].

How long were you running this cluster for? Can you check if perhaps /tmp/ray/session_2023-05-07_09-24-33_669967_36323/ was deleted after creation?