[Core] Exception in Ray Queue, probably during shutdown

I am using Ray 1.3.0. I have 4 actors popping msgs off a Ray Queue and one of them got exceptions as seen below in a catch-all try block. It looks like this happened during shutdown of Ray… the run processed all the files.

Maybe Ray Queue get() client should throw a RayActorError(“shutdown”) or something similar during shutdown.

Or should any RayActorError exception should be considered fatal in client code? (I’m not using a timeout.)

  File "/home/sunray/ws/segment/postprocess.py", line 166, in run
    g = self.work_queue.pop()
  File "/home/sunray/ws/segment/work_queue.py", line 58, in pop
    return self._queue.get()
  File "/home/sunray/miniconda3/envs/segment/lib/python3.6/site-packages/ray/util/queue.py", line 155, in get
    return ray.get(self.actor.get.remote(timeout))
  File "/home/sunray/miniconda3/envs/segment/lib/python3.6/site-packages/ray/_private/client_mode_hook.py", line 47, in wrapper
    return func(*args, **kwargs)
ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task.

This error can be raised when actor is dead, and ray.get cannot obtain the result from actor method because of that. Maybe we should improve our error message + document around the exception…

Thanks for your reply.
I think I will catch RayActorError and consider that the shutdown notice in the actor.

1 Like