[Core] Exception in Ray Queue, probably during shutdown

Ray Core
1

I am using Ray 1.3.0. I have 4 actors popping msgs off a Ray Queue and one of them got exceptions as seen below in a catch-all try block. It looks like this happened during shutdown of Ray… the run processed all the files.

Maybe Ray Queue get() client should throw a RayActorError(“shutdown”) or something similar during shutdown.

Or should any RayActorError exception should be considered fatal in client code? (I’m not using a timeout.)

  File "/home/sunray/ws/segment/postprocess.py", line 166, in run
    g = self.work_queue.pop()
  File "/home/sunray/ws/segment/work_queue.py", line 58, in pop
    return self._queue.get()
  File "/home/sunray/miniconda3/envs/segment/lib/python3.6/site-packages/ray/util/queue.py", line 155, in get
    return ray.get(self.actor.get.remote(timeout))
  File "/home/sunray/miniconda3/envs/segment/lib/python3.6/site-packages/ray/_private/client_mode_hook.py", line 47, in wrapper
    return func(*args, **kwargs)
ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task.
2

This error can be raised when actor is dead, and ray.get cannot obtain the result from actor method because of that. Maybe we should improve our error message + document around the exception…

3

Thanks for your reply.
I think I will catch RayActorError and consider that the shutdown notice in the actor.

1 Like