Raise errors when ray hangs?

Ray Core
#1

Hi,

In our workflows, ray will sometimes “get stuck” soon after a ray.get call, and resource utilization will essentially be zero on our clusters. Within ray, is there any way to detect when this happens or raise errors if it does?

This type of functionality would enable us to restart the job, which usually does the trick, or delete the cluster, which would save us money.

We could create our own monitor, but I wanted to check if something like this already exists.

Thanks.

#2

One possible way to detect hang is by adding timeout to ray.get , e.g. Ray Core API — Ray 2.1.0

Would this work for your use case?