Limiting Ray remote functions execution time

Ray Clusters
1

Is there a way to limit ray’s remote functions execution time? This might be very useful for protecting against “run-away” remote processes.

2

Oh it looks like its effectively supported through tmout on Ray.get. So the following code seems to do the trick:

try:
    obj = f.remote(200)
    res = ray.get(obj, timeout=4.0)
except Exception as err:
    print("Caught an exception", err)
    ray.cancel(obj, force=True, recursive=True)
    res = "Execution exception : " + str(err)
2 Likes