How to suppress TaskCancelledError without RAY_IGNORE_UNHANDLED_ERRORS?

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

When I call ray.cancel(r) on an inflight object ref, the worker prints an exception, even if I don’t call ray.get().

ERROR -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::fgh() (pid=13251, ip=
  File "<stdin>", line 7, in fgh
ray.exceptions.TaskCancelledError: Task: TaskID(62ffec03f52574bdffffffffffffffffffffffff03000000) was cancelled

I’d like to suppress this because my workload typically cancels a significant amount of tasks as a performance optimization. However, I don’t want to set ‘RAY_IGNORE_UNHANDLED_ERRORS=1’ because I don’t want to suppress actual unexpected worker errors, like workers dying.

Is there a way to suppress just TaskCancelledError? I did try catching KeyboardInterrupt and other exceptions within the task but that did not seem to work.

Hey @xcharleslin!

Do you have a small repro for this? I tried the below code but it doesn’t raise the canceled error if I don’t try to ray.get on it. (I used ray 2.2)

import ray

def f(t):
    import time
    return t

obj2 = f.remote(9999)

obj = f.remote(3)

Or if there is something I am missing here from your workflow?