How to suppress TaskCancelledError without RAY_IGNORE_UNHANDLED_ERRORS?

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

When I call ray.cancel(r) on an inflight object ref, the worker prints an exception, even if I don’t call ray.get().

ERROR -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::fgh() (pid=13251, ip=
  File "<stdin>", line 7, in fgh
ray.exceptions.TaskCancelledError: Task: TaskID(62ffec03f52574bdffffffffffffffffffffffff03000000) was cancelled

I’d like to suppress this because my workload typically cancels a significant amount of tasks as a performance optimization. However, I don’t want to set ‘RAY_IGNORE_UNHANDLED_ERRORS=1’ because I don’t want to suppress actual unexpected worker errors, like workers dying.

Is there a way to suppress just TaskCancelledError? I did try catching KeyboardInterrupt and other exceptions within the task but that did not seem to work.

Hey @xcharleslin!

Do you have a small repro for this? I tried the below code but it doesn’t raise the canceled error if I don’t try to ray.get on it. (I used ray 2.2)

import ray

def f(t):
    import time
    return t

obj2 = f.remote(9999)

obj = f.remote(3)

Or if there is something I am missing here from your workflow?

hey Ricky, try this:

import ray; import time; ray.init()

def f():
    while True:
        print("f() running")

ref = f.remote()


(f pid=146818) f() running
(f pid=146818) f() running
# pauses for a few seconds here
2023-02-02 16:00:32,819     ERROR -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::f() (pid=146818, ip=
  File "<stdin>", line 6, in f
ray.exceptions.TaskCancelledError: Task: TaskID(2f709301c413ada5ffffffffffffffffffffffff11000000) was cancelled

Hmmm I wasn’t able to reproduce it. Which ray version you are using? It might have been fixed in the nightly.