How to suppress TaskCancelledError without RAY_IGNORE_UNHANDLED_ERRORS?

Ray Core
1

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

When I call ray.cancel(r) on an inflight object ref, the worker prints an exception, even if I don’t call ray.get().

ERROR worker.py:400 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::fgh() (pid=13251, ip=192.168.1.15)
  File "<stdin>", line 7, in fgh
ray.exceptions.TaskCancelledError: Task: TaskID(62ffec03f52574bdffffffffffffffffffffffff03000000) was cancelled

I’d like to suppress this because my workload typically cancels a significant amount of tasks as a performance optimization. However, I don’t want to set ‘RAY_IGNORE_UNHANDLED_ERRORS=1’ because I don’t want to suppress actual unexpected worker errors, like workers dying.

Is there a way to suppress just TaskCancelledError? I did try catching KeyboardInterrupt and other exceptions within the task but that did not seem to work.

2

Hey @xcharleslin!

Do you have a small repro for this? I tried the below code but it doesn’t raise the canceled error if I don’t try to ray.get on it. (I used ray 2.2)

import ray

@ray.remote
def f(t):
    import time
    time.sleep(t)
    return t

obj2 = f.remote(9999)
ray.cancel(obj2)

obj = f.remote(3)
print(ray.get(obj))
print(ray.get(obj2))

Or if there is something I am missing here from your workflow?

3

hey Ricky, try this:

import ray; import time; ray.init()

@ray.remote
def f():
    while True:
        print("f() running")
        time.sleep(1)

ref = f.remote()
time.sleep(2)
ray.cancel(ref)

Output:

(f pid=146818) f() running
(f pid=146818) f() running
# pauses for a few seconds here
2023-02-02 16:00:32,819     ERROR worker.py:400 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::f() (pid=146818, ip=192.168.86.60)
  File "<stdin>", line 6, in f
ray.exceptions.TaskCancelledError: Task: TaskID(2f709301c413ada5ffffffffffffffffffffffff11000000) was cancelled
4

Hmmm I wasn’t able to reproduce it. Which ray version you are using? It might have been fixed in the nightly.