How severe does this issue affect your experience of using Ray?
- Medium: It contributes to significant difficulty to complete my task, but I can work around it.
When I call
ray.cancel(r) on an inflight object ref, the worker prints an exception, even if I don’t call ray.get().
ERROR worker.py:400 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::fgh() (pid=13251, ip=192.168.1.15) File "<stdin>", line 7, in fgh ray.exceptions.TaskCancelledError: Task: TaskID(62ffec03f52574bdffffffffffffffffffffffff03000000) was cancelled
I’d like to suppress this because my workload typically cancels a significant amount of tasks as a performance optimization. However, I don’t want to set ‘RAY_IGNORE_UNHANDLED_ERRORS=1’ because I don’t want to suppress actual unexpected worker errors, like workers dying.
Is there a way to suppress just TaskCancelledError? I did try catching KeyboardInterrupt and other exceptions within the task but that did not seem to work.