How to make `ray.cancel` less noisy?

Ray Core
1

I’m using ray.cancel to cancel remaining tasks in map-reduce operation when a particular value is found. It’s working correctly, but it’s very noisy, since SIGTERM signals are sent to the workers, which raises a KeyboardInterrupt error that is then sent back to stderr on the driver. This can be suppressed by setting RAY_IGNORE_UNHANDLED_ERRORS, but that’s severe overkill, since I still want to see errors from the workers.

Is there a way to cancel tasks without the noise? If not, I think that would be a useful feature.

2

I think this should be quiet by default, and if not, that could be a bug. Can you file an issue?

3

Hey @wmayner, the noise is happening since the task return value is not retrieved. You can both cancel the tasks and suppress the messages by following this pattern:

for ref in refs:
    ray.cancel(ref)
# Wait until all tasks have failed or been cancelled.
for ref in refs:
    try:
        ray.get(ref)
    except (ray.exceptions.RayTaskError, ray.exceptions.TaskCancelledError):
        pass
4

Thanks @ericl, this is a good workaround for now—but I think it would still be worthwhile to allow the user to avoid the additional overhead of ray.wait. I’ve raised a GitHub issue here: [Core] Suppress `ray.cancel` stderr noise from `KeyboardInterrupt` · Issue #24658 · ray-project/ray · GitHub