It's very Inconvenient that ObjectRef must be cloudpickled for later outband remote task cancelation

Ray Core
1

I known that there exist a reason for cloudpickle to save ObjectRef for later outband task cancelation, but it’s just not a convenient way to cancel task, from the point of view of a ray user.

2

Could you share more details about why it’s not convenient?

3

If I want to kill a remote task out of band, I have to cloudpickle ObjectRef in the first place and save it somewhere for later use which I feel inconvenient.

4

Is it correct that “out of band task” means you’d like to cancel a task after you lose the reference?

I think the primary problem is the “out-of-band task cancellation” is not an officially supported path, and cloudpickle is a workaround to achieve this. It basically is against the reference counting model Ray currently has.

I feel like the right approach is to use a high level lib that supports out-of-band tasks. @yic isn’t workflow lib the good use case for this? I can imagine it should support it.

5

Good point! True, I need another layer of abstraction.

6

@sangcho would you please elaborate more on using workflow to solve “out of band task cancellation” problem? I had exactly the same use case.

7

I meant this library; Ray Workflows: Durable Ray Task Graphs — Ray 3.0.0.dev0

But this is at alpha stage, and I am not sure about the development status now…

8

@blshao84 did you get a chance to check workflows to see if it is a fit?