Is is possible to obtain a task graph from an application?


Is is possible to dump the task graph of an application written with Ray? More specifically, I am interested in taking a look at the data dependency pattern between tasks, where each future was created and which tasks use their value, etc.

We do keep lineage information in the owner, but I think right now it’s purely for fault tolerance.

@sangcho is working on observability who might have a better idea about this.

Updates: synced offline and we don’t support this right now.

Thanks for your reply, @yic.

Is the lineage information dumped to disk at any point in the execution, be it for fault tolerance or any other reason?

I have been taking a look at the TaskManager class and noticed that the output ObjectReference's for a task are instantiated inside TaskManager::AddPendingTask. Do you think it would work if I instrumented this function to dump task information, along with its input and output ObjectReference's and use this information to “reconstruct” the graph offline using a script, for instance?

@leiteg lineage information is not dumped to disk. It’s memory data. We use that for reconstruction when data is lost.

@leiteg btw, if you have static graph, you can try Ray DAG (Building Computation Graphs with Ray DAG API — Ray 3.0.0.dev0)