Hello,
Is is possible to dump the task graph of an application written with Ray? More specifically, I am interested in taking a look at the data dependency pattern between tasks, where each future was created and which tasks use their value, etc.
We do keep lineage information in the owner, but I think right now it’s purely for fault tolerance.
@sangcho is working on observability who might have a better idea about this.
Updates: synced offline and we don’t support this right now.
Thanks for your reply, @yic.
Is the lineage information dumped to disk at any point in the execution, be it for fault tolerance or any other reason?
I have been taking a look at the TaskManager class and noticed that the output ObjectReference's for a task are instantiated inside TaskManager::AddPendingTask. Do you think it would work if I instrumented this function to dump task information, along with its input and output ObjectReference's and use this information to “reconstruct” the graph offline using a script, for instance?
@leiteg lineage information is not dumped to disk. It’s memory data. We use that for reconstruction when data is lost.
@leiteg btw, if you have static graph, you can try Ray DAG (Building Computation Graphs with Ray DAG API — Ray 3.0.0.dev0)
It is possible, but we have not prioritized supporting this features yet. I recommend you to create a Github issue to follow up!