My scenario is following: I have a Ray DAG which produces dataframes. I want to perform certain modifications to these dfs using Dask API lazily and do more modifications downstream using plain ray tasks/actors by continuing construction of existing DAG, so all of this needs to be a single DAG. Hence the question: rather than calling dask_dag.compute() for immediate execution, can I somehow translate this DAG to Ray’s tasks DAG and use it downstream? I’m sure Dask-on-Ray does something similar under the hood, can the functionality be exposed?
cc @yic
cc @Clark_Zinzow
Ping @yic @Clark_Zinzow , can you help please?
a good pattern might be to develop a generic representation of a dag and change the way it binds re: dask or ray