Hey,
So I was wondering if the pipeline API can be used for building pipelines where a few models have dependency to other models and need to be run sequentially where as for others there is no dependency and can be run in parallel. The current approach of Model Composition does help in making such a pipeline but unfortunately models that can be run in parallel also have to run sequentially. Would something like this be possible?
We take model dependency that support sequential (chaining) and parallel (broadcast, reduce) as first class pattern and it should be supported, just like how you can use ray tasks / actors to author it. Can you provide a bit more details of that pipeline you have in mind with some high-level DAG structure with classes ?