I have function(s) that expects input dataframe and config (dict). The function can return multiple dataframes of different sizes depending on the function.
I was trying to use Ray Data with map_groups but the return in map_groups expects dataframes of same size, which will not be the case in my functions.
Can use the above approach or Do I have to combine Ray Data with Ray Task to scale the computation?