Hi everyone,
I’m beginning to test Ray abilities to perform parallel operations for my use case and I’m not sure whether I do not know which module would fit best, or if the use case just isn’t suited for Ray.
The use case:
It is quite simple, the idea is to perform linear algebra operations (matrix sum, multiplication, inversion,…) to solve a system AX=B. The matrices are quite huge: A can easily be 10⁵x10⁵ and the raw data used to construct the system are 10⁶x10⁶.
I have some tight time constraints to perform the solving. That is why I’m looking for speed up via parallelization. I’ve already set up the algorithms to ease things by taking advantages of the shape of the matrix, now I’m looking for a framework to host them.
What I’ve tested:
I’ve first used Ray Core to test the operations but there are no integrated math libraries, and using numpy/scipy on top of Ray was proving ineffective.
I’ve then thought about using Ray Train to take the advantage of TensorFlow and its linear algebra library, but as I’m not a ML specialist at all, this idea seems a bit far-fetched.
The other idea I got was to use Dask on Ray since I’ve already set up my use case on Dask (which was . But in that case, what would this option provide me that Dask would not have? I’ve read the Ray documentation about it but I do not really get what this setup would provide Dask does not. Is Ray more robust than Dask (the latter already providing fault-tolerance solutions)? Would the computation not be slower if another layer is added?
Any tip is welcome, thanks for your time ! 