[Core] Ray Multiprocessing Pool Progress Tracker

Ray Core
1

Hi All,

I’m using Ray multiprocessing.Pool with slurm on our hpc cluster and I am super happy with it. I’d like to know what is the best way to track progress in an output file? i.e. parallel computation of 1000 chunks and currently have 10% of the chunks finished in x amount of time.

Thanks!!

2

Hey @jgjoseph you should be able to do something simple like:

with open("progress.txt", "w") as f:
    for result in Pool.imap_unordered(func, inputs):
        print(result, file=f)

imap_unordered will return results in any order as they finish, so you can track progress this way.

1 Like