Parallel text file processing

Ray Core
1

I needed some time figuring how to mimic async process data as it come i.e. like multiprocessing.imap_unordered() … finally it seems the only way is to abstract the inner loop in an Actor. This is correct, right I can’t do it with a task/function ?

But this also meant i had to abstract the data-feeder as a generator.

So here is what I came up with : Parallel text processing ++ · GitHub

u can add this to your Docs if you want!

2

@ray_user This is awesome! We should definitely make sure to include this on the docs. cc @will

BTW, can I ask, did it take you a long time to figure out how to do this? I suspect this may be hard for users to figure this out, but just wanted to check :slight_smile:

3

I was confused that I have to create Actor, coming from multiprocessing.

Different expectations I think :slight_smile: once I knew and found map_unordered it was good

1 Like