Running 500,000 tasks

1

I have this simple script to run some_function for more than 500,000 times. The ray cluster wouldn’t return result after all 500,000 are done but I would like to make it like a queue. How should I change the script? Your help is much appreciated. Thank you.

import ray

futures = []

# there are about 500,000  rows
for index, input_row in enumerate(endpoints_list):
    futures.append(some_func.remote(input_row))

print('##################### Futures done appending #####################')

while len(futures) > 0:
    done_id, futures = ray.wait(futures)
    results = ray.get(done_id[0])

    for result in results:
        print(result)
        csv_writer.output(result, result['status'])