When you run ray parallel iterator code in jupyter notebook . The worker processes are not killed, even after the script finishes execution. We can reproduce with simple code like this.
import ray
ray.init(address=‘auto’, _redis_password=‘5241590000000000’, ignore_reinit_error=True)
it = ray.util.iter.from_items([1, 2, 3, 4], num_shards=2)
for val in it.gather_async():
print(val)