Ray multiprocessing with multi pytorch model inference

Ray Core
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

Hi, I got stuck when I use ray to do multiprocessing inference. To be exactly, I have 8 GPU, and I want to do some inference with multiprocessing to speed up my work. On each process, a model will be put on a single GPU, which means there are 8 processes and 8 models. It is normally when I use concurrent.futures.ProcessPoolExecutor, but when I use ray, only one GPU is activated. Here is a snippet of my code.

@ray.remote(num_gpus=8)
def my_job(args):
    # do some jobs

def main():
    futures = [my_job.remote(j) for j in jobs]
    results = ray.get(futures)

if __name__ == "__main__":
    main()
2

Try setting num_gpus = 1. This means that each model will be put on 1 GPU.