Does ray support multi GPU inference with TensorRT?

Ray Libraries (Data, Train, Tune, Serve) Ray Serve
1

Does ray support multi GPU inference with TensorRT?

2

Hi @jinfagang, yes! You can just use TensorRT Python API in Ray Serve like local script because Ray Serve support arbitrary Python code. In order to use multiple GPU make sure to allocate GPUs via @serve.deployment(ray_actor_options={"GPU": 2}) syntax.