# Best practices to run multiple models in multiple GPUs in RayLLM

**URL:** <https://discuss.ray.io/t/best-practices-to-run-multiple-models-in-multiple-gpus-in-rayllm/13663>\
**Category:** Ray Train\
**Created:** [February 8, 2024, 3:57am UTC](https://discuss.ray.io/t/best-practices-to-run-multiple-models-in-multiple-gpus-in-rayllm/13663 "2024-02-08T03:57:11Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![lizzzcai](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/lizzzcai/32/3417_2.png) [@lizzzcai](https://discuss.ray.io/u/lizzzcai)\
**Post date:** [February 8, 2024, 3:57am UTC](https://discuss.ray.io/t/best-practices-to-run-multiple-models-in-multiple-gpus-in-rayllm/13663/1 "2024-02-08T03:57:11Z")

</div>

Hi team, I am trying out RayLLM with KubeRay recently following example [here](https://github.com/ray-project/ray-llm/tree/master/models/continuous_batching), I am trying to deploy a Mistral-7b model on 1xA10 GPU and 4x A10 GPU. I am expecting `4x` throughput but only get `1.x` improvement.

Would like to get some advice from experts here on the parameter tuning as I feel like the parameters in the above example are not optimised.

Few questions:

1. if I want to utilise 4 GPUs, I should set `4 workers with 1 replica each` or `1 worker with 4 replicas`, and what is the difference here?
2. if I want to run 2 models (different models) in a single GPU what are the best practices on the Ray Serve configuration for each of the models?

Thanks.