# How to do Load Balancing? **URL:** \ **Category:** Ray Clusters\ **Created:** [August 31, 2024, 12:10am UTC](https://discuss.ray.io/t/how-to-do-load-balancing/15705 "2024-08-31T00:10:08Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![ppljb](https://avatars.discourse-cdn.com/v4/letter/p/9de053/32.png) [@ppljb](https://discuss.ray.io/u/ppljb)\ **Post date:** [August 31, 2024, 12:10am UTC](https://discuss.ray.io/t/how-to-do-load-balancing/15705/1 "2024-08-31T00:10:08Z")
**How severe does this issue affect your experience of using Ray?** - High: It blocks me to complete my task. Hi there, I have a Ray cluster consisting of 5 nodes with 10 GPUs in total. Using Ray serve and vLLM I have deployed multiple LLM models on this cluster and each LLM exposes a REST endpoint via FastAPI. Currently, I have set `proxy_location: HeadOnly`. I want to ensure high availability and robustness to varying workload. I read already, that I probably change to run a proxy on each node in the cluster [Architecture — Ray 2.34.0](https://docs.ray.io/en/latest/serve/architecture.html) and enable AutoScaling. For now, this is not my concern and for testing purposes I manually spawned a new node to deploy one of my LLMs with 2 replicas. This worked so far, BUT my question is whether this contains load balancing or not because it looks like that currently all traffic is routed to the first replica and the second is idle all the time although the first replica could use some relief as it is under heavy fire and at \>90% usage. Am I just unlucky with my observations or must I implement load balancing on my own, e.g., using nginx, such that also the second replica is utilized? Unfortunately, the information I can find is somewhat contradictory: Here you explain load balancing [[High] Why doesn't parallelism work with data preprocessing? - #3 by shrekris](https://discuss.ray.io/t/high-why-doesnt-parallelism-work-with-data-preprocessing/12992/3) , the docs say I can use my own load balancing on top of Ray. Thank you very much Best regards ---
**Author:** ![shrekris](https://avatars.discourse-cdn.com/v4/letter/s/ce73a5/32.png) [@shrekris](https://discuss.ray.io/u/shrekris)\ **Post date:** [September 3, 2024, 5:31am UTC](https://discuss.ray.io/t/how-to-do-load-balancing/15705/2 "2024-09-03T05:31:39Z")
> it looks like that currently all traffic is routed to the first replica and the second is idle all the time although the first replica could use some relief as it is under heavy fire and at \>90% usage. Proxy actors forward requests to deployment replicas using a `ServeHandle`. Usually that means replicas get roughly even load because the `ServeHandle` performs power-of-two-choices. As an optimization, the `ServeHandles` in the proxy actor first perform power-of-two-choices across replicas only on the same node. That way, requests can be fulfilled without requiring cross-node communication. If no replicas are on the same node (or if all replicas on the same node are busy), then the proxy falls back to replicas on other nodes. Since you’re running with a proxy only on one node, this is likely why only one of your replicas is getting traffic. It’s not being saturated (i.e. the number of ongoing requests is always lower than `max_ongoing_requests`), so the proxy keeps sending that replica traffic. Could you enable proxy actors on all nodes, and balance requests across them? That way, the traffic is spread out more evenly. > must I implement load balancing on my own, e.g., using nginx, such that also the second replica is utilized? There are two places where load balancing should happen: 1. **Across proxy actors:** this must be implemented outside of Serve. For example, you could use `nginx` here to balance requests across all the different proxy actors. 2. **Across deployment replicas:** this happens out-of-the-box in the `ServeHandle`. When a `ServeHandle` receives a request, it selects a replica using power-of-two-choices and sends the request to that replica using a Ray actor call. ---
**Author:** ![ppljb](https://avatars.discourse-cdn.com/v4/letter/p/9de053/32.png) [@ppljb](https://discuss.ray.io/u/ppljb)\ **Post date:** [September 4, 2024, 10:49am UTC](https://discuss.ray.io/t/how-to-do-load-balancing/15705/3 "2024-09-04T10:49:18Z")
> As an optimization, the `ServeHandles` in the proxy actor first perform power-of-two-choices across replicas only on the same node. That way, requests can be fulfilled without requiring cross-node communication. If no replicas are on the same node (or if all replicas on the same node are busy), then the proxy falls back to replicas on other nodes. Ok, to be precise my 2 replicas run on different nodes. > Since you’re running with a proxy only on one node, this is likely why only one of your replicas is getting traffic. It’s not being saturated (i.e. the number of ongoing requests is always lower than `max_ongoing_requests` ), so the proxy keeps sending that replica traffic. Ah I found my error, `max_ongoing_requests` was set to 100 by default (Ray 2.22.0) With Ray 2.34.0 `max_ongoing_requests` is set to 5 and now it does the load-balancing > Could you enable proxy actors on all nodes, and balance requests across them? That way, the traffic is spread out more evenly. What will be the advantage of enabling proxy actors on all nodes if the `proxy_location: HeadOnly` also handles the load balancing? Thank you very much! Best regards ---
**Author:** ![shrekris](https://avatars.discourse-cdn.com/v4/letter/s/ce73a5/32.png) [@shrekris](https://discuss.ray.io/u/shrekris)\ **Post date:** [September 6, 2024, 3:43pm UTC](https://discuss.ray.io/t/how-to-do-load-balancing/15705/4 "2024-09-06T15:43:06Z")
> What will be the advantage of enabling proxy actors on all nodes if the `proxy_location: HeadOnly` also handles the load balancing? Running the proxy on all nodes provides fault tolerance in case the proxy on the head node becomes unavailable. ---
**Author:** ![ppljb](https://avatars.discourse-cdn.com/v4/letter/p/9de053/32.png) [@ppljb](https://discuss.ray.io/u/ppljb)\ **Post date:** [September 10, 2024, 5:11pm UTC](https://discuss.ray.io/t/how-to-do-load-balancing/15705/5 "2024-09-10T17:11:31Z")
> Running the proxy on all nodes provides fault tolerance in case the proxy on the head node becomes unavailable. Alright, thank you very much. ---
**Author:** ![Lisyra](https://avatars.discourse-cdn.com/v4/letter/l/71e660/32.png) [@Lisyra](https://discuss.ray.io/u/Lisyra)\ **Post date:** [January 2, 2026, 5:30pm UTC](https://discuss.ray.io/t/how-to-do-load-balancing/15705/6 "2026-01-02T17:30:31Z")
The true limiting was your max\_ongoing\_requests, therefore load balancing is now functional. Proxies on every node increase robustness and efficiency by reducing cross-node calls and adding fault tolerance, not by spreading traffic.