# How does routing work when using 'http-location=all'

**URL:** <https://discuss.ray.io/t/how-does-routing-work-when-using-http-location-all/11596>\
**Category:** Ray Serve\
**Created:** [July 28, 2023, 3:45am UTC](https://discuss.ray.io/t/how-does-routing-work-when-using-http-location-all/11596 "2023-07-28T03:45:14Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kyle\_Robinson](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kyle_robinson/32/4844_2.png) [@Kyle\_Robinson](https://discuss.ray.io/u/Kyle_Robinson)\
**Post date:** [July 28, 2023, 3:45am UTC](https://discuss.ray.io/t/how-does-routing-work-when-using-http-location-all/11596/1 "2023-07-28T03:45:14Z")

</div>

We are deploying an inference cluster using Serve on AWS, and are starting to distribute requests among all workers’ own http\_proxy servers. Requests are routed among all workers/head using our own external AWS load balancer. This load balancer is rather ‘stupid’ and has no context of the number of ongoing requests in each worker node. When a request arrives at a worker nodes http\_proxy, is it further routed to another worker? How do all of these http\_servers know which workers to route to?

---

<div class="post-metadata">

**Author:** ![Gene](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/gene/32/4701_2.png) [@Gene](https://discuss.ray.io/u/Gene)\
**Post date:** [July 28, 2023, 3:57am UTC](https://discuss.ray.io/t/how-does-routing-work-when-using-http-location-all/11596/2 "2023-07-28T03:57:21Z")

</div>

Short answer is the http\_proxy on the worker which LB routed to will randomly pick a replica to route the request. It can be on any worker/ head as long as they still accept requests.

`http_location` tells Ray where to start http\_proxy. If it’s set to all, then all nodes regardless of head or worker can accept requests on their http port. Once a request is hitting a node, the http\_proxy has the state of where is which replicas running. By default, there is this power of two choices algorithm, that basically just take two replicas and send the traffic to the one with less request queue length. You can read more [https://github.com/ray-project/ray/blob/master/python/ray/serve/\_private/router.py#L263](https://github.com/ray-project/ray/blob/master/python/ray/serve/_private/router.py#L263)