How to Expose Ray Serve API with proxy_location="EveryNode" Outside the Cluster

Ray Serve
1

1. Severity of the issue:

  • High – Completely blocking my work

2. Environment:

  • Ray version: 2.48.0
  • Python version: 3.12.11
  • Operating system: Ubuntu
  • Cloud/Infrastructure: AWS (Ray Autoscaler directly on EC2)

Issue Description:

Hello,

I’m currently deploying applications using Ray Serve and I’m having trouble understanding how to properly use proxy_location="EveryNode".

Specifically, when exposing the API outside the Ray cluster, I’m unsure how to determine which IP address to target. Since each node runs a proxy, do I need to know the IP of every single node?

Within the cluster, I can use get_deployment_handle, which makes things straightforward. However, I’d like to make the API accessible externally and I’m uncertain about the correct approach.

Do I need to set up an external load balancer? If so, how can I dynamically know which IPs to include or point to?

Thank you in advance for your help!

2

Yes, you’d need to setup an external load balancer. You can check out serving on Anyscale which takes care of this for you - Update a service | Anyscale Docs; or check the kuberay docs.

1 Like