Ray + VLLM - Need support on Proxy

1

We are using Ray - using ray cli
Started using ray start --head and joined on other nodes .Cluster is ok
We are using vllm for the inference
We are using proxy to download or connect to hugging face and its always failing on the second node .
How can we set http /https proxy using CLI when running or is there a waty we can set that with vllm-your help is much appreicated .Thanks alot

2

Hi @Rahul_MR There is a serve start command you can run to start Ray Serve with the proxy options you need. See the doc Ray Serve API — Ray 2.35.0

3

Bu is the “serve” creates a cluster or run application on top of an existing ray cluster ?
We want to use distributed inferencing using ray + vllm

@Gene ^^ & thank you

4

This “Serve” is the same “Serve” that automatically started when you run serve.run or serve.deploy to deploy your applications. It’s not a Ray cluster, it runs on existing Ray cluster to create all the necessary Serve components (controller, proxies, and replicas). By running the CLI first, you just create it explicitly with the config you want.

5

We tried some combination and did not work ,just to clarify that our approach is correct here is our config

  • our proxy
export http_proxy="http://userName:yourPassword@yourProxyURL.com:8080"
export https_proxy="http://userName:yourPassword@yourProxyURL.com:8080"

So we tried serve start —http-host “userName:yourPassword@yourProxyURL.com” --http-port 8080
We get an error as below

ProxyActor pid=3055068) INFO 2024-09-10 11:58:36,265 proxy 10.0.1.178 proxy.py:1179 - Proxy starting on node a33435baa9c5e27f1c4e6082078ccbe0cbebae39d6988dcc52870e4c (HTTP port: 8000).
(ProxyActor pid=3055068) ERROR 2024-09-10 11:58:36,270 proxy 10.0.1.178 proxy.py:1338 - Failed to bind Ray Serve HTTP proxy to ‘xxx:8080’. Please make sure your http-host and http-port are specified correctly.
(ProxyActor pid=3055068) Traceback (most recent call last):
(ProxyActor pid=3055068) File “/hxxon3.9/site-packages/ray/serve/_private/proxy.py”, line 1353, in run_http_server
@Gene -thanks alot

6

hmm I don’t think that’s how you configure the host. Serve usually bind to localhost, with the port you specified, then you can have your proxy routing requests to Serve’s port for your services.