I did a ray serve deployment in ray cluster using FastApi and serve batch.Ray serve deployment is running at http://127.0.0.1:8000/
i have the below client script to invoke the endpoint with the inputs.
Client script is as below.
import ray
import requests
@ray.remote
def send_query(value):
#print(f"value: {value}“)
resp = requests.post(”",json=value)
return resp.text
results = [send_query.remote(text) for text in names]
print(“Result returned:”, results)
for result in results:
print(result)
print(ray.get(result))
PS: Request is going to replica but client disconnects the request .As per the documentation Set Up FastAPI and HTTP — Ray 2.40.0, i added asyncio.shield also in the serve deployment script.it didn’t work with that as well.
Here are the configs:
max_ongoing_request=30
target_ongoing_request=2
i tried to add below config as well but i dont find a way to verify is this is really taking effect.
http_options:
host: 0.0.0.0
port: 8000
request_timeout_s: 300
keep_alive_timeout_s: 300
Points to be noted:
- First time when i give 1 request to the endpoint , it works fine without issues.
- Second time when i give 6 requests to the endpoint, it works fine without issues.
- Third time when i give 10 requests to the endpoint, its not working and giving connection refused error in the client script in the line ray.get(result).
After 3#, even if i send 2 request, its not working.
Here is the error log.
Tailing logs until the job exits (disable with --no-wait):
2025-01-02 13:23:30,092 INFO job_manager.py:530 – Runtime env is setting up.
2025-01-02 13:23:33,726 INFO client_builder.py:244 – Passing the following kwargs to ray.init() on the server: log_to_driver
SIGTERM handler is not set because current thread is not the main thread.
Result returned: [ClientObjectRef(238a00da985abb62ffffffffffffffffffffffff2a00000001000000), ClientObjectRef(689eb1a0c44aa227ffffffffffffffffffffffff2a00000001000000)]
ClientObjectRef(238a00da985abb62ffffffffffffffffffffffff2a00000001000000)
Traceback (most recent call last):
File “/tmp/ray/session_2025-01-01_19-24-17_406163_1/runtime_resources/working_dir_files/_ray_pkg_3182eb4dff41d75f/ray_serve_request.py”, line 64, in
print(ray.get(result))
File “/home/ray/anaconda3/lib/python3.9/site-packages/ray/_private/auto_init_hook.py”, line 21, in auto_init_wrapper
return fn(*args, **kwargs)
File “/home/ray/anaconda3/lib/python3.9/site-packages/ray/_private/client_mode_hook.py”, line 102, in wrapper
return getattr(ray, func.name)(*args, **kwargs)
File “/home/ray/anaconda3/lib/python3.9/site-packages/ray/util/client/api.py”, line 42, in get
return self.worker.get(vals, timeout=timeout)
File “/home/ray/anaconda3/lib/python3.9/site-packages/ray/util/client/worker.py”, line 433, in get
res = self._get(to_get, op_timeout)
File “/home/ray/anaconda3/lib/python3.9/site-packages/ray/util/client/worker.py”, line 461, in _get
raise err
ray.exceptions.RayTaskError(ConnectionError): ray::send_query() (pid=30499, ip=10.200.3.133)
File “/home/ray/anaconda3/lib/python3.9/site-packages/urllib3/util/connection.py”, line 95, in create_connection
raise err
File “/home/ray/anaconda3/lib/python3.9/site-packages/urllib3/util/connection.py”, line 85, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
ray::send_query() (pid=30499, ip=10.200.3.133)
File “/home/ray/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py”, line 715, in urlopen
httplib_response = self._make_request(
File “/home/ray/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py”, line 416, in _make_request
conn.request(method, url, **httplib_request_kw)
File “/home/ray/anaconda3/lib/python3.9/site-packages/urllib3/connection.py”, line 244, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File “/home/ray/anaconda3/lib/python3.9/http/client.py”, line 1285, in request
self._send_request(method, url, body, headers, encode_chunked)
File “/home/ray/anaconda3/lib/python3.9/http/client.py”, line 1331, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File “/home/ray/anaconda3/lib/python3.9/http/client.py”, line 1280, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File “/home/ray/anaconda3/lib/python3.9/http/client.py”, line 1040, in _send_output
self.send(msg)
File “/home/ray/anaconda3/lib/python3.9/http/client.py”, line 980, in send
self.connect()
File “/home/ray/anaconda3/lib/python3.9/site-packages/urllib3/connection.py”, line 205, in connect
conn = self._new_conn()
File “/home/ray/anaconda3/lib/python3.9/site-packages/urllib3/connection.py”, line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f52e010fe50>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
ray::send_query() (pid=30499, ip=10.200.3.133)
File “/home/ray/anaconda3/lib/python3.9/site-packages/requests/adapters.py”, line 486, in send
resp = conn.urlopen(
File “/home/ray/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py”, line 801, in urlopen
retries = retries.increment(
File “/home/ray/anaconda3/lib/python3.9/site-packages/urllib3/util/retry.py”, line 594, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host=‘127.0.0.1’, port=8000): Max retries exceeded with url: /test/serving_request/ (Caused by NewConnectionError(‘<urllib3.connection.HTTPConnection object at 0x7f52e010fe50>: Failed to establish a new connection: [Errno 111] Connection refused’))
During handling of the above exception, another exception occurred:
Please help me as i am totally stuck for the past 4 days.