Is ray serve queue FIFO?

Ray Libraries (Data, Train, Tune, Serve)
1

felt like it should be guaranteed FIFO since it was described as a queue, but just wanted to double check, we are observing some request getting in front of other request. but still investigating whether this is due to some request arrived later.

further debugging makes me think there might be something i didn’t fully understand on how ray serve handle queued requests.

let me describe what we have tested:

Client side:
a client send first 4 concurrent requests to ray serve http api, and wait for the response, each request would take ray serve approximately 60 sec to complete.

after roughly 1 minute, the client sends 2nd round of 4 concurrent requests.

Server side:
ray serve deployment exposing http endpoint on a single machine, configured as:

- name: process
  route_prefix: /process
  import_path: api_process:app_builder

  deployments:
  - name: ProcessDeployment
    max_ongoing_requests: 1
    max_queued_requests: 20
    autoscaling_config:
      min_replicas: 1
      initial_replicas: null
      max_replicas: 1
      target_num_ongoing_requests_per_replica: 1.0

the ProcessDeployment is the only api client calls, however under the hood, it would call other deploymentHandles to perform computations.

My expected behavior would be the server would process first 4 requests then work on next 4 requests, so in the order of A1,A2,A3,A4,B1,B2,B3,B4 where A and B represents two batches

However what we have observed was that the server was processing requests in the order of A1,A2,A3,B1,A4,B2,B3,B4, the first request of second batch got in the middle of the first batch, and this can be reproduced reliably.

Would there be anything I could possibly missed to setup ?

And i do see those messages got printed:

(ProxyActor pid=470667) WARNING 2024-11-08 05:53:50,270 proxy 10.128.0.54 5f9cb857-7c68-4aaf-86fa-0fca20003d98 /process/ pow_2_scheduler.py:698 - Failed to schedule request after 50 attempts over 44.30s. Retrying. Request ID: 5f9cb857-7c68-4aaf-86fa-0fca20003d98.

is this expected ?

Thank you very much