Programmatic lightweight update from rest call

Ray Serve
1

1. Severity of the issue: (select one)
Medium: Significantly affects my productivity but can find a workaround.
High: Completely blocks me.

2. Environment:

  • Ray version: 2.48.0
  • Python version: 3.9
  • OS: macos, linux
  • Cloud/Infrastructure:
  • Other libs/tools (if relevant):

3. What happened vs. what you expected:

  • Expected: See below.
  • Actual:

Hi Ray People!
The page Updating Applications In-Place — Ray 2.52.1 describes a command-line based lightweight update by updating to config in a the config file, then run serve deploy (command). How to do this same thing programmatically? I.E. From inside my running Deployment, How to trigger a lightweight update? Is there a code example on this?

This similar to the ask of 2-year-old question, Does Ray Serve support local model hot update/reload? - #2 by Sihan_Wang , but would like to know more:

  1. Is this capability (of triggering lightweight update from the code) available now? (Linked article is pretty old.) If do have the ability, what are the calls/actions that need to be made?
  2. If not available, Is there a reason for not having this feature?
  3. Is it based on file watch and not on a software variable-watch mechanism, like Observer pattern? It would seem that all it would take to allow programmatic trigger of lightweight update would be to expose the function called at the point where the file-change detection happens. :person_shrugging:

My scenario:

RayServe app defined in python code as MyDeployment class with serve.deployment and serve.ingress(app) decorators. Class implements reconfigure() as described in the page above and in reconfigure() model is (re)loaded. It also has an endpoint function “/update”, which when called is supposed to do lightweight “update”, resulting in load of a “new” model when the reconfigure() method is called, without re-deploying the replicas - That is the hope at least.

To trigger a lightweight update from inside my Deployment class, I’ve tried:

  1. Save the passed-in dict to reconfigure() in my Deployment class. Then change a value of “name” entry in the /update function hoping that perhaps that dict somehow has a watcher on it and the lightweight update would be triggered. Nope, no update triggered.

  2. Implement reconfigure() as #1, but additionally, call serve.run() passing in new updated name in the user_config dict. This causes a complete re-deploy in each of the replicas: NOT a lightweight update and also an exception due to shut down of the replica that is handling the incoming /update request (after a couple of waits by controller). The code for this server.run which is similar to my initial deploy except the user_config dict has the original value config_0:

    serve.run(
    MyDeployment.options(user_config={"name": newnm}).bind(),
    name="testapp",
    route_prefix="/"
)

  3. Using JobSubmissionClient.submit_job(“server deploy mydeploymod:testapp –name testapp”). This doesn’t do a lightweight update, but a full deploy, rolling one replica after another, which is expected. It does have the upside of controlling where incoming requests are routed in an orderly manner though.

Any direction / thoughts / insights on this topic helpful, Thanks!

2

Ray Serve does not currently provide a built-in API to trigger a lightweight config update (such as updating user_config and calling reconfigure) programmatically from within a running deployment or via Python code alone. Lightweight updates are only supported via updating the config file and running serve deploy, which is idempotent and only updates the changed config fields without restarting replicas, as described in the documentation. There is no watcher or observer pattern on the user_config dict; simply mutating it in Python will not trigger reconfigure, and calling serve.run imperatively will cause a full redeploy, not a lightweight update. This limitation is confirmed in both the documentation and community discussions, and there is no official API to trigger such updates from within a deployment as of Ray 2.48.0.

The only supported way to perform lightweight updates is to update the config file and redeploy using the CLI. Programmatic support for this feature has been requested by users, but as of now, it is not available, and there is no code example for in-place lightweight updates triggered from within a deployment or via a Python API. For more details, see the official docs and related discussions. Would you like a step-by-step breakdown of how the config-based update works, or more information on possible workarounds?

Sources:

Hint: Mention @RunLLM in the post for followups.