Ray server for use case with limited memory resources

Ray Serve
1

1. Severity of the issue: (select one)
High: Completely blocks me.

2. Environment:

  • Ray version: Latest (currently 2.56.1)
  • OS: Linux

Hello, I am exploring Ray serve to see if it fits the next use case. I hope somebody can help

I have a application that serves a lot of models in comparison to the hardware resources i have. Currently there is only 1 machine for the model server and maybe it will be upgraded to a couple of machine more(still not enough). I still consider using Ray because the app has other use cases with a bigger cluster where Ray fits clearly well.

So the main issue is that i cannot afford having loaded in memory all models in order to avoid OOM errors. I have seen in the docs that there are parameters for scaling down models already loaded (something about a timeout for scaling down to zero and initialize model wihtout loading the model until request on it). However that won’t guarantee me that i would get a OOM easy ( i maybe could set timeout very low but that would lead to the issue of loading a new model for every new request)

Currently i have a custom simple system with a http server where i manage a list of the current models being used and i unload-load the models based on new requests.

I would like to have something like that or with equivalent results in ray

I was thinking about some bussiness logic in the ray’s entry point deployment in the same way i have it in the current custom http server but even if it is valid it probably work only for cluster with 1 machine/node ( maybe is a good solution also for bigger cluster??? ). Also and most importantly it won’t work with this setup which is the one i plan to use because incompatibility of dependencies of the models to be deployed(diffent version of libraries, different version of python, different CUDA , etc ) Run Multiple Applications in Different Containers — Ray 2.56.0

2

Ray Serve supports your use case with model multiplexing, which allows you to serve many models without loading all of them into memory at once. Each deployment replica loads models on demand and evicts the least recently used ones when a configurable limit (max_num_models_per_replica) is reached, preventing OOM errors. You can also control the timeout for matching requests to replicas with a specific model loaded, balancing between latency and memory usage. This pattern is recommended for clusters with limited resources and works across multiple nodes, not just single-node setups. See the model multiplexing documentation for details and code examples.

For handling dependency isolation (different Python, CUDA, or library versions per model), Ray Serve supports running each application in a separate container using the image_uri field in the runtime environment. This allows you to deploy models with incompatible dependencies side-by-side on the same cluster, as described in the multi-app container guide. This approach is suitable for both single-node and multi-node clusters.

Would you like more detail or example code for either model multiplexing or multi-app container deployment?

Sources:

Hint: Mention @Herald in the post for followups.

3

Ok, i discarded multiplexing before because i didn’t know about image_uri for the issue with the dependencies. I will check it

Given the fast response i guess this is an AI bot but very useful anyways

thanks

4

I have checked the docs for the model multiplexing and i don’t see how it provides having each model in its own docker container

For example lets take this code of the docs

from ray import serve
import aioboto3
import torch
import starlette


@serve.deployment
class ModelInferencer:
    def __init__(self):
        self.bucket_name = "my_bucket"

    @serve.multiplexed(max_num_models_per_replica=3)
    async def get_model(self, model_id: str):
        session = aioboto3.Session()
        async with session.resource("s3") as s3:
            obj = await s3.Bucket(self.bucket_name)
            await obj.download_file(f"{model_id}/model.pt", f"model_{model_id}.pt")
            return torch.load(f"model_{model_id}.pt", weights_only=False)

    async def __call__(self, request: starlette.requests.Request):
        model_id = serve.get_multiplexed_model_id()
        model = await self.get_model(model_id)
        return model.forward(torch.rand(64, 3, 512, 512))


entry = ModelInferencer.bind()

the @serve.multiplexed is in a deployment and so according to the documentation(Handle Dependencies — Ray 2.56.0) i can setup a deployment in a docker container but not each of the models of the multiplexed deployment in their own container or im wrong ???

5

I managed to solve the issue by calling each model on its own process with python’s ‘suprocess.run()’

I provide a reduced version of my proof of concept just in case its useful for anybody with the same issue

Here the multiplexed method of the deployment

@serve.multiplexed(max_num_models_per_replica=1)
    async def get_model(self, model_id: str):

        if model_id == "m1":
            return Model1()

        elif model_id == "m2":
            return Model2()

Then each object internally calls subprocess.run() to run each model script with its own python and dependencies

subprocess.run([
   Path(M2_DIR, ".venv", "bin", "python3"),
   Path(M2_DIR, "src", "uv2", "model2.py"),
 ])

The model load-unload process works as expected