# Ray Serve LLM application

**URL:** <https://discuss.ray.io/t/ray-serve-llm-application/11874>\
**Category:** Uncategorized\
**Created:** [August 21, 2023, 1:22pm UTC](https://discuss.ray.io/t/ray-serve-llm-application/11874 "2023-08-21T13:22:07Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![CansuCandan](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/cansucandan/32/4000_2.png) [@CansuCandan](https://discuss.ray.io/u/CansuCandan)\
**Post date:** [August 21, 2023, 1:22pm UTC](https://discuss.ray.io/t/ray-serve-llm-application/11874/1 "2023-08-21T13:22:07Z")

</div>

Hi,

I want to apply ray.serve for my llm model.

It could be basic problem. But I stuck with this problem.

my app2.py:

```auto
import requests
from ray import serve
import starlette

@serve.deployment(route_prefix="/forecast")
class Ray_llm:
    async def __call__ (self, request: starlette.requests.Request):
        if "file" not in requests.files:
            return {"error": "No file part in the request"}, 400

        file = requests.files["file"]

        if file.filename == "":
            return {"error": "No selected file"}, 400

        query_text = requests.form.get("query", None)
        if not query_text:
            return {"error": "No query text provided"}, 400

        if file and query_text:

            response = send_to_llm(file, query_text)
            return response

def send_to_llm(file, querry_text):

    response = llm_caller(file, querry_text) # Send to model

    return response

app = Ray_llm.bind()
serve.run(app, port=8081)

```

Dockerfile:

```auto
FROM python:3.8-slim

WORKDIR /app

COPY requirements.txt .

RUN pip install --no-cache-dir -r requirements.txt
RUN pip install "ray[serve]"

EXPOSE 8081

COPY . .

CMD ["python", "app2.py"]

```

BuildAndRun.sh :

```auto
docker build -t llm_api .

docker run -p 8081:5000 llm_api

```

when I run BuildAndRun.sh my service does not alive to post request. How can I solve this problem?

ss:

 ![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/5/55d138321138d05531cba6d6027bc4c9a320260c.png)