# How to get Streaming output functions well?

**URL:** <https://discuss.ray.io/t/how-to-get-streaming-output-functions-well/10177>\
**Category:** Ray Serve\
**Created:** [April 11, 2023, 4:20am UTC](https://discuss.ray.io/t/how-to-get-streaming-output-functions-well/10177 "2023-04-11T04:20:00Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![jiaanguo](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jiaanguo/32/4203_2.png) [@jiaanguo](https://discuss.ray.io/u/jiaanguo)\
**Post date:** [April 11, 2023, 4:20am UTC](https://discuss.ray.io/t/how-to-get-streaming-output-functions-well/10177/1 "2023-04-11T04:20:00Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- High: It blocks me to complete my task.

```auto
app = FastAPI()

@serve.deployment(route_prefix="/", ray_actor_options={"num_gpus": 1})
@serve.ingress(app)
class ChatbotModelDeployment:

    @app.post("/")
    async def query(self, request: Request):
        data = await request.json()
        query = data.get("query", "")
        max_length = data.get("max_length", 2048)
        top_p = data.get("top_p", 0.9)
        temperature = data.get("temperature", 0.7)
        use_stream_chat = data.get("use_stream_chat", True)

        output = self._infer(query, None, max_length, top_p,
                             temperature, use_stream_chat)
        return StreamingResponse(output, media_type="text/plain")

chatbot_model_deployment = ChatbotModelDeployment.bind()

```

I am building a simple NLP model server with Ray Serve + FastAPI, however I could not build a streaming server that output tokens one by one. (The output is whole paragraph all at once). Any ideas on how to fix that? Many thanks in advance.

---

<div class="post-metadata">

**Author:** ![Akshay\_Malik](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/akshay_malik/32/3807_2.png) [@Akshay\_Malik](https://discuss.ray.io/u/Akshay_Malik)\
**Post date:** [April 11, 2023, 10:58pm UTC](https://discuss.ray.io/t/how-to-get-streaming-output-functions-well/10177/2 "2023-04-11T22:58:44Z")

</div>

Ray Serve does not support streaming responses yet. Please file a feature request here - [Issues · ray-project/ray · GitHub](https://github.com/ray-project/ray/issues)

---

<div class="post-metadata">

**Author:** ![Jules\_Damji](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jules_damji/32/4058_2.png) [@Jules\_Damji](https://discuss.ray.io/u/Jules_Damji)\
**Post date:** [April 11, 2023, 11:09pm UTC](https://discuss.ray.io/t/how-to-get-streaming-output-functions-well/10177/3 "2023-04-11T23:09:51Z")

</div>

@jiaanguo Since this functionality is not currently supported, can you file an issue as requested by @Akshay_Malik, and we can close this as resolved.

If you file an issue, include the link to the issue in here, please. Thanks!

---

<div class="post-metadata">

**Author:** ![jiaanguo](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jiaanguo/32/4203_2.png) [@jiaanguo](https://discuss.ray.io/u/jiaanguo)\
**Post date:** [April 12, 2023, 4:07am UTC](https://discuss.ray.io/t/how-to-get-streaming-output-functions-well/10177/4 "2023-04-12T04:07:29Z")

</div>

> <https://github.com/ray-project/ray/issues/34266>
>
> \### Description
> 
> I am building a chatbot with ray serves my model, currently m…y model supports streaming outputs like generating word tokens one after another. However, I could not get ray to send token back immediately in streaming manner. This would be a feature greatly required!
> 
> \`\`\`
> @serve.deployment(route\_prefix="/", ray\_actor\_options={"num\_gpus": 1})
> @serve.ingress(app)
> class ChatbotModelDeployment:
> 
> def \_infer(...):
> for output in self.\_model.stream\_chat(...):
> yield output+'\\n'
> 
> @app.post("/")
> async def query(self, request: Request):
> data = await request.json()
> query = data.get("query", "")
> 
> output = self.\_infer(query, ...)
> return StreamingResponse(output, media\_type="text/plain")
> 
>   
> chatbot\_model\_deployment = ChatbotModelDeployment.bind()
> 
> \`\`\`
> 
> Something like above that allows StreamingResponse from FastAPI, currently even I tried using Streaming Response, it still responses the whole paragraph once all tokens have been generated.
> 
> 
> \### Use case
> 
> Especially in nlp model deployment, need this feature to send token back directly rather than wait for the whole paragraph being generated.

I have created a new issue on github, link is above.

You could close this, many thanks.