# Streaming support for Ray actors **URL:** \ **Category:** Ray Core\ **Created:** [February 24, 2025, 1:33pm UTC](https://discuss.ray.io/t/streaming-support-for-ray-actors/21900 "2025-02-24T13:33:28Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![blublinsky](https://avatars.discourse-cdn.com/v4/letter/b/f05b48/32.png) [@blublinsky](https://discuss.ray.io/u/blublinsky)\ **Post date:** [February 24, 2025, 1:33pm UTC](https://discuss.ray.io/t/streaming-support-for-ray-actors/21900/1 "2025-02-24T13:33:28Z")
**How severe does this issue affect your experience of using Ray?** - Medium: It contributes to significant difficulty to complete my task, but I can work around it. Ray provides a great class RayGenerator ([Ray Generators — Ray 2.42.1](https://docs.ray.io/en/latest/ray-core/ray-generator.html)), which allows to run code as follows: ```auto import asyncio @ray.remote def task(): for i in range(5): time.sleep(1) yield i async def main(): async for ref in task.remote(): print(await ref) asyncio.run(main()) ``` Ray Also provides dynamic generators ([Dynamic generators — Ray 2.42.1](https://docs.ray.io/en/latest/ray-core/tasks/generators.html)), but, they are not supported by actors and do not provide async support Ray serve provides streaming responses ([Serve a Chatbot with Request and Response Streaming — Ray 2.42.1](https://docs.ray.io/en/latest/serve/tutorials/streaming.html)), which is exactly what I am looking for. Is there a chance to add streaming support for Ray actors or can you suggest the way to implement such functionality? ---
**Author:** ![rliaw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rliaw/32/24_2.png) [@rliaw](https://discuss.ray.io/u/rliaw)\ **Post date:** [February 26, 2025, 2:39am UTC](https://discuss.ray.io/t/streaming-support-for-ray-actors/21900/2 "2025-02-26T02:39:30Z")
Hmm, it should be supported in actors right? [https://docs.ray.io/en/latest/ray-core/ray-generator.html#generator-from-actor-tasks](https://docs.ray.io/en/latest/ray-core/ray-generator.html#generator-from-actor-tasks) ---
**Author:** ![blublinsky](https://avatars.discourse-cdn.com/v4/letter/b/f05b48/32.png) [@blublinsky](https://discuss.ray.io/u/blublinsky)\ **Post date:** [February 26, 2025, 8:45am UTC](https://discuss.ray.io/t/streaming-support-for-ray-actors/21900/3 "2025-02-26T08:45:31Z")
generators are, but dynamic generators no. As a result streaming, similar to Ray serve, is not supported ---
**Author:** ![rliaw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rliaw/32/24_2.png) [@rliaw](https://discuss.ray.io/u/rliaw)\ **Post date:** [February 26, 2025, 5:53pm UTC](https://discuss.ray.io/t/streaming-support-for-ray-actors/21900/4 "2025-02-26T17:53:28Z")
Can you help define what is “dynamic generators”? ---
**Author:** ![jjyao](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jjyao/32/1799_2.png) [@jjyao](https://discuss.ray.io/u/jjyao)\ **Post date:** [February 26, 2025, 6:10pm UTC](https://discuss.ray.io/t/streaming-support-for-ray-actors/21900/5 "2025-02-26T18:10:00Z")
Hi @blublinsky, streaming generator is supported by actors: [Ray Generators — Ray 2.42.1](https://docs.ray.io/en/latest/ray-core/ray-generator.html#generator-from-actor-tasks) ---
**Author:** ![blublinsky](https://avatars.discourse-cdn.com/v4/letter/b/f05b48/32.png) [@blublinsky](https://discuss.ray.io/u/blublinsky)\ **Post date:** [February 26, 2025, 8:28pm UTC](https://discuss.ray.io/t/streaming-support-for-ray-actors/21900/6 "2025-02-26T20:28:27Z")
The reference to documentation is above [Dynamic generators — Ray 2.42.1](https://docs.ray.io/en/latest/ray-core/tasks/generators.html). In simple terms, it allows to return back an AsyncGenerator, that can be used outside of the actor that recieved it. Dynamic generator != ray generator ---
**Author:** ![rliaw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rliaw/32/24_2.png) [@rliaw](https://discuss.ray.io/u/rliaw)\ **Post date:** [February 26, 2025, 9:57pm UTC](https://discuss.ray.io/t/streaming-support-for-ray-actors/21900/7 "2025-02-26T21:57:07Z")
@blublinsky Can you share exactly a code snippet, specifically in the context of an actor, for what you want to achieve? Pseudocode/code that doesn’t work is fine. I think the confusion on our side is that I don’t think we’ve intentionally named it differently to distinguish between generator vs dynamic generator. ---
**Author:** ![blublinsky](https://avatars.discourse-cdn.com/v4/letter/b/f05b48/32.png) [@blublinsky](https://discuss.ray.io/u/blublinsky)\ **Post date:** [February 27, 2025, 9:13am UTC](https://discuss.ray.io/t/streaming-support-for-ray-actors/21900/8 "2025-02-27T09:13:54Z")
Sure, here is my working code for Ray Serve. The deployment looks like: ```auto @serve.deployment(name="AsyncLLMEngine", ray_actor_options={"num_gpus": 1, "num_cpus": 4}) class AsyncLLMEngineDeployment: """ Ray serve deployment based on VLLM AsyncLLMEngine. We create this SO that we can easily scale this and wire it with other serving components """ def __init__ ( self, engine_args: AsyncEngineArgs, ): # create logger self.logger = logging.getLogger("async_llm_support") # create engine self.engine = AsyncLLMEngine.from_engine_args(engine_args) self.logger.debug("Created AsyncLLMEngine") .... async def generate( self, prompt: PromptType, sampling_params: SamplingParams, request_id: str, lora_request: Optional[LoRARequest] = None, trace_headers: Optional[Mapping[str, str]] = None, prompt_adapter_request: Optional[PromptAdapterRequest] = None, priority: int = 0, ) -> AsyncGenerator[RequestOutput, None]: """ Generate outputs for a request. This method is a coroutine. It adds the request into the waiting queue of the LLMEngine and streams the outputs from the LLMEngine to the caller. :param prompt: The prompt to the LLM. See :class:`~vllm.inputs.PromptType `for more details about the format of each input. :param sampling_params: the sampling parameters of the request. :param request_id: the unique id of the request :param lora_request: LoRA request to use for generation, if any :param trace_headers: OpenTelemetry trace headers :param prompt_adapter_request: Prompt Adapter request to use for generation, if any :param priority: the priority of the request. Only applicable with priority scheduling. :return: """ self.logger.info("AsyncLLMEngine - generate request") return self.engine.generate(prompt=prompt, sampling_params=sampling_params, request_id=request_id, lora_request=lora_request, trace_headers=trace_headers, prompt_adapter_request=prompt_adapter_request, priority=priority) .... ``` Now I can use this deployment as: ```auto results_generator = engine.options(stream=True).generate.remote( prompt=prompt, sampling_params=sampling_params, request_id=request_id, lora_request=lora_request, trace_headers=trace_headers, prompt_adapter_request=prompt_adapter_request, priority=priority ) final_output = None async for request_output in results_generator: final_output = request_output prompt = final_output.prompt text_outputs = [prompt + output.text for output in final_output.outputs] return text_outputs ``` I can not do the same thing for the plain Ray actors as `options(stream=True)` is deployment specific The main thing that I need to do is to return generator from the actor and then invoke it to get execution results ---
**Author:** ![blublinsky](https://avatars.discourse-cdn.com/v4/letter/b/f05b48/32.png) [@blublinsky](https://discuss.ray.io/u/blublinsky)\ **Post date:** [March 6, 2025, 6:51pm UTC](https://discuss.ray.io/t/streaming-support-for-ray-actors/21900/9 "2025-03-06T18:51:55Z")
They are very different things. Generator is a Ray actor method, implementing Python generator interface, while dynamic generator returns back a generator object, through which you can get data (bypassing an actor) ---
**Author:** ![blublinsky](https://avatars.discourse-cdn.com/v4/letter/b/f05b48/32.png) [@blublinsky](https://discuss.ray.io/u/blublinsky)\ **Post date:** [March 6, 2025, 6:54pm UTC](https://discuss.ray.io/t/streaming-support-for-ray-actors/21900/10 "2025-03-06T18:54:38Z")
@rliaw, @jjyao Is my explanation good enough? Any implementation suggestions? ---
**Author:** ![rliaw](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rliaw/32/24_2.png) [@rliaw](https://discuss.ray.io/u/rliaw)\ **Post date:** [March 6, 2025, 7:31pm UTC](https://discuss.ray.io/t/streaming-support-for-ray-actors/21900/11 "2025-03-06T19:31:37Z")
> [@blublinsky](#): > > They are very different things. Generator is a Ray actor method, implementing Python generator interface, while dynamic generator returns back a generator object, through which you can get data (bypassing an actor) Is the difference here just that in the latter, you don’t need to call ray.get()? ---
**Author:** ![blaine](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/blaine/32/8466_2.png) [@blaine](https://discuss.ray.io/u/blaine)\ **Post date:** [February 9, 2026, 11:40pm UTC](https://discuss.ray.io/t/streaming-support-for-ray-actors/21900/12 "2026-02-09T23:40:54Z")
I didn’t follow the discussion part about dynamic vs ray generators, but if I understand OP’s issue, it still exists. The example OP shared works. The example in the documentation for async actors also works, but if you notice, there’s a critical difference (I trimmed the example down): ```python @ray.remote class AsyncActor: async def f(self): for i in range(5): yield i actor = AsyncActor.remote() for ref in actor.f.remote(): print(ray.get(ref)) ``` The `async for` syntax is not supported. The only time it seems supported is if it was launched as part of a Ray Serve deployment or a Ray Job. This creates friction, because code that runs in Ray Serve isn’t exactly compatible using Ray Client. I confirmed this too: - Ray Client / a script: `actor.f.remote()` is type `ClientObjectRef` - Ray Serve: `actor.f.remote()` is type `ObjectRefGenerator`