Multiplexing and batching go together?

Ray Core Ray Workflows
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

Can multiplexing and batching be combined together? Looked into the documentation and could find any examples.

2

Hello there! I believe you can combine multiplexing and dynamic request batching in Ray Serve. The official docs don’t have an example of them together, but I think they can be used in the same deployment.

To get it working, you’d add both the @serve.multiplexed and @serve.batch decorators to your deployment’s methods together.

Multiplexed example: Model Multiplexing — Ray 2.46.0

Dynamic request batching example: Dynamic Request Batching — Ray 2.46.0

1 Like
3

Great thanks, I will give it an attempt

1 Like