How to post data to dynamic batch directly？

Ray Libraries (Data, Train, Tune, Serve) Ray Serve
1
  • High: It blocks me to complete my task.

Example from ray serve document,

from typing import List

import numpy as np

from ray import serve
from ray.serve.handle import DeploymentHandle


@serve.deployment
class Model:
    @serve.batch(max_batch_size=8, batch_wait_timeout_s=0.1)
    async def __call__(self, multiple_samples: List[int]) -> List[int]:
        # Use numpy's vectorized computation to efficiently process a batch.
        return np.array(multiple_samples) * 2

How can I use requests.post to invoke __call__ directly？

2

Hi @Hspix, if you use the Model you defined and run it with serve.run, you’ll be able to hit __call__ with requests.post:

@serve.deployment
class Model:
    @serve.batch(max_batch_size=8, batch_wait_timeout_s=0.1)
    async def __call__(self, multiple_samples: List[int]) -> List[int]:
        # Use numpy's vectorized computation to efficiently process a batch.
        return np.array(multiple_samples) * 2

serve.run(Model.bind())
r = requests.post("http://localhost:8000")

The multiple_samples argument in your __call__ function will be a list of starlette.requests.Request objects, not a list of integers.