# BatchPredictors for TensorRT/AITemplate models

**URL:** <https://discuss.ray.io/t/batchpredictors-for-tensorrt-aitemplate-models/9055>\
**Category:** Uncategorized\
**Created:** [January 20, 2023, 2:00pm UTC](https://discuss.ray.io/t/batchpredictors-for-tensorrt-aitemplate-models/9055 "2023-01-20T14:00:31Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![plum9](https://avatars.discourse-cdn.com/v4/letter/p/258eb7/32.png) [@plum9](https://discuss.ray.io/u/plum9)\
**Post date:** [January 20, 2023, 2:00pm UTC](https://discuss.ray.io/t/batchpredictors-for-tensorrt-aitemplate-models/9055/1 "2023-01-20T14:00:31Z")

</div>

Hi,

I was looking into using Ray AIR and Ray Cluster to perform offline batch inference on a large dataset. The model is accelerated using TensorRT or AIT in some cases. To improve on cost, latency and performance I wanted to use these with Ray AIR BatchPredictors.

My understanding is that the checkpoint can only be loaded for predefined ML frameworks. But how should I go about writing a simple wrapper to load weights for models optimized by XLAs.

Or should i write code using raw actors and actorpools?

Thanks

---

<div class="post-metadata">

**Author:** ![Huaiwei\_Sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/huaiwei_sun/32/1316_2.png) [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)\
**Post date:** [February 3, 2023, 6:20am UTC](https://discuss.ray.io/t/batchpredictors-for-tensorrt-aitemplate-models/9055/2 "2023-02-03T06:20:18Z")

</div>

It’s better to ask this question in Ray AIR category.

cc: @kai @matthewdeng