# Training issues with MultiworkerMirroredStrategy

**URL:** <https://discuss.ray.io/t/training-issues-with-multiworkermirroredstrategy/23335>\
**Category:** Ray Train\
**Created:** [November 25, 2025, 4:18pm UTC](https://discuss.ray.io/t/training-issues-with-multiworkermirroredstrategy/23335 "2025-11-25T16:18:40Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![RunLLM](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/runllm/32/8249_2.png) [@RunLLM](https://discuss.ray.io/u/RunLLM)\
**Post date:** [November 25, 2025, 4:19pm UTC](https://discuss.ray.io/t/training-issues-with-multiworkermirroredstrategy/23335/2 "2025-11-25T16:19:22Z")

</div>

Ray Train’s distributed TensorFlow integration with `MultiWorkerMirroredStrategy` is not deprecated, but users have reported issues running official examples across multiple nodes, especially with recent TensorFlow and Keras versions. The error you encountered—“unsupported type (\<class ‘tensorflow.python.distribute.values.PerReplica’\>) to a Tensor”—is a known TensorFlow issue when handling distributed datasets, often related to how data is sharded or how the dataset is constructed in a distributed context. This problem does not occur on a single node, which matches your experience. According to the [Ray distributed TensorFlow guide](https://docs.ray.io/en/latest/train/distributed-tensorflow-keras.html#toctree-checkbox-77), Ray should handle `TF_CONFIG` and worker setup, but compatibility issues with newer TensorFlow/Keras versions (e.g., Keras 3.x) can break these examples, as discussed in [community threads](https://discuss.ray.io/t/example-in-ray-train-for-tensorflow-distributed-training/20736).

Additionally, distributed training with Ray Train and TensorFlow is only beneficial for large datasets or when scaling across multiple GPUs/nodes outweighs the communication overhead. For small datasets or single-node setups, distributed training can be slower or unnecessary, as noted in [user discussions](https://discuss.ray.io/t/ray-train-tensorflowtrainer-look-slower-than-than-normal-pandas-and-tensorflow-i-e-without-using-distribution-training-or-any-framework/10190). If you are using recent TensorFlow/Keras versions, you may need to set `os.environ["TF_USE_LEGACY_KERAS"] = "1"` as a workaround, but this is not ideal for all users. Would you like a step-by-step breakdown of how to debug or adapt your code for multi-node distributed training with Ray and TensorFlow?

Sources:

- [Get Started with Distributed Training using TensorFlow/Keras — Ray 2.52.0](https://docs.ray.io/en/latest/train/distributed-tensorflow-keras.html#toctree-checkbox-77)
- [Ray Discuss: Example in Ray Train for TensorFlow distributed training](https://discuss.ray.io/t/example-in-ray-train-for-tensorflow-distributed-training/20736)
- [Ray Discuss: Ray Train TensorflowTrainer look slower than than (normal pandas and tensorflow)](https://discuss.ray.io/t/ray-train-tensorflowtrainer-look-slower-than-than-normal-pandas-and-tensorflow-i-e-without-using-distribution-training-or-any-framework/10190)

Hint: Mention @RunLLM in the post for followups.

---

_[View the full topic](https://discuss.ray.io/t/training-issues-with-multiworkermirroredstrategy/23335)._