# Training loop stuck at StreamSplitDataIterator
**URL:** https://discuss.ray.io/t/training-loop-stuck-at-streamsplitdataiterator/23069
**Category:** Ray Train
**Created:** [September 4, 2025, 2:50pm UTC](https://discuss.ray.io/t/training-loop-stuck-at-streamsplitdataiterator/23069 "2025-09-04T14:50:48Z")
**Posts on this page:** 2
**Page:** 1
### Author:  [@mitul93](https://discuss.ray.io/u/mitul93)
#### Post date: [September 4, 2025, 2:50pm UTC](https://discuss.ray.io/t/training-loop-stuck-at-streamsplitdataiterator/23069/1 "2025-09-04T14:50:48Z")
**1. Severity of the issue: (select one)**
High: Completely blocks me.
**2. Environment:**
- Ray version: 2.49.0
- Python version: 3.10.12
- OS: Ubuntu 24.04
- Cloud/Infrastructure: AWS
- Other libs/tools (if relevant):
**3. What happened vs. what you expected:**
- Expected: Ray training continue for 10 epoch
- Actual: Ray train stuck at the end of epoch
I’m using [Fine-tuning a face mask detection model with Faster R-CNN — Ray 2.49.1](https://docs.ray.io/en/latest/ray-overview/examples/object-detection/1.object_detection_train.html#main-content) example to train a RCNN model with minor modification.
If I use more than one workers in scaling\_config, the training loop gets stuck at following line.
The behavior is bit random. Sometimes it is able to finish 2 epochs and then stuck at the following warning and sometimes it is stuck after finishing 1 epoch.
```auto
...
...
StreamSplitDataIterator(epoch=1, split=1) blocked waiting on other clients for more than 30s. All clients must read from the DataIterator splits at the same time. This warning will not be printed again for this epoch.
```
Here is pseudo code
```auto
train_dataset = ray.data.read_parquet(dataset_info_file)
train_dataset = train_dataset.map(ReadDataset, concurrency=2, num_cpus=4)
train_loop_config = {"num_epochs": 10, "lr": 0.01, "batch_size": 4, "prefetch_batches":1, "enable_mlflow": False, "run_name" : run_name}
run_config=ray.train.RunConfig(
storage_path=s3_storage_path,
name=run_name,
)
scaling_config = ray.train.ScalingConfig(num_workers=2, use_gpu=True, resources_per_worker={"CPU": 3, "GPU" : 1})
trainer = TorchTrainer(
train_loop_per_worker=train_loop_per_worker,
train_loop_config=train_loop_config,
datasets={"train": train_dataset},
scaling_config=scaling_config,
run_config=run_config
)
result = trainer.fit()
```
I’m stuck here a little. Can someone guide me on debugging this problem?
Thank you.
---
### Author:  [@mitul93](https://discuss.ray.io/u/mitul93)
#### Post date: [September 5, 2025, 1:52pm UTC](https://discuss.ray.io/t/training-loop-stuck-at-streamsplitdataiterator/23069/2 "2025-09-05T13:52:53Z")
I found a problem in my code.
My code originally
```python
...
if ray.train.get_context().get_world_rank() == 0:
...
ray.train.report(metrics, checkpoint=checkpoint)
...
```
Documentation for `ray.train.report` states clearly that it should be called from all workers.
> **[ray.train.report — Ray 2.49.1](https://docs.ray.io/en/latest/train/api/doc/ray.train.report.html)**
`ray.train.report` was not called from non zero rank workers.
Following change fixed my problem.
```python
...
if ray.train.get_context().get_world_rank() == 0:
...
ray.train.report(metrics, checkpoint=checkpoint)
...
else:
ray.train.report(metrics, checkpoint=None)
...
```