Setting override_num_blocks=int(len(file_paths) / 100) with 200,000 items should indeed create ~2,000 blocks, which is much more than the number of actors. If you still see only a small number of actors working, the issue is likely not with block count, but with Ray Data’s scheduling, backpressure, or downstream consumption rate (source, source).
Check if downstream operators (e.g., training or further data processing) are consuming data fast enough, as Ray Data may scale down active actors if there is backpressure or idle time. Would you like more detail on how to monitor and address backpressure or actor scaling in Ray Data?
Sources:
- https://discuss.ray.io/t/data-map-batches-is-not-respecting-concurrency-from-the-beginning/16074
- https://github.com/ray-project/ray/issues/41504
Hint: Mention @RunLLM in the post for followups.