1. Severity of the issue: (select one)
None: I’m just curious or want clarification.
Low: Annoying but doesn’t hinder my work.
Medium: Significantly affects my productivity but can find a workaround.
High: Completely blocks me.
2. Environment:
Ray version: 2.58.0
Python version: 3.11
OS: Mac
Cloud/Infrastructure: (running on mac laptop)
Other libs/tools (if relevant): pyarrow 25.0.1
3. What happened vs. what you expected:
Expected:
When reading multiple Parquet files with ray.data.read_parquet(), if there’s a schema type mismatch between files, the error message doesn’t indicate which file(s) have the conflicting schema, making it difficult to debug datasets with many files.
The error should include the file paths and their respective schemas to help users identify which file(s) have the problematic data, for example:
pyarrow.lib.ArrowTypeError: Unable to merge: Field detections has incompatible types:
list<element: struct<…>> vs large_string
Schemas from sampled files:
File: s3://bucket/data/file1.parquet
Schema: detections: list<element: struct<…>>
Actual:
The error only shows the conflicting types but not the file paths:
It could be possible that Ray’s current error reporting for schema mismatches during ray.data.read_parquet() does not include the file paths or detailed per-file schemas, which makes debugging difficult when handling many files. The error typically only shows the conflicting types, not which files caused the issue, as you described. This limitation is mentioned in user discussions and issue reports, but there is no explicit documentation or API in the provided sources that shows Ray will print file paths alongside schema errors by default.
Though I’m not entirely certain, it might be that you would need to manually inspect files with PyArrow or similar tools to identify the problematic files, as suggested in community discussions. There is no evidence in the data sources that Ray currently provides enhanced error messages with file paths for schema mismatches out of the box. Would you like more detail on possible workarounds or how to manually check file schemas?
Yes, this is considered a usability bug. Ray currently does not include file paths or per-file schemas in schema mismatch errors when reading multiple Parquet files, making it hard to debug large datasets. Community discussions and GitHub issues confirm that users expect error messages to indicate which files have conflicting schemas, but this is not implemented by default in Ray as of version 2.58.0 (Ray GitHub Issue #39548).
The recommended workaround is to manually inspect files with PyArrow to identify schema differences, as suggested by Ray maintainers. There is no built-in Ray feature that automatically reports the problematic file paths in schema errors. Would you like more detail on how to implement such a check or track schema mismatches manually?