Write ray dataset to big query error

Ray Libraries (Data, Train, Tune, Serve)
1

When calling the ds.write_bigquery, with the latest ray data 2.34.0:

ds.write_bigquery(
    project_id=PROJECT_ID,
    dataset="batch_inference.test_ray_3",
)

it errors out: google.api_core.exceptions.BadRequest: 400 Table test_ray_3_2356b89e_2397_44cb_8233_276d99ea9147_source does not have a schema.; reason: invalid, message: Table test_ray_3_2356b89e_2397_44cb_8233_276d99ea9147_source does not have a schema.

2

This error looks to be related to BigQuery, and not directly from Ray Data.

Going by this GH issue, it looks like it’s related to when a table is first created without a schema, then trying to insert with a schema into a table. Can you check that the schema definition is consistent between the BigQuery table, and the Ray dataset being written? (Either omit the schema in both places, or use the same schema in both places).