I would like to rename a column in a dataset.
metadata = ray.data.from_items([{'index': index, 'timestamp': timestamp} for index, timestamp in zip(r, timestamps)])
image_data = ray.data.from_numpy(images)
full_ds = metadata.zip(image_data)
full_ds.schema()
Column Type
------ ----
index int64
timestamp double
data numpy.ndarray(shape=(720, 1280, 3), dtype=uint8)
def rename_row(row: Dict[str, Any]) -> Dict[str, Any]:
row['image'] = row.pop('data')
return row
renamed_ds = full_ds.map(rename_row)
renamed_ds.schema()
2023-09-24 19:35:09,776 INFO streaming_executor.py:92 -- Executing DAG InputDataBuffer[Input], InputDataBuffer[Input] -> ZipOperator[Zip] -> TaskPoolMapOperator[Map(rename_row)] -> LimitOperator[limit=1]
2023-09-24 19:35:09,777 INFO streaming_executor.py:93 -- Execution config: ExecutionOptions(resource_limits=ExecutionResources(cpu=None, gpu=None, object_store_memory=None), locality_with_output=False, preserve_order=True, actor_locality_enabled=True, verbose_progress=False)
2023-09-24 19:35:09,777 INFO streaming_executor.py:95 -- Tip: For detailed progress reporting, run `ray.data.DataContext.get_current().execution_options.verbose_progress = True`
(Map(rename_row) pid=1177384) Task failed with retryable exception: TaskID(6012944da5356edfffffffffffffffffffffffff01000000).
(Map(rename_row) pid=1177384) Traceback (most recent call last):
(Map(rename_row) pid=1177384) File "python/ray/_raylet.pyx", line 1191, in ray._raylet.execute_dynamic_generator_and_store_task_outputs
(Map(rename_row) pid=1177384) File "python/ray/_raylet.pyx", line 3684, in ray._raylet.CoreWorker.store_task_outputs
(Map(rename_row) pid=1177384) File "/mnt/md0/.anaconda3/envs/python311news/lib/python3.11/site-packages/ray/data/_internal/execution/operators/map_operator.py", line 415, in _map_task
(Map(rename_row) pid=1177384) for b_out in fn(iter(blocks), ctx):
(Map(rename_row) pid=1177384) File "/mnt/md0/.anaconda3/envs/python311news/lib/python3.11/site-packages/ray/data/_internal/planner/plan_udf_map_op.py", line 76, in do_map
(Map(rename_row) pid=1177384) yield from transform_fn(blocks, ctx, *fn_args, **fn_kwargs)
(Map(rename_row) pid=1177384) File "/mnt/md0/.anaconda3/envs/python311news/lib/python3.11/site-packages/ray/data/_internal/planner/map_rows.py", line 25, in fn
(Map(rename_row) pid=1177384) for row in block.iter_rows(public_row_format=True):
(Map(rename_row) pid=1177384) File "/mnt/md0/.anaconda3/envs/python311news/lib/python3.11/site-packages/ray/data/_internal/table_block.py", line 196, in __next__
(Map(rename_row) pid=1177384) return row.as_pydict()
(Map(rename_row) pid=1177384) ^^^^^^^^^^^^^^^
(Map(rename_row) pid=1177384) File "/mnt/md0/.anaconda3/envs/python311news/lib/python3.11/site-packages/ray/data/row.py", line 33, in as_pydict
(Map(rename_row) pid=1177384) return dict(self.items())
(Map(rename_row) pid=1177384) ^^^^^^^^^^^^^^^^^^
(Map(rename_row) pid=1177384) File "<frozen _collections_abc>", line 861, in __iter__
(Map(rename_row) pid=1177384) File "/mnt/md0/.anaconda3/envs/python311news/lib/python3.11/site-packages/ray/data/_internal/arrow_block.py", line 87, in __getitem__
(Map(rename_row) pid=1177384) return ArrowBlockAccessor._build_tensor_row(self._row, col_name=key)
(Map(rename_row) pid=1177384) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Map(rename_row) pid=1177384) File "/mnt/md0/.anaconda3/envs/python311news/lib/python3.11/site-packages/ray/data/_internal/arrow_block.py", line 201, in _build_tensor_row
(Map(rename_row) pid=1177384) element = element.as_py()
(Map(rename_row) pid=1177384) ^^^^^^^^^^^^^^^
(Map(rename_row) pid=1177384) File "/mnt/md0/.anaconda3/envs/python311news/lib/python3.11/site-packages/ray/air/util/tensor_extensions/arrow.py", line 193, in as_py
(Map(rename_row) pid=1177384) return self.type._extension_scalar_to_ndarray(self)
(Map(rename_row) pid=1177384) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Map(rename_row) pid=1177384) File "/mnt/md0/.anaconda3/envs/python311news/lib/python3.11/site-packages/ray/air/util/tensor_extensions/arrow.py", line 135, in _extension_scalar_to_ndarray
(Map(rename_row) pid=1177384) return _to_ndarray_helper(shape, value_type, offset, data_buffer)
(Map(rename_row) pid=1177384) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Map(rename_row) pid=1177384) File "/mnt/md0/.anaconda3/envs/python311news/lib/python3.11/site-packages/ray/air/util/tensor_extensions/arrow.py", line 892, in _to_ndarray_helper
(Map(rename_row) pid=1177384) return np.ndarray(shape, dtype=ext_dtype, buffer=data_buffer, offset=data_offset)
(Map(rename_row) pid=1177384) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Map(rename_row) pid=1177384) TypeError: buffer is too small for requested array
...
More task tracebacks
What should I do to increase the size of the buffer?
Also, is there a better way to rename a column?