when using Parallel Iterator on a Custom Object, which contains pyarrow data. We are facing Serialization Exceptions. PFA for the screenshot.
Ray version : dev -2.0.0
When we debug the ray code, inside cloudpickle_fast.py line no 668 , for the following snippet, it always returns Not implemented.
try: is_anyclass = issubclass(t, type) except TypeError: # t is not a class (old Boost; see SF #502085) is_anyclass = False if is_anyclass: return _class_reduce(obj) elif isinstance(obj, types.FunctionType): return self._function_reduce(obj) else: # fallback to save_global, including the Pickler's # dispatch_table return NotImplemented
Is there any workaround to bypass this issue or a fix can be provided based on evaluation. Any help is appreciated.