Hi ,
When we upgraded our environment from 1.2.0 to ray 2.0.0 , we are seeing issues like
“Cloudpickle Error: Unknown type {}”.format(type(obj))
_pickle.PicklingError: Cloudpickle Error: Unknown type <class ‘ray.actor.ActorHandle’>
PFA for screen shot
Please advice in solving the issue.
This issue is occurring when we iterate on Apache Arrow Objects. From ray 2.0.0 this issue occurs. Previous ray versions work fine. Will investigate further, any suggestions will help.
Hmm odd. @suquark do you happen to know any change that can cause this?
It looks like an ActorHandle instance is used in type hint parameters (or could be in some third-party libraries related to typing like typing_extensions or pydantic). This is unusual. If you are sure that the issue is not about your code, we can patch cloudpickle temporarily to suppress errors.
There is a recent change that updates the cloudpickle: Update cloudpickle to commit 6e0f571 by suquark · Pull Request #14693 · ray-project/ray · GitHub. This fixes some of our previous serialization issues and cloudpickle bugs. It modifies parametrized_type_hint_getinitargs and is likely to cause this issue.
@suquark This issue occurs when we do gather_async on parallel iterator , trying to verify the issue on our side. Can you confirm if the serialization is done on the data fetched by the worker process or anything else?
With the provided information, I cannot identify the root cause of the error. I tried some possible setups but I am unable to reproduce such errors. But I know that cloudpickle cannot serialize some weird objects well (especially typing objects, because python seriously considers typing object serialization only since python3.7+, for python <= 3.6, cloudpickle has to use numerous serialization heuristics to try its best, and it often fails under extreme cases). For your current situation, I can add more heuristics to cloudpickle in Ray to mitigate serialization issues.
I create a PR for the fix: [WIP] Cloudpickle workaround for false positive cases by suquark · Pull Request #15202 · ray-project/ray · GitHub
