I have some saved checkpoints which I am able to load and test locally. However, when I load these same checkpoints on Google Colab I receive the following error message:
Colab is fixed using Python 3.7 and the error occurs with Ray v1.8, v1.9 and v1.10. The source of the above error is not clear to me. Any suggestions would be greatly appreciated.
Yeah, indeed looks strange. Maybe a pickle version issue?
Could you send us more information so we can debug this? Like a small repro script that we could run in colab under the given python and ray versions?
Thank you for the quick reply. I have built a reproducible example after trying to understand the problem better. Here is the central issue. I am in a position where I need to analyse checkpoints on Colab (Python = 3.7, Ray = 1.9/1.10) which have been trained and saved with Python 3.8 and Ray 1.9/1.10.
I have attached the most basic cartpole example. If you run cartpole_train with >= Python 3.8 and then run cartpole_test inside Colab, you will receive the error.
NB. I have tried installing pickle5 and “import pickle5 as pickle” in Colab to no avail.
Is there any update on this? There is a similar issue here
corresponding to pickle incompatibility between Python 3.7 (Colab) and Python 3.8. However, all my attempts to upgrade Colab to use pickle5 failed.