Using ray.util.inspect_serializability tool, I am able to find that some ABC class and some other tensorflow callback are the culprit. TypeError: can't pickle _abc_data objects · Issue #332 · uqfoundation/dill · GitHub .
Might be a better way to just construct the object inside train_func so that there is no need to worry about serialization…