TypeError: can't pickle weakref objects when passing python object to trainable

1

Hi, I am seeing TypeError: can't pickle weakref objects error when passing object to trainable. The error implies that it cannot serialize this object.

Wondering what is the correct way to pass object to trainable? Any suggestions on how to bypass this error?

Example code:

        config = {"learning_rate" : tune.loguniform(1e-4, 1e-1)}
        trainer = Trainer(backend="tensorflow", num_workers=1, use_gpu=False)

        def train_func(config, model_base):
            model_base_object = model_base

            model_base_object.train_fn()
            tune.report(accuracy=xxx)
            return

        analysis = tune.run(
            tune.with_parameters(train_func, model_base=model_base_object),
            metric="accuracy",
            mode="max",
            num_samples=3,
            config=config
        )

Stacktrace:

...
    tune.with_parameters(train_func, model_base=copy.deepcopy(self)),
../build/training/environments/development-venv/lib/python3.7/site-packages/ray/tune/utils/trainable.py:348: in with_parameters
    parameter_registry.put(prefix + k, v)
../build/training/environments/development-venv/lib/python3.7/site-packages/ray/tune/registry.py:208: in put
    self.flush()
../build/training/environments/development-venv/lib/python3.7/site-packages/ray/tune/registry.py:220: in flush
    self.references[k] = ray.put(v)
../build/training/environments/development-venv/lib/python3.7/site-packages/ray/_private/client_mode_hook.py:105: in wrapper
    return func(*args, **kwargs)
../build/training/environments/development-venv/lib/python3.7/site-packages/ray/worker.py:1872: in put
    object_ref = worker.put_object(value, owner_address=serialize_owner_address)
../build/training/environments/development-venv/lib/python3.7/site-packages/ray/worker.py:305: in put_object
    serialized_value = self.get_serialization_context().serialize(value)
../build/training/environments/development-venv/lib/python3.7/site-packages/ray/serialization.py:413: in serialize
    return self._serialize_to_msgpack(value)
../build/training/environments/development-venv/lib/python3.7/site-packages/ray/serialization.py:392: in _serialize_to_msgpack
    metadata, python_objects
../build/training/environments/development-venv/lib/python3.7/site-packages/ray/serialization.py:353: in _serialize_to_pickle5
    raise e
../build/training/environments/development-venv/lib/python3.7/site-packages/ray/serialization.py:349: in _serialize_to_pickle5
    value, protocol=5, buffer_callback=writer.buffer_callback
../build/training/environments/development-venv/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle_fast.py:73: in dumps
    cp.dump(obj)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <ray.cloudpickle.cloudpickle_fast.CloudPickler object at 0x7f66d40cfbb0>
obj = [<TunerExample object at 0x7f66bc314d10>]

    def dump(self, obj):
        try:
>           return Pickler.dump(self, obj)
E           TypeError: can't pickle weakref objects

../build/training/environments/development-venv/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle_fast.py:620: TypeError
2

Using ray.util.inspect_serializability tool, I am able to find that some ABC class and some other tensorflow callback are the culprit. TypeError: can't pickle _abc_data objects · Issue #332 · uqfoundation/dill · GitHub .

Might be a better way to just construct the object inside train_func so that there is no need to worry about serialization…

1 Like