AttributeError: 'NoneType' object has no attribute 'call_retain' when deserializing an object ref

1

The following code recommended by a Ray team member in `ray.get` hangs on `ObjectRef`s passed out-of-band · Issue #9578 · ray-project/ray · GitHub no longer works if run against a remote ray cluster.

import ray

ray.init()

@ray.remote
def foo(serialized_object_ref):
    print(ray.get(ray.cloudpickle.loads(serialized_object_ref)))

object_ref = ray.put(42)
ray.get(foo.remote(ray.cloudpickle.dumps(object_ref)))
print("finished")

I’ve tested it on Ray 2.40. Is this a regression? What’s the currently recommended way of sending object refs out-of-bound?

Thanks

Stack trace:

Traceback (most recent call last):
  File ".../ser_fail.py", line 10, in <module>
    ray.get(foo.remote(ray.cloudpickle.dumps(object_ref)))
  File ".../python3.12/site-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File ".../python3.12/site-packages/ray/_private/client_mode_hook.py", line 102, in wrapper
    return getattr(ray, func.__name__)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../python3.12/site-packages/ray/util/client/api.py", line 42, in get
    return self.worker.get(vals, timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../python3.12/site-packages/ray/util/client/worker.py", line 433, in get
    res = self._get(to_get, op_timeout)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../python3.12/site-packages/ray/util/client/worker.py", line 461, in _get
    raise err
ray.exceptions.RayTaskError(AttributeError): ray::foo() (pid=790, ip=10.128.28.108)
  File ".../python_packages/future_ray_py_api/ser_fail.py", line 7, in foo
  File ".../python3.12/site-packages/ray/util/client/common.py", line 97, in __init__
    self._set_id(id)
  File ".../python3.12/site-packages/ray/util/client/common.py", line 188, in _set_id
    self._worker.call_retain(id)
    ^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'call_retain'
2

Hi there! Welcome to the Ray community serge :slight_smile:

To directly address your question:

The currently recommended way to send ObjectRefs out-of-band is to avoid explicit serialization using ray.cloudpickle. Instead, pass ObjectRefs through Ray’s native task arguments and return mechanisms, as this ensures compatibility. In newer versions of Ray, it’s recommended to not use ray.cloudpickle for this because then Ray won’t know how to garbage collect it and it also causes some unexpected behaviors.

If you absolutely must send ObjectRefs out-of-band, here’s what you should do:

  1. Enable out-of-band serialization by setting:
export RAY_allow_out_of_band_object_ref_serialization=1

You can read more about this here: Anti-pattern: Serialize ray.ObjectRef out of band — Ray 2.41.0

  1. Be mindful of object lifecycle issues. Ensure that the thread that’s deserializing the ObjectRef belongs to the same Ray cluster, and that the object is explicitly freed after use.

Some more docs that might be helpful:

3

Hi Christina,

Thank you for the quick reply. Can you check that the code I linked to (shared by a Ray team member) in `ray.get` hangs on `ObjectRef`s passed out-of-band · Issue #9578 · ray-project/ray · GitHub is supposed to work or not? I am getting the error I reported above regardless of RAY_allow_out_of_band_object_ref_serialization setting with that code.

Thanks