"RuntimeError: Lost reference to actor" when using get_actor directly (i.e. not storing in a temporary variable)

Ray Core
1

I’ve encountered a super puzzling lifecycle/scope(?) issue that took a long time to debug so wanted to share and gather some ideas form the community.

In my project I have a very basic setup (like this) with an actor that I converted into a named one so I don’t need to pass it though multiple layers of my code.

Doing so triggered this bizarre error:

>>> ray.get_actor("CounterActor").read.remote()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "###\envs\ray\lib\site-packages\ray\actor.py", line 111, in remote
    return self._remote(args, kwargs)
  File "###\envs\ray\lib\site-packages\ray\actor.py", line 152, in _remote
    return invocation(args, kwargs)
  File "###\envs\ray\lib\site-packages\ray\actor.py", line 140, in invocation
    raise RuntimeError("Lost reference to actor")
RuntimeError: Lost reference to actor

I’ve confirmed that my agent is still around because the same one passed via a variable was still happily accepting calls, and even tried to use “detached” lifecycle, just in case, with no effect.

Turned out that as long as I do something like this:

f = ray.get_actor('CounterActor')
f.read.remote()

everything works, but converting this to a oneliner doesn’t

ray.get_actor('CounterActor').read.remote()

so if anyone has ideas on what is going on, I would be happy for any clues :slight_smile:

2

You can reproduce this issue without named actors as follows.

import ray

ray.init()

@ray.remote
class Foo:
    def method(self):
        pass

ray.get(Foo.remote().method.remote())

The issue is tracked here Lost reference to actor exception for seemingly valid code. · Issue #6265 · ray-project/ray · GitHub.

Right now it’s a limitation that we have because the natural fix would introduce performance issues. There’s probably a good way to get around this though. We can probably also improve the error message in the short term.

3

Thank you, Robert!

I’ve missed the bug, which is embarrassing given that it’s the first Google result :slight_smile:

A better error message (or perhaps a warning in the docs) would be super helpful.

1 Like