Understanding async actors

Hello,
I found out when I introduce a dummy async method on a Ray Actor class but don’t call it, the order of execution becomes messed up.

import ray
import time

@ray.remote
class Foo:
    def __init__(self):
        self.value = 0

    def setter(self, value):
        self.value = value

    def getter(self):
        return self.value

    async def dummy(self):
        raise NotImplementedError


if __name__ == '__main__':
    ray.init()
    for i in range(10):
        bar = Foo.remote()
        bar.setter.remote(i)
        # time.sleep(0.1)
        print(ray.get(bar.getter.remote()))

The code should print numbers from 0 to 9, but actually prints out 10 zeros.
Commenting out the async method shows the expected behavior.
Uncommenting the sleep line to give the actor enough processing time also works.
(Tested on Ray 1.9.2 on Python 3.8, Ubuntu 18.04, on clean installation on fresh conda environment)

Without any prior knowledge on async actors, it is unnatural as the order of the execution is affected by some unrelated code. However, in the context of async actors, I’m not sure whether it is a feature or a bug.

Is it a correct behavior, that calling sync methods on an async actor may execute out of order?

To have sequential order, you can wait for the setter call to return before calling getter. e.g.

...
        bar = Foo.remote()
        ray.get(bar.setter.remote(i))
        print(ray.get(bar.getter.remote()))

Adding the async method increases the maximum concurrency from 1 to 1000, so you started to see calls executed out of order. Personally, I think it is better to explicitly sequence the calls with ray.get() or await, instead of relying on Actor’s single thread behavior.