Best practices for unit testing actors in large, complex codebase with many dependencies?

Ray Core Ray Client
1

I have some questions about best practices and limitations for unit testing Ray Actors, especially in non-toy examples.
For any sufficiently complex system, unit testing involves mocking large portions of that system, including environment variables, constants, network requests, and especially external dependencies like database. However, when unit testing Ray Actors, the only way for mocks to actually apply is to do the following:

  1. Run in local mode when initializing a local Ray cluster
  2. Create actors in a “JIT” fashion by forgoing the @ray.remote() decorator and instead turning your classes into actors at the last minute (ray.remote(**options)(ActorType).remote(**kwargs) )

Some questions about this

  • Is this correct about local mode? From what I can tell, when not running in local mode the Actors are actually run in another process after being pickled and sent via the GCS to the worker. Pickling and running remotely in this fashion avoids picking up any of the mocks/global fixtures that one generally relies on when unit testing
  • If this is the case, and given that local mode appears to be widely used for unit testing purposes (based on Ray github issues that mention it), why is local mode being deprecated? It seems like the only reasonable way to actually test a sufficiently complex system where mocks are required and especially if someone is trying to integrate actors into an existing complex codebase
  • Does the Ray team (or anyone else here who has built a sufficiently complex system with actors) have any other tips and tricks for unit testing and specifically mocking subcomponents of actors? We would love to run in non-local mode to better test the full system and have access to functionality like killing actors, the Ray state APIs, etc, but it seems impossible without just writing full integration tests that forgo the mocks
1 Like
3

Hi @jhallard

This is a great question. The problem with the local mode is that we are understaffed to maintain it, today it missing a few critical features (such as runtime-env, or features like iterators).

The best practice for the Ray team adopted today, is actually running integration tests where it starts multiple nodes on the same cluster without mocking. I guess you are already doing that today, but (ray/test_basic.py at master · ray-project/ray · GitHub) could be a starting point.