Test coverage for ray serve remote calls

Ray Libraries (Data, Train, Tune, Serve) Ray Serve
1

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

When writing tests I like to keep track of my code coverage using pytest-cov (which itself uses coverage). However, any code that gets executed behind a some_deployment.remote-call does not appear in my code coverage (even though I can guarantee it ran) which makes the coverage look atificially low (and dependending on the setup might lead to failed quality gates in my CI/CD-pipeline).

I tried manually starting the coverage as described here and I also tried to manually start the coverage in a worker setup function but none of this worked.

According to this issue one can get proper line coverage by setting local_mode=True. However, this does not work with async actors and also the local_mode is deprecated.

Is there any way to get proper line coverage for tests that do remote calls?

1 Like
2

There is also a github thread on this issue (sadly without a solution): [ray] How to get line test coverage for actor code? · Issue #9059 · ray-project/ray · GitHub

My latest attempt was to call coverage.process_startup during worker startup like so:

from collections.abc import Iterator

import pytest
import ray
import coverage

from ray.serve._private.client import ServeControllerClient
from ray.serve.context import _get_global_client



@pytest.fixture(scope="session")
def ray_serve_client() -> Iterator[ServeControllerClient | None]:
    ray.init(
        num_cpus=1,
        num_gpus=0,
        include_dashboard=False,
        runtime_env={
            "env_vars": {"COVERAGE_PROCESS_START": ".coveragerc"},
            "worker_process_setup_hook": coverage.process_startup,
        },
    )
    ray.serve.start(detached=True, http_options={"host": "0.0.0.0"})
    yield _get_global_client()

    ray.shutdown()

I can verify that coverage is actually properly started in the worker processes that are spawned during tests but the line coverage is still not correct.

Anyone got this to work yet?