Pytests repeating logs

We use in our code to start Tune experiments. When running our pytest tests on our code with 1.3, the logs seem to be repeating multiple times.

To be more specific, when the tests start running, the first test displays the logs correctly, once. The second test displays its results twice, the third thrice, and so on. Each test has its own ray.init with ignore_reinit_error set to True, and a call. Also, the pid eventually gets set to (pid=None).


Not really sure where to get started debugging this. Any help would be appreciated!

I’ve seen this before… how are you writing your tests?

Could you also run the Ray Tune pytest tests (that are included as part of the repo) to see if you see the same behavior?

We have a function that calls both ray.init and in the same code block and accepts a configuration. The tests, in a separate folder, call this function multiple times in a row with different configurations to test different models.

I also ran the Ray Tune tests with the same setup, and the log repeating issue crops up there too. It’s not immediately obvious because most training functions in the Ray Tune pytests don’t contain print statements, so explicitly adding them allowed this behaviour to be clearly seen. Especially in, which has a long chain of calls.

For some context, I’m trying these tests on a docker container launched by Ray, which means that there’s already a Ray cluster up and running in the background. However, just to make sure that this wasn’t causing the issue, I used ray.init with both address=“auto” and without it – the issue remained in both cases.