We use tune.run in our code to start Tune experiments. When running our pytest tests on our code with 1.3, the logs seem to be repeating multiple times.
To be more specific, when the tests start running, the first test displays the logs correctly, once. The second test displays its results twice, the third thrice, and so on. Each test has its own ray.init with ignore_reinit_error set to True, and a tune.run call. Also, the pid eventually gets set to (pid=None).
Not really sure where to get started debugging this. Any help would be appreciated!
We have a function that calls both ray.init and tune.run in the same code block and accepts a configuration. The tests, in a separate folder, call this function multiple times in a row with different configurations to test different models.
I also ran the Ray Tune tests with the same setup, and the log repeating issue crops up there too. It’s not immediately obvious because most training functions in the Ray Tune pytests don’t contain print statements, so explicitly adding them allowed this behaviour to be clearly seen. Especially in test_api.py, which has a long chain of tune.run calls.
For some context, I’m trying these tests on a docker container launched by Ray, which means that there’s already a Ray cluster up and running in the background. However, just to make sure that this wasn’t causing the issue, I used ray.init with both address=“auto” and without it – the issue remained in both cases.
