Exiting before cleanup is executed

When migrating from Ray 1.0.1 to Ray 1.1.0 or 1.2.0, I found that the program exits as soon as all the iterations are done in tune.run. The execution of the cleanup function is not fully completed.

I’ve actually faced this issue before in a previous version of Ray and @rliaw implemented a solution to fix it.

Here’s the issue I previously raised:

And here’s the PR that successfully fixed it:

In 1.2.0, I tried adding a time.sleep call after tune.run and that helped the cleanup function complete execution.

uh oh… this seems odd. Could you post an issue on Github and tag me?

Of course, I’ll do that