Tune job doesn't end (hanging) when run in localmode=True and v2.0.0dev

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

When running a tune job with ray v 2.0.0dev (Py3.8 on Windows), it never ends. I can see in the output, that it finished all iterations, but it never exits. When I run in debug mode (in IDE), and pause at the end (after all iterations finished), the current code execution is here: ray/import_thread.py at f60bd3afeefc78e30bd625e8ce98e064acd37b1f · ray-project/ray · GitHub
This problem doesn’t happen with ray v 1.2.0 !!

What can be the problem? Is it a known problem?

P.S It may be related to : Ray Tune training hangs

2

Hi @diman82. Thanks for asking. @kai can maybe help you.
If you do not get a response please let me know!

3

Hi @diman82, this is probably because we enabled placement group support per default, but placement groups are currently not supported in local model. See also this issue:

To fix this problem for the short term, please set the TUNE_PLACEMENT_GROUP_AUTO_DISABLED environment variable to 1, e.g. like this:

os.environ["TUNE_PLACEMENT_GROUP_AUTO_DISABLED"] = "1"

It might make sense for us to disable placement groups automatically when running in local mode until this is fixed. I’ll open an issue to track this.

4

@kai Thanks a lot, this workaround worked.

5

Off-topic question: How do you mark answers as solutions here? I couldn’t find this option in my questions. Or is it restricted to staff to mark answers as solutions?

6

@stefanbschneider Nope, I marked it by myself.
Maybe you need to get a specific badge for it…