Logs and results are cleared when the program ends

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

When setting a custom path for local_dir in RunConfig(), the logs and result records are cleared after the tuner.fit() run finishes.
If I don’t set local_dir , the logs and result records can be correctly saved when the program ends.
Sample code:

from ray import train, tune
from ray.air.config import RunConfig
from ray.tune.schedulers import ASHAScheduler


def train_fun(config):
    for epoch in range(100):
        acc = config["lr"] * config["batch_size"]
        train.report({"lr": config["lr"], "accuracy": acc})
    print("Finished Training")


if __name__ == "__main__":
    metric_str = "accuracy"
    config = {
        "lr": tune.loguniform(1e-4, 1e-1),
        "batch_size": tune.choice([2, 4, 8, 16]),
    }

    max_t = 100
    scheduler = ASHAScheduler(
        metric=metric_str,
        mode="max",
        max_t=max_t,
        grace_period=100,
        reduction_factor=2,
    )

    tuner = tune.Tuner(
        tune.with_resources(
            tune.with_parameters(train_fun),
            resources={"cpu": 2},
        ),
        tune_config=tune.TuneConfig(
            num_samples=2,
            scheduler=scheduler,
        ),
        param_space=config,
        run_config=RunConfig(
            name="ray_test",
            # local_dir="E:/temp_file/",
        ),
    )
    results = tuner.fit()
    # best_result = results.get_best_result(metric_str, "max")
    # print("Best trial config: {}".format(best_result.config))

I’m using ray 2.7.1 and python 3.10.11 on Windows 10 22H2.

2

Hey @congwen,

Do you mean the files are no longer found when they are expected under E:/temp_file/...? Ray Train/Tune doesn’t do any file deletion for the logfiles containing results – is it possible that this temp dir drive is automatically cleaning files up?

I will also make a disclaimer that Ray Train/Tune is not tested very comprehensively on windows – running on a devbox / linux subsystem may be a better alternative.

3

Thanks for your reply.
The files still exist under E:/temp_file/..., but their contents are cleared when the program ends. The contents are saved when the program is not finished running, but at the end of the run, the contents are suddenly purged. If I use the default local_dir, the contents in the files are preserved and not cleared.

4

@congwen Yeah, Ray Train/Tune don’t do anything special at the end. Could you try setting this to a different directory/drive?