Customized progress_reporter

1

Hello,

I am using Ray to tune hyperparameters. The tuning progress is reported every 30 seconds by default. I want to change the update interval to 60 seconds or longer. How can I realize that?

I have tried all the examples given in Tune Console Output (Reporters) — Ray 2.43.0

However, none of them work. Has anyone met the same problem?

Here is my code:

class ExperimentTerminationReporter(CLIReporter):
    def should_report(self, trials, done=False):
        """Reports only on experiment termination."""
        return done

class TrialTerminationReporter(CLIReporter):
    def __init__(self):
        super(TrialTerminationReporter, self).__init__()
        self.num_terminated = 0

    def should_report(self, trials, done=False):
        """Reports only on trial termination events."""
        old_num_terminated = self.num_terminated
        self.num_terminated = len([t for t in trials if t.status == Trial.TERMINATED])
        return self.num_terminated > old_num_terminated

search_space = {
        't': tune.uniform(50, 150),
        'map_weight1': tune.uniform(0, 1),
        'map_weight2': tune.uniform(0, 1),
        'map_weight3': tune.uniform(0, 1),
        'map_weight4': tune.uniform(0, 1),
        'alpha': tune.uniform(1, 10),
        'beta': tune.uniform(0, 1),
}
os.environ["RAY_CHDIR_TO_TRIAL_DIR"] = "0"
tuner = tune.Tuner(
        tune.with_resources(tune.with_parameters(stable_diffusion_func, args=args), resources={"cpu": args.num_cpu, "gpu": args.num_gpu}),
        param_space=search_space,
        tune_config=tune.TuneConfig(search_alg=OptunaSearch(), metric="f1_auc", mode="max", num_samples=args.num_runs),
        run_config=RunConfig(progress_reporter=ExperimentTerminationReporter()),
        # run_config=RunConfig(progress_reporter=reporter),
)
results = tuner.fit()
2

Per ray.tune.CLIReporter — Ray 2.37.0 the param max_report_frequency isn’t working for you @yj373 ?

3

No it doesn’t. The tuning progress is still reported every 30 seconds. Here is the code script

reporter = CLIReporter()
reporter.max_report_frequency = 60
tuner = tune.Tuner(
            tune.with_resources(tune.with_parameters(stable_cascade_func, args=args), resources={"cpu": args.num_cpu, "gpu": args.num_gpu}),
            param_space=search_space,
            tune_config=tune.TuneConfig(search_alg=OptunaSearch(), metric="f1_auc", mode="max", num_samples=args.num_runs),
            run_config=RunConfig(progress_reporter=reporter),
)
4

I also have the same situation on 2.38. I’ve reduced and increased max_report_frequency, however the report constantly is made every 30 seconds.

5

I think this is related to

and

Crawling trough the code I think the CLIReporter is not used anymore in favor of the new reporters that came with the new RAY_AIR_NEW_OUTPUT.