No progress when restoring previously saved tuner

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

The progress reporter tables are rendered correctly upon the first time the tuner got created and triggered by tuner.fit(). However, when running the following code in JupyterLab, the progress tables do not show up any more except the loggings, any idea?

tuner = tune.Tuner.restore(
    tuner_folder,
    trainable=objective,
    restart_errored=True,
    resume_unfinished=False,
)
tuner.fit()
2

@allenfrostline Are you able to reproduce the issue outside of Jupyter?

3

Not really as I’ve been only testing this on a Kaggle notebook. But it seems related to the use of JupyterNotebookReporter. The following is how I construct the tuner object and by commenting out the progress_reporter argument (and letting it default to whatever value’s set in source code), the tuner progress table shows up correctly after being restored:

# skipping definition of param_space, tuner_folder etc

os.environ['RAY_DEDUP_LOGS'] = '0'
os.environ['RAY_AIR_NEW_OUTPUT'] = '1'
os.environ['RAY_AIR_RICH_LAYOUT'] = '1'
os.environ['RAY_memory_monitor_refresh_ms'] = '0'

if os.path.exists(tuner_folder):
    logging.info(f'Loading tuner from {tuner_folder}')
    tuner = tune.Tuner.restore(
        tuner_folder,
        trainable=objective,
        restart_errored=True,
        resume_unfinished=True,
    )
    results = tuner.get_results()
    logging.info(f'In total {len(results)} results restored.')
    del results
    gc.collect()
else:
    scheduler = ASHAScheduler(
        grace_period=param_space['early_stopping_round'],
        reduction_factor=max_concurrent_trials,
        max_t=num_boost_round,
    )
    tune_config = tune.TuneConfig(
        mode='min',
        metric='mae',
        reuse_actors=False,
        scheduler=scheduler,
        num_samples=num_samples,
        search_alg=OptunaSearch(),
        max_concurrent_trials=max_concurrent_trials,
    )
    run_config = train.RunConfig(
        name='lgb',
        storage_path=ray_folder,
        checkpoint_config=train.CheckpointConfig(
            num_to_keep=5,
            checkpoint_score_order='min',
            checkpoint_score_attribute='mae',
        ),
#         progress_reporter=tune.JupyterNotebookReporter(
#             max_progress_rows=num_samples,
#             sort_by_metric=True,
#         ),
    )
    tuner = tune.Tuner(
        objective,
        param_space=param_space,
        tune_config=tune_config,
        run_config=run_config
    )

results = tuner.fit()

That being said, I wonder what progress_reporter I should use instead of a JupyterNotebookReporter if I need the sorting etc features?