[TUNE] Connecting to external cluster not showing JupyterNotebookReporter summary table

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

Hi everyone, this should be a straight forward question but when I connect to an external cluster via
ray.init(address=f"ray://{head_node_ip_address}:10001") and run a Tune job, I am not able to see the Jupyter notebook progress reporter.

My setup look something like this:

tuner = tune.Tuner(
    MyTrainable,
    tune_config=tune.TuneConfig(
        # configs here
    ),
    param_space=search_space,
    run_config = air.RunConfig(
        progress_reporter=JupyterNotebookReporter(overwrite=True),
        checkpoint_config=air.CheckpointConfig(
            checkpoint_at_end=False,
            checkpoint_frequency=0,
            num_to_keep=None,
        ),
        sync_config=sync_config,
        local_dir=log_dir,
        log_to_file=False,
        verbose=1,
    )
)

When I run .fit() I get this in the cells:

(TunerInternal pid=286391) <IPython.core.display.HTML object>
(TunerInternal pid=286391) <IPython.core.display.HTML object>
(TunerInternal pid=286391) <IPython.core.display.HTML object>
...
(TunerInternal pid=286391) <IPython.core.display.HTML object>
(TunerInternal pid=286391) <IPython.core.display.HTML object>
(TunerInternal pid=286391) <IPython.core.display.HTML object>

What I am expecting is this:

== Status ==
Current time: 2022-10-25 10:09:10 (running for 00:00:12.81)
...
Number of trials: 3/250 (1 PENDING, 2 RUNNING)
Trial Name status loc x y
TrainableFunc_f3139036 RUNNING 0.319237 0.319237
TrainableFunc_f3139036 RUNNING 0.44374 1.319237
TrainableFunc_f3139036 RUNNING 0.77797 2.319237
… … … … …
TrainableFunc_f3139036 RUNNING 0.77797 2.319237

What am I doing wrong here ?

Thanks!

2

Hey @max_ronda what Ray version are you using?

I believe this should be fixed in Ray 2.0 (in this PR: [Tune] Improve `JupyterNotebookReporter` by Yard1 · Pull Request #24444 · ray-project/ray · GitHub)

cc @Yard1

3

Hi @amogkam , I am using the latest, Ray 2.0. Also, when I don’t specify the progress reporter, it shows stats but not in table form, it just writes stats to the cell’s console, overpopulating it , and if running many trials, killing the browser. And this is when verbose=1. When I set verbose=2 or 3, it will write many more times.

Let me know if you have some other example I can try out !

Thanks!

4

Hey @max_ronda, thanks for bringing this to our attention. This is indeed a bug and we’ll fix it soon. Tracking here - [Tune] `Tuner` doesn't support Jupyter rich output · Issue #29863 · ray-project/ray · GitHub

2 Likes
5

Thanks @Yard1 ! Very desirable feature :slight_smile:

6

Hi @yard1 , were these changes added to newest release, 2.1 ?

7

Hey Max, they will be added to 2.2. You can also use the nightly version of Ray to take advantage of them right now!

1 Like
8

Thanks @Yard1 ! Much appreciated it ! I will give that a test :slight_smile:

9

Mind you, the fix was just merged so you may need to wait until tomorrow for the nightly version to include it!

1 Like