When is session.report() callable?

1

There are quite a few places in the doc about using session.report() in callback or tune (e.g., Running Tune experiments with SigOpt — Ray 2.1.0) which makes me feel that it should be usable whenever in a callback function. However, when trying to inject my own session.report() to log result, ray will fail saying session is not callable.

class MLflowLoggerCallback2(MLflowLoggerCallback):
    def log_trial_result(self, iteration: int, trial: "Trial", result: Dict):
        step = result.get(TIMESTEPS_TOTAL) or result[TRAINING_ITERATION]
        run_id = self._trial_runs[trial]
        self.mlflow_util.log_metrics(run_id=run_id, metrics_to_log=result, step=step)
        self.mlflow_util.log_metrics(run_id=run_id, metrics_to_log=result['custom_metrics'], step=step)
        from ray.air import session
        session.report(result)

The code above was an example where mlflow_util.log_metrics() worked but session.report() failes:

Traceback (most recent call last):
  File "/Users/zhan/Applications/anaconda3/envs/python38/lib/python3.8/site-packages/ray/tune/tuner.py", line 234, in fit
    return self._local_tuner.fit()
  File "/Users/zhan/Applications/anaconda3/envs/python38/lib/python3.8/site-packages/ray/tune/impl/tuner_internal.py", line 283, in fit
    analysis = self._fit_internal(trainable, param_space)
  File "/Users/zhan/Applications/anaconda3/envs/python38/lib/python3.8/site-packages/ray/tune/impl/tuner_internal.py", line 380, in _fit_internal
    analysis = run(
  File "/Users/zhan/Applications/anaconda3/envs/python38/lib/python3.8/site-packages/ray/tune/tune.py", line 722, in run
    runner.step()
  File "/Users/zhan/Applications/anaconda3/envs/python38/lib/python3.8/site-packages/ray/tune/execution/trial_runner.py", line 872, in step
    self._wait_and_handle_event(next_trial)
  File "/Users/zhan/Applications/anaconda3/envs/python38/lib/python3.8/site-packages/ray/tune/execution/trial_runner.py", line 851, in _wait_and_handle_event
    raise TuneError(traceback.format_exc())
ray.tune.error.TuneError: Traceback (most recent call last):
  File "/Users/zhan/Applications/anaconda3/envs/python38/lib/python3.8/site-packages/ray/tune/execution/trial_runner.py", line 839, in _wait_and_handle_event
    self._on_training_result(
  File "/Users/zhan/Applications/anaconda3/envs/python38/lib/python3.8/site-packages/ray/tune/execution/trial_runner.py", line 964, in _on_training_result
    self._process_trial_results(trial, result)
  File "/Users/zhan/Applications/anaconda3/envs/python38/lib/python3.8/site-packages/ray/tune/execution/trial_runner.py", line 1048, in _process_trial_results
    decision = self._process_trial_result(trial, result)
  File "/Users/zhan/Applications/anaconda3/envs/python38/lib/python3.8/site-packages/ray/tune/execution/trial_runner.py", line 1103, in _process_trial_result
    self._callbacks.on_trial_result(
  File "/Users/zhan/Applications/anaconda3/envs/python38/lib/python3.8/site-packages/ray/tune/callback.py", line 329, in on_trial_result
    callback.on_trial_result(**info)
  File "/Users/zhan/Applications/anaconda3/envs/python38/lib/python3.8/site-packages/ray/tune/logger/logger.py", line 130, in on_trial_result
    self.log_trial_result(iteration, trial, result)
  File "play_ray.py", line 353, in log_trial_result
    session.report(res)
  File "/Users/zhan/Applications/anaconda3/envs/python38/lib/python3.8/site-packages/ray/air/session.py", line 61, in report
    _get_session().report(metrics, checkpoint=checkpoint)
AttributeError: 'NoneType' object has no attribute 'report'

What is the scope that session.report() is usable?

2

We should probably improve the doc since it’s causing confusion.

But for a short answer here, session can be used in function trainable or train_loop_per_worker for DataParallelTrainer. In either case, it’s only used in a function that’s running in a remote actor.

LoggerCallback is different - it’s called by the Tune driver process. Hence is outside of the scope of using Session.

3

Ok that explains. Yeah I saw that session.report was supposed to be in a function serializable by ray. The example code was using tune.report() rather than session.report() though.

image
image1920×1461 158 KB

4

Got it, I will update that doc.