Train.report, tune.report and session.report does not work with ray.train specifically xgboost_ray? how to report custom metrics to the SearchGenerator?

Ray Libraries (Data, Train, Tune, Serve) Ray Train
1

I am getting

ValueError: Trial returned a result which did not include the specified metric(s) [‘val_ks’, ‘ks_diff’]thatSearchGeneratorexpects. Make sure your calls totune.report() include the metric, or set the TUNE_DISABLE_STRICT_METRIC_CHECKING environment variable to 1.

how can I report custom metrics from train_breast_cancer function in xgboost_ray/simple_tune.py at master · ray-project/xgboost_ray · GitHub ? I have tried train.report, session.report and tune.report nothing seems to work.

1 Like
2

Hi @Sanjay_Chouhan,

the metrics should be generated by XGBoost which will then automatically report them to Ray Tune.

You can do this using a customized metric function that you pass to xgboost_ray.train().

Here are the relevant docs for XGBoost: Custom Objective and Evaluation Metric — xgboost 1.7.3 documentation

And it could look like this

def ks(predt: np.ndarray, dtrain: xgb.DMatrix) -> Tuple[str, float]:
    # ...
    val_ks = ...
    return 'val_ks', val_ks


def train_breast_cancer(config, ray_params):
    # ...

    bst = train(
        params=config,
        dtrain=train_set,
        evals=[(test_set, "eval")],
        evals_result=evals_result,
        custom_metric=ks,
        ray_params=ray_params,
        verbose_eval=False,
        num_boost_round=10)