, and does not work with ray.train specifically xgboost_ray? how to report custom metrics to the SearchGenerator?

I am getting

ValueError: Trial returned a result which did not include the specified metric(s) [‘val_ks’, ‘ks_diff’]thatSearchGeneratorexpects. Make sure your calls include the metric, or set the TUNE_DISABLE_STRICT_METRIC_CHECKING environment variable to 1.

how can I report custom metrics from train_breast_cancer function in xgboost_ray/ at master · ray-project/xgboost_ray · GitHub ? I have tried, and nothing seems to work.

Hi @Sanjay_Chouhan,

the metrics should be generated by XGBoost which will then automatically report them to Ray Tune.

You can do this using a customized metric function that you pass to xgboost_ray.train().

Here are the relevant docs for XGBoost: Custom Objective and Evaluation Metric — xgboost 1.7.3 documentation

And it could look like this

def ks(predt: np.ndarray, dtrain: xgb.DMatrix) -> Tuple[str, float]:
    # ...
    val_ks = ...
    return 'val_ks', val_ks

def train_breast_cancer(config, ray_params):
    # ...

    bst = train(
        evals=[(test_set, "eval")],