Can SparkXGBoost tune with Ray tune?

Ray Tune
1

I use Spark XGBoost in Databricks but I try to combine with ray tune but it doesn’t work
Here is my code

def train_fn(config):

    train_df = spark.table("Train_df_ray")
    val_df = spark.table("Val_df_ray")

    feature_cols = [c for c in train_df.columns if c not in ("ID", "PERIOD", "Label")]

    assembler = VectorAssembler(
        inputCols=feature_cols,
        outputCol="features",
        handleInvalid="keep"
    )

    xgb = SparkXGBClassifier( # Correct class name
        features_col="features",
        label_col="Label",
        learning_rate=config["learning_rate"],
        max_depth=config["max_depth"],
        num_round=config["num_round"],
        eval_metric="logloss",
        seed=42,
        num_workers=4,
        missing=float("nan"),
    )

    pipeline = Pipeline(stages=[assembler, xgb])
    model = pipeline.fit(train_df)

    preds = model.transform(val_df)

    auc_eval = BinaryClassificationEvaluator(
        labelCol="TIKTOK_LABEL",
        rawPredictionCol="probability",
        metricName="areaUnderROC"
    )
    auc = auc_eval.evaluate(preds)

    session.report({"auc": auc})

and got error
It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063.