Ray Tune with Spark ML models

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

How severe does this issue affect your experience of using Ray?

  • None: Just asking a question out of curiosity

Hello. I’m new to Ray.
Is it possible to use Ray Tune with Spark ML models (e.g. DecisionTreeClassifier ) or SynapseML? Or better to use python version of LightGBM or sklearn DecisionTreeClassifier with Ray Tune?

2

Hi @Anatolii, sure, Ray Tune is agnostic to the type of trainable as long as it’s in Python. For instance, for sklearn you can use something like

def train(config):
    # ...
    model = sklearn.tree.DecisionTreeClassifier(criterion=..., splitter=..., max_depth=...)
    model = model.fit(data, labels)
    # ...
    preds = model.predict(test_data)
    tune.report(accuracy=accuracy_score(test_labels, preds))
1 Like
3

@kai, thanks for your answer. Does this mean that Ray Tune will work with the pyspark model?? Pyspark it is python + spark.

4

Sure, there shouldn’t be a problem with that - just wrap it in a function that accepts a config argument (see above) and train it there. If you have an example function, feel free to post it here and we can take a look together.

1 Like