Can Ray cluster be restarted during ray tune

1

I’m trying to use Ray tune to tune a heavy job’s hyper parameter, I want ray cluster to be re-started each trial is early stopped, or somehow bring the cluster into a clean state before trying another parameter set.

Any ideas?

2

does ray.shutdown() work for you?

3

not sure how to use it internally in ray tune

4

does

tuner1 = Tuner(...)
result1 = tuner1.fit()
ray.shutdown()
tuner2 = Tuner(...)
result2 = tuner2.fit()

work for you?
I don’t think you should call it internally in ray tune.

5

oh, actually, what I really want is have a clean state during each trail with the fit. My application is quite complex, I don’t think each trial can be stopped early and hand over a clean ray cluster for next trial.

6

could you elaborate more? Some pseudo code could be helpful.

7

Example copied from Tune: Scalable Hyperparameter Tuning — Ray 2.2.0

from ray import tune

# 1. Define an objective function.
def objective(config):
    score = config["a"] ** 2 + config["b"]
    return {"score": score}


# 2. Define a search space.
search_space = {
    "a": tune.grid_search([0.001, 0.01, 0.1, 1.0]),
    "b": tune.choice([1, 2, 3]),
}

# 3. Start a Tune run and print the best result.
tuner = tune.Tuner(objective, param_space=search_space)
results = tuner.fit()
# what I need, is during tuner.fit, where parameter space is explored,
# I want to ensure ray is in a clean state between each trail of parameter combination
# e.g., after trying {"a":0.001,"b":1}, shutdown ray, and then restart for next round of parameter testing

print(results.get_best_result(metric="score", mode="min").config)
8

I guess my question is what kind of “non-clean state” you are observing with each trial?

9

I haven’t tried with ray tune. But my current job run, whenever ends, before shutting down/restarting ray cluster, it seems object store is not empty. I’ll try it with tune to see if my concern is valid.