[Tune] ray tune with each trial running tf.distribute.experimental.MultiWorkerMirroredStrategy()

raytune_kuberay_user · August 18, 2022, 11:26pm

We want to use ray tune with each trial running tf.distribute.experimental.MultiWorkerMirroredStrategy()

We are using ray 1.12

I noticed that there are two ways to do this:

Use Trainer and then trainer.to_tune_trainable and pass it to tune.run()

ray-project/ray/blob/ray-1.12.0/python/ray/train/examples/tune_tensorflow_mnist_example.py

import argparse

import ray
from ray import tune
from ray.train import Trainer

from tensorflow_mnist_example import train_func


def tune_tensorflow_mnist(num_workers, num_samples):
    trainer = Trainer(backend="tensorflow", num_workers=num_workers)
    Trainable = trainer.to_tune_trainable(train_func)
    analysis = tune.run(
        Trainable,
        num_samples=num_samples,
        config={
            "lr": tune.loguniform(1e-4, 1e-1),
            "batch_size": tune.choice([32, 64, 128]),
            "epochs": 3,
        },

This file has been truncated. show original

Use DistributedTrainableCreator( and pass it to tune.run()
ray/tf_distributed_keras_example.py at ray-1.12.0 · ray-project/ray · GitHub

Which one do you recommend?
Will DistributedTrainableCreator be deprecated in the future say in ray 2.0?

rliaw · August 19, 2022, 8:26am

Both will be deprecated in 2.0.

The 2.0 usage will look something like: Ray AI Runtime (AIR) — Ray 2.0.0

For now, the Trainer workflow will probably be the closest thing.

Topic		Replies	Views
Example in ray.train for tensorflow distributed training?	1	6	November 24, 2024
Tuning a Keras model - no checkpoints saved Ray Tune	7	1462	March 1, 2023
Where do I find documentation on the tune.run method Ray Libraries (Data, Train, Tune, Serve)	3	2018	June 12, 2023
How to parallelize training multiple models Ray Tune	4	1073	April 5, 2021
What is the recommended and efficient way to make the best use of all ray features? Ray Libraries (Data, Train, Tune, Serve)	2	255	July 3, 2023

[Tune] ray tune with each trial running tf.distribute.experimental.MultiWorkerMirroredStrategy()

Related topics