Reproducibility issue

Ray Libraries (Data, Train, Tune, Serve)
1

I have troubles making my ray tune results reproducible. I went through the FAQ, basically all the google results that come up regarding this issue, but the results remain stochastic.

My code looks as follows:

# trainable
def trainable(config, data):
    
    # make it reproducible
    np.random.seed(config["seed"])
    seed(config["seed"])
    tf.random.set_seed(config["seed"])
    
    # initialize model
    model = tf.keras.Sequential([
        tf.keras.layers.Dropout(0.1, input_shape=(471,), seed=config['seed']),
        tf.keras.layers.Dense(32, activation=tf.keras.layers.LeakyReLU(alpha=0.1), input_shape=(471,), kernel_regularizer = regularizers.L1L2(l1 = 0.1, l2 = 0)), #manually change input shape
        tf.keras.layers.Dense(16, activation=tf.keras.layers.LeakyReLU(alpha=0.1), kernel_regularizer = regularizers.L1L2(l1 = 0.1, l2 = 0)),
        tf.keras.layers.Dense(8, activation=tf.keras.layers.LeakyReLU(alpha=0.1), kernel_regularizer = regularizers.L1L2(l1 = 0.1, l2 = 0)),
        tf.keras.layers.Dense(4, activation=tf.keras.layers.LeakyReLU(alpha=0.1), kernel_regularizer = regularizers.L1L2(l1 = 0.1, l2 = 0)),
        tf.keras.layers.Dense(1, activation=tf.keras.layers.LeakyReLU(alpha=0.1), kernel_regularizer = regularizers.L1L2(l1 = 0.1, l2 = 0))
    ])
    
    # compile model
    model.compile(
        optimizer=tf.keras.optimizers.Adam(learning_rate=config["learning_rate"]),
        loss='mse',
        metrics=[RSquare(), 'mse'],
    )

    # fit model
    model.fit(
        data[0], data[1],
        validation_data=(data[2], data[3]),
        epochs=3,
        batch_size=128,
        verbose=0,
        callbacks = [ReportCheckpointCallback(
            metrics={'val_mse':'val_mse'},
            checkpoint_on = 'epoch_end')]
    )

# define search space
space = {'learning_rate': tune.choice([0.1]), 'seed': 1234}

# tuning configuration
analysis = tune.Tuner( 
    tune.with_parameters(
        trainable,
        data=[X_train, Y_train, X_val, Y_val]
        ),
    param_space=space,
    tune_config=tune.TuneConfig(
        metric="val_mse",
        mode="min",
        num_samples=1,
        ),
    run_config=air.RunConfig(
        local_dir = path,
        name= name,
    ),
)

So setting seeds in the trainable function does not work. How do I resolve this issue?

2

Hi @F_S,

Just for reference, is this the guide you’re referring to? Ray Tune FAQ — Ray 2.3.0

Looks like you’re also following the recommendation from tf.keras.utils.set_random_seed  |  TensorFlow v2.11.0

A few questions: how is your training data being loaded? Is there any random shuffling happening before passing it into the training?

3

Hi there. Yes, that is the guide I am referring to.

Yes, the trainable itself is reproducible. I can easily verify that by just running the model separately.

My data is not randomly shuffled etc. It’s a simple dataframe that does not change at all. I found the following, very old, statement in regards to reproducability when using the functional AP in this Stack Overflow thread:

(This answer focuses on class API and ray version 0.8.7. Function API does not support reproducibility due to implementation specifics)

Does this still hold? Does this issue somehow relate to how tasks are distributed to workers? Anyways - looking forward to your reply!

4

@F_S, This thread may be a bit outdated – I don’t think Function vs. Class Trainable should affect reproducibility. Can you try using the tf.keras.utils.set_random_seed instead? Looks like they set an extra seed generator in the keras backend.

Keras model.fit shuffles by default, so you can also try turning that off with model.fit(shuffle=False).