Also log experiment folder in mlflow

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

Tune does an excellent job of logging config and metrics to mlflow. After several experiments, it is not easy to connect which folder in local_dir corresponds to which trial in mlflow. In mlflow, it logs the trial_name tag which are of the form MyTrainer_3d319_00002, MyTrainer_3d319_00001, etc. In local_dir folders are named differently, for example, MyTrainer_2022-03-12_08-04-27, MyTrainer_2022-03-12_08-27-22. This get even more challenging when we use the same experiment name and local_dir for multiple hyperparameter explorations as each exploration leads to multiple line items in mlflow.

So I was thinking to manually log the local directory where all trials are saved as a tag to mlflow. How can I achieve this in Trainable.setup function? Or is there a better way to solve my problem?

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.
2

Hey @Nitin_Pasumarthy, can you provide a simple repro for this?

3 
exp_name = "MyFirstExp"

tune.run(
    MyTrainer,
    resources_per_trial={"gpu": NUM_GPUS_PER_POD, "cpu": NUM_CPUS_PER_POD},
    num_samples=12,
    local_dir="./ray_results",
    config={
        ... # some config
    },
    callbacks=[
        MLflowLoggerCallback(
            tracking_uri="...",
            experiment_name=exp_name,
            save_artifact=True
        )
    ]
)

The exp folder that gets created under ./ray_results is different from the tag that is auto added to mlflow. This makes it challenging to map which local folder corresponds to which run on mlflow