Also log experiment folder in mlflow

Tune does an excellent job of logging config and metrics to mlflow. After several experiments, it is not easy to connect which folder in local_dir corresponds to which trial in mlflow. In mlflow, it logs the trial_name tag which are of the form MyTrainer_3d319_00002, MyTrainer_3d319_00001, etc. In local_dir folders are named differently, for example, MyTrainer_2022-03-12_08-04-27, MyTrainer_2022-03-12_08-27-22. This get even more challenging when we use the same experiment name and local_dir for multiple hyperparameter explorations as each exploration leads to multiple line items in mlflow.

So I was thinking to manually log the local directory where all trials are saved as a tag to mlflow. How can I achieve this in Trainable.setup function? Or is there a better way to solve my problem?

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

Hey @Nitin_Pasumarthy, can you provide a simple repro for this?

exp_name = "MyFirstExp"
    resources_per_trial={"gpu": NUM_GPUS_PER_POD, "cpu": NUM_CPUS_PER_POD},
        ... # some config

The exp folder that gets created under ./ray_results is different from the tag that is auto added to mlflow. This makes it challenging to map which local folder corresponds to which run on mlflow