When using Ray Tune and the Wandb integration I have difficulty stopping experiments, especially if “max_failures” for an experiment is not 0. If I’m running multiple experiments concurrently with tune I cannot use the “stop” button in the Wandb UI. If I press “stop” the one experiment will stop and then get restarted by tune and keep logging to Wandb but I will no longer be able to stop it. Ctr-C will also stop being responsive and I’ll have to kill the terminal. Ideally the stop button’s actions can be intercepted and the Wandb Callback or Mixin integration would then set the experiment to over. Is this something that is possible? It would also be nice to allow manual early stopping of bad experiments.
Great question! I think this is an extra integration point we need to do with wandb. I’ll reach out to them to ask for tips.
Cool, let me know what they say. I’d be happy to help and I could add it directly into my code too.