How to resume experiments after time_budget_s expired

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

Hello, I’ve been trying to use time_budget_s to stop my experiments and I’d like to have the option to still resume them in future afterwards. However, it seems that when stoppers are triggered, all trials are set to TERMINATED, which means subsequent runs won’t do anything.

I believe this is the same issue as described by FarzanT in this github issue: [tune] resume is not working properly · Issue #12389 · ray-project/ray · GitHub which is said to be fixed, but alas it does not seem like it is.

For now, my dirty workaround is to implement a timer in a separate thread and send a signal to the process after the elapsed time. Not great, but it works.