[Jobs] A lot of jobs have taken over the CPU and have been PENDING for over 5 hours now

A lot of jobs have taken over the CPU and have been PENDING for over 5 hours now. At the same time I found a bug and know that they will not start. But they seem to be in an endless loop trying to restart.

ray job stop ray_xxxxxxxx  # not working
ray job delete ray_xxxxxxx  # not working

How to stop these jobs?
How to set timeout on jobs in PENDING state?

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

@Sihan_Wang can you take a look?

2 Likes