Resource cannot be scheduled

Ray Clusters
1

How severe does this issue affect your experience of using Ray?

  • Low: It annoys or frustrates me for a moment.

Hi, everyone, thanks to the ray team, I am one step closer to the completion of the enhancements to my project, and here’s the situation.
I submit a task to cluster using a statement like this:

ray job submit --working-dir your_working_directory -- python script.py

and when I have a large task I get a WARNING, like this picture:

QQ截图20230111165040
QQ截图202301111650401585×698 50.8 KB

I have two questions:

  1. Following the hints in the documentation , I would also like to put the cpu to zero provided by head, but I do not know how to set.

  2. When I submit a task using ray submit, if I want to interrupt the execution of the task in the middle, how do I do it is the most appropriate, I currently use ctrl+z in the ssh control line to force stop task submission, but this will encounter a problem: is when this way to stop task submission, my ssh console (shell/bash) will become abnormally I used htop tool to check the cpu and memory usage, it is not high (about 20 percent), but the ssh operation input command becomes abnormally stuck, I input ray stop to terminate ray in a very stuck situation, after that it is immediately back to normal, no more stuck.

My head machine is a vm with 2 cpu and 8gb RAM, and the OS is ubuntu 22 LTS.

2
  1. Following the hints in the documentation , I would also like to put the cpu to zero provided by head, but I do not know how to set.

This should be set when you start a cluster. When you start the head node, you should have CLI command like ray start --head. You should add --num-cpus=0 to this command.

  1. When I submit a task using ray submit, if I want to interrupt the execution of the task in the middle, how do I do it is the most appropriate, I currently use ctrl+z in the ssh control line to force stop task submission, but this will encounter a problem: is when this way to stop task submission, my ssh console (shell/bash) will become abnormally I used htop tool to check the cpu and memory usage, it is not high (about 20 percent), but the ssh operation input command becomes abnormally stuck, I input ray stop to terminate ray in a very stuck situation, after that it is immediately back to normal, no more stuck.

We recommend you to use the job submission API and ray job stop API for this. Quickstart using the Ray Jobs CLI — Ray 3.0.0.dev0

1 Like
3

Thank you for your very accurate answer! Thank you for your time!

4

HI, about the second question reasons for getting stuck now I know why:
I should not use Ctrl+c to abort the process instead should use Ctrl+d Avoid Getting Stuck.