Hi, I am using a KubeRay cluster that auto scales the number of worker pods. If I start a job that increases the number of workers, but the pods is queued (e.g. volcaono queue), it is moved to the pending state. The ray stop command doesnt stop the job and waits for the worker to be started. I expected it to a) list the job as stopped (e.g. in the dashboard) and b) remove the auto scaled worker pod from the volcano queue.
ray job stop 09e5f53e0dd8386a
Job submission server address: <http://my-ingress-node>
Attempting to stop job '09e5f53e0dd8386a'
Waiting for job '09e5f53e0dd8386a' to exit (disable with --no-wait):
Job has not exited yet. Status: PENDING