Suppress "Warning: The following resource request cannot be scheduled right now"

Ray Clusters
1

I work at Amazon Machine Learning. I have multiple people using a single Ray cluster (with many nodes on AWS EC2). We’re using Ray to keep the hardware utilization high, by effectively using it as a FCFS queue (First Come, First Served) for submitted jobs.

When multiple people submit jobs to the cluster demanding more resources than available, Ray queues the requests as expected. However whenever the cluster is occupied and there are still pending jobs, all users get a warning: “Warning: The following resource request cannot be scheduled right now”.

This is annoying since the warning message repeats periodically. I would like to suppress the message (ideally, I would like it to only show up for the user who is submitting the job, but even suppressing for all users is acceptable).

How do I configure the Ray cluster to suppress this message? I am just running ray start --head or ray start --addressfrom each of the EC2 machines.

Example scenario: suppose the following submissions happen in order:

  1. Alice submits job(s) requiring 30% of cluster-resources
  2. Bob submits job(s) requiring 50% resources
  3. Carol submits job(s) requiring 60% of the resources.
  4. Dan submits job(s) requiring 10% of the resources.

In this case, what happens is:

  1. Alice’s jobs start
  2. Bob’s jobs start
  3. Some of Carol’s jobs start, while some are pending. However: Alice, Bob and Carol all get messages saying “Warning: The following resource request cannot be scheduled right now…”, which is annoying for Alice and Bob because both of their jobs are actually running.
  4. Dan’s jobs are pending.

I would ideally like to avoid the warning message here for all client connections, it should only be shown for Carol and Dan since they have pending jobs. If Alice/Bob sees the message “Warning: The following resource request cannot be scheduled right now”, they might decide to kill and restart their jobs, only for it to go back to the end of the queue (after Dan’s!). This is frustrating for a user.

I want to either:

  1. (Ideal UX) show the warning message only to client connections which actually have pending jobs
  2. (Acceptable UX) not show any warning messages to anyone.

How do I do this? Thanks.