Your issue is likely due to resource contention between your background executor and the Ray head node, especially around CPU scheduling and memory bandwidth. When a heavy background process runs, it can delay Ray’s internal heartbeats and dashboard updates, leading to “Worker heartbeat timeout” errors and dashboard sluggishness—even if the cluster isn’t fully loaded. This is a known problem: high disk or CPU I/O from other processes can cause Ray’s heartbeat or dashboard threads to be delayed, resulting in nodes being marked dead or the dashboard becoming unresponsive. Prioritizing Ray system processes (raylet, GCS server) using OS-level niceness or cgroup settings can help mitigate this by ensuring they get CPU time even under load. For example, you can use sudo renice -n -19 [raylet_pid] to increase the scheduling priority of Ray processes, as suggested in this Ray issue.
Additionally, for large or long-running clusters, it’s recommended to reserve extra resources for the head node and avoid running user workloads or heavy background jobs on it. Set num_cpus=0 for the head node to prevent Ray from scheduling tasks there, and consider running your background executor with a lower CPU priority (using nice or cpulimit) or pinning it to specific cores not used by Ray. This helps ensure Ray’s control plane and dashboard remain responsive and worker heartbeats are not missed due to resource starvation. See Ray best practices for large clusters and heartbeat timeout discussions for more details.
Would you like a step-by-step guide on how to set process priorities or isolate resources for your background executor and Ray head node?
Sources:
Hint: Mention @RunLLM in the post for followups.