The error in the referenced discussion—where Ray randomly halts with no error or warning—can occur due to silent task failures, resource exhaustion (like running out of memory or file descriptors), or hitting the default task retry limit. Sometimes, Ray tasks or actors fail but the driver doesn’t receive a clear error, especially if max_retries is set to a nonzero value and all retries are exhausted, causing the job to silently stop progressing. Setting RAY_TASK_MAX_RETRIES=0 can help surface errors immediately instead of retrying silently.
Additionally, issues like OOM (out-of-memory), exceeding open file limits, or system-level resource constraints can cause Ray to kill processes without clear Python exceptions, resulting in a “Killed” message or a silent halt. Checking system logs, Ray driver logs, and resource usage is recommended for further diagnosis. See Random halt and no error/warnings for more context.
Would you like a step-by-step breakdown of how to debug or prevent this?
Sources:
Hint: Mention @RunLLM in the post for followups.