How severe does this issue affect your experience of using Ray?
- Medium: It contributes to significant difficulty to complete my task, but I can work around it.
I have been using ray in a cloud-based cluster for some batch tasks (run thousands of homogeneous tasks, call the same remote function many times with different input). Everything worked great, until recently I start to scale the cluster up further (currently about 300 cores across 20 workers).
Occasionally, some workers stop running new tasks and their CPU utilization goes to 0%. In the worst case, the scheduler stops entirely and all workers become idle, with all remaining tasks stuck in the PENDING_NODE_ASSIGNMENT state.
The issue persists even if I manually kill all worker servers and add fresh new workers to the cluster. The only way to restore to a healthy state is to restart the head node, which causes all the pending task to be lost.
The cluster currently runs ray 2.3.0. I have checked head node logs (debug_state.txt, debug_state_gcs.txt, gcs_server.err, monitor.err, etc.) and they contain no apparent errors. (Previously, gcs_server.err reported “too many open files”; this has been addressed.)
Any pointer on how to debug further will be extremely appreciated! Thanks in advance.
===
Screenshot: job dashboard showing tasks stuck in pending.
Below is the diagnostics output from ray status, ray summary tasks, and ray summary objects.
======== Autoscaler status: 2023-03-20 16:17:02.803611 ========
Node status
---------------------------------------------------------------
Healthy:
1 node_37e23132b9b4e058bc249fd351e386c1db1a1bebdd95791f9042166d
1 node_bf1e7380dc822cfb3be95c9b60deafeb19571b1a672c7f93b544a05c
1 node_6f8998a6993ee6de627e147f610d5194ae67175e75fd8fe8a87172e4
Pending:
(no pending nodes)
Recent failures:
(no failures)
Resources
---------------------------------------------------------------
Usage:
0.0/164.0 CPU
0.0/800.0 Memory
0.00/788.691 GiB memory
0.00/4.727 GiB object_store_memory
Demands:
{'CPU': 2.0, 'Memory': 10.0}: 2234+ pending tasks/actors
======== Tasks Summary: 2023-03-20 16:17:07.231791 ========
Stats:
------------------------------------
total_actor_scheduled: 0
total_actor_tasks: 0
total_tasks: 4748
Table (group by func_name):
------------------------------------
FUNC_OR_CLASS_NAME STATE_COUNTS TYPE
0 run_on_docker FAILED: 725 NORMAL_TASK
FINISHED: 1612
PENDING_NODE_ASSIGNMENT: 2244
RUNNING: 160
1 check_summary FINISHED: 5 NORMAL_TASK
RUNNING: 1
2 check_uptime PENDING_NODE_ASSIGNMENT: 1 NORMAL_TASK
======== Object Summary: 2023-03-20 16:17:08.433759 ========
Stats:
------------------------------------
callsite_enabled: false
total_objects: 2987
total_size_mb: 950.1486234664917
Table (group by callsite)
------------------------------------
disabled
REF_TYPE_COUNTS TASK_STATE_COUNTS TOTAL_NUM_NODES TOTAL_NUM_WORKERS TOTAL_OBJECTS TOTAL_SIZE_MB
-- --------------------- ------------------------------------------- ----------------- ------------------- --------------- ---------------
0 LOCAL_REFERENCE: 2987 'Attempt #2: PENDING_NODE_ASSIGNMENT': 1764 1 2 2987 950.149
FINISHED: 581
PENDING_NODE_ASSIGNMENT: 481
SUBMITTED_TO_WORKER: 161
