# Pending tasks not starting up **URL:** \ **Category:** Kubernetes\ **Created:** [May 10, 2022, 8:33am UTC](https://discuss.ray.io/t/pending-tasks-not-starting-up/6090 "2022-05-10T08:33:18Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Filip\_K](https://avatars.discourse-cdn.com/v4/letter/f/7cd45c/32.png) [@Filip\_K](https://discuss.ray.io/u/Filip_K)\ **Post date:** [May 10, 2022, 8:33am UTC](https://discuss.ray.io/t/pending-tasks-not-starting-up/6090/1 "2022-05-10T08:33:18Z")
**How severe does this issue affect your experience of using Ray?** - Low: It annoys or frustrates me for a moment. Greetings! We sometimes (presumably when ray is under higher load) observe that our ray client creates a job but it never starts. We currently work around this by having timeout when waiting for job result. That naturally takes quite some time, as timeout needs to consider usual job duration as well. My 2 questions are: 1. Where to look for what is causing this problem? I checked ray-operator log (see below) but it didn’t help me. 2. Is there a direct way to query ray about jobs that are really running? Other than listing processes in underlying pods (which we do for our make-shift dashboard) or creating lockfile on shared storage from the job itself (which we’re considering to use). Thank you PS: [This is one complete status](https://drive.google.com/file/d/1w7En8d7CodjCoKun0VxIpzXu5mEWQX1l/view?usp=sharing) from our operator log where I see **10+ pending tasks/actors** but no pending nodes? Can you point us towards what might be wrong in our setup? ---
**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\ **Post date:** [May 12, 2022, 2:35am UTC](https://discuss.ray.io/t/pending-tasks-not-starting-up/6090/2 "2022-05-12T02:35:24Z")
The best bet now is to use `ray status`. We are aware of that this has limited functionality, and we are in the progress of supporting 2 from this REP. The project is in active progress; [State Observability API REP by rkooo567 · Pull Request #8 · ray-project/enhancements · GitHub](https://github.com/ray-project/enhancements/pull/8). For ```auto {'CPU': 1.0, 'worker1Gi': 1.0}: 10+ pending tasks/actors ``` For 1, I think there are 2 possibilities. 1. It is a bug from ray client. In this case, you can confirm it by not using ray client, but I am not sure if it is the option for you guys. 2. There’s a possibility none of node has {1 CPU + 1 worker1Gi} resources. For example, imagine 5 nodes A have {CPU: 1, worker1Gi: 1} and 5 nodes B that have {CPU: 1}. And if there are 5 tasks that use 1 CPU, it can use 1 CPU from A. In this case, the resources will be A: {CPU: 0, worker1Gi: 1} \* 5 B: {CPU: 1} \* 5 In this case although the cluster resources is {CPU: 10, worker1Gi: 5}, we cannot schedule 5 of {CPU: 1, worker1Gi: 1} tasks. We are also in progress exposing this information better from the same REP. But for now, you can verify it using `ray.state.state._available_resources_per_node()` (note that it is a private API, and it is encourage to use official APIs from the above REP once they are available). ---
**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\ **Post date:** [May 12, 2022, 3:18am UTC](https://discuss.ray.io/t/pending-tasks-not-starting-up/6090/3 "2022-05-12T03:18:15Z")
This part of the logs is concerning: `Cluster status: 23 nodes (2 failed to update)` If you’re using K8s, also take a look at [KubeRay](https://github.com/ray-project/kuberay) which works in more or less the same way, but is more stable. (The main thing KubeRay is missing at the moment is stable autoscaling support – that’s in progress.) ---
**Author:** ![Filip\_K](https://avatars.discourse-cdn.com/v4/letter/f/7cd45c/32.png) [@Filip\_K](https://discuss.ray.io/u/Filip_K)\ **Post date:** [May 12, 2022, 9:01am UTC](https://discuss.ray.io/t/pending-tasks-not-starting-up/6090/4 "2022-05-12T09:01:03Z")
Thanks for quick reply! `ray status` shows the same information as operator log status (part of it). Possibility of ray’s inner state observation would be something that would probably help us in this manner quite a lot. > 1. It is a bug from ray client. In this case, you can confirm it by not using ray client, but I am not sure if it is the option for you guys. We are using ray client. There are some other options to start tasks from python? > 1. There’s a possibility none of node has {1 CPU + 1 worker1Gi} resources… That’s when scaling should kick in, right? Ray operator should then launch new k8s pod with such resources (and underlying k8s node autoscaling would launch new node if there is pending pod). Or am I missing something? > But for now, you can verify it using `ray.state.state._available_resources_per_node()` (note that it is a private API, and it is encourage to use official APIs from the above REP once they are available). We tried and it resulted in following error: ```auto Traceback (most recent call last): File "", line 1, in File "/home/ray/anaconda3/lib/python3.9/site-packages/ray/state.py", line 678, in _available_resources_per_node self._check_connected() File "/home/ray/anaconda3/lib/python3.9/site-packages/ray/state.py", line 49, in _check_connected raise ray.exceptions.RaySystemError( ray.exceptions.RaySystemError: System error: Ray has not been started yet. You can start Ray with 'ray.init()'. ``` Even though `ray.init()` has been called and for example `ray.cluster_resources()` works ---
**Author:** ![Filip\_K](https://avatars.discourse-cdn.com/v4/letter/f/7cd45c/32.png) [@Filip\_K](https://discuss.ray.io/u/Filip_K)\ **Post date:** [May 12, 2022, 9:04am UTC](https://discuss.ray.io/t/pending-tasks-not-starting-up/6090/5 "2022-05-12T09:04:41Z")
We are using k8s and autoscaling is one of the main features that we use. Our workflow is that we run a lot of tasks in bursts couple times a day. ---
**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\ **Post date:** [May 13, 2022, 3:03am UTC](https://discuss.ray.io/t/pending-tasks-not-starting-up/6090/6 "2022-05-13T03:03:25Z")
A couple of things that would help here: The configs used to launch the cluster. If at all possible, a minimal reproduction of the problem. ---
**Author:** ![Filip\_K](https://avatars.discourse-cdn.com/v4/letter/f/7cd45c/32.png) [@Filip\_K](https://discuss.ray.io/u/Filip_K)\ **Post date:** [May 13, 2022, 8:24am UTC](https://discuss.ray.io/t/pending-tasks-not-starting-up/6090/7 "2022-05-13T08:24:29Z")
I’ll try to prepare reproduction. Is it okay to send it to you via Slack? ---
**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\ **Post date:** [May 13, 2022, 10:47pm UTC](https://discuss.ray.io/t/pending-tasks-not-starting-up/6090/8 "2022-05-13T22:47:21Z")
Slack works, but the [Ray GitHub](https://github.com/ray-project/ray/issues/new?assignees=&labels=bug%2Ctriage&template=bug-report.yml&title=%5B%3CRay+component%3A+Core%7CRLlib%7Cetc...%3E%5D+) is better!