# RayOutOfMemoryError: Why is autoscaler not creating new pods? **URL:** \ **Category:** Kubernetes\ **Created:** [March 29, 2022, 2:20pm UTC](https://discuss.ray.io/t/rayoutofmemoryerror-why-is-autoscaler-not-creating-new-pods/5587 "2022-03-29T14:20:09Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Haripraghash](https://avatars.discourse-cdn.com/v4/letter/h/96bed5/32.png) [@Haripraghash](https://discuss.ray.io/u/Haripraghash)\ **Post date:** [March 29, 2022, 2:20pm UTC](https://discuss.ray.io/t/rayoutofmemoryerror-why-is-autoscaler-not-creating-new-pods/5587/1 "2022-03-29T14:20:09Z")
I am running a ray cluster using helm charts. 1 Head pod. And worker pods are left to autoscale between 0 to 6. Worker node resource request and limits are set to 1 CPU and 2048Mi When I submit a job, 1 worker pod gets created. However, the ray CLI job logs show OOM. Unhandled error (suppress with RAY\_IGNORE\_UNHANDLED\_ERRORS=1): ray::ConsumerActor.run\_behaviour() (pid=586, ip=10.240.0.117, repr=\) ray.\_private.memory\_monitor.RayOutOfMemoryError: More than 95% of the memory on node ray-cluster-ray-worker-type-9w5p5 is used (1.96 / 2.0 GB). The top 10 memory consumers are: If the memory is not enough for the tasks to execute, should the auto scaler not kick in and create more pods for the tasks? ---
**Author:** ![Ameer\_Haj\_Ali](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/ameer_haj_ali/32/279_2.png) [@Ameer\_Haj\_Ali](https://discuss.ray.io/u/Ameer_Haj_Ali)\ **Post date:** [April 25, 2022, 11:47am UTC](https://discuss.ray.io/t/rayoutofmemoryerror-why-is-autoscaler-not-creating-new-pods/5587/2 "2022-04-25T11:47:49Z")
@Dmitri , can you please take a look? ---
**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\ **Post date:** [April 28, 2022, 2:59am UTC](https://discuss.ray.io/t/rayoutofmemoryerror-why-is-autoscaler-not-creating-new-pods/5587/3 "2022-04-28T02:59:03Z")
> If the memory is not enough for the tasks to execute, should the auto scaler not kick in and create more pods for the tasks? The autoscaler scales based on virtual resources specified on Ray task and actor annotations. It does not scale based on resource utilization. Note that for a tasks `@ray.remote` is equivalent to `@ray.remote(num_cpus=1)`. So, if you have workers with 1 CPU and you submit 6 tasks decorated with `@ray.remote`, the autoscaler will attempt to create 6 workers to run those tasks. (That’s is assuming max workers setting is set to at least 6 in the relevant config.) However, if each of those tasks is insanely memory consumptive, each task will fail with an OOM exception. The solution is to do some manual vertical scaling – try allocating more memory for your workers. ---
**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\ **Post date:** [April 28, 2022, 2:59am UTC](https://discuss.ray.io/t/rayoutofmemoryerror-why-is-autoscaler-not-creating-new-pods/5587/4 "2022-04-28T02:59:24Z")
See also this thread: > [@Tasks are completed but ray.exceptions.WorkerCrashedError](https://discuss.ray.io/t/tasks-are-completed-but-ray-exceptions-workercrashederror/5888/4): > > For heavy workloads, a large head node annotated with rayResources: { "CPU": 0 } is a good idea. If you’re seeing workers crashing, it might be a good idea to reduce the number of tasks that run concurrently on the workers by increasing the resource annotations on the task. For example if a task is annotated @ray.remote(num\_cpus=2) there will be at most 3 concurrent instances of that task on a CPU:6 worker node. @ray.remote is equivalent to @ray.remote(num\_cpus=1), which would result in up to…