I try to execute two process python on 8 worker ray and it returns this message: ERROR MAIN - Unhandled exception: Task was killed due to the node running low on memory. How I’m able to resolve this problem?

@tonidep Your task most likely was consuming memory, and the OOM monitor likely killed it. I’m not clear what do you mean by “execute two process python on 8 worker ray.” Do you mean you have two distinct tasks and a eight Ray node cluster? Note that tasks are scheduled on a particular node’s wo…

[image] Jules_Damji: Your task most likely was consuming memory, and the OOM monitor likely killed it. I’m not clear what do you mean by “execute two process python on 8 worker ray.” Do you mean you have two distinct tasks and a eight Ray node cluster? Note that tasks are scheduled on a part…

@tonidep I still don’t follow what do you mean by two module python. I execute these module from two different pc." 8GM is a bit low. You probably want each node in the Ray cluster with at least 16-32GB, and probably 8 cores. I run Ray on my m1 with 10 cores and 64GB, and hardly run into OOM unless…

[image] Jules_Damji: 8GM is a bit low. You probably want your Ray cluster with at least 16-32GB, and probably 8 cores. I run Ray on my m1 with 10 cores and 64GB. thanks @Jules_Damji . I try to grow up my image for the worker

Problem with 8 worker

tonidep April 7, 2023, 6:44am 5

thanks @Jules_Damji . I try to grow up my image for the worker

Problem node running low on memory

Topic		Replies	Views
Problem node running low on memory	3	2214	April 11, 2023
Why is the head dying regularly with OOM while the workers barely have any RAM usage?	3	761	July 5, 2023
Weird error logs when running Out Of Memory (OOM) Ray Core	6	3250	April 11, 2023
Memory error in distributed multiprocessing	11	806	February 23, 2021
(raylet) node_manager.cc Workers (tasks / actors) killed due to memory pressure (OOM)	2	420	March 6, 2024

Problem with 8 worker

Related topics