Hello, I am submitting a big job to my Ray cluster (43 nodes, 1952 CPUs). The job always runs for a few minutes and then workers start failing with the following exceptions and stack dumps. It is reporting a SIGBUS in plasma due to some memory allocation issue. I can reproduce 100% of the time. W…

Awesome! BTW @djakubiec you can install master using the latest wheels here: Installing Ray — Ray v2.0.0.dev0 Let us know if it works!

This is a known issue, and we are actively working on removing it rn actually! What’s the version of Ray are you using? If you use the master, it is probably already resolved.

Oh great news thanks! Sorry if there was a bug reference out there that I didn’t find. I am using Ray 1.4.0. I’ll figure out how to switch to master this weekend and let you know how that goes. Thank you!

Sounds good! This happened due to pretty complicated issue, so it’ll be great if you can verify our fix resolves your issue :)!

OK, well great news: it looks like the latest code (as of this morning) does in fact fix the SIGBUS failure! Cluster tasks used to start failing within 5 minutes of job startup. With the latest code it successfully ran through the whole (rather complicated) job and finished successfully in about 6…

@djakubiec that’s great news! [image] djakubiec: I think I am going to permanently migrate our cluster to the 2.0 beta. We have a variety of workloads to run so I think I can help report back to you on general stability. OK, that would be great. Keep in mind we basically use “2.0” as our “de…

[image] rliaw: I hope that works for you! Can I ask what you’re doing with Ray? We have several different machine learning projects built on sklearn and pandas, and we are looking to better leverage our cluster for these Python-based projects. ~2000 CPU cores spread across ~50 machines and ~…

Well, so after our initial success we’re running into a different problem now. We ramped up the script workload a bit and now it is hanging part way through – but I don’t see any obvious errors. Perhaps you could suggest where to look? We submit 204 tasks via myfunc.remote() and need to process …

And ooo… after I posted that last message I looked at my job output from the stuck job (it was still running but not doing anything) and it just popped out a few dozens exceptions like this from various worker nodes. SIGABRT this time – first time I’ve seen this error so I’m not sure if this is rel…

Hmm… There are probably other unknown issues since removing SIGBUS required us some protocol changes… Is it possible for you to provide us some self-contained repro? Otherwise, you might want to wait until [WIP] Fix PullManager handling of get requests and liveness issues by ericl · Pull Request #16…

Sorry, a self-contained repro is hard as the job is rather complex. I will say that since I last posted I did cure one case of of replicating a huge data structure 204 times for each of the tasks (which I condensed down to 1 shared object_ref). This now avoids the problem for me, though obviously …

Big cluster job failing due to SIGBUS in plasma

Ray Core

djakubiec July 12, 2021, 7:26pm 17

@sangcho I posted the issue to a new thread Periodic _MultiThreadedRendezvous failure leaves cluster in damaged state and mentioned you there. Thanks!

Topic		Replies	Views
(raylet) Some workers of the worker process(68497) have not registered within the timeout. The process is still alive, probably it's hanging during start Ray Clusters	4	2847	May 26, 2022
Ray Head restarting and leaving behind zombie processes Ray Clusters	0	190	March 12, 2024
Subset of tasks stuck in "PENDING_NODE_ASSIGNMENT" forever Ray Clusters	9	2455	May 25, 2023
Error while stopping a job in a ray cluster Check failed: addr_proto.worker_id() != "" Ray Clusters	0	23	June 30, 2024
Ray Actor Dying unexpectedly Ray Core	8	4251	October 21, 2022

Big cluster job failing due to SIGBUS in plasma

Related topics