How severe does this issue affect your experience of using Ray?
On a 8 core Linux server, in local mode after calling ray.init(), ray creates 8 (idle) workers (looks reasonable), but in each of them it also creates many threads, in this case total of 33 threads in each worker.
$ cat /proc/7511/cmdline
ray::IDLE
$ ls -l /proc/7511/task | wc -l
33
Is there anyway to limit how many threads each worker spawns? Thanks!
There’s no way to restrict the number of threads. But 33 seems more than I thought actually (I’d expect 10~ish threads per worker). What’s the output when you run Profiling for Ray Developers — Ray 3.0.0.dev0 to the process 7511?
Part 1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
__futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55b7c02f11e8) at ./nptl/futex-internal.c:57
57 ./nptl/futex-internal.c: No such file or directory.
Thread 32 (Thread 0x7fabddffb640 (LWP 5612) "worker.io"):
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7fabe800cfec) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7fabe800cfec) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7fabe800cfec, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3 0x00007fac4a093ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7fabe800cf88, cond=0x7fabe800cfc0) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_wait (cond=0x7fabe800cfc0, mutex=0x7fabe800cf88) at ./nptl/pthread_cond_wait.c:627
#5 0x00007fac4917cdf3 in boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#6 0x00007fac4917ea89 in boost::asio::detail::scheduler::run(boost::system::error_code&) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#7 0x00007fac4917ec9b in boost::asio::detail::posix_thread::func<boost::asio::thread_pool::thread_function>::run() () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#8 0x00007fac491769f1 in boost_asio_detail_posix_thread_function () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#9 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 31 (Thread 0x7fabde7fc640 (LWP 5610) "worker.io"):
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7fabe800cfe8) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7fabe800cfe8) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7fabe800cfe8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3 0x00007fac4a093ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7fabe800cf88, cond=0x7fabe800cfc0) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_wait (cond=0x7fabe800cfc0, mutex=0x7fabe800cf88) at ./nptl/pthread_cond_wait.c:627
#5 0x00007fac4917cdf3 in boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#6 0x00007fac4917ea89 in boost::asio::detail::scheduler::run(boost::system::error_code&) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#7 0x00007fac4917ec9b in boost::asio::detail::posix_thread::func<boost::asio::thread_pool::thread_function>::run() () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#8 0x00007fac491769f1 in boost_asio_detail_posix_thread_function () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#9 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 30 (Thread 0x7fabdeffd640 (LWP 5590) "grpc_global_tim"):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007fac471a6ed6 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#2 0x00007fac471a6e22 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#3 0x00007fac46ca6770 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#4 0x00007fac46f31397 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#5 0x00007fac46f62b03 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#6 0x00007fac46f37419 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#7 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 29 (Thread 0x7fabdf7fe640 (LWP 5588) "grpc_global_tim"):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007fac491ad15e in absl::lts_20220623::synchronization_internal::Waiter::Wait(absl::lts_20220623::synchronization_internal::KernelTimeout) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#2 0x00007fac491ad0a3 in AbslInternalPerThreadSemWait_lts_20220623 () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#3 0x00007fac488902db in absl::lts_20220623::CondVar::WaitCommon(absl::lts_20220623::Mutex*, absl::lts_20220623::synchronization_internal::KernelTimeout) [clone .cold] () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#4 0x00007fac4914dd2e in gpr_cv_wait () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#5 0x00007fac4910bf2c in timer_thread(void*) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#6 0x00007fac49149b26 in grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::{lambda(void*)#1}::_FUN(void*) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#7 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 28 (Thread 0x7fabdffff640 (LWP 5440) "task_event_buff"):
#0 0x00007fac4a125fde in epoll_wait (epfd=17, events=0x7fabdfffe520, maxevents=128, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1 0x00007fac4917ca39 in boost::asio::detail::epoll_reactor::run(long, boost::asio::detail::op_queue<boost::asio::detail::scheduler_operation>&) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#2 0x00007fac4917cced in boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#3 0x00007fac4917ea89 in boost::asio::detail::scheduler::run(boost::system::error_code&) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#4 0x00007fac4917ef42 in boost::asio::io_context::run() () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#5 0x00007fac48abfbde in std::thread::_State_impl<std::thread::_Invoker<std::tuple<ray::core::worker::TaskEventBufferImpl::Start(bool)::{lambda()#1}> > >::_M_run() () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#6 0x00007fac492a56c0 in execute_native_thread_routine () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#7 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 27 (Thread 0x7fac34ff9640 (LWP 5439) "client.poll0"):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007fac491ad1bf in absl::lts_20220623::synchronization_internal::Waiter::Wait(absl::lts_20220623::synchronization_internal::KernelTimeout) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#2 0x00007fac491ad0a3 in AbslInternalPerThreadSemWait_lts_20220623 () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#3 0x00007fac488902db in absl::lts_20220623::CondVar::WaitCommon(absl::lts_20220623::Mutex*, absl::lts_20220623::synchronization_internal::KernelTimeout) [clone .cold] () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#4 0x00007fac4914dd09 in gpr_cv_wait () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#5 0x00007fac4911a77f in pollset_work(grpc_pollset*, grpc_pollset_worker**, grpc_core::Timestamp) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#6 0x00007fac490e28bf in cq_next(grpc_completion_queue*, gpr_timespec, void*) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#7 0x00007fac48dee7a1 in grpc::CompletionQueue::AsyncNextInternal(void**, bool*, gpr_timespec) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#8 0x00007fac48a279fc in ray::rpc::ClientCallManager::PollEventsFromCompletionQueue(int) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#9 0x00007fac492a56c0 in execute_native_thread_routine () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#10 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#11 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 26 (Thread 0x7fac357fa640 (LWP 5436) "client.poll0"):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007fac491ad1bf in absl::lts_20220623::synchronization_internal::Waiter::Wait(absl::lts_20220623::synchronization_internal::KernelTimeout) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#2 0x00007fac491ad0a3 in AbslInternalPerThreadSemWait_lts_20220623 () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#3 0x00007fac488902db in absl::lts_20220623::CondVar::WaitCommon(absl::lts_20220623::Mutex*, absl::lts_20220623::synchronization_internal::KernelTimeout) [clone .cold] () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#4 0x00007fac4914dd09 in gpr_cv_wait () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#5 0x00007fac4911a77f in pollset_work(grpc_pollset*, grpc_pollset_worker**, grpc_core::Timestamp) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#6 0x00007fac490e28bf in cq_next(grpc_completion_queue*, gpr_timespec, void*) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#7 0x00007fac48dee7a1 in grpc::CompletionQueue::AsyncNextInternal(void**, bool*, gpr_timespec) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#8 0x00007fac48a279fc in ray::rpc::ClientCallManager::PollEventsFromCompletionQueue(int) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#9 0x00007fac492a56c0 in execute_native_thread_routine () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#10 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#11 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 25 (Thread 0x7fac35ffb640 (LWP 5435) "server.poll0"):
#0 0x00007fac4a125fde in epoll_wait (epfd=7, events=0x7fac496d6684 <g_epoll_set+4>, maxevents=100, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1 0x00007fac4911a8b9 in pollset_work(grpc_pollset*, grpc_pollset_worker**, grpc_core::Timestamp) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#2 0x00007fac490e28bf in cq_next(grpc_completion_queue*, gpr_timespec, void*) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#3 0x00007fac48dee7a1 in grpc::CompletionQueue::AsyncNextInternal(void**, bool*, gpr_timespec) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#4 0x00007fac48c84c74 in ray::rpc::GrpcServer::PollEventsFromCompletionQueue(int) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#5 0x00007fac492a56c0 in execute_native_thread_routine () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#6 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#7 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 24 (Thread 0x7fac367fc640 (LWP 5431) "grpcpp_sync_ser"):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007fac491ad1bf in absl::lts_20220623::synchronization_internal::Waiter::Wait(absl::lts_20220623::synchronization_internal::KernelTimeout) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#2 0x00007fac491ad0a3 in AbslInternalPerThreadSemWait_lts_20220623 () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#3 0x00007fac488902db in absl::lts_20220623::CondVar::WaitCommon(absl::lts_20220623::Mutex*, absl::lts_20220623::synchronization_internal::KernelTimeout) [clone .cold] () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#4 0x00007fac4914dd09 in gpr_cv_wait () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#5 0x00007fac490e0b0f in (anonymous namespace)::non_polling_poller_work(grpc_pollset*, grpc_pollset_worker**, grpc_core::Timestamp) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#6 0x00007fac490e28bf in cq_next(grpc_completion_queue*, gpr_timespec, void*) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#7 0x00007fac48dee7a1 in grpc::CompletionQueue::AsyncNextInternal(void**, bool*, gpr_timespec) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#8 0x00007fac48dd36c2 in grpc::Server::SyncRequestThreadManager::PollForWork(void**, bool*) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#9 0x00007fac48defd10 in grpc::ThreadManager::MainWorkLoop() () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#10 0x00007fac48deff64 in grpc::ThreadManager::WorkerThread::Run() () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#11 0x00007fac49149b26 in grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::{lambda(void*)#1}::_FUN(void*) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#12 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#13 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 23 (Thread 0x7fac36ffd640 (LWP 5430) "grpc_health_che"):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007fac491ad15e in absl::lts_20220623::synchronization_internal::Waiter::Wait(absl::lts_20220623::synchronization_internal::KernelTimeout) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#2 0x00007fac491ad0a3 in AbslInternalPerThreadSemWait_lts_20220623 () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#3 0x00007fac488902db in absl::lts_20220623::CondVar::WaitCommon(absl::lts_20220623::Mutex*, absl::lts_20220623::synchronization_internal::KernelTimeout) [clone .cold] () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#4 0x00007fac4914dd2e in gpr_cv_wait () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#5 0x00007fac490e0b0f in (anonymous namespace)::non_polling_poller_work(grpc_pollset*, grpc_pollset_worker**, grpc_core::Timestamp) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#6 0x00007fac490e28bf in cq_next(grpc_completion_queue*, gpr_timespec, void*) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#7 0x00007fac48dee7a1 in grpc::CompletionQueue::AsyncNextInternal(void**, bool*, gpr_timespec) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#8 0x00007fac48de7cb4 in grpc::DefaultHealthCheckService::HealthCheckServiceImpl::Serve(void*) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#9 0x00007fac49149b26 in grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::{lambda(void*)#1}::_FUN(void*) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#10 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#11 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 22 (Thread 0x7fac377fe640 (LWP 5427) "worker.io"):
#0 0x00007fac4a125fde in epoll_wait (epfd=10, events=0x7fac377fd520, maxevents=128, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1 0x00007fac4917ca39 in boost::asio::detail::epoll_reactor::run(long, boost::asio::detail::op_queue<boost::asio::detail::scheduler_operation>&) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#2 0x00007fac4917cced in boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#3 0x00007fac4917ea89 in boost::asio::detail::scheduler::run(boost::system::error_code&) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#4 0x00007fac4917ef42 in boost::asio::io_context::run() () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#5 0x00007fac48a5e38d in ray::core::CoreWorker::RunIOService() () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#6 0x00007fac492a56c0 in execute_native_thread_routine () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#7 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 21 (Thread 0x7fac37fff640 (LWP 5426) "client.poll0"):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007fac491ad1bf in absl::lts_20220623::synchronization_internal::Waiter::Wait(absl::lts_20220623::synchronization_internal::KernelTimeout) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#2 0x00007fac491ad0a3 in AbslInternalPerThreadSemWait_lts_20220623 () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#3 0x00007fac488902db in absl::lts_20220623::CondVar::WaitCommon(absl::lts_20220623::Mutex*, absl::lts_20220623::synchronization_internal::KernelTimeout) [clone .cold] () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#4 0x00007fac4914dd09 in gpr_cv_wait () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#5 0x00007fac4911a77f in pollset_work(grpc_pollset*, grpc_pollset_worker**, grpc_core::Timestamp) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#6 0x00007fac490e28bf in cq_next(grpc_completion_queue*, gpr_timespec, void*) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#7 0x00007fac48dee7a1 in grpc::CompletionQueue::AsyncNextInternal(void**, bool*, gpr_timespec) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#8 0x00007fac48a279fc in ray::rpc::ClientCallManager::PollEventsFromCompletionQueue(int) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#9 0x00007fac492a56c0 in execute_native_thread_routine () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#10 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#11 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 20 (Thread 0x7fac3c9ac640 (LWP 5425) "client.poll0"):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007fac491ad1bf in absl::lts_20220623::synchronization_internal::Waiter::Wait(absl::lts_20220623::synchronization_internal::KernelTimeout) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#2 0x00007fac491ad0a3 in AbslInternalPerThreadSemWait_lts_20220623 () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#3 0x00007fac488902db in absl::lts_20220623::CondVar::WaitCommon(absl::lts_20220623::Mutex*, absl::lts_20220623::synchronization_internal::KernelTimeout) [clone .cold] () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#4 0x00007fac4914dd09 in gpr_cv_wait () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#5 0x00007fac4911a77f in pollset_work(grpc_pollset*, grpc_pollset_worker**, grpc_core::Timestamp) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#6 0x00007fac490e28bf in cq_next(grpc_completion_queue*, gpr_timespec, void*) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#7 0x00007fac48dee7a1 in grpc::CompletionQueue::AsyncNextInternal(void**, bool*, gpr_timespec) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#8 0x00007fac48a279fc in ray::rpc::ClientCallManager::PollEventsFromCompletionQueue(int) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#9 0x00007fac492a56c0 in execute_native_thread_routine () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#10 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#11 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 19 (Thread 0x7fac3d1ad640 (LWP 5424) "grpc_global_tim"):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007fac491ad1bf in absl::lts_20220623::synchronization_internal::Waiter::Wait(absl::lts_20220623::synchronization_internal::KernelTimeout) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#2 0x00007fac491ad0a3 in AbslInternalPerThreadSemWait_lts_20220623 () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#3 0x00007fac488902db in absl::lts_20220623::CondVar::WaitCommon(absl::lts_20220623::Mutex*, absl::lts_20220623::synchronization_internal::KernelTimeout) [clone .cold] () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#4 0x00007fac4914dd09 in gpr_cv_wait () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#5 0x00007fac4910bf2c in timer_thread(void*) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#6 0x00007fac49149b26 in grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::{lambda(void*)#1}::_FUN(void*) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#7 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Part 2
Thread 18 (Thread 0x7fac3d9ae640 (LWP 5423) "resolver-execut"):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007fac491ad15e in absl::lts_20220623::synchronization_internal::Waiter::Wait(absl::lts_20220623::synchronization_internal::KernelTimeout) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#2 0x00007fac491ad0a3 in AbslInternalPerThreadSemWait_lts_20220623 () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#3 0x00007fac488902db in absl::lts_20220623::CondVar::WaitCommon(absl::lts_20220623::Mutex*, absl::lts_20220623::synchronization_internal::KernelTimeout) [clone .cold] () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#4 0x00007fac4914dd2e in gpr_cv_wait () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#5 0x00007fac491376e6 in grpc_core::Executor::ThreadMain(void*) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#6 0x00007fac49149b26 in grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::{lambda(void*)#1}::_FUN(void*) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#7 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 17 (Thread 0x7fac3e1af640 (LWP 5422) "default-executo"):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007fac491ad15e in absl::lts_20220623::synchronization_internal::Waiter::Wait(absl::lts_20220623::synchronization_internal::KernelTimeout) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#2 0x00007fac491ad0a3 in AbslInternalPerThreadSemWait_lts_20220623 () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#3 0x00007fac488902db in absl::lts_20220623::CondVar::WaitCommon(absl::lts_20220623::Mutex*, absl::lts_20220623::synchronization_internal::KernelTimeout) [clone .cold] () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#4 0x00007fac4914dd2e in gpr_cv_wait () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#5 0x00007fac491376e6 in grpc_core::Executor::ThreadMain(void*) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#6 0x00007fac49149b26 in grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::{lambda(void*)#1}::_FUN(void*) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#7 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 16 (Thread 0x7fac3f1b1640 (LWP 5421) "ray::IDLE"):
#0 0x00007fac4a0e5868 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=0x7fac3f1b0db0, rem=0x7fac3f1b0db0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
#1 0x00007fac4a0ea6e7 in __GI___nanosleep (req=<optimized out>, rem=<optimized out>) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
#2 0x00007fac491b5715 in AbslInternalSleepFor_lts_20220623 () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#3 0x00007fac491a089c in opencensus::stats::StatsExporterImpl::RunWorkerLoop() () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#4 0x00007fac492a56c0 in execute_native_thread_routine () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#5 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 15 (Thread 0x7fac3e9b0640 (LWP 5420) "ray::IDLE"):
#0 0x00007fac4a0e5868 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=0x7fac3e9afd80, rem=0x7fac3e9afd80) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
#1 0x00007fac4a0ea6e7 in __GI___nanosleep (req=<optimized out>, rem=<optimized out>) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
#2 0x00007fac491b5715 in AbslInternalSleepFor_lts_20220623 () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#3 0x00007fac4919bebf in opencensus::stats::DeltaProducer::RunHarvesterLoop() () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#4 0x00007fac492a56c0 in execute_native_thread_routine () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#5 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 14 (Thread 0x7fac3f9b2640 (LWP 5419) "ray::IDLE"):
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55b7c0a81868) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x55b7c0a81868) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55b7c0a81868, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3 0x00007fac4a093ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55b7c0a81808, cond=0x55b7c0a81840) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_wait (cond=0x55b7c0a81840, mutex=0x55b7c0a81808) at ./nptl/pthread_cond_wait.c:627
#5 0x00007fac4917cdf3 in boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#6 0x00007fac4917ea89 in boost::asio::detail::scheduler::run(boost::system::error_code&) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#7 0x00007fac4917ef42 in boost::asio::io_context::run() () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#8 0x00007fac48cc81eb in std::thread::_State_impl<std::thread::_Invoker<std::tuple<ray::IOServicePool::Run()::{lambda()#1}> > >::_M_run() () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#9 0x00007fac492a56c0 in execute_native_thread_routine () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#10 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#11 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 13 (Thread 0x7fac401b3640 (LWP 5399) "timer_manager"):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007fac471a6f3a in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#2 0x00007fac471a6e22 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#3 0x00007fac46ca6770 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#4 0x00007fac46f2a4b2 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#5 0x00007fac46f2aaa5 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#6 0x00007fac46f37467 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#7 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 12 (Thread 0x7fac409b4640 (LWP 5398) "event_engine"):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007fac471a6ed6 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#2 0x00007fac471a6e22 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#3 0x00007fac46ca6770 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#4 0x00007fac46f2cb8b in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#5 0x00007fac46f2db18 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#6 0x00007fac46f37467 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#7 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 11 (Thread 0x7fac411b5640 (LWP 5397) "event_engine"):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007fac471a6ed6 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#2 0x00007fac471a6e22 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#3 0x00007fac46ca6770 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#4 0x00007fac46f2cb8b in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#5 0x00007fac46f2db18 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#6 0x00007fac46f37467 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#7 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 10 (Thread 0x7fac419b6640 (LWP 5396) "event_engine"):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007fac471a6ed6 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#2 0x00007fac471a6e22 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#3 0x00007fac46ca6770 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#4 0x00007fac46f2cb8b in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#5 0x00007fac46f2db18 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#6 0x00007fac46f37467 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#7 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Part 3
Thread 9 (Thread 0x7fac421b7640 (LWP 5395) "event_engine"):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007fac471a6ed6 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#2 0x00007fac471a6e22 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#3 0x00007fac46ca6770 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#4 0x00007fac46f2cb8b in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#5 0x00007fac46f2db18 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#6 0x00007fac46f37467 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#7 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 8 (Thread 0x7fac429b8640 (LWP 5394) "event_engine"):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007fac471a6ed6 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#2 0x00007fac471a6e22 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#3 0x00007fac46ca6770 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#4 0x00007fac46f2cb8b in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#5 0x00007fac46f2db18 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#6 0x00007fac46f37467 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#7 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 7 (Thread 0x7fac431b9640 (LWP 5393) "event_engine"):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007fac471a6ed6 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#2 0x00007fac471a6e22 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#3 0x00007fac46ca6770 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#4 0x00007fac46f2cb8b in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#5 0x00007fac46f2db18 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#6 0x00007fac46f37467 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#7 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 6 (Thread 0x7fac439ba640 (LWP 5392) "event_engine"):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007fac471a6ed6 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#2 0x00007fac471a6e22 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#3 0x00007fac46ca6770 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#4 0x00007fac46f2cb8b in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#5 0x00007fac46f2db18 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#6 0x00007fac46f37467 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#7 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 5 (Thread 0x7fac441bb640 (LWP 5391) "event_engine"):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007fac471a6ed6 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#2 0x00007fac471a6e22 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#3 0x00007fac46ca6770 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#4 0x00007fac46f2cb8b in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#5 0x00007fac46f2db18 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#6 0x00007fac46f37467 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#7 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 4 (Thread 0x7fac449bc640 (LWP 5390) "grpc_global_tim"):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007fac471a6f3a in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#2 0x00007fac471a6e22 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#3 0x00007fac46ca6770 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#4 0x00007fac46f31374 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#5 0x00007fac46f62b03 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#6 0x00007fac46f37419 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#7 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 3 (Thread 0x7fac451bd640 (LWP 5389) "resolver-execut"):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007fac471a6ed6 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#2 0x00007fac471a6e22 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#3 0x00007fac46ca6770 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#4 0x00007fac46f31397 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#5 0x00007fac46f4de2d in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#6 0x00007fac46f37419 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#7 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 2 (Thread 0x7fac459be640 (LWP 5388) "default-executo"):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007fac471a6ed6 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#2 0x00007fac471a6e22 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#3 0x00007fac46ca6770 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#4 0x00007fac46f31397 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#5 0x00007fac46f4de2d in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#6 0x00007fac46f37419 in ?? () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so
#7 0x00007fac4a094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8 0x00007fac4a126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 1 (Thread 0x7fac4a2f7000 (LWP 5277) "ray::IDLE"):
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55b7c02f11e8) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x55b7c02f11e8) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55b7c02f11e8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3 0x00007fac4a093ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55b7c02f1188, cond=0x55b7c02f11c0) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_wait (cond=0x55b7c02f11c0, mutex=0x55b7c02f1188) at ./nptl/pthread_cond_wait.c:627
#5 0x00007fac4917cdf3 in boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#6 0x00007fac4917ea89 in boost::asio::detail::scheduler::run(boost::system::error_code&) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#7 0x00007fac4917ef42 in boost::asio::io_context::run() () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#8 0x00007fac48a3934c in ray::core::CoreWorker::RunTaskExecutionLoop() () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#9 0x00007fac48a76e4c in ray::core::CoreWorkerProcessImpl::RunWorkerTaskExecutionLoop() () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#10 0x00007fac48a76ffd in ray::core::CoreWorkerProcess::RunTaskExecutionLoop() () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#11 0x00007fac488de207 in __pyx_pw_3ray_7_raylet_10CoreWorker_7run_task_loop(_object*, _object*) () from /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_raylet.so
#12 0x000055b7bf95be03 in method_vectorcall_NOARGS (func=<method_descriptor at remote 0x7fac4998f1f0>, args=0x7fac45a3bc38, nargsf=<optimized out>, kwnames=0x0) at ../Objects/descrobject.c:432
#13 0x000055b7bf9408cb in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fac45a3bc38, callable=<method_descriptor at remote 0x7fac4998f1f0>, tstate=0x55b7c02b1640) at ../Include/cpython/abstract.h:114
#14 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fac45a3bc38, callable=<method_descriptor at remote 0x7fac4998f1f0>) at ../Include/cpython/abstract.h:123
#15 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7fff920fbbe0, tstate=<optimized out>) at ../Python/ceval.c:5891
#16 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:4198
#17 0x000055b7bf9581ec in _PyEval_EvalFrame (throwflag=0, f=Frame 0x7fac45a3bac0, for file /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_private/worker.py, line 844, in main_loop (self=<Worker(node=<Node(_default_worker=True, head=False, kernel_fate_share=False, all_processes={}, removal_lock=<_thread.lock at remote 0x7fac46b23380>, _node_ip_address='10.229.127.202', _raylet_ip_address='10.229.127.202', _resource_spec=None, _localhost='127.0.0.1', _ray_params=<RayParams(redis_address='None', gcs_address='10.229.127.202:65201', num_cpus=None, num_gpus=None, memory=None, object_store_memory=None, resources={}, redis_max_memory=None, redis_port=None, redis_shard_ports=None, object_manager_port=None, node_manager_port=45171, gcs_server_port=None, node_ip_address='10.229.127.202', node_name=None, raylet_ip_address='10.229.127.202', min_worker_port=None, max_worker_port=None, worker_port_list=None, ray_client_server_port=None, driver_mode=None, redirect_output=None, external_addresses=None, num_redis_shards=None, redis_max_clients=None, redis_password=Non...(truncated), tstate=0x55b7c02b1640) at ../Include/internal/pycore_ceval.h:46
#18 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, con=0x7fac45dfc290, tstate=0x55b7c02b1640) at ../Python/ceval.c:5065
#19 _PyFunction_Vectorcall (func=<function at remote 0x7fac45dfc280>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:342
#20 0x000055b7bf9408cb in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fac49f29da8, callable=<function at remote 0x7fac45dfc280>, tstate=0x55b7c02b1640) at ../Include/cpython/abstract.h:114
#21 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fac49f29da8, callable=<function at remote 0x7fac45dfc280>) at ../Include/cpython/abstract.h:123
#22 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7fff920fbdb0, tstate=<optimized out>) at ../Python/ceval.c:5891
#23 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:4198
#24 0x000055b7bf93ced6 in _PyEval_EvalFrame (throwflag=0, f=Frame 0x7fac49f29c40, for file /home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_private/workers/default_worker.py, line 258, in <module> (), tstate=0x55b7c02b1640) at ../Include/internal/pycore_ceval.h:46
#25 _PyEval_Vector (tstate=0x55b7c02b1640, con=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../Python/ceval.c:5065
#26 0x000055b7bfa33366 in PyEval_EvalCode (co=<code at remote 0x7fac49b3b3c0>, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_private/workers/default_worker.py') at remote 0x7fac49b21030>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fac49f905e0>, '__file__': '/home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_private/workers/default_worker.py', '__cached__': None, 'argparse': <module at remote 0x7fac499f1760>, 'base64': <module at remote 0x7fac499f3880>, 'json': <module at remote 0x7fac49a62bb0>, 'time': <module at remote 0x7fac49b07ab0>, 'sys': <module at remote 0x7fac49f7e250>, 'ray': <module at remote 0x7fac49a7d850>, 'ray_constants': <module at remote 0x7fac47f820c0>, 'RayParams': <type at remote 0x55b7c0950ed0>, 'configure_log_file': <function at remote 0x7fac467da440>, 'get_worker_log_file_name': <function at remote 0x7fac467d9e10>, 'parser': <ArgumentParser(description='Parse addresses for the worke...(truncated), locals=<optimized out>) at ../Python/ceval.c:1134
#27 0x000055b7bfa60108 in run_eval_code_obj (tstate=0x55b7c02b1640, co=0x7fac49b3b3c0, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_private/workers/default_worker.py') at remote 0x7fac49b21030>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fac49f905e0>, '__file__': '/home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_private/workers/default_worker.py', '__cached__': None, 'argparse': <module at remote 0x7fac499f1760>, 'base64': <module at remote 0x7fac499f3880>, 'json': <module at remote 0x7fac49a62bb0>, 'time': <module at remote 0x7fac49b07ab0>, 'sys': <module at remote 0x7fac49f7e250>, 'ray': <module at remote 0x7fac49a7d850>, 'ray_constants': <module at remote 0x7fac47f820c0>, 'RayParams': <type at remote 0x55b7c0950ed0>, 'configure_log_file': <function at remote 0x7fac467da440>, 'get_worker_log_file_name': <function at remote 0x7fac467d9e10>, 'parser': <ArgumentParser(description='Parse addresses for the worke...(truncated), locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_private/workers/default_worker.py') at remote 0x7fac49b21030>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fac49f905e0>, '__file__': '/home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_private/workers/default_worker.py', '__cached__': None, 'argparse': <module at remote 0x7fac499f1760>, 'base64': <module at remote 0x7fac499f3880>, 'json': <module at remote 0x7fac49a62bb0>, 'time': <module at remote 0x7fac49b07ab0>, 'sys': <module at remote 0x7fac49f7e250>, 'ray': <module at remote 0x7fac49a7d850>, 'ray_constants': <module at remote 0x7fac47f820c0>, 'RayParams': <type at remote 0x55b7c0950ed0>, 'configure_log_file': <function at remote 0x7fac467da440>, 'get_worker_log_file_name': <function at remote 0x7fac467d9e10>, 'parser': <ArgumentParser(description='Parse addresses for the worke...(truncated)) at ../Python/pythonrun.c:1291
#28 0x000055b7bfa58f5b in run_mod (mod=<optimized out>, filename=<optimized out>, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_private/workers/default_worker.py') at remote 0x7fac49b21030>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fac49f905e0>, '__file__': '/home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_private/workers/default_worker.py', '__cached__': None, 'argparse': <module at remote 0x7fac499f1760>, 'base64': <module at remote 0x7fac499f3880>, 'json': <module at remote 0x7fac49a62bb0>, 'time': <module at remote 0x7fac49b07ab0>, 'sys': <module at remote 0x7fac49f7e250>, 'ray': <module at remote 0x7fac49a7d850>, 'ray_constants': <module at remote 0x7fac47f820c0>, 'RayParams': <type at remote 0x55b7c0950ed0>, 'configure_log_file': <function at remote 0x7fac467da440>, 'get_worker_log_file_name': <function at remote 0x7fac467d9e10>, 'parser': <ArgumentParser(description='Parse addresses for the worke...(truncated), locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_private/workers/default_worker.py') at remote 0x7fac49b21030>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fac49f905e0>, '__file__': '/home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_private/workers/default_worker.py', '__cached__': None, 'argparse': <module at remote 0x7fac499f1760>, 'base64': <module at remote 0x7fac499f3880>, 'json': <module at remote 0x7fac49a62bb0>, 'time': <module at remote 0x7fac49b07ab0>, 'sys': <module at remote 0x7fac49f7e250>, 'ray': <module at remote 0x7fac49a7d850>, 'ray_constants': <module at remote 0x7fac47f820c0>, 'RayParams': <type at remote 0x55b7c0950ed0>, 'configure_log_file': <function at remote 0x7fac467da440>, 'get_worker_log_file_name': <function at remote 0x7fac467d9e10>, 'parser': <ArgumentParser(description='Parse addresses for the worke...(truncated), flags=<optimized out>, arena=<optimized out>) at ../Python/pythonrun.c:1312
#29 0x000055b7bfa5fe55 in pyrun_file (fp=fp@entry=0x55b7c028f370, filename=filename@entry='/home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_private/workers/default_worker.py', start=start@entry=257, globals=globals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_private/workers/default_worker.py') at remote 0x7fac49b21030>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fac49f905e0>, '__file__': '/home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_private/workers/default_worker.py', '__cached__': None, 'argparse': <module at remote 0x7fac499f1760>, 'base64': <module at remote 0x7fac499f3880>, 'json': <module at remote 0x7fac49a62bb0>, 'time': <module at remote 0x7fac49b07ab0>, 'sys': <module at remote 0x7fac49f7e250>, 'ray': <module at remote 0x7fac49a7d850>, 'ray_constants': <module at remote 0x7fac47f820c0>, 'RayParams': <type at remote 0x55b7c0950ed0>, 'configure_log_file': <function at remote 0x7fac467da440>, 'get_worker_log_file_name': <function at remote 0x7fac467d9e10>, 'parser': <ArgumentParser(description='Parse addresses for the worke...(truncated), locals=locals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_private/workers/default_worker.py') at remote 0x7fac49b21030>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fac49f905e0>, '__file__': '/home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_private/workers/default_worker.py', '__cached__': None, 'argparse': <module at remote 0x7fac499f1760>, 'base64': <module at remote 0x7fac499f3880>, 'json': <module at remote 0x7fac49a62bb0>, 'time': <module at remote 0x7fac49b07ab0>, 'sys': <module at remote 0x7fac49f7e250>, 'ray': <module at remote 0x7fac49a7d850>, 'ray_constants': <module at remote 0x7fac47f820c0>, 'RayParams': <type at remote 0x55b7c0950ed0>, 'configure_log_file': <function at remote 0x7fac467da440>, 'get_worker_log_file_name': <function at remote 0x7fac467d9e10>, 'parser': <ArgumentParser(description='Parse addresses for the worke...(truncated), closeit=closeit@entry=1, flags=0x7fff920fc058) at ../Python/pythonrun.c:1208
#30 0x000055b7bfa5f338 in _PyRun_SimpleFileObject (fp=0x55b7c028f370, filename='/home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_private/workers/default_worker.py', closeit=1, flags=0x7fff920fc058) at ../Python/pythonrun.c:456
#31 0x000055b7bfa5f033 in _PyRun_AnyFileObject (fp=0x55b7c028f370, filename='/home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_private/workers/default_worker.py', closeit=1, flags=0x7fff920fc058) at ../Python/pythonrun.c:90
#32 0x000055b7bfa502de in pymain_run_file_obj (skip_source_first_line=0, filename='/home/ubuntu/workspace/.venv/lib/python3.10/site-packages/ray/_private/workers/default_worker.py', program_name='/home/ubuntu/workspace/.venv/bin/python') at ../Modules/main.c:353
#33 pymain_run_file (config=0x55b7c0294f30) at ../Modules/main.c:372
#34 pymain_run_python (exitcode=0x7fff920fc054) at ../Modules/main.c:587
#35 Py_RunMain () at ../Modules/main.c:666
#36 0x000055b7bfa2632d in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at ../Modules/main.c:720
#37 0x00007fac4a029d90 in __libc_start_call_main (main=main@entry=0x55b7bfa262f0 <main>, argc=argc@entry=17, argv=argv@entry=0x7fff920fc268) at ../sysdeps/nptl/libc_start_call_main.h:58
#38 0x00007fac4a029e40 in __libc_start_main_impl (main=0x55b7bfa262f0 <main>, argc=17, argv=0x7fff920fc268, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff920fc258) at ../csu/libc-start.c:392
#39 0x000055b7bfa26225 in _start ()
[Inferior 1 (process 5277) detached]
So there are
worker.io
grpc_global_tim
task_event_buff
client.poll0
server.poll0
grpcpp_sync_ser
grpc_health_che
resolver-execut
default-execut
ray :: IDLE
timer_manager
event_engine
threads.
For comparison, on a 128 core server
worker.io
grpc_global_tim
task_event_buff
client.poll0
server.poll0
grpcpp_sync_ser
grpc_health_che
resolver-execut
default-execut
ray :: IDLE
timer_manager
event_engine
total of 86 threads in one ray IDLE process.
And there are 128 ray IDLE processes, a total of over 10000 threads spawned just by calling ray.init().
@sangcho Ping, please take a look at the gdb output, thanks!
@messense I think the thread number seems ok. It’s not an issue as long as it’s not causing any trouble for the workloads.
It does cause trouble for the workloads, tasks become very slow since the Linux kernel has to schedule lots of threads which has a quite big overhead.
I know, but I don’t think it’s a big issue. You can do some profiling with heavy workloads and check the overhead of context switching cost.
Most of the threads should just be idle.
The event_engine comes from the gRPC which we can’t do a lot of things.
worker.io is a worker which means you have a pending ray worker there. It’s also idle. This is for performance purpose because later we can short the worker start time.
@messense can you create an issue and send me a link here? Sorry, I was off last week.
@yic I think 33 worker.io seems a bit sketchy given it is from a single process (based on my code reading, there should be only 1 worker.io per process?). event_engine also seems to be proportional to host machine’s CPU (and it should be controllable).
I agree we don’t need to fix this unless the workload has a problem, but I think it is a bit hard to estimate the impact (there’s a possibility lots of threads actually cause context switching overhead?)
@FengLi666 is it possible to share the output before/after 2.2? That will help us understand better what part of threads have been increased
@FengLi666 how did you generator the thread_names.csv?
ps -eLo tid,comm | sed 1d | awk '{print $1","$2}' > thread_names.csv
Btw this is probably fixed in the master. Can you double check?