# Too many threads in ray worker **URL:** \ **Category:** Ray Core\ **Created:** [June 2, 2023, 2:01am UTC](https://discuss.ray.io/t/too-many-threads-in-ray-worker/10881 "2023-06-02T02:01:03Z")\ **Posts on this page:** 6\ **Page:** 2
**Author:** ![heliar-k](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/heliar-k/32/5005_2.png) [@heliar-k](https://discuss.ray.io/u/heliar-k)\ **Post date:** [September 5, 2023, 5:56am UTC](https://discuss.ray.io/t/too-many-threads-in-ray-worker/10881/21 "2023-09-05T05:56:10Z")
I also encounter the same issue on our K8S cluster. Could you point out which merged PR solve this? ---
**Author:** ![Karapo](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/karapo/32/5014_2.png) [@Karapo](https://discuss.ray.io/u/Karapo)\ **Post date:** [September 6, 2023, 9:53am UTC](https://discuss.ray.io/t/too-many-threads-in-ray-worker/10881/22 "2023-09-06T09:53:56Z")
I also found similar issues in my case. I’m attempting to run ray within a capsule of the job scheduler we have here. Even if I limit the num\_cpus and memory ray should have access to ```auto ray.init(num_cpus=2, _memory=1.2e+11) # Hard coded for a capsule ``` It still spawns a huge number of threads per process (~1300): ![Screenshot from 2023-09-06 11-47-15](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/d/d8e8c0d2c83e9eccb3dbac3ee03d6bce3b4a1ec9.png) Even though these processes are likely not to be scheduled and do not consume a lot of memory, I worry about the overhead I might cause to the job scheduler, especially because I plan to run multiple trials and the total # of threads is just going to steadily increase. Is there an environment variable I can set? Or a cheap trick to avoid it (even if I have to hard-code it)? ---
**Author:** ![beantowel](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/beantowel/32/5253_2.png) [@beantowel](https://discuss.ray.io/u/beantowel)\ **Post date:** [November 2, 2023, 3:21am UTC](https://discuss.ray.io/t/too-many-threads-in-ray-worker/10881/23 "2023-11-02T03:21:17Z")
Ray creates too many threads. Since the user’s default ulimit of NPROC is usually 4096, it breaks other processes of the same user, I can’t even open a `top` command in the shell when Ray is running 32 tasks \* 100+ threads per task. ---
**Author:** ![max\_ronda](https://avatars.discourse-cdn.com/v4/letter/m/f0a364/32.png) [@max\_ronda](https://discuss.ray.io/u/max_ronda)\ **Post date:** [November 22, 2023, 6:18pm UTC](https://discuss.ray.io/t/too-many-threads-in-ray-worker/10881/24 "2023-11-22T18:18:55Z")
Has there been any work around this? I am encountering this issue using Ray rllib. I was not able to ssh into the machine running my job and thought I was just using too many rollout workers, but even after reducing the number of rollout workers, ray is keeping me from ssh’ing into that machine. It is very strange because the ray dashboard seems okay. Any solutions around this? ---
**Author:** ![messense](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/messense/32/4519_2.png) [@messense](https://discuss.ray.io/u/messense)\ **Post date:** [May 29, 2024, 2:17am UTC](https://discuss.ray.io/t/too-many-threads-in-ray-worker/10881/25 "2024-05-29T02:17:17Z")
I have found a workaround: [[Core] Too many threads in ray worker · Issue #36936 · ray-project/ray · GitHub](https://github.com/ray-project/ray/issues/36936#issuecomment-2134496892) Set `RAY_num_server_call_thread` env var to a smaller value. ---
**Author:** ![thvaisa](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/thvaisa/32/6359_2.png) [@thvaisa](https://discuss.ray.io/u/thvaisa)\ **Post date:** [January 8, 2025, 9:38pm UTC](https://discuss.ray.io/t/too-many-threads-in-ray-worker/10881/26 "2025-01-08T21:38:10Z")
Unfortunately, this might cause another round of refactoring for me. Single ray worker (running ray actor) had 110 threads and even after RAY\_num\_server\_call\_thread=1 about 55 threads. This is too much because we have 120 actors on a single server. We noticed that something was wrong when a 5-second task sent to a single actor took 10-15 seconds occasionally, whereas the same task was 5 seconds with a Dask equivalent cluster. I was able to get stable 5-second tasks when cutting ray actors to just 1 and launching individual tasks inside the actor with Python’s own multiprocessing. The task is external library call that does heavy matrix operations. I suspect the huge number of threads causes context switches and trashing of caches which has a detrimental effect on our algorithm. I have tried to bind cores, niceness, holding gil lock, play with NUMA (we have 4 sockets in a server). If anyone knows a way to reduce the number of threads even more, I would be interested in knowing about it. [Previous page](https://discuss.ray.io/t/too-many-threads-in-ray-worker/10881.md?page=1)