# Increasing amount of Backup\_poller errors

**URL:** <https://discuss.ray.io/t/increasing-amount-of-backup-poller-errors/7203>\
**Category:** Ray Core\
**Created:** [August 15, 2022, 7:26pm UTC](https://discuss.ray.io/t/increasing-amount-of-backup-poller-errors/7203 "2022-08-15T19:26:59Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![andyjh122](https://avatars.discourse-cdn.com/v4/letter/a/bbe5ce/32.png) [@andyjh122](https://discuss.ray.io/u/andyjh122)\
**Post date:** [August 15, 2022, 7:26pm UTC](https://discuss.ray.io/t/increasing-amount-of-backup-poller-errors/7203/1 "2022-08-15T19:26:59Z")

</div>

Hi, I’m running optuna with Ray 1.13, grpc 1.43.0 and getting increasing number of backup\_poller errors that eventually hang my trials.

Also it seems like over time training slows down significantly.

I’ve looked in Ray issues didn’t see anything specifically related, wondering if anyone has seen this.

Thank you!  
Andrew

E0815 14:26:06.967181484 1842238 backup\_poller.cc:134] Run client channel backup poller: {“created”:“@1660587966.967090153”,“description”:“pollset\_work”,“file”:“src/core/lib/iomgr/ev\_epollex\_linux.cc”,“file\_line”:320,“referenced\_errors”:[{“created”:“@1660587966.967085911”,“description”:“Bad file descriptor”,“errno”:9,“file”:“src/core/lib/iomgr/ev\_epollex\_linux.cc”,“file\_line”:950,“os\_error”:“Bad file descriptor”,“syscall”:“epoll\_wait”}]}

---

<div class="post-metadata">

**Author:** ![jjyao](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jjyao/32/1799_2.png) [@jjyao](https://discuss.ray.io/u/jjyao)\
**Post date:** [August 17, 2022, 12:13am UTC](https://discuss.ray.io/t/increasing-amount-of-backup-poller-errors/7203/2 "2022-08-17T00:13:46Z")

</div>

Do you have a repro script?

---

<div class="post-metadata">

**Author:** ![andyjh122](https://avatars.discourse-cdn.com/v4/letter/a/bbe5ce/32.png) [@andyjh122](https://discuss.ray.io/u/andyjh122)\
**Post date:** [August 18, 2022, 6:08pm UTC](https://discuss.ray.io/t/increasing-amount-of-backup-poller-errors/7203/3 "2022-08-18T18:08:10Z")

</div>

Thank you jjyao! I’ll put a repro script together.

I managed to trace this to be specific to agent train.trainer.

Train trainer spins up TensorboardX process, ray util, and QueueFeederThread but it seems like ray.shutdown() after a train does not shut it down until the main process ends, since we are doing a training trial loop with optuna (each trial loop has a ray.init and ray.shutdown, these 3 processes never close on shutdown), and when trainer is called again a new set of three threads are spanned, dont get shut down and fork bomb.

Andrew

---

<div class="post-metadata">

**Author:** ![jjyao](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jjyao/32/1799_2.png) [@jjyao](https://discuss.ray.io/u/jjyao)\
**Post date:** [August 19, 2022, 7:33am UTC](https://discuss.ray.io/t/increasing-amount-of-backup-poller-errors/7203/4 "2022-08-19T07:33:44Z")

</div>

@xwjiang2010 seems a ML question.

---

<div class="post-metadata">

**Author:** ![xwjiang2010](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/xwjiang2010/32/1476_2.png) [@xwjiang2010](https://discuss.ray.io/u/xwjiang2010)\
**Post date:** [August 19, 2022, 5:09pm UTC](https://discuss.ray.io/t/increasing-amount-of-backup-poller-errors/7203/5 "2022-08-19T17:09:01Z")

</div>

yeah a repro script would be helpful. happy to take a look then!

---

<div class="post-metadata">

**Author:** ![andyjh122](https://avatars.discourse-cdn.com/v4/letter/a/bbe5ce/32.png) [@andyjh122](https://discuss.ray.io/u/andyjh122)\
**Post date:** [August 23, 2022, 8:06pm UTC](https://discuss.ray.io/t/increasing-amount-of-backup-poller-errors/7203/6 "2022-08-23T20:06:20Z")

</div>

Working on getting a repro script together.

We were able to fix the two leaking threads by:

1. Uninstalling tensorboardX
2. setting “log\_sys\_usage” to False

ray/utils.py was the offending code at least on #2, is ignoring ray.shutdown

Maybe something with psutils?