How to handle `Connection closed by server` gracefully

Ray Core
1

When the ray-head is stopped the clients that are connected to it print out some error messages and then exits with error code 1.

2021-09-28 09:45:46,184 INFO worker.py:800 -- Connecting to existing Ray cluster at address: 10.0.0.63:6379
2021-09-28 09:45:48,184 WARNING worker.py:1189 -- The autoscaler failed with the following error:
Terminated with signal 15
  File "/home/birger/.local/lib/python3.8/site-packages/ray/autoscaler/_private/monitor.py", line 416, in <module>
    monitor.run()
  File "/home/birger/.local/lib/python3.8/site-packages/ray/autoscaler/_private/monitor.py", line 317, in run
    self._run()
  File "/home/birger/.local/lib/python3.8/site-packages/ray/autoscaler/_private/monitor.py", line 234, in _run
    time.sleep(AUTOSCALER_UPDATE_INTERVAL_S)

2021-09-28 09:45:48,204 ERROR worker.py:1191 -- listen_error_messages_raylet: Connection closed by server.
2021-09-28 09:45:48,204 ERROR worker.py:468 -- print_logs: Connection closed by server.
2021-09-28 09:45:48,205 ERROR import_thread.py:88 -- ImportThread: Connection closed by server.

I would like to handle this error in our code, such that the application can exit gracefully. Is there any way in which that can be done?

2

Do you want to give something like this a try

import ray
import time
try:
ā”Š   with ray.init("ray://"):
ā”Š   ā”Š   time.sleep(10)
except ConnectionError as e :
ā”Š   print("ERROR", e)
1 Like
3

Thanks :pray: That did indeed work :smiley: :clap:

1 Like