How to handle `Connection closed by server` gracefully

When the ray-head is stopped the clients that are connected to it print out some error messages and then exits with error code 1.

2021-09-28 09:45:46,184 INFO -- Connecting to existing Ray cluster at address:
2021-09-28 09:45:48,184 WARNING -- The autoscaler failed with the following error:
Terminated with signal 15
  File "/home/birger/.local/lib/python3.8/site-packages/ray/autoscaler/_private/", line 416, in <module>
  File "/home/birger/.local/lib/python3.8/site-packages/ray/autoscaler/_private/", line 317, in run
  File "/home/birger/.local/lib/python3.8/site-packages/ray/autoscaler/_private/", line 234, in _run

2021-09-28 09:45:48,204 ERROR -- listen_error_messages_raylet: Connection closed by server.
2021-09-28 09:45:48,204 ERROR -- print_logs: Connection closed by server.
2021-09-28 09:45:48,205 ERROR -- ImportThread: Connection closed by server.

I would like to handle this error in our code, such that the application can exit gracefully. Is there any way in which that can be done?

Do you want to give something like this a try

import ray
import time
ā”Š   with ray.init("ray://"):
ā”Š   ā”Š   time.sleep(10)
except ConnectionError as e :
ā”Š   print("ERROR", e)

1 Like

Thanks :pray: That did indeed work :smiley: :clap:

1 Like