Unable to connect with ray cluster

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

When trying to connect with remote on-premise ray cluster we are receiving Connection timeout error from ray client,

Traceback (most recent call last):
File “/home/test.py”, line 2, in
File “/home/ray/lib/python3.9/site-packages/ray/_private/client_mode_hook.py”, line 103, in wrapper
return func(*args, **kwargs)
File “/home/lib/python3.9/site-packages/ray/_private/worker.py”, line 1379, in init
ctx = builder.connect()
File “/home/ray/lib/python3.9/site-packages/ray/client_builder.py”, line 173, in connect
client_info_dict = ray.util.client_connect.connect(
File “/home/ray/lib/python3.9/site-packages/ray/util/client_connect.py”, line 55, in connect
conn = ray.connect(
File “/home/ray/lib/python3.9/site-packages/ray/util/client/init.py”, line 250, in connect
conn = self.get_context().connect(*args, **kw_args)
File “/home/ray/lib/python3.9/site-packages/ray/util/client/init.py”, line 92, in connect
self.client_worker = Worker(
File “/home/ray/lib/python3.9/site-packages/ray/util/client/worker.py”, line 139, in init
File “/home/ray/lib/python3.9/site-packages/ray/util/client/worker.py”, line 260, in _connect_channel
raise ConnectionError(“ray client connection timeout”)
ConnectionError: ray client connection timeout

python version : 3.9.14
ray : 2.8.1

@Jules_Damji @architkulkarni @arturn