Accessing Ray running on the cluster

I am running Ray cluster using operator rayproject/ray:nightly and nodes with image rayproject/ray:nightly-py37-cpu.
If I use port-forward I can happily connect to the cluster using:
ray.util.connect('127.0.0.1:10001')
But when I am trying to expose the service using either loadbalancer or nodeport, I am getting the following error:

Got Error from data channel – shutting down: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.UNKNOWN
details = “Exception iterating responses: "
debug_error_string = “{“created”:”@1617666164.593330000”,“description”:“Error received from peer ipv4:150.239.10.227:10001”,“file”:“src/core/lib/surface/call.cc”,“file_line”:1062,“grpc_message”:"Exception iterating responses: “,“grpc_status”:2}”

Exception in thread Thread-48:
Traceback (most recent call last):
File “/usr/local/Cellar/python@3.7/3.7.10_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py”, line 926, in _bootstrap_inner
self.run()
File “/usr/local/Cellar/python@3.7/3.7.10_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py”, line 870, in run
self._target(*self._args, **self._kwargs)
File “/usr/local/lib/python3.7/site-packages/ray/util/client/dataclient.py”, line 90, in _data_main
raise e
File “/usr/local/lib/python3.7/site-packages/ray/util/client/dataclient.py”, line 64, in _data_main
for response in resp_stream:
File “/usr/local/lib/python3.7/site-packages/grpc/_channel.py”, line 416, in next
return self._next()
File “/usr/local/lib/python3.7/site-packages/grpc/_channel.py”, line 706, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.UNKNOWN
details = “Exception iterating responses: "
debug_error_string = “{“created”:”@1617666164.593330000”,“description”:“Error received from peer ipv4:150.239.10.227:10001”,“file”:“src/core/lib/surface/call.cc”,“file_line”:1062,“grpc_message”:"Exception iterating responses: “,“grpc_status”:2}”

I think this is similar to the: Exposing dashboard, client and ray serve in kubernetes through NodePort

It looks like the client does not like going through the intermediary

Oh it seem to be broken even for the port-forward now. Do I need to update client? I have the one installed using https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-2.0.0.dev0-cp37-cp37m-macosx_10_13_intel.whl about a week ago

And this was it. It works now. Sorry about confusion

1 Like