I have two node ray cluster version 1.12.1. On red hat machines.
When starting the worker node. The dashboard disappears and i get this error
022-08-16 12:09:05,121 ERROR node_head.py:259 – Error updating node stats of 62316dd4e5a2fb61c9bf6de71b4b346de80594b2e3559ad36929c074.
Traceback (most recent call last):
File “/home/elevoai/inst/elevo-0.0.1/elevo_venv/lib64/python3.8/site-packages/ray/dashboard/modules/node/node_head.py”, line 250, in _update_node_stats
reply = await stub.GetNodeStats(
File “/home/elevoai/inst/elevo-0.0.1/elevo_venv/lib64/python3.8/site-packages/grpc/aio/_call.py”, line 290, in await
raise _create_rpc_error(self._cython_call._initial_metadata,
grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = “failed to connect to all addresses”
debug_error_string = “{“created”:”@1660676945.121393603",“description”:“Failed to pick subchannel”,“file”:“src/core/ext/filters/client_channel/client_channel.cc”,“file_line”:3134,“referenced_errors”:[{“created”:“@1660676945.121392625”,“description”:“failed to connect to all addresses”,“file”:“src/core/lib/transport/error_utils.cc”,“file_line”:163,“grpc_status”:14}]}"
gcs log continuosly shows
gcs log shows
Error message: failed to connect to all addresses; RPC Error details:
[2022-08-16 12:09:47,066 I 653273 653273] (gcs_server) gcs_resource_report_poller.cc:143: Couldn’t get resource request from raylet 62316dd4e5a2fb61c9bf6de71b4b346de80594b2e3559ad36929c074: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details:
[2022-08-16 12:09:47,167 I 653273 653273] (gcs_server) gcs_resource_report_poller.cc:143: Couldn’t get resource request from raylet 62316dd4e5a2fb61c9bf6de71b4b346de80594b2e3559ad36929c074: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details:
Is it a connectivity issue?