I’ve been successfully using Ray 1.5 for a little while. After upgrading to Ray 1.6 and 1.7 I receive the following error message when starting a Ray worker:
global_state_accessor.cc:394: This node has an IP address of ****, while we can not found the matched Raylet address. This maybe come from when you connect the Ray cluster with a different IP address or connect a container.
The worker is started from CLI as shown below. I’ve been playing with the --node-ip-address arg, but it seems to make no difference.
# --node-ip-address $(hostname -i)
Both head (server) and worker (client) run from within a singularity container. I believe this shouldn’t matter, it certainly hasn’t with Ray 1.5. I’d be grateful for any pointers how to debug this.