Client can't find Raylet address in Ray > 1.5

Hi Forum,

I’ve been successfully using Ray 1.5 for a little while. After upgrading to Ray 1.6 and 1.7 I receive the following error message when starting a Ray worker:

global_state_accessor.cc:394: This node has an IP address of ****, while we can not found the matched Raylet address. This maybe come from when you connect the Ray cluster with a different IP address or connect a container.

The worker is started from CLI as shown below. I’ve been playing with the --node-ip-address arg, but it seems to make no difference.

ray start
–redis-password ${RAY_REDIS_PASSWORD}
–address ****:${RAY_PORT}
# --node-ip-address $(hostname -i)
–block

Both head (server) and worker (client) run from within a singularity container. I believe this shouldn’t matter, it certainly hasn’t with Ray 1.5. I’d be grateful for any pointers how to debug this.

Thanks.