Thanks for pointing me in the right direction Christina!
I had to allow connections on port 6379 for ray status --address 10.212.154.239:6379 to work
Curiously, I also had to allow connections on port 41823 for ray memory --address 10.212.154.239:6379 to work. However, it seems this port is chosen randomly at start, so after restarting my cluster the command failed again. Do you know perhaps which of the ray start options mentioned here would control this port so I can make it constant?
I should say it’s a rather unexpected and impractical behaviour that ray status and ray job submit expect address in a different format even though they will both use the address from the same environment variable if not given on command line. This makes it impossible to avoid the --address option for a subset of commands.
Is someone finds that helpful, as an alternative to specifying --address, I noticed all of these commands work without --address option or RAY_ADDRESS env var when run on the head node, which can be done from outside the cluster via ray exec, for example ray exec config.yaml 'ray status'. Some may find this more convenient than having to remember what to pass as --address.
Thank you for your help finding workarounds to my troubles!