Started ray cluster with status saying it's up but can't connect

Ray Core
1

Hi,

  • I’m running a ray cluster locally by called ray start --head
  • The ray runtime starts
  • I give it some time and run ray status and see the details to confirm it’s running
  • I run a line to start my python script to connect to the server
  • I see Connecting to existing ray cluster at address : x.x.x.x:6379 but also
  • WARNING worker.py:1578 -- SIGTERM handler is not set because current thread is not the main thread.
  • global_state_accessor.cc:506: Some processes that the driver needs to connect to have not registered with GCS, so retrying. Have you run 'ray start' on this node?

I’ve tried the following fixes:

  • Stopped the ray cluster and restarted
  • Cleared the tmp folder
  • Check ray cluster status after starting the cluster using ray status (it returns info that suggests it is)
  • Checked if ray is initialized using ray._is_ray_initialized (it isn’t)
  • Checked if I can connect to the ip address + ports (I can successfully)
  • Checked firewall problems (not an issue)

So I’m very confused why it says ray isn’t initialized when the status suggests it is.

Is there a way I can debug this easily? I’m used to the vscode python debugger but open to alternative methods!

Any help would greatly be appreciated!

2

Found the bug, closing thread

3

@MRay for posterity, what did the issue end up being?

4

Ray cluster was running 2.41.x and not the latest ray version (a ray update fixed it)

This caused the OS (latest macOS) to conflict

There was no debugging message (that I’m aware of) that easily flagged it