I am currently testing a hybrid infrastructure where the head node of the ray cluster is running on an AWS EC2 instance and the worker node is my local computer.
When I connect the worker node using the public IP address of the EC2 head node, no error message appears but when I use the “ray status” command, I can only see the head node. The connection to the cluster seems to be working because “ray status” on my local computer returns the correct resources of the head node, but nothing about my local worker node.
Also, I can successfully connect to the cluster with a python application using the “ray.init(address=…)” command and I can see both the head node AND worker node when I run the “ray.nodes()” python command. But again, running the distributed app only result in the head node working and not the worker node.
Could it be an issue with the autoscaler not seeing the worker?