Ray Cluster tutorial on AWS cluster ran into IP issue unexpectedly

Ray Clusters
1

Doing a Ray cluster tutorial as per this

With this config

# Cloud-provider specific configuration.
provider:
    type: aws
    region: us-east-1
    security_group:
        GroupName: ray_client_security_group
        IpPermissions:
              - FromPort: 10001
                ToPort: 10001
                IpProtocol: TCP
                IpRanges:
                    # This will enable inbound access from ALL IPv4 addresses.
                    - CidrIp: 0.0.0.0/0
              - FromPort: 30000
                ToPort: 50000
                IpProtocol: TCP
                IpRanges:
                    # This will enable inbound access from ALL IPv4 addresses.
                    - CidrIp: 0.0.0.0/0

max_workers: 3

available_node_types:
  ray.head.default:
      node_config:
        InstanceType: t2.micro
        BlockDeviceMappings:
            - DeviceName: /dev/sda1
              Ebs:
                  VolumeSize: 100
      resources: {"CPU": 2}
  ray.worker.default:
      node_config:
        InstanceType: t2.micro
        InstanceMarketOptions:
            MarketType: spot
      resources: {"CPU": 2}
      min_workers: 3

and run ray up see the instances up and running

image
image989×226 28 KB

ran ray monitor on it nothing unmarkable show up

Resources
---------------------------------------------------------------
Usage:
 0.0/8.0 CPU
 0.00/2.455 GiB memory
 0.00/0.783 GiB object_store_memory

Demands:
 (no resource demands)

==> /tmp/ray/session_latest/logs/monitor.out <==
2022-05-22 01:06:53,606 INFO log_timer.py:30 -- AWSNodeProvider: Set tag ray-node-status=up-to-date on ['i-0f9b73ee0deafdfee', 'i-0d4b03c3f68f4d512', 'i-0262137b5ca046257']  [LogTimer=245ms]
2022-05-22 01:06:53,727 INFO log_timer.py:30 -- AWSNodeProvider: Set tag ray-runtime-config=16edc8dc4ea5834b481e1d61ff93998df246d490 on ['i-0f9b73ee0deafdfee', 'i-0d4b03c3f68f4d512', 'i-0262137b5ca046257']  [LogTimer=121ms]
2022-05-22 01:06:53,727 INFO updater.py:189 -- New status: up-to-date
2022-05-22 01:06:53,728 INFO updater.py:189 -- New status: up-to-date
2022-05-22 01:06:53,727 INFO updater.py:189 -- New status: up-to-date
^C

However when I them submit a job locally pointing to the head ray.init(address='ray://18.207.255.90:10001')

I get the following error

  File "python/ray/includes/global_state_accessor.pxi", line 162, in ray._raylet.GlobalStateAccessor.get_node_to_connect_for_driver
RuntimeError: b"This node has an IP address of 172.31.77.110, and Ray expects this IP address to be either the GCS address or one of the Raylet addresses. Connected to GCS at 172.31.77.110 and found raylets at 172.31.77.202, 172.31.73.36, 172.31.79.43 but none of these match this node's IP 172.31.77.110. Are any of these actually a different IP address for the same node?You might need to provide --node-ip-address to specify the IP address that the head should use when sending to this node."

Shared connection to 18.207.255.90 closed.
Error: Command failed:

I also have port 30K to 50K open

Any pointer apprecaited