Hi Team,
we are trying to run Ray on one of the containers in AWS ECS Fargate. We wanted to connect to the ray container from another container(rest).
Ray container started well. the below is the log
2023-03-22T17:48:26.110+05:30 2023-03-22 12:18:21,888 INFO usage_lib.py:516 – Usage stats collection is enabled by default without user confirmation because this terminal is detected to be non-interactive. To disable this, add --disable-usage-stats to the command that starts the cluster, or run the following command: ray disable-usage-stats before starting the cluster. See Usage Stats Collection — Ray 3.0.0.dev0 for more details.
2023-03-22T17:48:26.110+05:30 2023-03-22 12:18:21,888 INFO scripts.py:702 – Local node IP: 10.X.X.X
2023-03-22T17:48:26.110+05:30 2023-03-22 12:18:26,108 SUCC scripts.py:739 – --------------------
2023-03-22T17:48:26.110+05:30 2023-03-22 12:18:26,108 SUCC scripts.py:740 – Ray runtime started.
2023-03-22T17:48:26.110+05:30 2023-03-22 12:18:26,108 SUCC scripts.py:741 – --------------------
2023-03-22T17:48:26.110+05:30 2023-03-22 12:18:26,108 INFO scripts.py:743 – Next steps
2023-03-22T17:48:26.110+05:30 2023-03-22 12:18:26,108 INFO scripts.py:744 – To connect to this Ray runtime from another node, run
2023-03-22T17:48:26.110+05:30 2023-03-22 12:18:26,108 INFO scripts.py:747 – ray start --address=‘10.186.166.231:6379’
2023-03-22T17:48:26.110+05:30 2023-03-22 12:18:26,108 INFO scripts.py:763 – Alternatively, use the following Python code:
2023-03-22T17:48:26.110+05:30 2023-03-22 12:18:26,108 INFO scripts.py:765 – import ray
2023-03-22T17:48:26.110+05:30 2023-03-22 12:18:26,109 INFO scripts.py:769 – ray.init(address=‘auto’)
2023-03-22T17:48:26.110+05:30 2023-03-22 12:18:26,109 INFO scripts.py:781 – To connect to this Ray runtime from outside of the cluster, for example to
2023-03-22T17:48:26.110+05:30 2023-03-22 12:18:26,109 INFO scripts.py:785 – connect to a remote cluster from your laptop directly, use the following
2023-03-22T17:48:26.110+05:30 2023-03-22 12:18:26,109 INFO scripts.py:789 – Python code:
2023-03-22T17:48:26.110+05:30 2023-03-22 12:18:26,109 INFO scripts.py:791 – import ray
2023-03-22T17:48:26.110+05:30 2023-03-22 12:18:26,109 INFO scripts.py:792 – ray.init(address=‘ray://<head_node_ip_address>:10001’)
2023-03-22T17:48:26.110+05:30 2023-03-22 12:18:26,109 INFO scripts.py:801 – To see the status of the cluster, use
2023-03-22T17:48:26.110+05:30 2023-03-22 12:18:26,109 INFO scripts.py:802 – ray status
2023-03-22T17:48:26.110+05:30 2023-03-22 12:18:26,109 INFO scripts.py:812 – If connection fails, check your firewall settings and network configuration.
2023-03-22T17:48:26.110+05:30 2023-03-22 12:18:26,109 INFO scripts.py:820 – To terminate the Ray runtime, run
The connection to the Ray container from rest container is the below code:-(getting the ips of the containers and initialising it)
ip_address=‘10.X.X.X’
ray_address = “ray://” + ip_address+“:10001”
print(ray_address)
try:
ray.init(address=ray_address)
print(“ray initialized”)
except Exception as e:
print(“ray initialization failed”,e)
when the above piece of code is run in rest container, we get the below error:
ray initialization failed ray client connection timeout
Not sure what is happening here. any suggestion or help is appreciated.
it works in local machine in containers but not on the ECS fargate.
Ray version 2.2