Hi, we have a Kubernedes cluster. Our YAML file specifies a “kind: Deployment” which is a docker container with our flask app and a “kind: RayCluster” which is a Ray cluster with a head and two workers. We are trying to ray.init from our flask app to the Ray cluster. I have been able to connect to the Ray cluster by running “kubectl exec -it [FLASK POD NAME] – /bin/bash”, then “python”, then “import ray”, “ray.init(address=‘[RAY IP OF RAY HEAD]:6379’, _node_ip_address=‘[RAY IP OF RAY HEAD]’)” (I have tried connecting through a service or without _node_ip_address and neither works, but I just want to connect right now in any way that works). However, from my actually flask app inside my Docker container, with the same ray.init command I get this error:
(base) kubectl logs [POD NAME]
- Tip: There are .env or .flaskenv files present. Do “pip install python-dotenv” to use them.
- Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead. - Debug mode: off
2023-06-28 02:54:51,158 INFO worker.py:1452 – Connecting to existing Ray cluster at address: [IP]…
2023-06-28 02:54:51,170 INFO worker.py:1627 – Connected to Ray cluster. View the dashboard at [IP]
[2023-06-28 02:55:01,218 C 7 7] raylet_client.cc:60: Could not connect to socket /tmp/ray/session_2023-06-27_06-11-52_301872_8/sockets/raylet
*** StackTrace Information ***
/usr/local/lib/python3.8/site-packages/ray/_raylet.so(+0xdc551a) [0x7f150e97051a] ray::operator<<()
/usr/local/lib/python3.8/site-packages/ray/_raylet.so(+0xdc7002) [0x7f150e972002] ray::SpdLogMessage::Flush()
/usr/local/lib/python3.8/site-packages/ray/_raylet.so(_ZN3ray6RayLogD1Ev+0x37) [0x7f150e972317] ray::RayLog::~RayLog() // more stack trace lines.
Any ideas? I am using Ray 2.5.1, Python 3.8.13.