We’re running Ray with external Redis for GCS persistence and start the head with ray start --head --redis-password <...> (and --redis-username default where applicable). Can you confirm the connection/auth behavior below?
Does the Ray head (GCS / core) establish a long-lived/persistent TCP connection to the external Redis that stays open indefinitely (e.g., via keepalive) until the Ray process/pod dies, assuming the network remains healthy?
On initial connect, does Ray send AUTH (e.g., AUTH default <password> or equivalent) once per TCP connection, and then rely on that authenticated session for the lifetime of the connection (i.e., no periodic re-AUTH)?
If the TCP connection drops (idle timeout, failover, network blip), does Ray automatically reconnect and re-AUTH using the configured password/username?
If there’s a doc link in Ray you recommend citing for this, please share.
Yes, Ray’s GCS (head node) establishes persistent TCP connections to the external Redis for both synchronous and asynchronous operations, and these connections remain open as long as the process is alive and the network is healthy. On initial connection, Ray sends the AUTH command (e.g., AUTH default <password>) once per TCP connection, and the authenticated session is used for the lifetime of that connection—there is no periodic re-AUTH unless the connection is lost and re-established. If the TCP connection drops (due to idle timeout, failover, or network issues), Ray will automatically attempt to reconnect and re-AUTH using the configured username and password. This reconnection logic is implemented to ensure continued operation after transient failures or Redis failover events.
You can see this behavior in the Ray source code, where the connection and authentication logic is handled in the RedisContext class, and reconnection (with re-AUTH) is triggered if the connection is lost. For more details, see redis_context.cc and the related pull request discussion. Official documentation on GCS fault tolerance and external Redis setup is available here.
Would you like a step-by-step breakdown of the connection and reconnection logic?
Yes, Ray head (GCS / core) establishes a long-lived/persistent TCP connection to the external Redis that stays open indefinitely.
Yes, Ray sends AUTH once per TCP connection and then relies on that authenticated session for the lifetime of the connection, and there is no periodic re-AUTH.
No, Ray doesn’t reconnect and re-AUTH. It takes GCS down and relies on external orchestration to restart the GCS again.