How severe does this issue affect your experience of using Ray?
- High: It blocks me to complete my task.
I have the following env vars for both Ray head and worker K8S yamls by following the TLS section in the Ray doc.
- name: RAY_USE_TLS
value: "1"
- name: RAY_TLS_SERVER_CERT
value: "/etc/tls/tls.crt"
- name: RAY_TLS_SERVER_KEY
value: "/etc/tls/tls.key"
- name: RAY_TLS_CA_CERT
value: "/etc/tls/ca.crt"
However, the head gave me an error that
details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:10.0.38.74:6379: Peer name 10.0.38.74 is not in peer certificate"
debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:10.0.38.74:6379: Peer name 10.0.38.74.118 is not in peer certificate {grpc_status:14, created_time:“2023-03-10T10:08:02.43567248-08:00"}}"
I have added in SAN:
DNS:localhost and IP:127.0.0.1
In the above logs, it tries to connect using pod IP (10.0.38.74). But we cannot define in SAN pod IP because the pod ip is dynamic and will be changed during the k8s restart. Also, we cannot use K8S cert manager.
Wondering if there is any detailed example or solution to solve this issue.