Actor died while training because of raylet lost connection

Ray Clusters Kubernetes
1

hey ,
we have a setup of Kubernetes cluster with kuberay
while training a model to some epochs actor died.
it is happening once in a while. neither found the issue nor was able to replicate the issue.
Find the attached cluster logs
ray_logs.txt