[Clusters] Preemptible machines stop unexpectedly in GCP

Hi Ray team!

I am experiencing a problem when using preemptible machines in GCP.
Workers seem to stop unexpectedly with the head presenting this message

The node with node id: NODE and ip: IP has been marked dead because the detector has missed too many heartbeats from it. This can happen when a raylet crashes unexpectedly or has lagging heartbeats.

This was solved when I moved to using non-preemptible.
Are you familiar with this problem? Anything I can do about this?

Thank you!

Hmm, I haven’t seen this issue before; maybe GCP changes your machine IP addresses randomly?