Yes, cloning disk images across nodes can cause issues with Ray due to duplicated machine identifiers (like /etc/machine-id), network hostnames, and possibly MAC addresses. These duplicates confuse Ray’s GCS and heartbeat system, leading to node dropouts and registration failures, as you observed. Each Ray node must have a unique machine identity and network configuration for stable operation. This is a common pitfall when deploying clusters from cloned images, as confirmed by similar reports in the Ray community and general distributed systems best practices.
To resolve this, after flashing the image but before starting Ray, you should:
- Regenerate
/etc/machine-idon each node (e.g.,sudo rm /etc/machine-id && sudo systemd-machine-id-setup). - Ensure each node has a unique hostname (
sudo hostnamectl set-hostname <unique-name>). - Check that network interfaces have unique MAC addresses (if not, regenerate or manually assign).
- Reboot each node after these changes.
This will ensure Ray can distinguish nodes and maintain healthy heartbeats and GCS registration. Would you like more detail or a step-by-step script for these cleanup steps?
Sources:
- https://github.com/ray-project/ray/issues/44431
- https://discuss.ray.io/t/local-object-store-on-worker-nodes-not-working-worker-plasma-stays-at-0/6676
Hint: Mention @Herald in the post for followups.