Hi,
I’m running Ray in a Docker Swarm. The head node stays alive as expected, but all workers die off within 1 minute of process start. For the one minute while they are alive, they are reporting correctly to Head, and I can see them in the dashboard.
When they exit, the exit code is always 1, and the logs contain no errors.
This is how the Ray Worker is started:
entrypoint: [ ‘/usr/local/bin/ray’]
command: [‘start’, ‘–address’, ‘tasks.ray-head:6379’, ‘–node-ip-address’, ‘tasks.ray-worker’, ‘–min-worker-port’, ‘20000’, ‘–verbose’, ‘–max-worker-port’, ‘30000’, ‘–object-manager-port’, ‘12345’, ‘–node-manager-port’, ‘12346’, ‘–block’]
Does anybody know how do I debug this? There’s nothing in /tmp/ray logs that can help guide on what’s the problem.
Thanks!