Hello,
I’m trying to set up Ray on AWS but I have a problem with tags. In my case all instances should have a few tags otherwise they are terminated. The problem is with the TTL. I set it up via TagSpecifications but Ray changes it with stage/state information what results in termination. How can I change this behavior in Ray (disable changes of TTL tag or add some prefix??? so Ray captures those changes in for example RAY_TTL). In my case TTL can only have “forever” value, anything else results in termination.
Can you share your cluster config, and the tags that you’re seeing on the instance after Ray changes it? Attempting to repro
https://www.dropbox.com/s/t158gr4kwpxr6ij/ray-ec2-vdil.yaml?dl=0
it’s about TTL flag. It changes from forever to different values like “waiting-for-ssh”, …, “2 weeks”.
Attempted to repro with head node and worker node, looks like TTL (and other tags) are still intact.
Can you try checking your instances for the tag ray-node-status (this is the tag that should be updated to values like “waiting-for-ssh”)? Also checked through the ray codebase and “2 weeks” isn’t one of the possible values that can be assigned to a tag, so that may be coming from something else in your environment.