# Autoscaler does not seem to watch head node

**URL:** <https://discuss.ray.io/t/autoscaler-does-not-seem-to-watch-head-node/1410>\
**Category:** Kubernetes\
**Created:** [March 26, 2021, 4:26pm UTC](https://discuss.ray.io/t/autoscaler-does-not-seem-to-watch-head-node/1410 "2021-03-26T16:26:56Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![blublinsky](https://avatars.discourse-cdn.com/v4/letter/b/f05b48/32.png) [@blublinsky](https://discuss.ray.io/u/blublinsky)\
**Post date:** [March 26, 2021, 4:26pm UTC](https://discuss.ray.io/t/autoscaler-does-not-seem-to-watch-head-node/1410/1 "2021-03-26T16:26:56Z")

</div>

I have installed k8 Ray cluster using Ray up.  
In one of my experiments a head node failed due to out of memory. It looks like nothing is watching this node, because, it keeps sitting in this state forever. Should autoscaler watch this node as well?  
Also Ray down, fails to clean up any pod not in ready state. This pods have to be deleted manually.

Are those bugs?

---

<div class="post-metadata">

**Author:** ![blublinsky](https://avatars.discourse-cdn.com/v4/letter/b/f05b48/32.png) [@blublinsky](https://discuss.ray.io/u/blublinsky)\
**Post date:** [March 26, 2021, 5:06pm UTC](https://discuss.ray.io/t/autoscaler-does-not-seem-to-watch-head-node/1410/2 "2021-03-26T17:06:11Z")

</div>

Also, ray down does not remove the service associated with head pod. Also Ray up sometimes does create this service, but sometimes it does not.

---

<div class="post-metadata">

**Author:** ![eoakes](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/eoakes/32/7825_2.png) [@eoakes](https://discuss.ray.io/u/eoakes)\
**Post date:** [March 26, 2021, 5:54pm UTC](https://discuss.ray.io/t/autoscaler-does-not-seem-to-watch-head-node/1410/3 "2021-03-26T17:54:12Z")

</div>

@Dmitri could you clarify the behavior here please?

---

<div class="post-metadata">

**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\
**Post date:** [March 26, 2021, 6:27pm UTC](https://discuss.ray.io/t/autoscaler-does-not-seem-to-watch-head-node/1410/4 "2021-03-26T18:27:44Z")

</div>

> [@blublinsky](#):
>
> Also, ray down does not remove the service associated with head pod. Also Ray up sometimes does create this service, but sometimes it does not.

Currently, Ray down does not delete the service and Ray up creates the service if it’s not already present.  
There’s an issue open to change this behavior so that Ray down deletes the service

> <https://github.com/ray-project/ray/issues/14700>
>
> When creating a cluster on Kubernetes, Ray will allocate a service routing traff…ic to the head node when the user adds this to their cluster config:
> 
> \`\`\`
> services:
> - apiVersion: v1
> kind: Service
> metadata:
> name: local-cluster-ray-head
> spec:
> selector:
> component: local-cluster-ray-head
> ports:
> - name: client
> protocol: TCP
> port: 10001
> targetPort: 10001
> - name: dashboard
> protocol: TCP
> port: 8265
> targetPort: 8265
> \`\`\`
> 
> However, when calling \`ray down cluster.yaml\`, this service (unlike the pods) will not be removed. The expected behavior is that all resources created by \`ray up\` should be properly cleaned up after calling \`ray down\`.
> 
> cc @richardliaw

---

<div class="post-metadata">

**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\
**Post date:** [March 26, 2021, 6:35pm UTC](https://discuss.ray.io/t/autoscaler-does-not-seem-to-watch-head-node/1410/5 "2021-03-26T18:35:29Z")

</div>

This so

> [@blublinsky](#):
>
> I have installed k8 Ray cluster using Ray up.  
> In one of my experiments a head node failed due to out of memory. It looks like nothing is watching this node, because, it keeps sitting in this state forever. Should autoscaler watch this node as well?  
> Also Ray down, fails to clean up any pod not in ready state. This pods have to be deleted manually.
> 
> Are those bugs?

Ray down should remove all pods created by Ray up, regardless of their status. If this didn’t work for you, it would be great if you could file an issue on the Ray github with bug reproduction details!

Unfortunately, we currently don’t implement error handling to deal with Ray head failure. In fact, when launching clusters on Kubernetes with the cluster launcher (`ray up`), the autoscaler runs on the head node so there’s no way for the autoscaler to recover the head node.  
In the future, the Ray K8s Operator will implement sensible logic to deal with head failure –  
this issue is tracked here

> <https://github.com/ray-project/ray/issues/14726>
>
> \### What is the problem?
> \- Ray version: 1.2.0
> \- OS version: Ubuntu 20.04.2 LTS… (Focal Fossa)
> \- Python version: 3.7.7
> 
> \### Reproduction (REQUIRED)
> Example cluster config for ray operator:
> \`\`\`
> apiVersion: cluster.ray.io/v1
> kind: RayCluster
> metadata:
> namespace: ray
> name: kf-prod01
> spec:
> # The maximum number of workers nodes to launch in addition to the head node.
> maxWorkers: 3
> # The autoscaler will scale up the cluster faster with higher upscaling speed.
> # E.g., if the task requires adding more nodes then autoscaler will gradually
> # scale up the cluster in chunks of upscaling\_speed\*currently\_running\_nodes.
> # This number should be \> 0.
> upscalingSpeed: 1.0
> # If a node is idle for this many minutes, it will be removed.
> idleTimeoutMinutes: 5
> # Specify the pod type for the ray head node (as configured below).
> headPodType: head-node
> # Specify the allowed pod types for this ray cluster and the resources they provide.
> podTypes:
> - name: head-node
> # Minimum number of Ray workers of this Pod type.
> minWorkers: 0
> # Maximum number of Ray workers of this Pod type. Takes precedence over minWorkers.
> maxWorkers: 0
> podConfig:
> apiVersion: v1
> kind: Pod
> metadata:
> # Automatically generates a name for the pod with this prefix.
> generateName: ray-head-
> spec:
> restartPolicy: Never
> nodeSelector:
> cloud.google.com/gke-nodepool: default-pool
> # This volume allocates shared memory for Ray to use for its plasma
> # object store. If you do not provide this, Ray will fall back to
> # /tmp which cause slowdowns if is not a shared memory volume.
> volumes:
> - name: dshm
> emptyDir:
> medium: Memory
> containers:
> - name: ray-node
> imagePullPolicy: Always
> image: rayproject/ray:1.2.0
> # Do not change this command - it keeps the pod alive until it is
> # explicitly killed.
> command: \["/bin/bash", "-c", "--"\]
> args: \['trap : TERM INT; sleep infinity & wait;'\]
> ports:
> - containerPort: 6379 # Redis port
> - containerPort: 10001 # Used by Ray Client
> - containerPort: 12345 # Ray internal communication.
> - containerPort: 12346 # Ray internal communication.
> - containerPort: 8265 # Used by Ray Dashboard
> 
> # This volume allocates shared memory for Ray to use for its plasma
> # object store. If you do not provide this, Ray will fall back to
> # /tmp which cause slowdowns if is not a shared memory volume.
> volumeMounts:
> - mountPath: /dev/shm
> name: dshm
> resources:
> requests:
> cpu: 2000m
> memory: 4Gi
> limits:
> cpu: 2000m
> # The maximum memory that this pod is allowed to use. The
> # limit will be detected by ray and split to use 10% for
> # redis, 30% for the shared memory object store, and the
> # rest for application memory. If this limit is not set and
> # the object store size is not set manually, ray will
> # allocate a very large object store in each pod that may
> # cause problems for other pods.
> memory: 4Gi
> setupCommands:
> - pip install pipdate==0.5.2
> - name: worker-node
> # Minimum number of Ray workers of this Pod type.
> minWorkers: 2
> # Maximum number of Ray workers of this Pod type. Takes precedence over minWorkers.
> maxWorkers: 3
> # User-specified custom resources for use by Ray.
> # (Ray detects CPU and GPU from pod spec resource requests and limits, so no need to fill those here.)
> # rayResources: {"foo": 1, "bar": 1}
> podConfig:
> apiVersion: v1
> kind: Pod
> metadata:
> # Automatically generates a name for the pod with this prefix.
> generateName: ray-worker-
> spec:
> restartPolicy: Never
> nodeSelector:
> cloud.google.com/gke-nodepool: cpu-worker-pool01
> volumes:
> - name: dshm
> emptyDir:
> medium: Memory
> containers:
> - name: ray-node
> imagePullPolicy: Always
> image: rayproject/ray:1.2.0
> command: \["/bin/bash", "-c", "--"\]
> args: \["trap : TERM INT; sleep infinity & wait;"\]
> # This volume allocates shared memory for Ray to use for its plasma
> # object store. If you do not provide this, Ray will fall back to
> # /tmp which cause slowdowns if is not a shared memory volume.
> volumeMounts:
> - mountPath: /dev/shm
> name: dshm
> resources:
> requests:
> cpu: 2000m
> memory: 4Gi
> limits:
> cpu: 2000m
> # The maximum memory that this pod is allowed to use. The
> # limit will be detected by ray and split to use 10% for
> # redis, 30% for the shared memory object store, and the
> # rest for application memory. If this limit is not set and
> # the object store size is not set manually, ray will
> # allocate a very large object store in each pod that may
> # cause problems for other pods.
> memory: 4Gi
> setupCommands:
> - pip install --ignore-installed ruamel.yaml==0.16.12
> - pip install --no-cache-dir torch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0
> - pip install --no-cache-dir scipy==1.6.0 deprecated==1.2.11 gsutil==4.59 pytorch-lightning==1.1.8 ray\[tune\]==1.2.0
> #- pip install rollsroyce-0.0.1.post0.dev141+g301b4af-py2.py3-none-any.whl
> #- pip install --no-cache-dir pytorch-lightning==1.1.8
> # Commands to start Ray on the head node. You don't need to change this.
> # Note dashboard-host is set to 0.0.0.0 so that Kubernetes can port forward.
> headStartRayCommands:
> - ray stop
> - ulimit -n 65536; ray start --head --no-monitor --dashboard-host 0.0.0.0 --object-manager-port=12345 --node-manager-port=12346 --ray-client-server-port 10001
> # Commands to start Ray on worker nodes. You don't need to change this.
> workerStartRayCommands:
> - ray stop
> - ulimit -n 65536; ray start --address=$RAY\_HEAD\_IP:6379 --object-manager-port=12345 --node-manager-port=12346
> \---
> apiVersion: v1
> kind: Service
> metadata:
> namespace: ray
> name: ray-head
> spec:
> # This selector must match the head node pod's selector.
> selector:
> component: kf-prod01-ray-head
> ports:
> - name: client
> protocol: TCP
> port: 10001
> targetPort: 10001
> # Redis ports.
> - name: redis-primary
> port: 6379
> targetPort: 6379
> # Ray internal communication ports.
> - name: object-manager
> protocol: TCP
> port: 12345
> targetPort: 12345
> - name: node-manager
> protocol: TCP
> port: 12346
> targetPort: 12346
> \---
> apiVersion: v1
> kind: Service
> metadata:
> namespace: ray
> name: ray-dashboard
> spec:
> type: LoadBalancer
> # This selector must match the head node pod's selector.
> selector:
> component: kf-prod01-ray-head
> ports:
> - name: dashboard
> protocol: TCP
> port: 8265
> targetPort: 8265
> \`\`\`
> Apply the cluster config by:
> \`\`\`
> kubectl -n ray apply -f cluster.yaml
> \`\`\`
> Then check the translated cluster config inside the ray operator:
> \`\`\`
> (base) ray@ray-operator-pod:~$ cat ~/ray\_cluster\_configs/kf-prod01\_config.yaml
> auth: {}
> available\_node\_types:
> head-node:
> max\_workers: 0
> min\_workers: 0
> node\_config:
> apiVersion: v1
> kind: Pod
> metadata:
> generateName: ray-head-
> labels:
> component: kf-prod01-ray-head
> ownerReferences:
> - &id001
> apiVersion: cluster.ray.io/v1
> blockOwnerDeletion: true
> controller: true
> kind: RayCluster
> name: kf-prod01
> uid: e21f7f17-6ade-4f50-a3cc-ba09651e6047
> spec:
> containers:
> - args:
> - 'trap : TERM INT; sleep infinity & wait;'
> command:
> - /bin/bash
> - -c
> - --
> image: rayproject/ray:1.2.0
> imagePullPolicy: Always
> name: ray-node
> ports:
> - containerPort: 6379
> protocol: TCP
> - containerPort: 10001
> protocol: TCP
> - containerPort: 12345
> protocol: TCP
> - containerPort: 12346
> protocol: TCP
> - containerPort: 8265
> protocol: TCP
> resources:
> limits:
> cpu: 2000m
> memory: 4Gi
> requests:
> cpu: 2000m
> memory: 4Gi
> volumeMounts:
> - mountPath: /dev/shm
> name: dshm
> nodeSelector:
> cloud.google.com/gke-nodepool: default-pool
> restartPolicy: Never
> volumes:
> - emptyDir:
> medium: Memory
> name: dshm
> resources:
> CPU: 2
> worker\_setup\_commands:
> - pip install pipdate==0.5.2
> worker-node:
> max\_workers: 3
> min\_workers: 2
> node\_config:
> apiVersion: v1
> kind: Pod
> metadata:
> generateName: ray-worker-
> ownerReferences:
> - \*id001
> spec:
> containers:
> - args:
> - 'trap : TERM INT; sleep infinity & wait;'
> command:
> - /bin/bash
> - -c
> - --
> image: rayproject/ray:1.2.0
> imagePullPolicy: Always
> name: ray-node
> resources:
> limits:
> cpu: 2000m
> memory: 4Gi
> requests:
> cpu: 2000m
> memory: 4Gi
> volumeMounts:
> - mountPath: /dev/shm
> name: dshm
> nodeSelector:
> cloud.google.com/gke-nodepool: cpu-worker-pool01
> restartPolicy: Never
> volumes:
> - emptyDir:
> medium: Memory
> name: dshm
> resources:
> CPU: 2
> worker\_setup\_commands:
> - pip install --ignore-installed ruamel.yaml==0.16.12
> - pip install --no-cache-dir torch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0
> - pip install --no-cache-dir scipy==1.6.0 deprecated==1.2.11 gsutil==4.59 pytorch-lightning==1.1.8
> ray\[tune\]==1.2.0
> cluster\_name: kf-prod01
> cluster\_synced\_files: \[\]
> file\_mounts: {}
> file\_mounts\_sync\_continuously: false
> head\_node: {}
> head\_node\_type: head-node
> head\_setup\_commands: \[\]
> head\_start\_ray\_commands:
> \- ray stop
> \- ulimit -n 65536; ray start --head --no-monitor --dashboard-host 0.0.0.0 --object-manager-port=12345
> --node-manager-port=12346 --ray-client-server-port 10001
> idle\_timeout\_minutes: 5
> initialization\_commands: \[\]
> max\_workers: 3
> provider:
> namespace: ray
> services:
> - apiVersion: v1
> kind: Service
> metadata:
> name: kf-prod01-ray-head
> namespace: ray
> spec:
> ports:
> - name: client
> port: 10001
> protocol: TCP
> targetPort: 10001
> - name: dashboard
> port: 8265
> protocol: TCP
> targetPort: 8265
> selector:
> component: kf-prod01-ray-head
> type: kubernetes
> use\_internal\_ips: true
> setup\_commands: \[\]
> upscaling\_speed: 1
> worker\_nodes: {}
> worker\_setup\_commands: \[\]
> worker\_start\_ray\_commands:
> \- ray stop
> \- ulimit -n 65536; ray start --address=$RAY\_HEAD\_IP:6379 --object-manager-port=12345
> --node-manager-port=12346
> \`\`\`
> Looks like the \`setupCommands\` in the operator config is translated into \`worker\_setup\_commands\` for head node, work node, as well as cluster level \`worker\_setup\_commands\`.
> 
> Looks like the problem is \[here\](https://github.com/ray-project/ray/blob/ray-1.2.0/python/ray/operator/operator\_utils.py#L30), the same config map is used by both head node and worker node.

The Ray operator runs the autoscaler in a pod separate from the Ray cluster.

Here’s the documentation on the cluster launcher and operator.

> <https://github.com/ray-project/ray/issues/14726>
>
> \### What is the problem?
> \- Ray version: 1.2.0
> \- OS version: Ubuntu 20.04.2 LTS… (Focal Fossa)
> \- Python version: 3.7.7
> 
> \### Reproduction (REQUIRED)
> Example cluster config for ray operator:
> \`\`\`
> apiVersion: cluster.ray.io/v1
> kind: RayCluster
> metadata:
> namespace: ray
> name: kf-prod01
> spec:
> # The maximum number of workers nodes to launch in addition to the head node.
> maxWorkers: 3
> # The autoscaler will scale up the cluster faster with higher upscaling speed.
> # E.g., if the task requires adding more nodes then autoscaler will gradually
> # scale up the cluster in chunks of upscaling\_speed\*currently\_running\_nodes.
> # This number should be \> 0.
> upscalingSpeed: 1.0
> # If a node is idle for this many minutes, it will be removed.
> idleTimeoutMinutes: 5
> # Specify the pod type for the ray head node (as configured below).
> headPodType: head-node
> # Specify the allowed pod types for this ray cluster and the resources they provide.
> podTypes:
> - name: head-node
> # Minimum number of Ray workers of this Pod type.
> minWorkers: 0
> # Maximum number of Ray workers of this Pod type. Takes precedence over minWorkers.
> maxWorkers: 0
> podConfig:
> apiVersion: v1
> kind: Pod
> metadata:
> # Automatically generates a name for the pod with this prefix.
> generateName: ray-head-
> spec:
> restartPolicy: Never
> nodeSelector:
> cloud.google.com/gke-nodepool: default-pool
> # This volume allocates shared memory for Ray to use for its plasma
> # object store. If you do not provide this, Ray will fall back to
> # /tmp which cause slowdowns if is not a shared memory volume.
> volumes:
> - name: dshm
> emptyDir:
> medium: Memory
> containers:
> - name: ray-node
> imagePullPolicy: Always
> image: rayproject/ray:1.2.0
> # Do not change this command - it keeps the pod alive until it is
> # explicitly killed.
> command: \["/bin/bash", "-c", "--"\]
> args: \['trap : TERM INT; sleep infinity & wait;'\]
> ports:
> - containerPort: 6379 # Redis port
> - containerPort: 10001 # Used by Ray Client
> - containerPort: 12345 # Ray internal communication.
> - containerPort: 12346 # Ray internal communication.
> - containerPort: 8265 # Used by Ray Dashboard
> 
> # This volume allocates shared memory for Ray to use for its plasma
> # object store. If you do not provide this, Ray will fall back to
> # /tmp which cause slowdowns if is not a shared memory volume.
> volumeMounts:
> - mountPath: /dev/shm
> name: dshm
> resources:
> requests:
> cpu: 2000m
> memory: 4Gi
> limits:
> cpu: 2000m
> # The maximum memory that this pod is allowed to use. The
> # limit will be detected by ray and split to use 10% for
> # redis, 30% for the shared memory object store, and the
> # rest for application memory. If this limit is not set and
> # the object store size is not set manually, ray will
> # allocate a very large object store in each pod that may
> # cause problems for other pods.
> memory: 4Gi
> setupCommands:
> - pip install pipdate==0.5.2
> - name: worker-node
> # Minimum number of Ray workers of this Pod type.
> minWorkers: 2
> # Maximum number of Ray workers of this Pod type. Takes precedence over minWorkers.
> maxWorkers: 3
> # User-specified custom resources for use by Ray.
> # (Ray detects CPU and GPU from pod spec resource requests and limits, so no need to fill those here.)
> # rayResources: {"foo": 1, "bar": 1}
> podConfig:
> apiVersion: v1
> kind: Pod
> metadata:
> # Automatically generates a name for the pod with this prefix.
> generateName: ray-worker-
> spec:
> restartPolicy: Never
> nodeSelector:
> cloud.google.com/gke-nodepool: cpu-worker-pool01
> volumes:
> - name: dshm
> emptyDir:
> medium: Memory
> containers:
> - name: ray-node
> imagePullPolicy: Always
> image: rayproject/ray:1.2.0
> command: \["/bin/bash", "-c", "--"\]
> args: \["trap : TERM INT; sleep infinity & wait;"\]
> # This volume allocates shared memory for Ray to use for its plasma
> # object store. If you do not provide this, Ray will fall back to
> # /tmp which cause slowdowns if is not a shared memory volume.
> volumeMounts:
> - mountPath: /dev/shm
> name: dshm
> resources:
> requests:
> cpu: 2000m
> memory: 4Gi
> limits:
> cpu: 2000m
> # The maximum memory that this pod is allowed to use. The
> # limit will be detected by ray and split to use 10% for
> # redis, 30% for the shared memory object store, and the
> # rest for application memory. If this limit is not set and
> # the object store size is not set manually, ray will
> # allocate a very large object store in each pod that may
> # cause problems for other pods.
> memory: 4Gi
> setupCommands:
> - pip install --ignore-installed ruamel.yaml==0.16.12
> - pip install --no-cache-dir torch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0
> - pip install --no-cache-dir scipy==1.6.0 deprecated==1.2.11 gsutil==4.59 pytorch-lightning==1.1.8 ray\[tune\]==1.2.0
> #- pip install rollsroyce-0.0.1.post0.dev141+g301b4af-py2.py3-none-any.whl
> #- pip install --no-cache-dir pytorch-lightning==1.1.8
> # Commands to start Ray on the head node. You don't need to change this.
> # Note dashboard-host is set to 0.0.0.0 so that Kubernetes can port forward.
> headStartRayCommands:
> - ray stop
> - ulimit -n 65536; ray start --head --no-monitor --dashboard-host 0.0.0.0 --object-manager-port=12345 --node-manager-port=12346 --ray-client-server-port 10001
> # Commands to start Ray on worker nodes. You don't need to change this.
> workerStartRayCommands:
> - ray stop
> - ulimit -n 65536; ray start --address=$RAY\_HEAD\_IP:6379 --object-manager-port=12345 --node-manager-port=12346
> \---
> apiVersion: v1
> kind: Service
> metadata:
> namespace: ray
> name: ray-head
> spec:
> # This selector must match the head node pod's selector.
> selector:
> component: kf-prod01-ray-head
> ports:
> - name: client
> protocol: TCP
> port: 10001
> targetPort: 10001
> # Redis ports.
> - name: redis-primary
> port: 6379
> targetPort: 6379
> # Ray internal communication ports.
> - name: object-manager
> protocol: TCP
> port: 12345
> targetPort: 12345
> - name: node-manager
> protocol: TCP
> port: 12346
> targetPort: 12346
> \---
> apiVersion: v1
> kind: Service
> metadata:
> namespace: ray
> name: ray-dashboard
> spec:
> type: LoadBalancer
> # This selector must match the head node pod's selector.
> selector:
> component: kf-prod01-ray-head
> ports:
> - name: dashboard
> protocol: TCP
> port: 8265
> targetPort: 8265
> \`\`\`
> Apply the cluster config by:
> \`\`\`
> kubectl -n ray apply -f cluster.yaml
> \`\`\`
> Then check the translated cluster config inside the ray operator:
> \`\`\`
> (base) ray@ray-operator-pod:~$ cat ~/ray\_cluster\_configs/kf-prod01\_config.yaml
> auth: {}
> available\_node\_types:
> head-node:
> max\_workers: 0
> min\_workers: 0
> node\_config:
> apiVersion: v1
> kind: Pod
> metadata:
> generateName: ray-head-
> labels:
> component: kf-prod01-ray-head
> ownerReferences:
> - &id001
> apiVersion: cluster.ray.io/v1
> blockOwnerDeletion: true
> controller: true
> kind: RayCluster
> name: kf-prod01
> uid: e21f7f17-6ade-4f50-a3cc-ba09651e6047
> spec:
> containers:
> - args:
> - 'trap : TERM INT; sleep infinity & wait;'
> command:
> - /bin/bash
> - -c
> - --
> image: rayproject/ray:1.2.0
> imagePullPolicy: Always
> name: ray-node
> ports:
> - containerPort: 6379
> protocol: TCP
> - containerPort: 10001
> protocol: TCP
> - containerPort: 12345
> protocol: TCP
> - containerPort: 12346
> protocol: TCP
> - containerPort: 8265
> protocol: TCP
> resources:
> limits:
> cpu: 2000m
> memory: 4Gi
> requests:
> cpu: 2000m
> memory: 4Gi
> volumeMounts:
> - mountPath: /dev/shm
> name: dshm
> nodeSelector:
> cloud.google.com/gke-nodepool: default-pool
> restartPolicy: Never
> volumes:
> - emptyDir:
> medium: Memory
> name: dshm
> resources:
> CPU: 2
> worker\_setup\_commands:
> - pip install pipdate==0.5.2
> worker-node:
> max\_workers: 3
> min\_workers: 2
> node\_config:
> apiVersion: v1
> kind: Pod
> metadata:
> generateName: ray-worker-
> ownerReferences:
> - \*id001
> spec:
> containers:
> - args:
> - 'trap : TERM INT; sleep infinity & wait;'
> command:
> - /bin/bash
> - -c
> - --
> image: rayproject/ray:1.2.0
> imagePullPolicy: Always
> name: ray-node
> resources:
> limits:
> cpu: 2000m
> memory: 4Gi
> requests:
> cpu: 2000m
> memory: 4Gi
> volumeMounts:
> - mountPath: /dev/shm
> name: dshm
> nodeSelector:
> cloud.google.com/gke-nodepool: cpu-worker-pool01
> restartPolicy: Never
> volumes:
> - emptyDir:
> medium: Memory
> name: dshm
> resources:
> CPU: 2
> worker\_setup\_commands:
> - pip install --ignore-installed ruamel.yaml==0.16.12
> - pip install --no-cache-dir torch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0
> - pip install --no-cache-dir scipy==1.6.0 deprecated==1.2.11 gsutil==4.59 pytorch-lightning==1.1.8
> ray\[tune\]==1.2.0
> cluster\_name: kf-prod01
> cluster\_synced\_files: \[\]
> file\_mounts: {}
> file\_mounts\_sync\_continuously: false
> head\_node: {}
> head\_node\_type: head-node
> head\_setup\_commands: \[\]
> head\_start\_ray\_commands:
> \- ray stop
> \- ulimit -n 65536; ray start --head --no-monitor --dashboard-host 0.0.0.0 --object-manager-port=12345
> --node-manager-port=12346 --ray-client-server-port 10001
> idle\_timeout\_minutes: 5
> initialization\_commands: \[\]
> max\_workers: 3
> provider:
> namespace: ray
> services:
> - apiVersion: v1
> kind: Service
> metadata:
> name: kf-prod01-ray-head
> namespace: ray
> spec:
> ports:
> - name: client
> port: 10001
> protocol: TCP
> targetPort: 10001
> - name: dashboard
> port: 8265
> protocol: TCP
> targetPort: 8265
> selector:
> component: kf-prod01-ray-head
> type: kubernetes
> use\_internal\_ips: true
> setup\_commands: \[\]
> upscaling\_speed: 1
> worker\_nodes: {}
> worker\_setup\_commands: \[\]
> worker\_start\_ray\_commands:
> \- ray stop
> \- ulimit -n 65536; ray start --address=$RAY\_HEAD\_IP:6379 --object-manager-port=12345
> --node-manager-port=12346
> \`\`\`
> Looks like the \`setupCommands\` in the operator config is translated into \`worker\_setup\_commands\` for head node, work node, as well as cluster level \`worker\_setup\_commands\`.
> 
> Looks like the problem is \[here\](https://github.com/ray-project/ray/blob/ray-1.2.0/python/ray/operator/operator\_utils.py#L30), the same config map is used by both head node and worker node.

---

<div class="post-metadata">

**Author:** ![blublinsky](https://avatars.discourse-cdn.com/v4/letter/b/f05b48/32.png) [@blublinsky](https://discuss.ray.io/u/blublinsky)\
**Post date:** [March 26, 2021, 8:01pm UTC](https://discuss.ray.io/t/autoscaler-does-not-seem-to-watch-head-node/1410/6 "2021-03-26T20:01:07Z")

</div>

Did you guys consider to install head node as a deployment of 1 to allow deployment to restart it in the case of failures.