# GCP Cluster Worker Nodes fail to Initialize

**URL:** <https://discuss.ray.io/t/gcp-cluster-worker-nodes-fail-to-initialize/12423>\
**Category:** Ray Clusters\
**Created:** [October 12, 2023, 1:46am UTC](https://discuss.ray.io/t/gcp-cluster-worker-nodes-fail-to-initialize/12423 "2023-10-12T01:46:15Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Vrushank\_Desai](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/vrushank_desai/32/5092_2.png) [@Vrushank\_Desai](https://discuss.ray.io/u/Vrushank_Desai)\
**Post date:** [October 12, 2023, 1:46am UTC](https://discuss.ray.io/t/gcp-cluster-worker-nodes-fail-to-initialize/12423/1 "2023-10-12T01:46:16Z")

</div>

**How severe does this issue affect your experience of using Ray?**  
High: It blocks me to complete my task.

Hi, as of last few weeks I have had increasing number of errors faced during spool up of worker-nodes on GCP by the Ray autoscaler. It has gotten to the point where I basically can’t scale clusters past a handful of worker nodes. I am able to reproduce this using an almost exact duplicate of the Ray example YAML for GCP, a script that runs dummy remote tasks, and across new conda envs on the local node launching the cluster with Ray versions 2.3.1, 2.7.0, 2.7.1. The consistency across Ray versions makes me think some sort of issue with a recent GCP change. Things I have tried to fix this issue include:

-as mentioned, trying multiple Ray versions on new environments  
-different GCP instance types  
-different GCP instance images with differnent Python versions  
-pip installing various versions of google-python-api-client in cluster init commands

My monitor.err in all cases is full of Exceptions related to read time-outs and SSL Errors. It’s worth noting that a co-worker ran the same job on AWS with a similar config and had no issues scaling to 50-nodes in 6 mins, while I can’t get past 5 nodes after 1hr+. This bug has made Ray unusable for me so any help would be very much appreciated!

Logs from four different attempts - [Ray Error Logs - Google Drive](https://drive.google.com/drive/folders/1hjdjgnHpHmPP5PVc33BYYtoBIv9PebaG?usp=sharing)

YAML config -

> **[barebones\_config.yml](https://drive.google.com/file/d/1h1dc3fQgVpAAu51YzMxQN9voEjPnpdxV/view?usp=sharing)**
>
> Google Drive file.

Dummy Python payload that I ran on head node -

> **[dummy\_ray.py](https://drive.google.com/file/d/1aB3rPld7F7xCTLlJ0hFk3UjiQln1fZl-/view?usp=sharing)**
>
> Google Drive file.

---

<div class="post-metadata">

**Author:** ![PaulFenton](https://avatars.discourse-cdn.com/v4/letter/p/779978/32.png) [@PaulFenton](https://discuss.ray.io/u/PaulFenton)\
**Post date:** [November 11, 2023, 1:43pm UTC](https://discuss.ray.io/t/gcp-cluster-worker-nodes-fail-to-initialize/12423/2 "2023-11-11T13:43:44Z")

</div>

@Vrushank_Desai could you try filling-in and un-commenting this section?:

```auto
    # serviceAccounts:
    # - email: ray-autoscaler-sa-v1@<project_id>.iam.gserviceaccount.com
    # scopes:
    # - https://www.googleapis.com/auth/cloud-platform

```

I had a variety of issues following this full example to get things running on GCP and ended up building my config up from the smaller GCP example.

---

<div class="post-metadata">

**Author:** ![denmarc](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/denmarc/32/6529_2.png) [@denmarc](https://discuss.ray.io/u/denmarc)\
**Post date:** [March 8, 2024, 2:30pm UTC](https://discuss.ray.io/t/gcp-cluster-worker-nodes-fail-to-initialize/12423/3 "2024-03-08T14:30:16Z")

</div>

Any development on this issue? I am facing the same problem. Ray Tune used to be very helpful but now it is quite useless on GCP.

---

<div class="post-metadata">

**Author:** ![denmarc](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/denmarc/32/6529_2.png) [@denmarc](https://discuss.ray.io/u/denmarc)\
**Post date:** [September 11, 2024, 9:55pm UTC](https://discuss.ray.io/t/gcp-cluster-worker-nodes-fail-to-initialize/12423/4 "2024-09-11T21:55:50Z")

</div>

I am still looking for a solution to this problem.

---

<div class="post-metadata">

**Author:** ![Sam\_Chan](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sam_chan/32/5011_2.png) [@Sam\_Chan](https://discuss.ray.io/u/Sam_Chan)\
**Post date:** [September 12, 2024, 3:49am UTC](https://discuss.ray.io/t/gcp-cluster-worker-nodes-fail-to-initialize/12423/5 "2024-09-12T03:49:26Z")

</div>

@denmarc what ray version are you on; and can you supply a repro script and some dets on your setup on GCP?

---

<div class="post-metadata">

**Author:** ![denmarc](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/denmarc/32/6529_2.png) [@denmarc](https://discuss.ray.io/u/denmarc)\
**Post date:** [October 10, 2024, 8:00pm UTC](https://discuss.ray.io/t/gcp-cluster-worker-nodes-fail-to-initialize/12423/6 "2024-10-10T20:00:03Z")

</div>

Hi @Sam_Chan, I updated to the latest version (2.37.0) and the previous problem seems to be solved, now Ray will launch and communicate with the worker nodes, but their initialization still seems to be faulty, as I reported [here](https://discuss.ray.io/t/only-the-first-few-worker-nodes-sync-files-file-mount/15953). It seems no worker commands are being executed but for the first one or two nodes.