# Cannot start Ray runtime on GCP **URL:** \ **Category:** Ray Core\ **Created:** [February 25, 2021, 4:28am UTC](https://discuss.ray.io/t/cannot-start-ray-runtime-on-gcp/1011 "2021-02-25T04:28:13Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![kipnisal](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kipnisal/32/534_2.png) [@kipnisal](https://discuss.ray.io/u/kipnisal)\ **Post date:** [February 25, 2021, 4:28am UTC](https://discuss.ray.io/t/cannot-start-ray-runtime-on-gcp/1011/1 "2021-02-25T04:28:13Z")
Running ray up cluster.yaml Getting the following: > [7/7] Starting the Ray runtime > Could not terminate ` **/home/ray/anaconda3/lib/python3.7/site-packages/ray/core/src/ray/raylet/raylet --raylet_socket_name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/raylet --store_socket_name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/plasma_store --object_manager_port=8076 --min_worker_port=10000 --max_worker_port=10999 --node_manager_port=34998 --node_ip_address=10.138.0.2 --redis_address=10.138.0.2 --redis_port=6379 --num_initial_workers=2 --maximum_startup_concurrency=2 --static_resource_list=node:10.138.0.2,1.0,CPU,2,memory,87,object_store_memory,30 --config_list=plasma_store_as_thread,True "--python_worker_command=/home/ray/anaconda3/bin/python /home/ray/anaconda3/lib/python3.7/site-packages/ray/workers/default_worker.py --node-ip-address=10.138.0.2 --node-manager-port=34998 --object-store-name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/plasma_store --raylet-name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/raylet --redis-address=10.138.0.2:6379 --config-list=plasma_store_as_thread,True --temp-dir=/tmp/ray --metrics-agent-port=56424 --redis-password=5241590000000000" --java_worker_command= --cpp_worker_command= --redis_password=5241590000000000 --temp_dir=/tmp/ray --session_dir=/tmp/ray/session_2021-02-24_16-25-42_086193_4322 --metrics-agent-port=56424 --metrics_export_port=55267 "--agent_command=/home/ray/anaconda3/bin/python -u /home/ray/anaconda3/lib/python3.7/site-packages/ray/new_dashboard/agent.py --node-ip-address=10.138.0.2 --redis-address=10.138.0.2:6379 --metrics-export-port=55267 --dashboard-agent-port=56424 --node-manager-port=34998 --object-store-name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/plasma_store --raylet-name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/raylet --temp-dir=/tmp/ray --log-dir=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/logs --redis-password=5241590000000000" --object_store_memory=2284963430 --plasma_directory=/dev/shm --head_node** ` due to psutil.AccessDenied (pid=6479, name=‘raylet’) > > Could not terminate ` **/home/ray/anaconda3/lib/python3.7/site-packages/ray/core/src/plasma/plasma_store_server -s /tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/plasma_store -m 2284963430 -d /dev/shm -z** ` due to psutil.AccessDenied (pid=6478, name=‘plasma\_store\_server’) > > Could not terminate ` **/home/ray/anaconda3/lib/python3.7/site-packages/ray/core/src/ray/gcs/gcs_server --redis_address=10.138.0.2 --redis_port=6379 --config_list=plasma_store_as_thread,True --gcs_server_port=0 --metrics-agent-port=56424 --node-ip-address=10.138.0.2 --redis_password=5241590000000000** ` due to psutil.AccessDenied (pid=6457, name=‘gcs\_server’) > > Could not terminate ` **/home/ray/anaconda3/bin/python -u /home/ray/anaconda3/lib/python3.7/site-packages/ray/log_monitor.py --redis-address=10.138.0.2:6379 --logs-dir=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/logs --redis-password 5241590000000000** ` due to psutil.AccessDenied (pid=6480, name=‘/home/ray/anaco’) > > Could not terminate ` **"/home/ray/anaconda3/lib/python3.7/site-packages/ray/core/src/ray/thirdparty/redis/src/redis-server *:6379" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""** ` due to psutil.AccessDenied (pid=6446, name=‘redis-server’) > > Could not terminate ` **"/home/ray/anaconda3/lib/python3.7/site-packages/ray/core/src/ray/thirdparty/redis/src/redis-server *:17690" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""** ` due to psutil.AccessDenied (pid=6452, name=‘redis-server’) > > Could not terminate ` **/home/ray/anaconda3/lib/python3.7/site-packages/ray/core/src/ray/raylet/raylet --raylet_socket_name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/raylet --store_socket_name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/plasma_store --object_manager_port=8076 --min_worker_port=10000 --max_worker_port=10999 --node_manager_port=34998 --node_ip_address=10.138.0.2 --redis_address=10.138.0.2 --redis_port=6379 --num_initial_workers=2 --maximum_startup_concurrency=2 --static_resource_list=node:10.138.0.2,1.0,CPU,2,memory,87,object_store_memory,30 --config_list=plasma_store_as_thread,True "--python_worker_command=/home/ray/anaconda3/bin/python /home/ray/anaconda3/lib/python3.7/site-packages/ray/workers/default_worker.py --node-ip-address=10.138.0.2 --node-manager-port=34998 --object-store-name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/plasma_store --raylet-name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/raylet --redis-address=10.138.0.2:6379 --config-list=plasma_store_as_thread,True --temp-dir=/tmp/ray --metrics-agent-port=56424 --redis-password=5241590000000000" --java_worker_command= --cpp_worker_command= --redis_password=5241590000000000 --temp_dir=/tmp/ray --session_dir=/tmp/ray/session_2021-02-24_16-25-42_086193_4322 --metrics-agent-port=56424 --metrics_export_port=55267 "--agent_command=/home/ray/anaconda3/bin/python -u /home/ray/anaconda3/lib/python3.7/site-packages/ray/new_dashboard/agent.py --node-ip-address=10.138.0.2 --redis-address=10.138.0.2:6379 --metrics-export-port=55267 --dashboard-agent-port=56424 --node-manager-port=34998 --object-store-name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/plasma_store --raylet-name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/raylet --temp-dir=/tmp/ray --log-dir=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/logs --redis-password=5241590000000000" --object_store_memory=2284963430 --plasma_directory=/dev/shm --head_node** ` due to psutil.AccessDenied (pid=6479, name=‘raylet’) > > Could not terminate ` **/home/ray/anaconda3/bin/python -u /home/ray/anaconda3/lib/python3.7/site-packages/ray/log_monitor.py --redis-address=10.138.0.2:6379 --logs-dir=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/logs --redis-password 5241590000000000** ` due to psutil.AccessDenied (pid=6480, name=‘/home/ray/anaco’) > > Could not terminate ` **/home/ray/anaconda3/bin/python -u /home/ray/anaconda3/lib/python3.7/site-packages/ray/new_dashboard/dashboard.py --host=localhost --port=8265 --redis-address=10.138.0.2:6379 --temp-dir=/tmp/ray --log-dir=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/logs --redis-password 5241590000000000** ` due to psutil.AccessDenied (pid=6477, name=‘/home/ray/anaco’) > > Could not terminate ` **/home/ray/anaconda3/lib/python3.7/site-packages/ray/core/src/ray/raylet/raylet --raylet_socket_name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/raylet --store_socket_name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/plasma_store --object_manager_port=8076 --min_worker_port=10000 --max_worker_port=10999 --node_manager_port=34998 --node_ip_address=10.138.0.2 --redis_address=10.138.0.2 --redis_port=6379 --num_initial_workers=2 --maximum_startup_concurrency=2 --static_resource_list=node:10.138.0.2,1.0,CPU,2,memory,87,object_store_memory,30 --config_list=plasma_store_as_thread,True "--python_worker_command=/home/ray/anaconda3/bin/python /home/ray/anaconda3/lib/python3.7/site-packages/ray/workers/default_worker.py --node-ip-address=10.138.0.2 --node-manager-port=34998 --object-store-name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/plasma_store --raylet-name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/raylet --redis-address=10.138.0.2:6379 --config-list=plasma_store_as_thread,True --temp-dir=/tmp/ray --metrics-agent-port=56424 --redis-password=5241590000000000" --java_worker_command= --cpp_worker_command= --redis_password=5241590000000000 --temp_dir=/tmp/ray --session_dir=/tmp/ray/session_2021-02-24_16-25-42_086193_4322 --metrics-agent-port=56424 --metrics_export_port=55267 "--agent_command=/home/ray/anaconda3/bin/python -u /home/ray/anaconda3/lib/python3.7/site-packages/ray/new_dashboard/agent.py --node-ip-address=10.138.0.2 --redis-address=10.138.0.2:6379 --metrics-export-port=55267 --dashboard-agent-port=56424 --node-manager-port=34998 --object-store-name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/plasma_store --raylet-name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/raylet --temp-dir=/tmp/ray --log-dir=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/logs --redis-password=5241590000000000" --object_store_memory=2284963430 --plasma_directory=/dev/shm --head_node** ` due to psutil.AccessDenied (pid=6479, name=‘raylet’) > > Could not terminate ` **/home/ray/anaconda3/bin/python -u /home/ray/anaconda3/lib/python3.7/site-packages/ray/new_dashboard/agent.py --node-ip-address=10.138.0.2 --redis-address=10.138.0.2:6379 --metrics-export-port=55267 --dashboard-agent-port=56424 --node-manager-port=34998 --object-store-name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/plasma_store --raylet-name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/raylet --temp-dir=/tmp/ray --log-dir=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/logs --redis-password=5241590000000000** ` due to psutil.AccessDenied (pid=6533, name=‘/home/ray/anaco’) ---
**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\ **Post date:** [February 25, 2021, 5:49pm UTC](https://discuss.ray.io/t/cannot-start-ray-runtime-on-gcp/1011/2 "2021-02-25T17:49:50Z")
@rliaw have you seen this issue before? ---
**Author:** ![kipnisal](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kipnisal/32/534_2.png) [@kipnisal](https://discuss.ray.io/u/kipnisal)\ **Post date:** [February 25, 2021, 6:03pm UTC](https://discuss.ray.io/t/cannot-start-ray-runtime-on-gcp/1011/3 "2021-02-25T18:03:59Z")
It’s the first I’m trying to set up a cluster on GCP, so no, I haven’t. ---
**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\ **Post date:** [February 25, 2021, 8:56pm UTC](https://discuss.ray.io/t/cannot-start-ray-runtime-on-gcp/1011/4 "2021-02-25T20:56:14Z")
Oh sorry. I was asking other guys haha (since I know he recently answer to the GCP related issue). Also cc @Ameer_Haj_Ali @kipnisal always feel free to ping me again if people don’t respond to you! ---
**Author:** ![Ameer\_Haj\_Ali](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/ameer_haj_ali/32/279_2.png) [@Ameer\_Haj\_Ali](https://discuss.ray.io/u/Ameer_Haj_Ali)\ **Post date:** [February 26, 2021, 12:26am UTC](https://discuss.ray.io/t/cannot-start-ray-runtime-on-gcp/1011/5 "2021-02-26T00:26:25Z")
can you please paste the cluster yaml? does it also happen on the ray nightly? ---
**Author:** ![Ameer\_Haj\_Ali](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/ameer_haj_ali/32/279_2.png) [@Ameer\_Haj\_Ali](https://discuss.ray.io/u/Ameer_Haj_Ali)\ **Post date:** [February 26, 2021, 12:27am UTC](https://discuss.ray.io/t/cannot-start-ray-runtime-on-gcp/1011/6 "2021-02-26T00:27:46Z")
CC @ijrsvt, perhaps this is related to the other gpc issue? ---
**Author:** ![kipnisal](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kipnisal/32/534_2.png) [@kipnisal](https://discuss.ray.io/u/kipnisal)\ **Post date:** [February 26, 2021, 12:29am UTC](https://discuss.ray.io/t/cannot-start-ray-runtime-on-gcp/1011/7 "2021-02-26T00:29:38Z")
Here is the cluster yaml: ``` cluster_name: default min_workers: 3 max_workers: 4 provider: type: gcp region: us-west1 availability_zone: us-west1-a project_id: slurm-experiment # Globally unique project id auth: ssh_user: ubuntu setup_commands: - sudo apt update - sudo apt-get install -y cmake - pip install ray[all] numpy pandas TwoSampleHC ``` ---
**Author:** ![kipnisal](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kipnisal/32/534_2.png) [@kipnisal](https://discuss.ray.io/u/kipnisal)\ **Post date:** [March 2, 2021, 12:38am UTC](https://discuss.ray.io/t/cannot-start-ray-runtime-on-gcp/1011/8 "2021-03-02T00:38:36Z")
Hi @sangcho, I’m still struggling with this issue. Are there any directions where to look? ---
**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\ **Post date:** [March 2, 2021, 12:54am UTC](https://discuss.ray.io/t/cannot-start-ray-runtime-on-gcp/1011/9 "2021-03-02T00:54:20Z")
Can you check if the ubuntu user has the access to this folder? `’/home/ray/anaco’` ---
**Author:** ![kipnisal](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kipnisal/32/534_2.png) [@kipnisal](https://discuss.ray.io/u/kipnisal)\ **Post date:** [March 2, 2021, 4:18am UTC](https://discuss.ray.io/t/cannot-start-ray-runtime-on-gcp/1011/10 "2021-03-02T04:18:08Z")
I don’t understand the question. What user? how can I check that? I’m trying to follow the instructions on [https://docs.ray.io/en/releases-0.8.5/autoscaling.html](https://docs.ray.io/en/releases-0.8.5/autoscaling.html) Below is the output of a different attempt. This time I added ``` docker: image: "kipnisal/phase_transition" container_name: "phase_transition_exp" pull_before_run: True run_options: [] # Extra options to pass into "docker run" ``` to the cluster.yaml file and removed what was under `setup\_commands’ (the Docker container includes instructions to install ray and the other requirements). ``` [6/7] Running setup commands (0/3) **wget https://repo.anaconda.com**... bash: wget: command not found bash: /root/anaconda3.sh: No such file or directory rm: cannot remove '/root/anaconda3.sh': No such file or directory Shared connection to 35.203.137.46 closed. 2021-03-01 19:57:08,101 INFO node_provider.py:20 -- wait_for_compute_zone_operation: Waiting for operation operation-1614657427526-5bc85ba66d52d-91feb0eb-0eabe74f to finish... 2021-03-01 19:57:13,843 INFO node_provider.py:32 -- wait_for_compute_zone_operation: Operation operation-1614657427526-5bc85ba66d52d-91feb0eb-0eabe74f finished. New status: **update-failed** !!! SSH command failed. !!! Failed to setup head node. zsh: exit 1 ray up cluster.yaml ``` What is unclear to me is why ray is looking for anaconda? ---
**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\ **Post date:** [March 2, 2021, 4:46am UTC](https://discuss.ray.io/t/cannot-start-ray-runtime-on-gcp/1011/11 "2021-03-02T04:46:14Z")
> [@kipnisal](#): > > psutil.AccessDenied (pid=6533, name=’/home/ray/anaco’) I asked due to this line; If you launch the autoscaler, the node’s user is ubuntu. (Look at your cluster.yaml) ```auto auth: ssh_user: ubuntu ``` And it seems like this ssh user doesn’t have access to the folder `/home/ray/anaco` in a glance. Also, what version of ray are you using? Also, @Dmitri @Ameer_Haj_Ali can you guys handle this issue? ---
**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\ **Post date:** [March 3, 2021, 4:56am UTC](https://discuss.ray.io/t/cannot-start-ray-runtime-on-gcp/1011/12 "2021-03-03T04:56:51Z")
This is suboptimal behavior… It’s happening because setup commands are getting pulled from here: > I’d actually recommend taking a look at this file as a starting point and modifying to suit your use-case: > ---
**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\ **Post date:** [March 3, 2021, 4:59am UTC](https://discuss.ray.io/t/cannot-start-ray-runtime-on-gcp/1011/13 "2021-03-03T04:59:31Z")
Also, if you could post an issue on the Ray github with the last error message you saw, that would be very helpful.