Cannot start Ray runtime on GCP

kipnisal · February 25, 2021, 4:28am

Running
ray up cluster.yaml

Getting the following:

[7/7] Starting the Ray runtime
Could not terminate **/home/ray/anaconda3/lib/python3.7/site-packages/ray/core/src/ray/raylet/raylet --raylet_socket_name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/raylet --store_socket_name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/plasma_store --object_manager_port=8076 --min_worker_port=10000 --max_worker_port=10999 --node_manager_port=34998 --node_ip_address=10.138.0.2 --redis_address=10.138.0.2 --redis_port=6379 --num_initial_workers=2 --maximum_startup_concurrency=2 --static_resource_list=node:10.138.0.2,1.0,CPU,2,memory,87,object_store_memory,30 --config_list=plasma_store_as_thread,True "--python_worker_command=/home/ray/anaconda3/bin/python /home/ray/anaconda3/lib/python3.7/site-packages/ray/workers/default_worker.py --node-ip-address=10.138.0.2 --node-manager-port=34998 --object-store-name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/plasma_store --raylet-name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/raylet --redis-address=10.138.0.2:6379 --config-list=plasma_store_as_thread,True --temp-dir=/tmp/ray --metrics-agent-port=56424 --redis-password=5241590000000000" --java_worker_command= --cpp_worker_command= --redis_password=5241590000000000 --temp_dir=/tmp/ray --session_dir=/tmp/ray/session_2021-02-24_16-25-42_086193_4322 --metrics-agent-port=56424 --metrics_export_port=55267 "--agent_command=/home/ray/anaconda3/bin/python -u /home/ray/anaconda3/lib/python3.7/site-packages/ray/new_dashboard/agent.py --node-ip-address=10.138.0.2 --redis-address=10.138.0.2:6379 --metrics-export-port=55267 --dashboard-agent-port=56424 --node-manager-port=34998 --object-store-name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/plasma_store --raylet-name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/raylet --temp-dir=/tmp/ray --log-dir=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/logs --redis-password=5241590000000000" --object_store_memory=2284963430 --plasma_directory=/dev/shm --head_node** due to psutil.AccessDenied (pid=6479, name=‘raylet’)

Could not terminate **/home/ray/anaconda3/lib/python3.7/site-packages/ray/core/src/plasma/plasma_store_server -s /tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/plasma_store -m 2284963430 -d /dev/shm -z** due to psutil.AccessDenied (pid=6478, name=‘plasma_store_server’)

Could not terminate **/home/ray/anaconda3/lib/python3.7/site-packages/ray/core/src/ray/gcs/gcs_server --redis_address=10.138.0.2 --redis_port=6379 --config_list=plasma_store_as_thread,True --gcs_server_port=0 --metrics-agent-port=56424 --node-ip-address=10.138.0.2 --redis_password=5241590000000000** due to psutil.AccessDenied (pid=6457, name=‘gcs_server’)

Could not terminate **/home/ray/anaconda3/bin/python -u /home/ray/anaconda3/lib/python3.7/site-packages/ray/log_monitor.py --redis-address=10.138.0.2:6379 --logs-dir=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/logs --redis-password 5241590000000000** due to psutil.AccessDenied (pid=6480, name=‘/home/ray/anaco’)

Could not terminate **"/home/ray/anaconda3/lib/python3.7/site-packages/ray/core/src/ray/thirdparty/redis/src/redis-server *:6379" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""** due to psutil.AccessDenied (pid=6446, name=‘redis-server’)

Could not terminate **"/home/ray/anaconda3/lib/python3.7/site-packages/ray/core/src/ray/thirdparty/redis/src/redis-server *:17690" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""** due to psutil.AccessDenied (pid=6452, name=‘redis-server’)

Could not terminate **/home/ray/anaconda3/lib/python3.7/site-packages/ray/core/src/ray/raylet/raylet --raylet_socket_name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/raylet --store_socket_name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/plasma_store --object_manager_port=8076 --min_worker_port=10000 --max_worker_port=10999 --node_manager_port=34998 --node_ip_address=10.138.0.2 --redis_address=10.138.0.2 --redis_port=6379 --num_initial_workers=2 --maximum_startup_concurrency=2 --static_resource_list=node:10.138.0.2,1.0,CPU,2,memory,87,object_store_memory,30 --config_list=plasma_store_as_thread,True "--python_worker_command=/home/ray/anaconda3/bin/python /home/ray/anaconda3/lib/python3.7/site-packages/ray/workers/default_worker.py --node-ip-address=10.138.0.2 --node-manager-port=34998 --object-store-name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/plasma_store --raylet-name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/raylet --redis-address=10.138.0.2:6379 --config-list=plasma_store_as_thread,True --temp-dir=/tmp/ray --metrics-agent-port=56424 --redis-password=5241590000000000" --java_worker_command= --cpp_worker_command= --redis_password=5241590000000000 --temp_dir=/tmp/ray --session_dir=/tmp/ray/session_2021-02-24_16-25-42_086193_4322 --metrics-agent-port=56424 --metrics_export_port=55267 "--agent_command=/home/ray/anaconda3/bin/python -u /home/ray/anaconda3/lib/python3.7/site-packages/ray/new_dashboard/agent.py --node-ip-address=10.138.0.2 --redis-address=10.138.0.2:6379 --metrics-export-port=55267 --dashboard-agent-port=56424 --node-manager-port=34998 --object-store-name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/plasma_store --raylet-name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/raylet --temp-dir=/tmp/ray --log-dir=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/logs --redis-password=5241590000000000" --object_store_memory=2284963430 --plasma_directory=/dev/shm --head_node** due to psutil.AccessDenied (pid=6479, name=‘raylet’)

Could not terminate **/home/ray/anaconda3/bin/python -u /home/ray/anaconda3/lib/python3.7/site-packages/ray/log_monitor.py --redis-address=10.138.0.2:6379 --logs-dir=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/logs --redis-password 5241590000000000** due to psutil.AccessDenied (pid=6480, name=‘/home/ray/anaco’)

Could not terminate **/home/ray/anaconda3/bin/python -u /home/ray/anaconda3/lib/python3.7/site-packages/ray/new_dashboard/dashboard.py --host=localhost --port=8265 --redis-address=10.138.0.2:6379 --temp-dir=/tmp/ray --log-dir=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/logs --redis-password 5241590000000000** due to psutil.AccessDenied (pid=6477, name=‘/home/ray/anaco’)

Could not terminate **/home/ray/anaconda3/lib/python3.7/site-packages/ray/core/src/ray/raylet/raylet --raylet_socket_name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/raylet --store_socket_name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/plasma_store --object_manager_port=8076 --min_worker_port=10000 --max_worker_port=10999 --node_manager_port=34998 --node_ip_address=10.138.0.2 --redis_address=10.138.0.2 --redis_port=6379 --num_initial_workers=2 --maximum_startup_concurrency=2 --static_resource_list=node:10.138.0.2,1.0,CPU,2,memory,87,object_store_memory,30 --config_list=plasma_store_as_thread,True "--python_worker_command=/home/ray/anaconda3/bin/python /home/ray/anaconda3/lib/python3.7/site-packages/ray/workers/default_worker.py --node-ip-address=10.138.0.2 --node-manager-port=34998 --object-store-name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/plasma_store --raylet-name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/raylet --redis-address=10.138.0.2:6379 --config-list=plasma_store_as_thread,True --temp-dir=/tmp/ray --metrics-agent-port=56424 --redis-password=5241590000000000" --java_worker_command= --cpp_worker_command= --redis_password=5241590000000000 --temp_dir=/tmp/ray --session_dir=/tmp/ray/session_2021-02-24_16-25-42_086193_4322 --metrics-agent-port=56424 --metrics_export_port=55267 "--agent_command=/home/ray/anaconda3/bin/python -u /home/ray/anaconda3/lib/python3.7/site-packages/ray/new_dashboard/agent.py --node-ip-address=10.138.0.2 --redis-address=10.138.0.2:6379 --metrics-export-port=55267 --dashboard-agent-port=56424 --node-manager-port=34998 --object-store-name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/plasma_store --raylet-name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/raylet --temp-dir=/tmp/ray --log-dir=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/logs --redis-password=5241590000000000" --object_store_memory=2284963430 --plasma_directory=/dev/shm --head_node** due to psutil.AccessDenied (pid=6479, name=‘raylet’)

Could not terminate **/home/ray/anaconda3/bin/python -u /home/ray/anaconda3/lib/python3.7/site-packages/ray/new_dashboard/agent.py --node-ip-address=10.138.0.2 --redis-address=10.138.0.2:6379 --metrics-export-port=55267 --dashboard-agent-port=56424 --node-manager-port=34998 --object-store-name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/plasma_store --raylet-name=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/sockets/raylet --temp-dir=/tmp/ray --log-dir=/tmp/ray/session_2021-02-24_16-25-42_086193_4322/logs --redis-password=5241590000000000** due to psutil.AccessDenied (pid=6533, name=‘/home/ray/anaco’)

sangcho · February 25, 2021, 5:49pm

@rliaw have you seen this issue before?

kipnisal · February 25, 2021, 6:03pm

It’s the first I’m trying to set up a cluster on GCP, so no, I haven’t.

sangcho · February 25, 2021, 8:56pm

Oh sorry. I was asking other guys haha (since I know he recently answer to the GCP related issue).

Also cc @Ameer_Haj_Ali

@kipnisal always feel free to ping me again if people don’t respond to you!

Ameer_Haj_Ali · February 26, 2021, 12:26am

can you please paste the cluster yaml?
does it also happen on the ray nightly?

Ameer_Haj_Ali · February 26, 2021, 12:27am

CC @ijrsvt, perhaps this is related to the other gpc issue?

kipnisal · February 26, 2021, 12:29am

Here is the cluster yaml:

cluster_name: default

min_workers: 3
max_workers: 4


provider:
    type: gcp
    region: us-west1
    availability_zone: us-west1-a
    project_id: slurm-experiment # Globally unique project id

auth:
    ssh_user: ubuntu

setup_commands:
  - sudo apt update
  - sudo apt-get install -y cmake
  - pip install ray[all] numpy pandas TwoSampleHC

kipnisal · March 2, 2021, 12:38am

Hi @sangcho,
I’m still struggling with this issue.
Are there any directions where to look?

sangcho · March 2, 2021, 12:54am

Can you check if the ubuntu user has the access to this folder? ’/home/ray/anaco’

kipnisal · March 2, 2021, 4:18am

I don’t understand the question. What user? how can I check that?

I’m trying to follow the instructions on https://docs.ray.io/en/releases-0.8.5/autoscaling.html

Below is the output of a different attempt. This time I added

docker:
  image: "kipnisal/phase_transition"
  container_name: "phase_transition_exp"
  pull_before_run: True
  run_options: []  # Extra options to pass into "docker run"

to the cluster.yaml file and removed what was under `setup_commands’ (the Docker container includes instructions to install ray and the other requirements).

[6/7] Running setup commands

(0/3) **wget https://repo.anaconda.com** ...

bash: wget: command not found

bash: /root/anaconda3.sh: No such file or directory

rm: cannot remove '/root/anaconda3.sh': No such file or directory

Shared connection to 35.203.137.46 closed.

2021-03-01 19:57:08,101 INFO node_provider.py:20 -- wait_for_compute_zone_operation: Waiting for operation operation-1614657427526-5bc85ba66d52d-91feb0eb-0eabe74f to finish...

2021-03-01 19:57:13,843 INFO node_provider.py:32 -- wait_for_compute_zone_operation: Operation operation-1614657427526-5bc85ba66d52d-91feb0eb-0eabe74f finished.

New status: **update-failed**

!!!

SSH command failed.

!!!

Failed to setup head node.

zsh: exit 1 ray up cluster.yaml

What is unclear to me is why ray is looking for anaconda?

sangcho · March 2, 2021, 4:46am

I asked due to this line;

If you launch the autoscaler, the node’s user is ubuntu. (Look at your cluster.yaml)

auth:
    ssh_user: ubuntu

And it seems like this ssh user doesn’t have access to the folder /home/ray/anaco in a glance.

Also, what version of ray are you using? Also, @Dmitri @Ameer_Haj_Ali can you guys handle this issue?

Dmitri · March 3, 2021, 4:56am

This is suboptimal behavior…
It’s happening because setup commands are getting pulled from here:

github.com

ray-project/ray/blob/master/python/ray/autoscaler/gcp/defaults.yaml

# An unique identifier for the head node and workers of this cluster.
cluster_name: default

# The maximum number of workers nodes to launch in addition to the head
# node.
max_workers: 2

# The autoscaler will scale up the cluster faster with higher upscaling speed.
# E.g., if the task requires adding more nodes then autoscaler will gradually
# scale up the cluster in chunks of upscaling_speed*currently_running_nodes.
# This number should be > 0.
upscaling_speed: 1.0

# This executes all commands on all nodes in the docker container,
# and opens all the necessary ports to support the Ray cluster.
# Empty string means disabled.
docker: {}

# If a node is idle for this many minutes, it will be removed.
idle_timeout_minutes: 5

This file has been truncated. show original

I’d actually recommend taking a look at this file as a starting point and modifying to suit your use-case:

github.com

ray-project/ray/blob/master/python/ray/autoscaler/gcp/example-full.yaml

# An unique identifier for the head node and workers of this cluster.
cluster_name: default

# The maximum number of workers nodes to launch in addition to the head
# node.
max_workers: 2

# The autoscaler will scale up the cluster faster with higher upscaling speed.
# E.g., if the task requires adding more nodes then autoscaler will gradually
# scale up the cluster in chunks of upscaling_speed*currently_running_nodes.
# This number should be > 0.
upscaling_speed: 1.0

# This executes all commands on all nodes in the docker container,
# and opens all the necessary ports to support the Ray cluster.
# Empty string means disabled.
docker:
  image: "rayproject/ray-ml:latest-gpu" # You can change this to latest-cpu if you don't need GPU support and want a faster startup
    # image: rayproject/ray:latest-gpu   # use this one if you don't need ML dependencies, it's faster to pull
  container_name: "ray_container"

This file has been truncated. show original

Dmitri · March 3, 2021, 4:59am

Also, if you could post an issue on the Ray github with the last error message you saw, that would be very helpful.

Topic		Replies	Views
Ray status no process right after runtime comes up	0	362	July 30, 2021
Intial setup for ray on a HPC Ray Serve	4	226	January 20, 2024
Some Issues When I Start My Ray Cluster in centos 7 Ray Clusters	4	539	January 28, 2022
Error while trying to connec to ray cluster from docker Ray Clusters	4	333	July 15, 2021
2023-03-19 13:38:56,574 WARNING utils.py:1445 -- Unable to connect to GCS at gaowei0155.69.142.146:8901 Ray Core	1	302	March 21, 2023

Cannot start Ray runtime on GCP

Related Topics