Best practices with secrets on GCP

Ray Clusters
1

How severe does this issue affect your experience of using Ray?

  • None: Just asking a question out of curiosity

I am using GCP regurlarly to run my Ray workloads. So far I have not yet used private repositories hosted on GitHub. Are there any good practices how to store the secrets and use them, e.g. Google Cloud Secret Manager with corresponding commands in the node setups (are there examples you could point me to)?

Thanks for any insights

2

So I have created my secrets with Google Cloud Secret Manager that give me access to private git repos. I added the following commands to the head_setup_commands in the cluster.yaml:

head_setup_commands:
  - pip install google-api-python-client==1.7.8
  - ssh-keygen -F github.com || ssh-keyscan -t rsa github.com > ~/.ssh/known_hosts.github
  - gcloud secrets versions access 1 --secret="my_secret_name" --out-file="my_secret_key"
  - mv my_secret_key ~/.ssh/my_secret_key
  - eval "$(ssh-agent -s)"
  - ssh-add ~/.ssh/my_secret_key
  - git clone git@github.com:my_private_repo.git

I get an error:

New status: update-failed
  !!!
  {'message': 'SSH command failed.'}
  SSH command failed.
  !!!
  
  Failed to setup head node.

More detailed:

(2/8) ssh-keygen -F github.com || ssh-keyscan -t rsa github.com > ~/.ssh/known_hosts.github
    Running `docker exec -it  ray_container /bin/bash -c 'bash --login -c -i '"'"'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (ssh-keygen -F github.com || ssh-keyscan -t rsa github.com > /home/ray/.ssh/known_hosts.github)'"'"'' `
Cannot stat /home/ray/.ssh/known_hosts: No such file or directory
# github.com:22 SSH-2.0-babeld-f33ee42f
Shared connection to 34.82.224.46 closed.
(3/8) gcloud secrets versions access 1 --secret="my_secret" --out-file="my_secret_key"
    Running `docker exec -it  ray_container /bin/bash -c 'bash --login -c -i '"'"'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (gcloud secrets versions access 1 --secret="my_secret" --out-file="my_secret_key")'"'"'' `
bash: gcloud: command not found
Shared connection to 34.82.224.46 closed.

I guess these errors occur especially because the commands are executed inside the Docker (because on the VM directly they do work). The Docker appears to have no access to the secrets via gcloud and it has not the .ssh directory at where I assume it to be.

Question
How is it possible to clone private repos using secrets (or not, if an easier workflow exists)?

@Dmitri do you have an idea here?

3

I solved my issue now by creating a Personal GitHub Access Token and using this in the bash commands in the setup:

head_setup_commands:
  - pip install google-api-python-client==1.7.8
  - if [[ -d <REPO_NAME> ]]; then clone https://<GIT_USER_NAME>:<GIT_ACCESS_TOKEN>@github.com/<GIT_USER_NAME>/<REPO_NAME>.git; fi

This is not the most secure solution, but it works (and we still have TLS). I am still interested in a more GCP-conform way to do it.