Best practices with secrets on GCP

How severe does this issue affect your experience of using Ray?

  • None: Just asking a question out of curiosity

I am using GCP regurlarly to run my Ray workloads. So far I have not yet used private repositories hosted on GitHub. Are there any good practices how to store the secrets and use them, e.g. Google Cloud Secret Manager with corresponding commands in the node setups (are there examples you could point me to)?

Thanks for any insights

So I have created my secrets with Google Cloud Secret Manager that give me access to private git repos. I added the following commands to the head_setup_commands in the cluster.yaml:

  - pip install google-api-python-client==1.7.8
  - ssh-keygen -F || ssh-keyscan -t rsa > ~/.ssh/known_hosts.github
  - gcloud secrets versions access 1 --secret="my_secret_name" --out-file="my_secret_key"
  - mv my_secret_key ~/.ssh/my_secret_key
  - eval "$(ssh-agent -s)"
  - ssh-add ~/.ssh/my_secret_key
  - git clone

I get an error:

New status: update-failed
  {'message': 'SSH command failed.'}
  SSH command failed.
  Failed to setup head node.

More detailed:

(2/8) ssh-keygen -F || ssh-keyscan -t rsa > ~/.ssh/known_hosts.github
    Running `docker exec -it  ray_container /bin/bash -c 'bash --login -c -i '"'"'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (ssh-keygen -F || ssh-keyscan -t rsa > /home/ray/.ssh/known_hosts.github)'"'"'' `
Cannot stat /home/ray/.ssh/known_hosts: No such file or directory
# SSH-2.0-babeld-f33ee42f
Shared connection to closed.
(3/8) gcloud secrets versions access 1 --secret="my_secret" --out-file="my_secret_key"
    Running `docker exec -it  ray_container /bin/bash -c 'bash --login -c -i '"'"'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (gcloud secrets versions access 1 --secret="my_secret" --out-file="my_secret_key")'"'"'' `
bash: gcloud: command not found
Shared connection to closed.

I guess these errors occur especially because the commands are executed inside the Docker (because on the VM directly they do work). The Docker appears to have no access to the secrets via gcloud and it has not the .ssh directory at where I assume it to be.

How is it possible to clone private repos using secrets (or not, if an easier workflow exists)?

@Dmitri do you have an idea here?

I solved my issue now by creating a Personal GitHub Access Token and using this in the bash commands in the setup:

  - pip install google-api-python-client==1.7.8
  - if [[ -d <REPO_NAME> ]]; then clone https://<GIT_USER_NAME>:<GIT_ACCESS_TOKEN><GIT_USER_NAME>/<REPO_NAME>.git; fi

This is not the most secure solution, but it works (and we still have TLS). I am still interested in a more GCP-conform way to do it.