Testing code changes in workspace with minikube

Ray Core
1

Hello,

I have code changes, specifically additional loggers in autoscaler which I would like to deploy and test in minikube env, I have the followed building steps mentioned here:

https://docs.ray.io/en/master/development.html#building-ray-python-only

The example cluster that I am deploying pulls ray-nightly containers, can you please help how can I test code changes in my workspace by deploying on minikube ?

2

Maybe you could try building your own docker image and then using that for the docker setup? I think you can use the following to create your own image:

cd ray
./build-docker.sh
1 Like
3

Hi @rliaw ,

I see below docker images -

$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
e82dcdd7e122 2 minutes ago 1.27GB
rayproject/base-deps nightly 23c7768bc874 3 minutes ago 582MB
2f32d7c0bb50 3 minutes ago 582MB
rayproject/ray-deps nightly d1d33793e182 2 hours ago 1.27GB
ubuntu focal 7e0aa2d69a15 5 days ago 72.7MB

Can you please help, which images should I get added to cluster-config

Thanks

4

sharing stdout if this helps:

$ ./build-docker.sh --build-development-image

  • GPU=
  • BASE_IMAGE=ubuntu:focal
  • WHEEL_URL=https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-1.3.0-cp37-cp37m-manylinux2014_x86_64.whl
  • PYTHON_VERSION=3.7.7
  • [[ 1 -gt 0 ]]
  • key=–build-development-image
  • case $key in
  • BUILD_DEV=YES
  • shift
  • [[ 0 -gt 0 ]]
    ++ mktemp -d
  • WHEEL_DIR=/tmp/tmp.X6jJerD8cO
  • wget --quiet https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-1.3.0-cp37-cp37m-manylinux2014_x86_64.whl -P /tmp/tmp.X6jJerD8cO
    ++ basename ‘/tmp/tmp.X6jJerD8cO/*.whl’
  • WHEEL=’/tmp/tmp.X6jJerD8cO/*.whl’
  • for IMAGE in “base-deps” “ray-deps” “ray”
    ++ basename ‘/tmp/tmp.X6jJerD8cO/*.whl’
  • cp ‘/tmp/tmp.X6jJerD8cO/.whl’ 'docker/base-deps/.whl’
    cp: cannot stat ‘/tmp/tmp.X6jJerD8cO/*.whl’: No such file or directory
  • ‘[’ ‘]’
    ++ basename ‘/tmp/tmp.X6jJerD8cO/*.whl’
  • docker build --build-arg GPU= --build-arg BASE_IMAGE=ubuntu:focal --build-arg 'WHEEL_PATH=.whl’ --build-arg PYTHON_VERSION=3.7.7 -t rayproject/base-deps:nightly docker/base-deps
    Sending build context to Docker daemon 7.168kB
    Step 1/14 : ARG BASE_IMAGE=“ubuntu:focal”
    Step 2/14 : FROM ${BASE_IMAGE}
    —> 7e0aa2d69a15
    Step 3/14 : ARG AUTOSCALER=“autoscaler”
    —> Using cache
    —> 30ec13c2a17b
    Step 4/14 : ENV TZ=America/Los_Angeles
    —> Using cache
    —> e1e31965cba0
    Step 5/14 : ENV PATH “/home/ray/anaconda3/bin:$PATH”
    —> Using cache
    —> b8ee584764a3
    Step 6/14 : ARG DEBIAN_FRONTEND=noninteractive
    —> Using cache
    —> 8bed87666b18
    Step 7/14 : ARG PYTHON_VERSION=3.7.7
    —> Using cache
    —> e96e2c8b687b
    Step 8/14 : ARG RAY_UID=1000
    —> Using cache
    —> 2b56c72e05bf
    Step 9/14 : ARG RAY_GID=100
    —> Using cache
    —> c8877bffdc32
    Step 10/14 : RUN apt-get update -y && apt-get install -y sudo tzdata && useradd -ms /bin/bash -d /home/ray ray --uid $RAY_UID --gid $RAY_GID && usermod -aG sudo ray && echo ‘ray ALL=NOPASSWD: ALL’ >> /etc/sudoers && rm -rf /var/lib/apt/lists/     && apt-get clean
    —> Using cache
    —> 59745b084856
    Step 11/14 : USER $RAY_UID
    —> Using cache
    —> 718e349cbd98
    Step 12/14 : ENV HOME=/home/ray
    —> Using cache
    —> d336f35c5dfe
    Step 13/14 : RUN sudo apt-get update -y && sudo apt-get upgrade -y && sudo apt-get install -y git wget cmake g++ zlib1g-dev $(if [ “$AUTOSCALER” = “autoscaler” ]; then echo tmux screen rsync openssh-client gnupg; fi) && wget --quiet “https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh” -O /tmp/miniconda.sh && /bin/bash /tmp/miniconda.sh -b -u -p $HOME/anaconda3 && $HOME/anaconda3/bin/conda init && echo ‘export PATH=$HOME/anaconda3/bin:$PATH’ >> /home/ray/.bashrc && rm /tmp/miniconda.sh && $HOME/anaconda3/bin/conda install -y libgcc python=$PYTHON_VERSION && $HOME/anaconda3/bin/conda clean -y --all && $HOME/anaconda3/bin/pip install --no-cache-dir flatbuffers cython==0.29.0 numpy==1.15.4 psutil blist atari-py && $HOME/anaconda3/bin/pip uninstall -y dask && sudo apt-get autoremove -y cmake g++ zlib1g-dev && (if [ “$AUTOSCALER” = “autoscaler” ]; then wget -O - -q https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && sudo touch /etc/apt/sources.list.d/kubernetes.list && echo “deb http://apt.kubernetes.io/ kubernetes-xenial main” | sudo tee -a /etc/apt/sources.list.d/kubernetes.list && sudo apt-get update && sudo apt-get install kubectl; else sudo apt-get autoremove -y wget; fi;) && sudo rm -rf /var/lib/apt/lists/* && sudo apt-get clean
    —> Using cache
    —> 5dd5fd9773da
    Step 14/14 : WORKDIR $HOME
    —> Using cache
    —> 23c7768bc874
    [Warning] One or more build-args [GPU WHEEL_PATH] were not consumed
    Successfully built 23c7768bc874
    Successfully tagged rayproject/base-deps:nightly
    ++ basename ‘/tmp/tmp.X6jJerD8cO/*.whl’
  • rm ‘docker/base-deps/.whl’
    rm: cannot remove 'docker/base-deps/    .whl’: No such file or directory
  • for IMAGE in “base-deps” “ray-deps” “ray”
    ++ basename ‘/tmp/tmp.X6jJerD8cO/*.whl’
  • cp ‘/tmp/tmp.X6jJerD8cO/.whl’ 'docker/ray-deps/.whl’
    cp: cannot stat ‘/tmp/tmp.X6jJerD8cO/*.whl’: No such file or directory
  • ‘[’ ‘]’
    ++ basename ‘/tmp/tmp.X6jJerD8cO/*.whl’
  • docker build --build-arg GPU= --build-arg BASE_IMAGE=ubuntu:focal --build-arg ‘WHEEL_PATH=.whl’ --build-arg PYTHON_VERSION=3.7.7 -t rayproject/ray-deps:nightly docker/ray-deps
    Sending build context to Docker daemon 5.12kB
    Step 1/8 : ARG GPU=""
    Step 2/8 : FROM rayproject/base-deps:nightly"$GPU"
    —> 23c7768bc874
    Step 3/8 : ARG AUTOSCALER=“autoscaler”
    —> Using cache
    —> c4694dacad30
    Step 4/8 : ARG WHEEL_PATH
    —> Using cache
    —> 597aa98d1c12
    Step 5/8 : ENV LC_ALL=C.UTF-8
    —> Using cache
    —> 7480ecf550db
    Step 6/8 : ENV LANG=C.UTF-8
    —> Using cache
    —> 2f32d7c0bb50
    Step 7/8 : COPY $WHEEL_PATH .
    COPY failed: no source files were specified
    ++ basename '/tmp/tmp.X6jJerD8cO/    .whl’
  • rm ‘docker/ray-deps/.whl’
    rm: cannot remove 'docker/ray-deps/    .whl’: No such file or directory
  • for IMAGE in “base-deps” “ray-deps” “ray”
    ++ basename ‘/tmp/tmp.X6jJerD8cO/*.whl’
  • cp ‘/tmp/tmp.X6jJerD8cO/.whl’ 'docker/ray/.whl’
    cp: cannot stat ‘/tmp/tmp.X6jJerD8cO/*.whl’: No such file or directory
  • ‘[’ ‘]’
    ++ basename ‘/tmp/tmp.X6jJerD8cO/*.whl’
  • docker build --build-arg GPU= --build-arg BASE_IMAGE=ubuntu:focal --build-arg ‘WHEEL_PATH=.whl’ --build-arg PYTHON_VERSION=3.7.7 -t rayproject/ray:nightly docker/ray
    Sending build context to Docker daemon 4.096kB
    Step 1/7 : ARG GPU
    Step 2/7 : FROM rayproject/ray-deps:nightly"$GPU"
    —> d1d33793e182
    Step 3/7 : ARG WHEEL_PATH
    —> Using cache
    —> ecd74df56f5b
    Step 4/7 : ENV LC_ALL=C.UTF-8
    —> Using cache
    —> f5b1c7822a3b
    Step 5/7 : ENV LANG=C.UTF-8
    —> Using cache
    —> e82dcdd7e122
    Step 6/7 : COPY $WHEEL_PATH .
    COPY failed: no source files were specified
    ++ basename '/tmp/tmp.X6jJerD8cO/    .whl’
  • rm ‘docker/ray/.whl’
    rm: cannot remove 'docker/ray/    .whl’: No such file or directory
  • ‘[’ YES ‘]’
  • git rev-parse HEAD
    ++ git rev-parse HEAD
  • git archive -o ./docker/development/ray.tar 2a02b97f1ae552debd2071985e23e2da418ed906
  • ‘[’ ‘]’
  • docker build --no-cache -t rayproject/development docker/development
    Sending build context to Docker daemon 35.68MB
    Step 1/13 : FROM rayproject/ray-deps:latest
    latest: Pulling from rayproject/ray-deps
    345e3491a907: Already exists
    57671312ef6f: Already exists
    5e9250ddb7d0: Already exists
    eb719956b105: Pull complete
    8cd8a3afa23c: Pull complete
    cd3cf60c1c0b: Pull complete
    11f3db4e7797: Pull complete
    Digest: sha256:bf96885ce5b196af9373ba258c9be74a5e1b43afe65411ad061bdbfcd05828c4
    Status: Downloaded newer image for rayproject/ray-deps:latest
    —> b791cde6d299
    Step 2/13 : ADD ray.tar /ray
    —> 0bda8bb7c522
    Step 3/13 : ADD git-rev /ray/git-rev
    —> 7c89be128584
    Step 4/13 : RUN sudo apt-get update && sudo apt-get install -y curl unzip cmake gcc g++ && sudo apt-get clean
    —> Running in d1e61a6781a0
    Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [109 kB]
    Get:3 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
    Get:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9383 B]
    Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [783 kB]
    Get:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
    Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [46.2 kB]
    Get:7 http://archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB]
    Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB]
    Get:9 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [21.7 kB]
    Get:10 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [690 kB]
    Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [243 kB]
    Get:12 http://archive.ubuntu.com/ubuntu focal/restricted amd64 Packages [33.4 kB]
    Get:13 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [177 kB]
    Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1275 kB]
    Get:15 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [29.7 kB]
    Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [958 kB]
    Get:17 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [274 kB]
    Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1198 kB]
    Get:19 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [4305 B]
    Fetched 17.7 MB in 3s (5429 kB/s)
    Reading package lists…
    Reading package lists…
    Building dependency tree…
    Reading state information…
    Some packages could not be installed. This may mean that you have
    requested an impossible situation or if you are using the unstable
    distribution that some required packages have not yet been created
    or been moved out of Incoming.
    The following information may help to resolve the situation:

The following packages have unmet dependencies:
g++ : Depends: g+±9 (>= 9.3.0-3~) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
The command ‘/bin/sh -c sudo apt-get update && sudo apt-get install -y curl unzip cmake gcc g++ && sudo apt-get clean’ returned a non-zero code: 100

  • rm ./docker/development/ray.tar ./docker/development/git-rev
  • ‘[’ ‘]’
  • rm -rf /tmp/tmp.X6jJerD8cO

I see that command failed

5

Hello I have install gcc 9 on ubuntu 18 but I am still unable to build the container:

$ gcc --version
gcc (Ubuntu 9.3.0-11ubuntu0~18.04.1) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.5 LTS
Release: 18.04
Codename: bionic

I get the same error as posted above :

The following packages have unmet dependencies:
g++ : Depends: g+±9 (>= 9.3.0-3~) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
The command ‘/bin/sh -c sudo apt-get update && sudo apt-get install -y curl unzip cmake gcc g++ && sudo apt-get clean’ returned a non-zero code: 100

Any help is appreciated!

6

Alternatively, you could do start a new container using a pre-built image, modify some of the ray source code, then do docker commit to create a new image with your modifications.

1 Like