# Running a ray tune example from within a singularity container **URL:** \ **Category:** Uncategorized\ **Created:** [May 4, 2023, 9:03pm UTC](https://discuss.ray.io/t/running-a-ray-tune-example-from-within-a-singularity-container/10522 "2023-05-04T21:03:36Z")\ **Posts on this page:** 1\ **Showing post:** 2
**Author:** ![jchen6727](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jchen6727/32/4416_2.png) [@jchen6727](https://discuss.ray.io/u/jchen6727)\ **Post date:** [May 12, 2023, 2:52pm UTC](https://discuss.ray.io/t/running-a-ray-tune-example-from-within-a-singularity-container/10522/2 "2023-05-12T14:52:20Z")
Would like to bump, we have new on prem HPC using singularity that it would be cost effective for me to migrate my code to. However, I am not confident that a ray implementation will work in general, so… > > > \`\`\` > $ singularity --version > singularity version 3.5.3 > \`\`\` > \### Expected behav…ior > After building a sif off of a docker NGC base, cuda driver files should remain intact as they were on the docker image. > \### Actual behavior > The cudu driver files are 0 bytes on the singularity image: > > \`\`\` > \# Docker > $ srun --pty docker run -it --rm native\_horovod\_ray2 bash -c 'ls -al /usr/lib/x86\_64-linux-gnu/libcuda.so\*' > lrwxrwxrwx 1 root root 12 Apr 14 17:37 /usr/lib/x86\_64-linux-gnu/libcuda.so -\> libcuda.so.1 > lrwxrwxrwx 1 root root 20 Apr 14 17:37 /usr/lib/x86\_64-linux-gnu/libcuda.so.1 -\> libcuda.so.455.45.01 > \-rw-r--r-- 1 root root 16331696 Dec 27 19:02 /usr/lib/x86\_64-linux-gnu/libcuda.so.418.181.07 > \-rwxr-xr-x 1 root root 21070200 Nov 5 23:13 /usr/lib/x86\_64-linux-gnu/libcuda.so.455.45.01 > > \# Singularity > $ singularity run native\_horray2.sif bash -c 'ls -alL /usr/lib/x86\_64-linux-gnu/libcuda.so\*' > \-rwxr-xr-x 1 root root 0 Apr 14 17:37 /usr/lib/x86\_64-linux-gnu/libcuda.so > \-rwxr-xr-x 1 root root 0 Apr 14 17:37 /usr/lib/x86\_64-linux-gnu/libcuda.so.1 > \-rw-r--r-- 1 root root 0 Apr 14 17:37 /usr/lib/x86\_64-linux-gnu/libcuda.so.418.181.07 > \-rwxr-xr-x 1 root root 0 Apr 14 17:37 /usr/lib/x86\_64-linux-gnu/libcuda.so.455.45.01 > \`\`\` > > \### Steps to reproduce this behavior > \`\`\` > cat Dockerfile > FROM nvidia/cuda:10.1-cudnn7-devel-ubuntu18.04 > CMD nvidia-smi > > \# TensorFlow version is tightly coupled to CUDA and cuDNN so it should be selected carefully > ENV TENSORFLOW\_VERSION=2.3.2 > ENV PYTORCH\_VERSION=1.6.0 > ENV TORCHVISION\_VERSION=0.7.0 > ENV CUDNN\_VERSION=7.6.5.32-1+cuda10.1 > ENV NCCL\_VERSION=2.7.8-1+cuda10.1 > ENV MXNET\_VERSION=1.6.0.post0 > > ENV PYSPARK\_PACKAGE=pyspark==2.4.7 > ENV SPARK\_PACKAGE=spark-2.4.7/spark-2.4.7-bin-hadoop2.7.tgz > > \# Python 3.7 is supported by Ubuntu Bionic out of the box > ARG python=3.7 > ENV PYTHON\_VERSION=${python} > > \# Set default shell to /bin/bash > SHELL \["/bin/bash", "-cu"\] > > RUN apt-get update && apt-get install -y --allow-downgrades --allow-change-held-packages --no-install-recommends \\ > build-essential \\ > cmake \\ > g++-7 \\ > git \\ > curl \\ > vim \\ > wget \\ > ca-certificates \\ > libcudnn7=${CUDNN\_VERSION} \\ > libnccl2=${NCCL\_VERSION} \\ > libnccl-dev=${NCCL\_VERSION} \\ > libjpeg-dev \\ > libpng-dev \\ > python${PYTHON\_VERSION} \\ > python${PYTHON\_VERSION}-dev \\ > python${PYTHON\_VERSION}-distutils \\ > librdmacm1 \\ > libibverbs1 \\ > ibverbs-providers > > RUN ln -s /usr/bin/python${PYTHON\_VERSION} /usr/bin/python > > RUN curl -O https://bootstrap.pypa.io/get-pip.py && \\ > python get-pip.py && \\ > rm get-pip.py > > \# Install TensorFlow, Keras, PyTorch and MXNet > RUN pip install future typing packaging > RUN pip install tensorflow==${TENSORFLOW\_VERSION} \\ > keras \\ > h5py > > RUN PYTAGS=$(python -c "from packaging import tags; tag = list(tags.sys\_tags())\[0\]; print(f'{tag.interpreter}-{tag.abi}')") && \\ > pip install https://download.pytorch.org/whl/cu101/torch-${PYTORCH\_VERSION}%2Bcu101-${PYTAGS}-linux\_x86\_64.whl \\ > https://download.pytorch.org/whl/cu101/torchvision-${TORCHVISION\_VERSION}%2Bcu101-${PYTAGS}-linux\_x86\_64.whl > RUN pip install mxnet-cu101==${MXNET\_VERSION} > > \# Install Spark stand-alone cluster. > RUN wget --progress=dot:giga https://archive.apache.org/dist/spark/${SPARK\_PACKAGE} -O - | tar -xzC /tmp; \\ > archive=$(basename "${SPARK\_PACKAGE}") bash -c "mv -v /tmp/\\${archive/%.tgz/} /spark" > > \# Install PySpark. > RUN apt-get update -qq && apt install -y openjdk-8-jdk-headless > RUN pip install ${PYSPARK\_PACKAGE} > > \# Install Open MPI > RUN mkdir /tmp/openmpi && \\ > cd /tmp/openmpi && \\ > wget https://www.open-mpi.org/software/ompi/v4.0/downloads/openmpi-4.0.0.tar.gz && \\ > tar zxf openmpi-4.0.0.tar.gz && \\ > cd openmpi-4.0.0 && \\ > ./configure --enable-orterun-prefix-by-default && \\ > make -j $(nproc) all && \\ > make install && \\ > ldconfig && \\ > rm -rf /tmp/openmpi > > RUN pip install ray > > \# Install Horovod, temporarily using CUDA stubs > RUN ldconfig /usr/local/cuda/targets/x86\_64-linux/lib/stubs && \\ > HOROVOD\_GPU\_OPERATIONS=NCCL HOROVOD\_WITH\_TENSORFLOW=1 HOROVOD\_WITH\_PYTORCH=1 HOROVOD\_WITH\_MXNET=1 \\ > pip install --no-cache-dir horovod\[all-frameworks\] && \\ > ldconfig > > \# Install OpenSSH for MPI to communicate between containers > RUN apt-get install -y --no-install-recommends openssh-client openssh-server && \\ > mkdir -p /var/run/sshd > > \# Allow OpenSSH to talk to containers without asking for confirmation > RUN cat /etc/ssh/ssh\_config | grep -v StrictHostKeyChecking \> /etc/ssh/ssh\_config.new && \\ > echo " StrictHostKeyChecking no" \>\> /etc/ssh/ssh\_config.new && \\ > mv /etc/ssh/ssh\_config.new /etc/ssh/ssh\_config > > \# Download examples > RUN apt-get install -y --no-install-recommends subversion && \\ > svn checkout https://github.com/horovod/horovod/trunk/examples && \\ > rm -rf /examples/.svn > > WORKDIR "/examples" > \`\`\` > docker build . --rm=false --tag native\_horovod\_ray2 > singularity build native\_horray2.sif native\_horray2/ > singularity build --sandbox native\_horray2 docker-archive://native\_horray2.tar > docker run -it --rm native\_horovod\_ray2 bash -c 'ls -al /usr/lib/x86\_64-linux-gnu/libcuda.so\*' > singularity run native\_horray2.sif bash -c 'ls -alL /usr/lib/x86\_64-linux-gnu/libcuda.so\*' > > \### What OS/distro are you running > RHEL 7.9 > > \### How did you install Singularity > From source using ansible, https://github.com/abims-sbr/ansible-singularity/blob/master/tasks/main.yml > [@RayExecutors Launched Using with Singularity Don't See GPUs](https://discuss.ray.io/t/rayexecutors-launched-using-with-singularity-dont-see-gpus/1846): > > I set up a SLURM job to do distributed training using Ray running with Singularity containers. The SLURM command launches a process that does the typical Ray commands to set up RayExecutors. What happens is that this process sees the GPUs without issue. However, the RayExecutor worker processes cannot see the GPUs. It is detailed extensively in this issue opened against Singularity: I closed that issue because it does not look like it’s a Singularity problem. It appears that the worker p… --- _[View the full topic](https://discuss.ray.io/t/running-a-ray-tune-example-from-within-a-singularity-container/10522)._