Running a ray tune example from within a singularity container

jchen6727 · May 12, 2023, 2:52pm

Would like to bump, we have new on prem HPC using singularity that it would be cost effective for me to migrate my code to.

However, I am not confident that a ray implementation will work in general, so…

github.com/apptainer/singularity

Singularity Built From NGC Base Yields "unable to find libcuda.so.1"

opened 07:43PM - 14 Apr 21 UTC

closed 07:20PM - 21 Apr 21 UTC

cupdike

Question cannot reproduce

``` $ singularity --version singularity version 3.5.3 ``` ### Expected behav…ior After building a sif off of a docker NGC base, cuda driver files should remain intact as they were on the docker image. ### Actual behavior The cudu driver files are 0 bytes on the singularity image: ``` # Docker $ srun --pty docker run -it --rm native_horovod_ray2 bash -c 'ls -al /usr/lib/x86_64-linux-gnu/libcuda.so*' lrwxrwxrwx 1 root root 12 Apr 14 17:37 /usr/lib/x86_64-linux-gnu/libcuda.so -> libcuda.so.1 lrwxrwxrwx 1 root root 20 Apr 14 17:37 /usr/lib/x86_64-linux-gnu/libcuda.so.1 -> libcuda.so.455.45.01 -rw-r--r-- 1 root root 16331696 Dec 27 19:02 /usr/lib/x86_64-linux-gnu/libcuda.so.418.181.07 -rwxr-xr-x 1 root root 21070200 Nov 5 23:13 /usr/lib/x86_64-linux-gnu/libcuda.so.455.45.01 # Singularity $ singularity run native_horray2.sif bash -c 'ls -alL /usr/lib/x86_64-linux-gnu/libcuda.so*' -rwxr-xr-x 1 root root 0 Apr 14 17:37 /usr/lib/x86_64-linux-gnu/libcuda.so -rwxr-xr-x 1 root root 0 Apr 14 17:37 /usr/lib/x86_64-linux-gnu/libcuda.so.1 -rw-r--r-- 1 root root 0 Apr 14 17:37 /usr/lib/x86_64-linux-gnu/libcuda.so.418.181.07 -rwxr-xr-x 1 root root 0 Apr 14 17:37 /usr/lib/x86_64-linux-gnu/libcuda.so.455.45.01 ``` ### Steps to reproduce this behavior ``` cat Dockerfile FROM nvidia/cuda:10.1-cudnn7-devel-ubuntu18.04 CMD nvidia-smi # TensorFlow version is tightly coupled to CUDA and cuDNN so it should be selected carefully ENV TENSORFLOW_VERSION=2.3.2 ENV PYTORCH_VERSION=1.6.0 ENV TORCHVISION_VERSION=0.7.0 ENV CUDNN_VERSION=7.6.5.32-1+cuda10.1 ENV NCCL_VERSION=2.7.8-1+cuda10.1 ENV MXNET_VERSION=1.6.0.post0 ENV PYSPARK_PACKAGE=pyspark==2.4.7 ENV SPARK_PACKAGE=spark-2.4.7/spark-2.4.7-bin-hadoop2.7.tgz # Python 3.7 is supported by Ubuntu Bionic out of the box ARG python=3.7 ENV PYTHON_VERSION=${python} # Set default shell to /bin/bash SHELL ["/bin/bash", "-cu"] RUN apt-get update && apt-get install -y --allow-downgrades --allow-change-held-packages --no-install-recommends \ build-essential \ cmake \ g++-7 \ git \ curl \ vim \ wget \ ca-certificates \ libcudnn7=${CUDNN_VERSION} \ libnccl2=${NCCL_VERSION} \ libnccl-dev=${NCCL_VERSION} \ libjpeg-dev \ libpng-dev \ python${PYTHON_VERSION} \ python${PYTHON_VERSION}-dev \ python${PYTHON_VERSION}-distutils \ librdmacm1 \ libibverbs1 \ ibverbs-providers RUN ln -s /usr/bin/python${PYTHON_VERSION} /usr/bin/python RUN curl -O https://bootstrap.pypa.io/get-pip.py && \ python get-pip.py && \ rm get-pip.py # Install TensorFlow, Keras, PyTorch and MXNet RUN pip install future typing packaging RUN pip install tensorflow==${TENSORFLOW_VERSION} \ keras \ h5py RUN PYTAGS=$(python -c "from packaging import tags; tag = list(tags.sys_tags())[0]; print(f'{tag.interpreter}-{tag.abi}')") && \ pip install https://download.pytorch.org/whl/cu101/torch-${PYTORCH_VERSION}%2Bcu101-${PYTAGS}-linux_x86_64.whl \ https://download.pytorch.org/whl/cu101/torchvision-${TORCHVISION_VERSION}%2Bcu101-${PYTAGS}-linux_x86_64.whl RUN pip install mxnet-cu101==${MXNET_VERSION} # Install Spark stand-alone cluster. RUN wget --progress=dot:giga https://archive.apache.org/dist/spark/${SPARK_PACKAGE} -O - | tar -xzC /tmp; \ archive=$(basename "${SPARK_PACKAGE}") bash -c "mv -v /tmp/\${archive/%.tgz/} /spark" # Install PySpark. RUN apt-get update -qq && apt install -y openjdk-8-jdk-headless RUN pip install ${PYSPARK_PACKAGE} # Install Open MPI RUN mkdir /tmp/openmpi && \ cd /tmp/openmpi && \ wget https://www.open-mpi.org/software/ompi/v4.0/downloads/openmpi-4.0.0.tar.gz && \ tar zxf openmpi-4.0.0.tar.gz && \ cd openmpi-4.0.0 && \ ./configure --enable-orterun-prefix-by-default && \ make -j $(nproc) all && \ make install && \ ldconfig && \ rm -rf /tmp/openmpi RUN pip install ray # Install Horovod, temporarily using CUDA stubs RUN ldconfig /usr/local/cuda/targets/x86_64-linux/lib/stubs && \ HOROVOD_GPU_OPERATIONS=NCCL HOROVOD_WITH_TENSORFLOW=1 HOROVOD_WITH_PYTORCH=1 HOROVOD_WITH_MXNET=1 \ pip install --no-cache-dir horovod[all-frameworks] && \ ldconfig # Install OpenSSH for MPI to communicate between containers RUN apt-get install -y --no-install-recommends openssh-client openssh-server && \ mkdir -p /var/run/sshd # Allow OpenSSH to talk to containers without asking for confirmation RUN cat /etc/ssh/ssh_config | grep -v StrictHostKeyChecking > /etc/ssh/ssh_config.new && \ echo " StrictHostKeyChecking no" >> /etc/ssh/ssh_config.new && \ mv /etc/ssh/ssh_config.new /etc/ssh/ssh_config # Download examples RUN apt-get install -y --no-install-recommends subversion && \ svn checkout https://github.com/horovod/horovod/trunk/examples && \ rm -rf /examples/.svn WORKDIR "/examples" ``` docker build . --rm=false --tag native_horovod_ray2 singularity build native_horray2.sif native_horray2/ singularity build --sandbox native_horray2 docker-archive://native_horray2.tar docker run -it --rm native_horovod_ray2 bash -c 'ls -al /usr/lib/x86_64-linux-gnu/libcuda.so*' singularity run native_horray2.sif bash -c 'ls -alL /usr/lib/x86_64-linux-gnu/libcuda.so*' ### What OS/distro are you running RHEL 7.9 ### How did you install Singularity From source using ansible, https://github.com/abims-sbr/ansible-singularity/blob/master/tasks/main.yml

Topic		Replies	Views
RuntimeError : Socket Timeout (ProcessGoupGloo) Ray Libraries (Data, Train, Tune, Serve)	3	816	January 11, 2023
Runtime error while training Ray Train	1	428	August 26, 2022
Ray init failed, but ray status success Ray Core	4	605	March 16, 2024
Getting Started with Ray does not work on any computer I try it Ray Tune	4	1357	September 13, 2023
Error: RuntimeError: No rendezvous handler for env:// Ray Train	5	651	April 5, 2023

Running a ray tune example from within a singularity container

Related Topics