The University of Arizona
    For questions, please open a UAService ticket and assign to the Tools Team.
Page tree

Versions Compared


  • This line was added.
  • This line was removed.
  • Formatting was changed.


Image Modified

Image Added


Table of Contents

Containers Overview

Shipping containers have frequently

A container is a packaged unit of software that contains code and all its dependencies including, but not limited to, system tools, libraries, settings, and data. This makes applications and pipelines portable and reproducible, allowing for a consistent environment that can run on multiple platforms.

Shipping containers have frequently been used as an analogy

for computing containers

because the container is standard,


does not care what is put inside, and


will be carried on any ship; or in the case of computing containers, it can run on many

systems different from the one it was created on

different systems.

Docker is widely used by researchers

for reasons we won't get into here.  See their documentation.  Docker images

, however, Docker images require root privileges which means they cannot be run in


an HPC environment

.  This has to do with the privileges required to run a Docker container


 This issue is addressed with Singularity.  It is a container technology that completely contains

Singularity addresses this by completely containing the authority so that all privileges needed at runtime stay inside the container.


This makes it ideal for the shared environment of a supercomputer.

 And even

Even better

is that

, a Docker image can be encapsulated inside a Singularity image.

 So the documentation here instructs how to take either

Some ideal use cases that can be supported by Singularity on HPC include:

  • You already use Docker and want to run your jobs on HPC.
  • You want to preserve your environment so a system change will not affect your work.
  • You need newer or different libraries than are offered on the system.
  • Someone else developed a workflow using a different version of Linux.
  • You prefer to use a Linux distribution other than CentOS (e.g. Ubuntu).
  • You want a container with a database server like MariaDB.

The documentation here provides instructions on how to either take a Docker image and run it from Singularity, or create an image using Singularity only. 

Image Added

Accessing Singularity


Singularity enables users to have full control of their environment. Singularity containers can be used to package entire scientific workflows, software and libraries, and even data. This means that you don’t have to ask your cluster admin to install anything for you - you can put it in a Singularity container and run. Did you already invest in Docker? The Singularity software can import your Docker images without having Docker installed or being a superuser. Need to share your code? Put it in a Singularity container and your collaborator won’t have to go through the pain of installing missing dependencies. Do you need to run a different operating system entirely? You can “swap out” the operating system on your host for a different one within a Singularity container. As the user, you are in control of the extent to which your container interacts with its host. There can be seamless integration, or little to no communication at all.  They have extensive documentation at their website.

Image Removed

Here are some of the use cases we support using Singularity:

  • You already use Docker and want to run your jobs on HPC
  • You want to preserve your environment so that a system change will not affect your work
  • You need newer or different libraries than are offered on HPC systems
  • Someone else developed the workflow using a different version of linux
  • You prefer to use a Linux distribution other than CentOS, perhaps Ubuntu 
  • You want a container with a database server like MariaDB. 

Singularity Commands

On Puma, Singularity is installed in the operating system, so the commands will work without any extra effort.

On Ocelote and ElGato, you will first need to module load singularity

Code Block
$ singularity --help

Linux container platform optimized for High Performance Computing (HPC) 

   singularity [global options...]


  Singularity containers provide an application virtualization layer enabling 
 of compute via both application and environment portability. With one is 
 capable of building a root file system that runs on any Linux system where 
 Singularity is installed.

  -d, --debug              print debugging information (highest verbosity)
  -h, --help               help for singularity
  -q, --quiet              suppress normal output
  -s, --silent             only print errors
  -t, --tokenfile string   path to the file holding your sylabs
                           authentication token (default
  -v, --verbose            print additional information
      --version            version for singularity

Available Commands:
  build          Build a new Singularity container
  capability  Manage Linux capabilities on containers
  exec          Execute a command within container
  help          Help about any command
  inspect     Display metadata for container if available
  instance   Manage containers running in the background
  keys          Manage OpenPGP key stores
  pull           Pull a container from a URI
  push         Push a container to a Library URI
  run            Launch a runscript within container
  run-help    Display help for container if available
  search      Search the library
  shell        Run a Bourne shell within container
  sign         Attach cryptographic signatures to container
  test          Run defined tests for this particular container
  verify       Verify cryptographic signatures on container
  version     Show application version


  $ singularity help <command>

      Additional help for any Singularity subcommand can be seen by appending
      the subcommand name to the above command.

Singularity Changes in Version 3.x

For existing users two of the biggest changes are; that the file type of images now is ".sif" for Singularity Image Format for cryptographically signed and verifiable container images; and that your container may not run outside your home directory unless you include binding to other directories like /xdisk

Singularity Hub

Singularity Hub lets you build and keep containers at their Hub.  You maintain your recipes there and each time you need to pull one, it gets built there and then you retrieve the container.

This is very convenient for the scenario where you do not have access to root authority to build the container. The build takes place through the Hub.

This also lets you share containers

Singularity Remote Builder (root access)

An earlier limitation of Singularity was the requirement for access to a root account to build a container.  You will not have root access on a HPC cluster.  Singularity 3.0 introduced the ability to build a container in the cloud negating the root restriction.

Here is an example:

  •  Log into
  • Generate an access token (API key)
  • From your working directory: module load singularity
  • singularity remote login and paste in the API key
  • singularity build --remote ~/nersc.sif nersc.recipe
  • This will produce INFO:  Build complete:  /home/u13/netid/nersc.sif  where
  • nersc.recipe is
    Code Block
    BootStrap: docker From: nersc/ubuntu-mpi:14.04

    on HPC

    Singularity is installed on the operating systems of all HPC compute nodes, so can be easily accessed either from an interactive session or batch script without worrying about software modules. 

    Building a Container

    Local Builds

    Building a container locally requires root authority which users do not have on HPC. This means you must use a Mac or Linux workstation where you have sudo privileges and Singularity installed. The Sylabs website has instructions that can help users get started on building their own containers. Additionally, Nvidia provides an HPC Container Maker which lets you build a recipe without having to know all the syntax. You will just include the blocks you need (e.g., Cuda or Infiniband) and it will create the recipe that you can use for a build on your local workstation.

    Remote Builds

    To bypass the issue of needing root privileges to build your container, Singularity Hub lets you build and keep containers in the cloud as well as share them with other users. You maintain your recipes there and each time you need to pull one, it gets built remotely and is retrieved to your workstation. This conveniently allows you to build containers directly from HPC. 

    As an example, if you want to build a container in your account, first go to, generate an access token (API key), and save it to your clipboard. Next, log in to an interactive terminal session and find your recipe file. In this example, we'll use the recipe:

    Code Block
    BootStrap: docker
    From: nersc/ubuntu-mpi:14.04
        echo "This is what happens when you run the container..."

    Then, assuming the recipe is stored in our home directory, we can build it remotely using:

    Code Block
    $ singularity remote login   # paste in your API key at the prompt
    $ singularity build --remote ~/nersc.sif ~/nersc.recipe

    This will produce a .sif file in your home directory that is ready for use.

    Singularity, Nvidia and GPU's



    Image Added


    One of the most significant use cases for Singularity is to support machine learning workflows. For information on using GPUs on HPC, see our GPU documentation.

    Pulling Nvidia Images

    The NVIDIA GPU Cloud (NGC) provides GPU-accelerated HPC and deep learning containers for scientific computing.  NVIDIA tests HPC container compatibility with the Singularity runtime through a rigorous QA process. Application-specific information may vary so it is recommended that you follow the container-specific documentation before running with Singularity. If the container documentation does not include Singularity information, then the container has not yet been tested under Singularity.

    Pulling Images

    • The containers from nvidia that are in /contrib have been modified to include path bindings to /xdisk and /groups. They also include the path to the Nvidia commands like nvidia-smi.
    • Because login nodes are small and do not provide software, singularity images should be pulled and executed on a compute node.

    The general form to pull and convert a NGC image to a local Singularity image file is:

    Code Block
    $ singularity build <local_image> docker://<registry>/<app:tag>

    This Singularity build command will download the app:tag NGC Docker image, convert it to Singularity format, and save it to the local filename local_image. For example, to pull the namd NGC container tagged with version 2.12-171025 to a local file named namd.simg saved to your home directory:

    Code Block
    $ singularity build ~/namd.simg docker://


    Directory access:

    Singularity containers are themselves ostensibly read only. In order to provide application input and output host directories are generally bound to the container, this is accomplished through the Singularity -B flag. The format of this flag is -B <host_src_dir>:<container_dst_dir>. Once a host directory, host_src_dir, is bound into the container you may interact with this directory from within the container, located at container_dst_dir, the same as you would outside the container.

    You may also make use of the --pwd <container_dir> flag, which will be used to set the present working directory of the command to be run within the container.

    Ocelote does not support filesystem overlay and as such the container_dst_dir must exist within the image for a bind to be successful. To get around the inability to bind arbitrary directories $HOME and /tmp are mounted in automatically and may be used for application I/O.

    GPU support:

    All NGC containers are optimized for NVIDIA GPU acceleration so you will always want to add the --nv flag to enable NVIDIA GPU support within the container.

    Standard run command:

    The Singularity command below represents the canonical form that will be used on the Ocelote cluster.

    Code Block
    $ singularity exec --nv --pwd <work_dir> <image.simg> <cmd>   # <work_dir> should be set to either $HOME or /tmp

    Containers Available on HPC

    We support the use of HPC and ML/DL containers available on NVIDIA GPU Cloud (NGC). Many of the popular HPC applications including NAMD, LAMMPS and GROMACS containers are optimized for performance and available to run in Singularity on Ocelote or Puma. The containers and respective README files can be found in /contrib/singularity/nvidia. 

    • The Nvidia images have been modified to include bindings for your /xdisk and /groups directories if you want to run your jobs there
    • The filename has a tag at the end that represents when it was made. For example, 18.01 is January 2018.

    nvidia-caffe.18.09-py2.simgCaffe is a deep learning framework made with expression, speed, and modularity in mind. It was originally developed by the Berkeley Vision and Learning Center (BVLC) 

    PyTorch is a Python package that provides two high-level features:

    • Tensor computation (like numpy) with strong GPU acceleration
    • Deep Neural Networks built on a tape-based autograd system


    MXNet is a deep learning framework designed for both efficiency and flexibility. It allows you to mix the flavors of symbolic programming and imperative programming to maximize efficiency and productivity.
    nvidia-tensorflow.18.09-py3.simgTensorFlow is an open source software library for numerical computation using data flow graphs. TensorFlow was originally developed by researchers and engineers working on the Google Brain team within Google's Machine Intelligence research organization for the purposes of conducting machine learning and deep neural networks research.
    nvidia-theano.18.08.simgTheano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently.


    Simple Example

    The lolcow image is often used as the standard "hello world!" introduction to containers and is described in Singularity's documentation. To follow their example, first start by logging into an interactive terminal session and pull the image:

    Code Block
    $  singularity pull docker://godlovedc/lolcow
    INFO:    Converting OCI blobs to SIF format
    INFO:    Starting build...
    Getting image source signatures
    Writing manifest to image destination
    Storing signatures
    INFO:    Creating SIF file...
    INFO:    Build complete: /home/uxx/netid/.singularity/cache/oci-tmp/a692b57abc43035b197b10390ea2c12855d21649f2ea2cc28094d18b93360eeb/lolcow_latest.sif

    This will pull the image from Docker Hub and save it in your home in a hidden directory .singularity. Next, run the image simply using singularity run

    Code Block
    $ singularity run lolcow_latest.sif
    / Perilous to all of us are the devices \
    | of an art deeper than we ourselves    |
    | possess.                              |
    |                                       |
    | -- Gandalf the Grey [J.R.R. Tolkien,  |
    \ "Lord of the Rings"]                  /
            \   ^__^
             \  (oo)\_______
                (__)\       )\/\
                    ||----w |
                    ||     ||

    Running Singularity in a Batch Job

    Running a job with Singularity is as easy as running other jobs, simply include your resource requests, and include any commands necessary to execute your workflow. For more detailed information on creating and running jobs, see our SLURM documentation or Puma Quick Start. An example script might look like:

    Code Block
    #SBATCH --job-name singularity-job
    #SBATCH --account=your_pi
    #SBATCH --partition=standard
    #SBATCH --nodes=1
    #SBATCH --ntasks=1
    #SBATCH --time=01:00:00
    singularity exec --nv dockerTF.img python

    Example Recipe Files

    CentOS with Tensorflow

    Code Block
    BootStrap: yum
    OSVersion: 7
    Include: yum
    # best to build up container using kickstart mentality.
    # ie, to add more packages to image,
    # re-run bootstrap command again.
    # bootstrap on existing image will build on top of it, not overwriting it/restarting from scratch
    # singularity .def file is like kickstart file
    # unix commands can be run, but if there is any error, the bootstrap process ends
       # commands to be executed on host outside container during bootstrap
         # commands to be executed inside container during bootstrap
         # add python and install some packages
        yum -y install vim wget python3 epel-release
         # install tensorflow
        pip3 install --upgrade pip
        pip3 install tensorflow-gpu==2.0.0-rc1
         # create bind points for storage.
        mkdir /xdisk
        mkdir /groups
        exit 0
    # %runscript
       # commands to be executed when the container runs
    # %test
       # commands to be executed within container at close of bootstrap process
       python --version

    To build and test a container from the recipe from an interactive session on a GPU node:

    Code Block
    $ singularity build centosTflow.sif centosTflow.def # Remember, you will need to either build this on a workstation where you have root privileges or will need to user a --remote build
    $ singularity exec --nv centosTFlow.simg python3

    As a tensorflow example, you could use the following script:

    Code Block
    #Linear Regression Example with TensorFlow v2 library 
    from __future__ import absolute_import, division, print_function
    import tensorflow as tf
    import numpy as np
    rng = np.random
    # Parameters.
    learning_rate = 0.01
    training_steps = 1000
    display_step = 50
    # Training Data.
    X = np.array([3.3,4.4,5.5,6.71,6.93,4.168,9.779,6.182,7.59,2.167,
    Y = np.array([1.7,2.76,2.09,3.19,1.694,1.573,3.366,2.596,2.53,1.221,
    n_samples = X.shape[0]
    # Weight and Bias, initialized randomly.
    W = tf.Variable(rng.randn(), name="weight")
    b = tf.Variable(rng.randn(), name="bias")
    # Linear regression (Wx + b).
    def linear_regression(x):
        return W * x + b
    # Mean square error.
    def mean_square(y_pred, y_true):
        return tf.reduce_sum(tf.pow(y_pred-y_true, 2)) / (2 * n_samples)
    # Stochastic Gradient Descent Optimizer.
    optimizer = tf.optimizers.SGD(learning_rate)
    # Optimization process. 
    def run_optimization():
    # Wrap computation inside a GradientTape for automatic differentiation.
        with tf.GradientTape() as g:
            pred = linear_regression(X)
            loss = mean_square(pred, Y)
        # Compute gradients.
        gradients = g.gradient(loss, [W, b])    
        # Update W and b following gradients.
        optimizer.apply_gradients(zip(gradients, [W, b]))
    # Run training for the given number of steps.
    for step in range(1, training_steps + 1):
        # Run the optimization to update W and b values.
        if step % display_step == 0:
            pred = linear_regression(X)
            loss = mean_square(pred, Y)
            print("step: %i, loss: %f, W: %f, b: %f" % (step, loss, W.numpy(), b.numpy()))


    Singularity supports MPI pretty well since, by default, the network is the same inside and outside the container. The more complicated bit is making sure that the container has the right set of MPI libraries. MPI is an open specification, but there are several different implementations (OpenMPI, MVAPICH2, and Intel MPI to name three) with some non-overlapping feature sets. If the host and container are running different MPI implementations, or even different versions of the same implementation, hilarity may ensue. 

    The general rule is that you want the version MPI inside the container to be the same version or newer than the host. You may be thinking that this is not good for the portability of your container and you are right. Containerizing MPI applications is not terribly difficult with Singularity, but it comes at the cost of additional requirements for the host system. 

    In this example, the infiniband pieces are installed and then the MVAPICH version of MPI. When the job is run, the script will need to load the correct module with the matching version of MVAPICH.

    Code Block
    titleMPI Recipe File
    BootStrap: debootstrap
    OSVersion: xenial
        echo "This is what happens when you run the container..."
    Singularity Tutorials, Examples, and Applications

    Singularity, Nvidia and GPU's

    One of the most significant use cases for Singularity is to support machine learning workflows.  The details are in the GPU section. 
    You can register at their NGC GPU Cloud site and pull your own containers.  You can do this from HPC. Follow these instructions:
    Singularity Tutorials
        echo "Hello from inside the container"
        sed -i 's/$/ universe/' /etc/apt/sources.list
        apt update
        apt -y --allow-unauthenticated install vim build-essential wget gfortran bison libibverbs-dev libibmad-dev libibumad-dev librdmacm-dev libmlx5-dev libmlx4-dev
        tar xvf mvapich2-2.1.tar.gz
        cd mvapich2-2.1
        ./configure --prefix=/usr/local
        make -j4
        make install
        /usr/local/bin/mpicc examples/hellow.c -o /usr/bin/hellow