Shipping containers have frequently been used as an analogy for computing containers because the container is standard, it does not care what is put inside, and it will be carried on any ship, or in the case of computing containers can run on many systems different from the one it was created on.  Hence the logo for Docker, the most common container platform.

Docker is widely used by researchers for reasons we won't get into here.  See their documentation

Docker images cannot be run in a HPC environment.  This has to do with the privileges required to run a Docker container.  This issue is addressed with Singularity.  It is a container technology that completely contains the authority so that when the image is run all privileges stay inside the container.  This makes it ideal for the shared environment of a supercomputer.  And even better is that a Docker image can be encapsulated inside a Singularity image.  So the documentation here instructs how to take either a Docker image and run it from Singularity or create an image using Singularity only. 





Singularity Overview

Singularity enables users to have full control of their environment. Singularity containers can be used to package entire scientific workflows, software and libraries, and even data. This means that you don’t have to ask your cluster admin to install anything for you - you can put it in a Singularity container and run. Did you already invest in Docker? The Singularity software can import your Docker images without having Docker installed or being a superuser. Need to share your code? Put it in a Singularity container and your collaborator won’t have to go through the pain of installing missing dependencies. Do you need to run a different operating system entirely? You can “swap out” the operating system on your host for a different one within a Singularity container. As the user, you are in control of the extent to which your container interacts with its host. There can be seamless integration, or little to no communication at all.


Here are some of the use cases we support using Singularity:

  • You already use Docker and want to run your jobs on HPC
  • You want to preserve your environment so that a system change will not affect your work
  • You need newer or different libraries than are offered on HPC systems
  • Someone else developed the workflow using a different version of linux
  • You prefer to use something other than Red Hat / CentOS, like Ubuntu 

Singularity Commands

$ singularity --help
USAGE: singularity [global options...] <command> [command options...] ...

GLOBAL OPTIONS:
    -d|--debug    Print debugging information
    -h|--help     Display usage summary
    -s|--silent   Only print errors
    -q|--quiet    Suppress all normal output
       --version  Show application version
    -v|--verbose  Increase verbosity +1
    -x|--sh-debug Print shell wrapper debugging information

GENERAL COMMANDS:
    help       Show additional help for a command or container                  
    selftest   Run some self tests for singularity install                      

CONTAINER USAGE COMMANDS:
    exec       Execute a command within container                               
    run        Launch a runscript within container                              
    shell      Run a Bourne shell within container                              
    test       Launch a testscript within container                             

CONTAINER MANAGEMENT COMMANDS:
    apps       List available apps within a container                           
    bootstrap  *Deprecated* use build instead                                   
    build      Build a new Singularity container                                
    check      Perform container lint checks                                    
    inspect    Display a container's metadata                                   
    mount      Mount a Singularity container image                              
    pull       Pull a Singularity/Docker container to $PWD                      

COMMAND GROUPS:
    image      Container image command group                                    
    instance   Persistent instance command group                                


CONTAINER USAGE OPTIONS:
    see singularity help <command>

For any additional help or support visit the Singularity
website: http://singularity.lbl.gov/



Singularity Changes in Version 2.4


The changes to Singularity have both made it easier to use and more tightly integrated with existing hubs like Docker to support recreatable research.  A significant change is that a container is created for the size it needs and not a pre-defined (guessed at) size. And then it is compressed to take up even less space.
https://github.com/singularityware/singularity/releases



Singularity Hub


Singularity Hub lets you build and keep containers at their Hub.  You maintain your recipes there and each time you need to pull one, it gets built there and then you retrieve the container.

This is very convenient for the scenario where you do not have access to root authority to build the container. The build takes place through the Hub.

This also lets you share containers
https://github.com/singularityhub/singularityhub.github.io/wiki/Build-A-Container#naming-recipes


Singularity Tutorials

There are tutorials located here


Singularity, Nvidia and GPU's

One of the most significant use cases for Singularity is to support machine learning workflows.  The details are in the GPU section. 

Singularity Python and Machine Learning

More tutorials