Ocelote has 46 new compute nodes with Nvidia P100 GPU's.  These are available to researchers on campus.  There will be fairshare limitations but the intention is for them to be as widely available as possible. There are still compute nodes on El Gato with 70 nodes provisioned with Nvidia Tesla K20's. 

Specifications

Image Removed

Cuda Modules

Currently the following Cuda modules are available on Ocelote:

/cm/shared/modulefiles

Panel
borderColor #9c9fb5 bgColor #fcfcfc borderWidth 2 borderStyle solid
Cluster Information Deck of Cards startHidden false id cluster information Card default true id puma label Puma title Puma Puma Puma has a different arrangement for GPU nodes than Ocelote and ElGato. Whereas the older clusters have one GPU per node, Puma has four. This has a financial advantage for providing GPU's with lower overall cost, and a technical advantage of allowing jobs that can use multiple GPU's to run faster than spanning multiple nodes.  This capability comes from using a newer operating system.   Each node has four Nvidia V100S model GPUs. They are provisioned with 32GB memory compared to 16GB on the P100's. Image Added Card id ocelote label Ocelote title Ocelote Ocelote Ocelote has 45 compute nodes with Nvidia P100 GPUs that are available to researchers on campus. The limitation is a maximum of 10 concurrent jobs. One node with a V100 is also available. Since there is only one, you can feel free to use it for testing and comparisons to the P100, but production work should be run on the P100's. There is also one node with two P100's for testing jobs that use two GPU's. This one should be used to compare with running a job on two nodes.   Image Added

$ module avail cuda

-------------------- /opt/ohpc/pub/moduledeps/gnu8-openmpi3 --------------------
   cp2k-cuda/7.1.0

-------------------------- /opt/ohpc/pub/modulefiles ---------------------------
   cuda11-dnn/8.0.2    cuda11-sdk/20.7    cuda11/11.0

About two times a year we host the Xsede Workshop on Programming GPU's with OpenACC.  Watch for announcements to the HPC-Info list.

Machine Learning 

*** Nvidia Provided GPU Codes ***

Nvidia builds the popular set of ML and DL frameworks which is not a trivial task. They have made them available to us and they will be updated regularly.  They are currently located at:
/unsupported/singularity/nvidia  

Current list:

PyTorch is a Python package that provides two high-level features:

• Tensor computation (like numpy) with strong GPU acceleration
• Deep Neural Networks built on a tape-based autograd system

nvidia-mxnet.18.06.simg

Each is provided in a Singularity container.

The file name has a tag at the end that represents when it was made, so 18.01 is January 2018

USAGE

Copy the file you wish to use to your directory.  Your home path as well as /extra and /xdisk have been bound to the image, so those are your choices.

For interactive use, start an interactive job on a GPU node modifying this command:

$ qsub -I -N jobname -m bea -W group_list=GROUP-NAME -q windfall -l select=1:ncpus=28:mem=168gb:ngpus=1 -l cput=1:0:0 -l walltime=1:0:0

You must change the group_list and you should change the other attributes as desired.

On the compute node assigned to you, as an example you can run:

$ module load singularity
$ singularity exec --nv nvidia-tensorflow.18.01-py3.simg python tensorflow_example.py

You need to include the --nv and note it has two dashes.  This will bind the Cuda libraries.
The example file is included in this directory. "tensorflow_example.py"

For batch use, you will include these two lines in your submission script

module load singularity
singularity exec --nv nvidia-tensorflow.18.01-py3.simg python tensorflow_example.py

There are more detailed examples here

Singularity

For more information on Singularity, see their web site at:

http://singularity.lbl.gov/user-guide

There are tutorials for Singularity on HPC here

Training

We host workshops from the Pittsburgh Supercomputer Center which is a NSF funded location.  We are working with Nvidia to offer a workshop in the April 2018 timeframe.