More detailed information on system resources can be found on our Compute Resources page.
Singularity containers are available as modules on HPC for GPU-supported workflows. For more information, see our documentation on Containers.
Information on how to request GPUs using SLURM can be found in our SLURM Documentation.
For a list of training resources related to GPU workflows, see our Training documentation.
Puma has a different arrangement for GPU nodes than Ocelote and ElGato. Whereas the older clusters have one GPU per node, Puma has four. This has a financial advantage for providing GPU's with lower overall cost, and a technical advantage of allowing jobs that can use multiple GPU's to run faster than spanning multiple nodes. This capability comes from using a newer operating system.
Each node has four Nvidia V100S model GPUs. They are provisioned with 32GB memory compared to 16GB on the P100's.
Ocelote has 45 compute nodes with Nvidia P100 GPUs that are available to researchers on campus. The limitation is a maximum of 10 concurrent jobs. One node with a V100 is also available. Since there is only one, you can feel free to use it for testing and comparisons to the P100, but production work should be run on the P100's. There is also one node with two P100's for testing jobs that use two GPU's. This one should be used to compare with running a job on two nodes.
ElGato has 90 nodes with one or two Nvidia K20 GPU's. Whilst they are older, well quite old actually, they still support the latest version of Cuda and are useful for testing. Especially when the newer GPU's are busy
Nvidia Nsight Compute (the interactive kernel profiler) is not available. In response to a security alert (CVE-2018-6260) this capability is only available with root authority which users do not have.
The latest Cuda module available on the system is 11.0 and is the only version until newer ones come along. The Cuda driver version can be queried with the
nvidia-smi command. To see the modules available, in an interactive session simply run:
$ module avail cuda
The OpenACC API is a collection of compiler directives and runtime routines that allow you to specify loops and regions of code in standard C, C++, and Fortran that you can offload from a host CPU to the GPU.
We provide two methods of support for OpenACC
Many applications have been optimized to run faster on GPU's. These include:
|NAMD||Installed as a module|
|VASP||A restricted license version is installed; only available to licensed users|
|GROMACS||Installed as a module|
|LAMMPS||Installed as a module|
|ABAQUS||Installed as a module and available as an application through Open OnDemand|
|GAUSSIAN||Installed as a module. See these notes.|
|MATLAB||Installed as a module and available as an application through Open OnDemand. Review the GPU Coder on their website|
|ANSYS Fluent||Installed as a module and available as an application through Open OnDemand|
|RELION||Available as a Singularity container or as a module.|
|ML and DL Frameworks||See the section below.|
Nvidia Rapids is only available on Ocelote currently.
Tensorflow and tensorboard are available on both Puma (python/3.8) and Ocelote (ocelote/3.6)
The minimum version of Python that is supported is 3.6 so you will module load python/3.6 on Ocelote, for all these functions. This will get you to installed packages including:
|numba||RAPIDS: numba is for Cuda programming|
|cuml||RAPIDS: Cuda Machine Learning has many ML algorithms like K-means, PCA and SVM|
|cudf||RAPIDS: Cuda Dataframes supports loading and manipulating datasets|
|tensorflow||TensorFlow is an open source software library for numerical computation using data flow graphs.|
|torch||PyTorch supports tensor computation and deep neural networks|
|caffe2||A deep learning framework|
|tensorrt||Inference server for deep learning|
|tensorboard||Visualization tool for machine learning|