The University of Arizona
    For questions, please open a UAService ticket and assign to the Tools Team.
Page tree

Versions Compared


  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents


El Gato

was implemented


During the quarterly maintenance cycle on April 27, 2022 the ElGato K20s were removed because they are no longer supported by Nvidia.

Implemented at the start of 2014 and there is no planned date at this point to discontinue it.  El , it has been reprovisioned with CentOS 7 and new compilers and libraries.  From July 2021 it has been using Slurm for job submission. El Gato is a large GPU /PHI cluster, purchased by an NSF MRI grant by researchers in Astronomy and SISTA.  30% of this system is available for general campus research, including the Nvidia GPU's and Intel Phi's.Ocelote was implemented


Implemented in the middle of 2016.  It is designed to support all workloads on the standard nodes except:

Ocelote now has 46 nodes with Nvidia P100's that are available for "standard" and "windfall" queues. See details at Running Jobs
  1. Large memory workloads that do not run within the 192GB RAM of each node can also be beyond 188GB are handled with either the large memory node or virtual SMP nodes
  2. GPU's are available as a buy-in option or windfall and are now available standard on Ocelote.
  1. which has 2TB of memory.
  2. GPU workload is supported on 46 nodes with Nvidia P100 GPU's.


Implemented in 2020, Puma is the biggest cat yet. Similar to Ocelote, it has standard CPU nodes (with 94 cores and 512 GB of memory per node), GPU nodes (with Nvidia V100) and two high-memory nodes (3 TB). Local scratch storage increased to ~1.4 TB. Puma runs on CentOS 7.

Free vs Buy-In

The HPC resources at UA are differentiated from many other universities in that there is central funding for a significant portion of the available resources. Each PI receives a standard monthly allocation of hours at no charge.  There is no charge to the allocation for windfall usage and that has proven to be very valuable for researchers with substantial compute requirements.  

Research groups can 'Buy-In the cases where there is funding for buy-in, those resources are dedicated to the group providing the funding.  When those resources are not in use they are available to windfall users' (adding additional compute nodes) to the base HPC systems as funding becomes available. Buy-In research groups will have highest priority on the resources they add to the system.  If the expansion resources are not fully utilized by the Buy-In group they will be made available to all users as windfall.

Details on allocations

Details on buy-in

Test Environment

HPC has a test / trial environment as well as the primary clusters detailed below.  This environment is intended to be used for projects that are six months or less in duration and cannot be run on the production systems. Reasons for not being able to be run on the production systems include requiring root access, and hardware or software requirements that cannot be met by one of the production systems. If you have a project in mind that we might be able to support, contact 

FeatureDetailNodes16CPUXeon Westmere-EP X5650
Dual 6-coreMemory128GBDisk10TB (5 x 2TB)NetworkGbE and QDR IB

Compute System Details

Compute System Details


During the quarterly maintenance cycle on April 27, 2022 the ElGato K20s and Ocelote K80s were removed because they are no longer supported by Nvidia.



El Gato




IBM System X iDataPlex dx360 M4

Lenovo NeXtScale nx360 M5Penguin Altus XE2242

Year Purchased


2016 -2018


Distributed Memory

Serial, SMP, Distributed and Large Memory* 


Xeon Ivy Bridge E5-2650
Dual 8-core

Xeon Haswell E5-2695 Dual 14-core
Xeon Broadwell E5-2695 Dual 14-core

Processor Speed (GHz)




140 Nvidia K20x
 40 Intel Phi

46 Nvidia P100
15 Nvidia K80 (windfall only) 

Node Count



Cores / Node



Total Cores



Memory / Node (GB)

 64 or 256

High memory node and vSMPnode - 2TB

Total Memory (TB)


~840GB(2018 P100 nodes)2020

Node Count



236 CPU-only
2 High-memory

Total System Memory (TB)




2x Xeon E5-2650v2 8-core (Ivy Bridge)

2x Xeon E5-2695v3 14-core (Haswell)
2x Xeon E5-2695v4 14-core (Broadwell)
4x Xeon E7-4850v2 12-core (Ivy Bridge)

2x AMD EPYC 7642 48-core (Rome)

Cores / Node (schedulable)


28c (48c - High-memory node)94c

Total Cores



Processor Speed


2.3GHz (2.4GHz - Broadwell CPUs)2.4GHz

Memory / Node

256GB - GPU nodes
64GB - CPU-only nodes

192GB (2TB - High-memory node)

512GB (3TB - High-memory nodes)


46 NVIDIA P100 (16GB)


/tmp~840 GB spinning
/tmp is part of root filesystem
~840 GB spinning
/tmp is part of root filesystem

Max Performance

~1440 TB NVMe

HPL Rmax (TFlop/s)




 RedHat 6.4Centos 7

 CentOS 6.7CentOS 7


FDR Inifinband

FDR Infiniband for node-node
10Gb 10 Gb Ethernet node-storage

Application Support

MPI, Serial, GPU, Phi

Parallel, MPI, OpenMP, Serial

* The new cluster includes a large memory node with 2TB of RAM available on 48 cores.  More details on the Large Memory Node

*Virtual SMP software implements large memory images.

Image Removed

Image Removed

1x 25Gb/s Ethernet RDMA (RoCEv2)
1x 25Gb/s Ethernet to storage

* Includes high-memory and GPU node CPUs

Example Resource Requests

See our User Guide under Running Jobs with SLURM → Example Resource Requests.

Image Added