The University of Arizona
    For questions, please open a UAService ticket and assign to the Tools Team.
Page tree
Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 104 Next »


All three clusters, Puma, Ocelote, and ElGato, use SLURM for resource management and job scheduling. 

Additional SLURM Resources and Examples

Official SchedMD User DocumentationOfficial SchedMD user documentation. Includes detailed information on SLURM directives and commands.
PBS ⇔ SLURM Rosetta StoneTable for converting some common PBS job directives to SLURM syntax.
Puma Quick StartHPC Quick Start guide. If you have never submitted a batch job before, this is a great place to start.
Job ExamplesBasic SLURM example scripts. Includes PBS scripts for comparison. 
Even More Job Examples!Growing repository of example SLURM submission scripts

SLURM and System Commands

Native Slurm Commands
sbatchSubmits a batch script for executionsbatch script.slurm
srunRun parallel jobs. Can be in place of mpirun/mpiexec. Can be used interactively as well as in batch scriptssrun -n 1 --mpi=pmi2 a.out
sallocRequests a session to  work on a compute node interactivelysee: Interactive Sessions section below
squeueChecks the status of pending and running jobs

squeue --job $JOBID
squeue --user $NETID

scancelCancel a running or pending job

scancel $JOBID
scancel -u $NETID

scontrol holdPlace a hold on a job to prevent it from being executedscontrol hold $JOBID
scontrol releaseReleases a hold placed on a job allowing it to be executedscontrol release $JOBID
System Commands
vaDisplays your group membership, your account usage, and CPU allocation. Short for "view allocation"va
interactiveShortcut for quickly requesting an interactive jobinteractive -a $GROUP_NAME
job-historyRetrieves a running or completed job's history in a user-friendly formatjob-history $JOBID
nodes-busyDisplay a visualization of nodes on a cluster and their usagenodes-busy --help
system-busyDisplay a text-based summary of a cluster's usagesystem-busy
cluster-busyDisplay a visualization of all three cluster's overall usagecluster-busy --help

Batch Job Directives

Command Purpose
#SBATCH --account=group_nameSpecify the account where hours are charged. Don't know your group name? Run the command "va" to see which groups you belong to
#SBATCH --partition=partition_nameSet the job partition. This determines your job's priority and the hours charged. See Job Partition Requests below for additional information
#SBATCH --time=DD-HH:MM:SSSet the job's runtime limit in days, hours, minutes, and seconds
#SBATCH --nodes=N

Allocate N nodes to your job.

For non-MPI enabled jobs, this should be set to "–-nodes=1" to ensure access to all requested resources and prevent memory errors.

#SBATCH --ntasks=N

ntasks specifies the number of tasks (or processes) the job will run. For MPI jobs, this is the number of MPI processes.  Most of the time, you can use ntasks to specify the number of CPUs your job needs. However, in some odd cases you might run into issues. For example, see: Using Matlab

By default, you will be allocated one CPU/task. This can be increased by including the additional directive --cpus-per-task.

The number of CPUs a job is allocated is cpus/task * ntasks, or M*N

#SBATCH --cpus-per-task=M
#SBATCH --mem=Ngb

Select N gb of memory per node. If "gb" is not included, this value defaults to MB. Directives --mem and --mem-per-cpu are mutually exclusive.

#SBATCH --mem-per-cpu=NgbSelect N GB of memory per CPU. Valid values can be found in the Node Types/Example Resource Requests section below. If "gb" is not included, this value defaults to MB.
#SBATCH --gres=gpu:NOptional: Request N GPUs.
#SBATCH --constraint=hi_memOptional: Request a high memory node (Ocelote and Puma only).
#SBATCH --array=N-MSubmits an array job from indices N to M
#SBATCH --job-name=JobNameOptional: Specify a name for your job. This will not automatically affect the output filename.
#SBATCH -e output_filename.err
#SBATCH -o output_filename.out
Optional: Specify output filename(s). If -e is missing, stdout and stderr will be combined.
#SBATCH --open-mode=appendOptional: Append your job's output to the specified output filename(s). 
#SBATCH --mail-type=BEGIN|END|FAIL|ALLOptional: Request email notifications. Beware of mail bombing yourself.
#SBATCH --mail-user=email@address.xyzOptional: Specify email address. If this is missing, notifications will go to your UArizona email address by default.
#SBATCH --exclusiveOptional: Request exclusive access to node.
#SBATCH --export=VAROptional: Export a comma-delimited list of environment variables to a job. 
#SBATCH --export=all (default)Optional: Export your working environment to your job.
#SBATCH --export=noneOptional: Do not export working environment to your job.

Job Partition Requests

PartitionSLURM Details
standard#SBATCH --account=<PI GROUP>
#SBATCH --partition=standard
Consumes your group's standard allocation. These jobs cannot be interrupted.
windfall#SBATCH --partition=windfallDoes not consume your group's standard allocation. Jobs may be interrupted and restarted by higher-priority jobs. The --account flag needs to be omitted or an error will occur. 
high_priority#SBATCH --account=<PI GROUP>
#SBATCH --partition=standard
#SBATCH --qos=user_qos_<PI GROUP>
Available for groups who have purchased compute resources. The partition flag is left as standard and requires the additional --qos flag. Replace <PI GROUP> with your group's name. 

SLURM Output Filename Patterns

SLURM offers ways to make your job's output filenames customizable through the use of character replacements. A table is provided below as a guide with some examples. Variables may be used or combined as desired. Note: character replacements may also be used with other SBATCH directives such as error filename, input filename, and job name.

VariableMeaningExample Slurm Directive(s)Output
%AA job array's main job ID

#SBATCH --array=1-2
#SBATCH -o %A.out
#SBATCH --open-mode=append

%aA job array's index number#SBATCH --array=1-2
#SBATCH -o %A_%a.out
%JJob ID plus stepid#SBATCH -o %J.out12345.out
%jJob ID#SBATCH -o %j.out12345.out
%NHostname of the first compute node allocated to the job#SBATCH -o %N.outr1u11n1.out
%uUsername#SBATCH -o %u.outnetid.out
%xJobname#SBATCH --job-name=JobName
#SBATCH -o %x.out

Node Types/Example Resource Requests

Standard Nodes

ClusterMax CPUsMem/CPUMax MemSample Request Statement
ElGato164gb62gb#SBATCH --nodes=1
#SBATCH --ntasks=16
#SBATCH --mem-per-cpu=4gb

#SBATCH --nodes=1
#SBATCH --ntasks=28
#SBATCH --mem-per-cpu=6gb

Puma945gb470gb#SBATCH --nodes=1
#SBATCH --ntasks=94
#SBATCH --mem-per-cpu=5gb

GPU Nodes

GPU jobs are requested using the generic resource, or --gres, SLURM directive. In general, the directive to request N GPUs will be of the form:

#SBATCH --gres=gpu:N

ClusterMax CPUsMem/CPUMax MemSample Request Statement
ElGato11616gb250gb#SBATCH --nodes=1
#SBATCH --ntasks=16
#SBATCH --mem-per-cpu=16gb
#SBATCH --gres=gpu:1

#SBATCH --nodes=1
#SBATCH --ntasks=28
#SBATCH --mem-per-cpu=6gb
#SBATCH --gres=gpu:pascal:1


#SBATCH --nodes=1
#SBATCH --ntasks=94
#SBATCH --mem-per-cpu=5gb
#SBATCH --gres=gpu:1

1 Up to two GPUs may be requested on a single node with --gres=gpu:2
2 There is only one GPU node with two GPUs on it available on Ocelote. To request it, use --gres=gpu:2
3 there are two models of GPU's on Ocelote: K80 and P100.  If you don't specify which one, you could get either. We recommend including "pascal" to get the P100
4 Up to four GPUs may be requested on Puma on a single GPU node with --gres=gpu:1, 2, 3, or 4

High Memory Nodes

When requesting a high memory node, include both the memory/CPU and constraint directives:

#SBATCH --mem-per-cpu=XXgb
#SBATCH --constraint=hi_mem

ClusterMax CPUsMem/CPUMax MemSample Request Statement

#SBATCH --nodes=1
#SBATCH --ntasks=48
#SBATCH --mem-per-cpu=41gb
#SBATCH --constraint=hi_mem

Puma9432gb3000gb#SBATCH --nodes=1
#SBATCH --ntasks=94
#SBATCH --mem-per-cpu=32gb
#SBATCH --constraint=hi_mem

Interactive Jobs

When you are on a login node, you can request an interactive session on a compute node. This is useful for checking available modules, testing submission scripts, compiling software, and running programs directly from the command line. We have a built-in shortcut command that will allow you to quickly and easily request a session by simply entering: interactive

When you request a session, the full salloc command being executed will be displayed for verification/copying/editing/pasting purposes. For example:

(ocelote) [netid@junonia ~]$ interactive
Run "interactive -h for help customizing interactive use"
Submitting with /usr/local/bin/salloc --job-name=interactive --mem-per-cpu=4GB --nodes=1    --ntasks=1 --time=01:00:00 --account=windfall --partition=windfall
salloc: Pending job allocation 531843
salloc: job 531843 queued and waiting for resources
salloc: job 531843 has been allocated resources
salloc: Granted job allocation 531843
salloc: Waiting for resource configuration
salloc: Nodes i16n1 are ready for job
[netid@i16n1 ~]$ 

Notice in the example above how the command prompt changes once your session starts. When you're on a login node, your prompt will show "junonia" or "wentletrap". Once you're in an interactive session, you'll see the name of the compute node you're connected to. 

If no options are supplied to the command interactive, your job will automatically run using the windfall partition for one hour using one CPU. To use the standard partition, include the flag "-a" followed by your group's name. To see all the customization options:

(ocelote) [netid@junonia ~]$ interactive -h
Usage: /usr/local/bin/interactive [-x] [-g] [-N nodes] [-m memory per core] [-n ncpus per node] [-Q optional qos] [-t hh::mm:ss] [-a account to charge]

You may also create your own salloc commands using any desired SLURM directives for maximum customization.

MPI Jobs


For openmpi the important variables are set by default, so you do not need to include them in your scripts.

Default OpenMPI variables
export OMPI_MCA_btl_openib_cpc_include=rdmacm
export OMPI_MCA_btl_openib_if_include=bnxt_re1
export OMPI_MCA_btl_openib_rroce_enable=1
export OMPI_MCA_btl=vader,self,openib
export OMPI_MCA_oob_tcp_if_include=eth1

Intel MPI

For Intel MPI, these variables are set for you:

Default Intel MPI variables
export I_MPI_FABRICS=shm:ofi
export FI_PROVIDER=verbs
export FI_VERBS_IFACE=eth1

Because the modules gnu8 and openmpi3 are loaded by default, these should be manually unloaded in your submission script:

module unload openmpi3 gnu8

If you're using Intel MPI with mpirun and are getting errors, try replacing mpirun -np $NPROCESSES with:

srun -n $NPROCESSES --mpi=pmi2
  • No labels