Allocation and Limits

Disk Storage

When you obtain a new HPC account, you will be provided with the following storage:

/home/uxx/netid - 15GB (backed up nightly)
/extra/netid - 200GB (no backups) Ocelote and ElGato only
/tmp - some clusters have /tmp space which is on a disk in each compute node. Ocelote has about 800GB available on each node. It is often faster and more efficient to use this during your jobs and then do the final write to the shared array.

Additional storage:

/xdisk/netid - 200GB to 1TB available on request. The time limit is 45 days with one extension. This allocation is deleted after 45 days if no extension is requested. There are directory level quotas so any files in this directory count against this quota not matter who created it. The data is not backed up.
/rsgrps/netid - rented / purchased space (no backups)

File count limit:
As of this year file count limits have been imposed. The lack of limits created performance problems on the storage array for reasons that get very detailed. This means you may not be able to create files while there is space seemingly available.

/extra and /rsgrps are limited to 600 files / GB
/home and /xdisk do not have limits

We strongly recommend that you do some regular housekeeping of your allocated space. Millions of files are hard to keep organized and even more difficult to migrate. Archiving or using a tool like tar will help keep our disk arrays efficient and potentially free up more space for you to use.

xdisk

Use this link for details on xdisk usage

extra

/extra is something new with Ocelote. When you log in to Ocelote for the first time, an allocation of 200GB will be created for you. It takes an hour or two to show up, and then it is permanent. Remember that it is not backed up like /home. The number of files within the 200GB is limited to 120,000.

uquota is the command to display how much space you have used / remaining

Job Limits

Job Time Limits

Each group is allocated a base of 24,000 hours of compute time, this allocation is refreshed monthly. The allocation can be used on either the htc/cluster/smp clusters or on the new cluster, Ocelote., or a combination.

The command va will display your remaining time

ElGato has a different allocation method of time as it is funded through an NSF MRI grant with usage time provided for campus researchers outside of the grant recipients. Reference the www.elgato.arizona.edu web site.

PBS Batch Queue Limits

The batch queues on the different systems have the following memory, time and core limits.

Queue	Description
standard	Used to consume the monthly allocation of hours provided to each group
windfall	Used when standard is depleted but subject to preemption
high_priority	Used by 'buy-in' users for purchased nodes

System	Queue Name	# of Compute Nodes	Max Wallclock Hrs / Job	Largest job	Total cores in use / group	Largest job / memory	Max # of Running Jobs	Max Queued Jobs
Ocelote	standard	331	240	1344** cores	1344**	8064GB	500	3000
	windfall	331	240	1344** cores		8064GB	500	3000
	high_pri	36	720	1344** cores		8064GB	500	5000

Cluster (8400)	standard	124	240	504 cores	1008 (user limit) ***	1008 GB	512	500
	windfall	229 (all)	240	256 cores	512 (user limit)	512 GB	512
	cluster_high	105 (buy-in)	720	512 cores	buy-in	1024 GB	1148

SMP (UV1000)	standard		240	256 cores	512***	512 GB	512	500
	windfall		240	256 cores		512 GB	256
	smp_high		720	512 cores		1280 GB	240

HTC	standard	104	720	256 cores	512***	512 GB	512	500
	windfall	104	720	256 cores		512 GB	512
	htc_high	10	720	512 cores		512 GB	240

** This represents 48 physical nodes, and 9.2TB of memory
*** This limit is shared by all members of a group across all queues. So you can use the system 1000 core limit by one user on the standard queue or share it across multiple users or queues.

Page tree