HPC Endpoint

The endpoint for HPC can be found by searching UA HPC Filesystems under the Collections tab. If you do not see the UA HPC Filesystems collections, uncheck any checked filter in Quick Filters. Select the result UA HPC Filesystems with the subheading Managed Mapped Collection.

Storage Rental Endpoint

The endpoint for rental storage (found on the filexfer nodes under /rental) can be found by searching UA Rental Storage Filesystem under the Collections tab.

Google Drive Endpoint

1. Under the Endpoints tab (see: left-hand side menu) Search for UA Google Drive and select the result UA Google Drive with the subheading Managed Mapped Collection.           
2. After clicking the result, navigate to the Credentials tab.                  
   
3. This should bring you to a page to register your credentials. Click Continue.
   
4. Select your university account on the next page to proceed.
   
5. Select Allow to give Globus access to your files.
   
6. Once your credentials are set up, navigate back to the UA Google Drive endpoint, go to the Collections tab and select Add a Guest Collection.
7. Under Directory, enter /My Drive to set the correct working directory for your collection. Next, give your collection a descriptive Display Name so you can identify it. When you're ready, click Create Collection.
   
8. Once your collection is ready, you can find it under the Collections tab under Shareable By You. Clicking the name will open it in the File Manager window allowing you to initiate transfers and browse the contents.                                 
   
   

AWS S3 Endpoint (UITS Subsidized Tier 2 Storage)

1. Under the Collections tab, enter UA AWS S3 in the search bar. In the results, you should see the name UA AWS S3 show up with the description Managed Mapped Collection. Click the endpoint's name to proceed.                     
2. Next, select the Credentials tab. If you are prompted for Authentication/Consent, click Continue
   
3. If requested, authenticate by selecting your Arizona email address, then Allow. 
   
4. You will then be returned to the Credentials tab. From there, link to your AWS S3 Bucket by entering your public and private keys in the provided fields
   
   
   
5. Once you've added your keys, navigate back to the UA AWS S3 collection, go to the Collections tab, and click Add a Guest Collection on the right
   

6. Under Create New Guest Collection, click Browse next to the Directory field to find your group's AWS bucket. You will find it under /ua-rt-t2-faculty_netid/ where faculty_netid is the NetID of the faculty member who requested the bucket. Under Display Name, enter a descriptive name that you can use to identify your bucket. Once you've completed the process, click Create Collection.

   If you encounter Authentication/Consent Required after clicking Browse, click Continue, select your university credentials, and click Allow. That should bring you back to the Browse window.

   

7. To find and use your new collection, navigate to the Collections tab, go to Shareable By You, and select the name. That will open your collection in the File Manager window allowing you to view the contents and initiate transfers.
   

Mac/Linux

You will need to use an SSH v2 compliant terminal to move files to/from HPC. For more information on using SCP, use man scp.

Moving a File or Directory to the HPC:

In your terminal, navigate to the desired working directory on your local machine (laptop or desktop usually). To move a file or directory to a designated subdirectory in your account on HPC:

$ scp -rp filenameordirectory NetId@filexfer.hpc.arizona.edu:subdirectory 

Getting a File or Directory From the HPC:

In your terminal, navigate to the desired working directory on your local machine. The copy a remote file from HPC to your current directory:

$ scp -rp NetId@filexfer.hpc.arizona.edu:filenameordirectory .

** the space folllowed by a period at the end means the destination is the current directory** 

Wildcards

Wildcards can be used for multiple file transfers (e.g. all files with .dat extension). Note the backslash " \ " preceding *

$ scp NetId@filexfer.hpc.arizona.edu: subdirectory /\*. dat

Windows

Windows users can use software like WinSCP to make SCP transfers. To use WinSCP, first download/install the software from: https://winscp.net/eng/download.php

To connect, enter filexfer.hpc.arizona.edu in the Host Name field, enter your NetID under User name, and enter your password. Accept by clicking Login. You'll be prompted to Duo Authenticate:

Command Line

iRODS 4 is installed as a standard package to the operating system on every node. This means you will not have to "module load irods". You will still need to "iinit" the first time (see below). iRODS is also available on the filexfer node for use.

Initializing iRODS

Running iinit for any system using iRODS 4.x, unlike its iRODS3 counterpart, does not help you set up the environment. Instead, you need to run create_irods_env with suitable options for the iRODS host, zone, username,etc manually.

For example:

$ create_irods_env -a native -h someserver.somewhere.net -z MYZONE

will suffice to create an appropriate ~/.irods/irods_environment.json file to allow you to run iinit; we took the default -p 1247, -u <your NetId> in the above example by omitting -p and -u.  You only need to do this step ONE time; subsequent times you will just run iinit and it will asked for your password. Note create_irods_env will NOT overwrite or alter an existing ~/.irods/irods_environment.json file.

Once the  ~/.irods/irods_environment.json file is created properly, you should be able to sign in to the iRods server your selected using iinit, viz:

$ iinit	
Enter your current ... password:	# enter your iRODS server password here

At this point you can use other iRods commands such as icp to move files.

Examples

In the following examples:

• my-files-to-transfer/ is the example name of the directory or folder for bulk transfers.
• my-file-to-transfer.txt is the example name for single file transfers.
• Any filename may be used for the checkpoint-file.

Bulk Files Transfer Example

iput -P -b -r -T --retries 3 -X checkpoint-file my-files-to-transfer/

Single Large File Transfer Example

iput -P -T --retries 3 --lfrestart checkpoint-lf-file my-file-to-transfer.txt

Graphical Interface with Cyberduck

Cyberduck is an open source cross-platform, high-throughput and parallel data transfer transfer program that supports multiple transfer protocols (FTP, SFTP, WebDAV, Cloud files, Amazon S3, etc.). It serves as an alternative to the iDrop Java applet, and has been extensively tested with large data transfers (60-70 GB). This allows users to transfer large files, depending on the user's available bandwidth and network settings.

Cyberduck versions are available for Mac OS (10.6 and higher on Intel 64-bit) and Windows (Windows XP, Windows Vista, Windows 7, or Windows 8). LINUX users should use iDrop Desktop or iCommands. Cyberduck version 4.7.1 (released July 7, 2015) and later supports the iRODS protocol.

To use Cyberduck with iRODS:

1. Install or Update Cyberduck. If Cyberduck has already been installed, you may update it under the dropdown Cyberduck → Check for Updates. If you need to install Cyberduck for the first time, go to https://cyberduck.io/, download the installer that is appropriate for your operating system and install. See Cyberduck Preferences for more information on installation.
2. Configure Cyberduck for use with iRODS:
   
   1. Open Cyberduck and click Open Connection
   2. In the first drop down field, enter a profile name
   3. Create the connection entering your iRODS server under Server, 1247 under Port, and your username under Username.
3. Transfer Files by opening another connection and dragging/dropping files.

[netid@sdmz-dtn-4 ~]$ rclone config
Name                 Type
====                 ====

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> n
name> GoogleDrive
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
...
12 / Google Drive
   \ "drive"
...
Storage> 12
Google Application Client Id
client_id> 
client_secret> 
scope> 1
root_folder_id> 
service_account_file> 
Edit advanced config? (y/n)
y) Yes
n) No
y/n> n

The next prompt will ask if you would like to use auto config, select N here or the configuration will not be successful. You will be given a URL. Copy and paste this into your web browser and follow the prompts to allow rclone to have access to your UArizona Google Drive account. When you are done, it will give you a verification code. Copy and paste this back into the terminal to proceed.

Remote config
Use auto config?
 * Say Y if not sure
 * Say N if you are working on a remote or headless machine
y) Yes
n) No
y/n> N
If your browser doesn't open automatically go to the following link: <long URL>
Log in and authorize rclone for access
Enter verification code> <your verification code here>
Configure this as a team drive?
y) Yes
n) No
y/n> n
--------------------
[GoogleDrive]
type = drive
scope = drive
token = {"access_token": <lots of token data>}
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:

Name                 Type
====                 ====
GoogleDrive          drive

Your Google Drive connection is now active and you can use rclone to make transfers. For information on rclone commands, see: https://rclone.org/commands/