This lesson is in the early stages of development (Alpha version)

Introduction to High-Performance Computing

Why use a Cluster?

Overview

Teaching: 15 min
Exercises: 5 min
Questions

  • Why would I be interested in High Performance Computing (HPC)?

  • What can I expect to learn from this course?

Objectives

  • Describe what an HPC system is

  • Identify how an HPC system could benefit you.

Frequently, research problems that use computing can outgrow the capabilities of the desktop or laptop computer where they started:

In all these cases, access to more (and larger) computers is needed. Those computers should be usable at the same time, solving many researchers’ problems in parallel.

Jargon Busting Presentation

Open the HPC Jargon Buster in a new tab. To present the content, press C to open a clone in a separate window, then press P to toggle presentation mode.

I’ve Never Used a Server, Have I?

Take a minute and think about which of your daily interactions with a computer may require a remote server or even cluster to provide you with results.

Some Ideas

  • Checking email: your computer (possibly in your pocket) contacts a remote machine, authenticates, and downloads a list of new messages; it also uploads changes to message status, such as whether you read, marked as junk, or deleted the message. Since yours is not the only account, the mail server is probably one of many in a data center.
  • Searching for a phrase online involves comparing your search term against a massive database of all known sites, looking for matches. This “query” operation can be straightforward, but building that database is a monumental task! Servers are involved at every step.
  • Searching for directions on a mapping website involves connecting your (A) starting and (B) end points by traversing a graph in search of the “shortest” path by distance, time, expense, or another metric. Converting a map into the right form is relatively simple, but calculating all the possible routes between A and B is expensive.

Checking email could be serial: your machine connects to one server and exchanges data. Searching by querying the database for your search term (or endpoints) could also be serial, in that one machine receives your query and returns the result. However, assembling and storing the full database is far beyond the capability of any one machine. Therefore, these functions are served in parallel by a large, “hyperscale” collection of servers working together.

Key Points

  • High Performance Computing (HPC) typically involves connecting to very large computing systems elsewhere in the world.

  • These other systems can be used to do work that would either be impossible or much slower on smaller systems.

  • HPC resources are shared by multiple users.

  • The standard method of interacting with such systems is via a command line interface.

Connecting to a remote HPC system

Overview

Teaching: 25 min
Exercises: 10 min
Questions

  • How do I log in to a remote HPC system?

Objectives

  • Configure secure access to a remote HPC system.

  • Connect to a remote HPC system.

Secure Connections

The first step in using a cluster is to establish a connection from our laptop to the cluster. When we are sitting at a computer (or standing, or holding it in our hands or on our wrists), we have come to expect a visual display with icons, widgets, and perhaps some windows or applications: a graphical user interface, or GUI. Since computer clusters are remote resources that we connect to over slow or intermittent interfaces (WiFi and VPNs especially), it is more practical to use a command-line interface, or CLI, to send commands as plain-text. If a command returns output, it is printed as plain text as well. The commands we run today will not open a window to show graphical results.

If you have ever opened the Windows Command Prompt or macOS Terminal, you have seen a CLI. If you have already taken The Carpentries’ courses on the UNIX Shell or Version Control, you have used the CLI on your local machine extensively. The only leap to be made here is to open a CLI on a remote machine, while taking some precautions so that other folks on the network can’t see (or change) the commands you’re running or the results the remote machine sends back. We will use the Secure SHell protocol (or SSH) to open an encrypted network connection between two machines, allowing you to send & receive text and data without having to worry about prying eyes.

/hpc-intro/Connect%20to%20cluster

SSH clients are usually command-line tools, where you provide the remote machine address as the only required argument. If your username on the remote system differs from what you use locally, you must provide that as well. If your SSH client has a graphical front-end, such as PuTTY or MobaXterm, you will 7set these arguments before clicking “connect.” From the terminal, you’ll write something like ssh userName@hostname, where the argument is just like an email address: the “@” symbol is used to separate the personal ID from the address of the remote machine.

When logging in to a laptop, tablet, or other personal device, a username, password, or pattern are normally required to prevent unauthorized access. In these situations, the likelihood of somebody else intercepting your password is low, since logging your keystrokes requires a malicious exploit or physical access. For systems like login12.myriad.ucl.ac.uk running an SSH server, anybody on the network can log in, or try to. Since usernames are often public or easy to guess, your password is often the weakest link in the security chain. Many clusters therefore forbid password-based login, requiring instead that you generate and configure a public-private key pair with a much stronger password. Even if your cluster does not require it, the next section will guide you through the use of SSH keys and an SSH agent to both strengthen your security and make it more convenient to log in to remote systems.

Better Security With SSH Keys

The Lesson Setup provides instructions for installing a shell application with SSH. If you have not done so already, please open that shell application with a Unix-like command line interface to your system.

SSH keys are an alternative method for authentication to obtain access to remote computing systems. They can also be used for authentication when transferring files or for accessing remote version control systems (such as GitHub). In this section you will create a pair of SSH keys:

Private keys are your secure digital passport

A private key that is visible to anyone but you should be considered compromised, and must be destroyed. This includes having improper permissions on the directory it (or a copy) is stored in, traversing any network that is not secure (encrypted), attachment on unencrypted email, and even displaying the key on your terminal window.

Protect this key as if it unlocks your front door. In many ways, it does.

Regardless of the software or operating system you use, please choose a strong password or passphrase to act as another layer of protection for your private SSH key.

Considerations for SSH Key Passwords

When prompted, enter a strong password that you will remember. There are two common approaches to this:

  1. Create a memorable passphrase with some punctuation and number-for-letter substitutions, 32 characters or longer. Street addresses work well; just be careful of social engineering or public records attacks.
  2. Use a password manager and its built-in password generator with all character classes, 25 characters or longer. KeePass and BitWarden are two good options.
  3. Nothing is less secure than a private key with no password. If you skipped password entry by accident, go back and generate a new key pair with a strong password.

SSH Keys on Linux, Mac, MobaXterm, and Windows Subsystem for Linux

Once you have opened a terminal, check for existing SSH keys and filenames since existing SSH keys are overwritten.

[user@laptop ~]$ ls ~/.ssh/

If ~/.ssh/id_ed25519 already exists, you will need to specify a different name for the new key-pair.

Generate a new public-private key pair using the following command, which will produce a stronger key than the ssh-keygen default by invoking these flags:

[user@laptop ~]$ ssh-keygen -a 100 -f ~/.ssh/id_ed25519 -t ed25519

When prompted, enter a strong password with the above considerations in mind. Note that the terminal will not appear to change while you type the password: this is deliberate, for your security. You will be prompted to type it again, so don’t worry too much about typos.

Take a look in ~/.ssh (use ls ~/.ssh). You should see two new files:

Use RSA for Older Systems

If key generation failed because ed25519 is not available, try using the older (but still strong and trustworthy) RSA cryptosystem. Again, first check for an existing key:

[user@laptop ~]$ ls ~/.ssh/

If ~/.ssh/id_rsa already exists, you will need to specify choose a different name for the new key-pair. Generate it as above, with the following extra flags:

  • -b sets the number of bits in the key. The default is 2048. EdDSA uses a fixed key length, so this flag would have no effect.
  • -o (no default): use the OpenSSH key format, rather than PEM.
[user@laptop ~]$ ssh-keygen -a 100 -b 4096 -f ~/.ssh/id_rsa -o -t rsa

When prompted, enter a strong password with the above considerations in mind.

Take a look in ~/.ssh (use ls ~/.ssh). You should see two new files:

  • your private key (~/.ssh/id_rsa): do not share with anyone!
  • the shareable public key (~/.ssh/id_rsa.pub): if a system administrator asks for a key, this is the one to send. It is also safe to upload to websites such as GitHub: it is meant to be seen.

SSH Keys on PuTTY

If you are using PuTTY on Windows, download and use puttygen to generate the key pair. See the PuTTY documentation for details.

Take a look in the folder you specified. You should see two new files:

SSH Agent for Easier Key Handling

An SSH key is only as strong as the password used to unlock it, but on the other hand, typing out a complex password every time you connect to a machine is tedious and gets old very fast. This is where the SSH Agent comes in.

Using an SSH Agent, you can type your password for the private key once, then have the Agent remember it for some number of hours or until you log off. Unless some nefarious actor has physical access to your machine, this keeps the password safe, and removes the tedium of entering the password multiple times.

Just remember your password, because once it expires in the Agent, you have to type it in again.

SSH Agents on Linux, macOS, and Windows

Open your terminal application and check if an agent is running:

[user@laptop ~]$ ssh-add -l

Add your key to the agent, with session expiration after 8 hours:

[user@laptop ~]$ ssh-add -t 8h ~/.ssh/id_ed25519

Enter passphrase for .ssh/id_ed25519: 
Identity added: .ssh/id_ed25519
Lifetime set to 86400 seconds

For the duration (8 hours), whenever you use that key, the SSH Agent will provide the key on your behalf without you having to type a single keystroke.

SSH Agent on PuTTY

If you are using PuTTY on Windows, download and use pageant as the SSH agent. See the PuTTY documentation.

Transfer Your Public Key

Use the secure copy tool to send your public key to the cluster.

[user@laptop ~]$ scp ~/.ssh/id_ed25519.pub [email protected]:~/

If you are working outside the UCL network you will need to connect through our Gateway machine before accessing Myriad or other local clusters, so you will need to copy your public key to the Gateway machine too:

[user@laptop ~]$ scp ~/.ssh/id_ed25519.pub [email protected]:~/

Log In to the Cluster

Go ahead and open your terminal or graphical SSH client, then log in to the cluster. Replace yourUsername with your username or the one supplied by the instructors.

Again, if you are working outside the UCL network you will need to connect to Gateway machine before accessing Myriad or other local clusters

[user@laptop ~]$ ssh [email protected]

Once you are inside the UCL network (either on the UCL Gateway or physically on site), you can SSH into Myriad:

[user@laptop ~]$ ssh [email protected]

You may be asked for your password. Watch out: the characters you type after the password prompt are not displayed on the screen. Normal output will resume once you press Enter.

You may have noticed that the prompt changed when you logged into the remote system using the terminal (if you logged in using PuTTY this will not apply because it does not offer a local terminal). This change is important because it can help you distinguish on which system the commands you type will be run when you pass them into the terminal. This change is also a small complication that we will need to navigate throughout the workshop. Exactly what is displayed as the prompt (which conventionally ends in $) in the terminal when it is connected to the local system and the remote system will typically be different for every user. We still need to indicate which system we are entering commands on though so we will adopt the following convention:

Looking Around Your Remote Home

Very often, many users are tempted to think of a high-performance computing installation as one giant, magical machine. Sometimes, people will assume that the computer they’ve logged onto is the entire computing cluster. So what’s really happening? What computer have we logged on to? The name of the current computer we are logged onto can be checked with the hostname command. (You may also notice that the current hostname is also part of our prompt!)

[yourUsername@login12 ~]$  hostname

login12.myriad.ucl.ac.uk

So, we’re definitely on the remote machine. Next, let’s find out where we are by running pwd to print the working directory.

[yourUsername@login12 ~]$  pwd

/home/yourUsername

Great, we know where we are! Let’s see what’s in our current directory:

[yourUsername@login12 ~]$  ls

id_ed25519.pub

The system administrators may have configured your home directory with some helpful files, folders, and links (shortcuts) to space reserved for you on other filesystems. If they did not, your home directory may appear empty. To double-check, include hidden files in your directory listing:

[yourUsername@login12 ~]$  ls -a

  .            .bashrc           id_ed25519.pub
  ..           .ssh

In the first column, . is a reference to the current directory and .. a reference to its parent (/home). You may or may not see the other files, or files like them: .bashrc is a shell configuration file, which you can edit with your preferences; and .ssh is a directory storing SSH keys and a record of authorized connections.

Install Your SSH Key

There May Be a Better Way

Policies and practices for handling SSH keys vary between HPC clusters: follow any guidance provided by the cluster administrators or documentation. In particular, if there is an online portal for managing SSH keys, use that instead of the directions outlined here.

If you transferred your SSH public key with scp, you should see id_ed25519.pub in your home directory. To “install” this key, it must be listed in a file named authorized_keys under the .ssh folder.

If the .ssh folder was not listed above, then it does not yet exist: create it.

[yourUsername@login12 ~]$  mkdir ~/.ssh

Now, use cat to print your public key, but redirect the output, appending it to the authorized_keys file:

[yourUsername@login12 ~]$  cat ~/id_ed25519.pub >> ~/.ssh/authorized_keys

That’s all! Disconnect, then try to log back into the remote: if your key and agent have been configured correctly, you should not be prompted for the password for your SSH key.

[yourUsername@login12 ~]$  logout

[user@laptop ~]$ ssh [email protected]

Key Points

  • An HPC system is a set of networked machines.

  • HPC systems typically provide login nodes and a set of worker nodes.

  • The resources found on independent (worker) nodes can vary in volume and type (amount of RAM, processor architecture, availability of network mounted filesystems, etc.).

  • Files saved on one node are available on all nodes.

Exploring Remote Resources

Overview

Teaching: 25 min
Exercises: 10 min
Questions

  • How does my local computer compare to the remote systems?

  • How does the login node compare to the compute nodes?

  • Are all compute nodes alike?

Objectives

  • Survey system resources using nproc, free, and the queuing system

  • Compare & contrast resources on the local machine, login node, and worker nodes

  • Learn about the various filesystems on the cluster using df

  • Find out who else is logged in

  • Assess the number of idle and occupied nodes

Look Around the Remote System

If you have not already connected to Myriad, please do so now:

[user@laptop ~]$  ssh [email protected]

Take a look at your home directory on the remote system:

[yourUsername@login12 ~]$  ls

What’s different between your machine and the remote?

Open a second terminal window on your local computer and run the ls command (without logging in to Myriad). What differences do you see?

Solution

You would likely see something more like this:

[user@laptop ~]$ ls

Applications Documents    Library      Music        Public
Desktop      Downloads    Movies       Pictures

The remote computer’s home directory shares almost nothing in common with the local computer: they are completely separate systems!

Most high-performance computing systems run the Linux operating system, which is built around the UNIX Filesystem Hierarchy Standard. Instead of having a separate root for each hard drive or storage medium, all files and devices are anchored to the “root” directory, which is /:

[yourUsername@login12 ~]$  ls /

bin   etc   lib64  proc  sbin     sys  var
boot  home  mnt    root  scratch  tmp  working
dev   lib   opt    run   srv      usr

The “home” directory is the one where we generally want to keep all of our files. Other folders on a UNIX OS contain system files, and get modified and changed as you install new software or upgrade your OS.

Using HPC filesystems

On HPC systems, you have a number of places where you can store your files. These differ in both the amount of space allocated and whether or not they are backed up.

  • Home – often a network filesystem, data stored here is available throughout the HPC system, and often backed up periodically. Files stored here are typically slower to access, the data is actually stored on another computer and is being transmitted and made available over the network!
  • Scratch – typically faster than the networked Home directory, but not usually backed up, and should not be used for long term storage.
  • Work – sometimes provided as an alternative to Scratch space, Work is a fast file system accessed over the network. Typically, this will have higher performance than your home directory, but lower performance than Scratch; it may not be backed up. It differs from Scratch space in that files in a work file system are not automatically deleted for you: you must manage the space yourself.

Nodes

Individual computers that compose a cluster are typically called nodes (although you will also hear people call them servers, computers and machines). On a cluster, there are different types of nodes for different types of tasks. The node where you are right now is called the login node, head node, landing pad, or submit node. A login node serves as an access point to the cluster.

As a gateway, the login node should not be used for time-consuming or resource-intensive tasks. You should be alert to this, and check with your site’s operators or documentation for details of what is and isn’t allowed. It is well suited for uploading and downloading files, setting up software, and running tests. Generally speaking, in these lessons, we will avoid running jobs on the login node.

Who else is logged in to the login node?

[yourUsername@login12 ~]$  who

This may show only your user ID, but there are likely several other people (including fellow learners) connected right now.

Dedicated Transfer Nodes

If you want to transfer larger amounts of data to or from the cluster, some systems offer dedicated nodes for data transfers only. The motivation for this lies in the fact that larger data transfers should not obstruct operation of the login node for anybody else. Check with your cluster’s documentation or its support team if such a transfer node is available. As a rule of thumb, consider all transfers of a volume larger than 500 MB to 1 GB as large. But these numbers change, e.g., depending on the network connection of yourself and of your cluster or other factors.

The real work on a cluster gets done by the compute (or worker) nodes. compute nodes come in many shapes and sizes, but generally are dedicated to long or hard tasks that require a lot of computational resources.

All interaction with the compute nodes is handled by a specialized piece of software called a scheduler (the scheduler used in this lesson is called SGE). We’ll learn more about how to use the scheduler to submit jobs next, but for now, it can also tell us more information about the compute nodes.

For example, we can view all of the compute nodes by running the command nodetypes.

[yourUsername@login12 ~]$  nodetypes

    4 type * nodes: 36 cores, 188.4G RAM
    7 type B nodes: 36 cores,   1.5T RAM
   66 type D nodes: 36 cores, 188.4G RAM
    9 type E nodes: 36 cores, 188.4G RAM
    1 type F nodes: 36 cores, 188.4G RAM
    3 type H nodes: 36 cores, 172.7G RAM
   53 type H nodes: 36 cores, 188.4G RAM
    3 type I nodes: 36 cores,   1.5T RAM
    2 type J nodes: 36 cores, 188.4G RAM

A lot of the nodes are busy running work for other users: we are not alone here!

There are also specialized machines used for managing disk storage, user authentication, and other infrastructure-related tasks. Although we do not typically logon to or interact with these machines directly, they enable a number of key features like ensuring our user account and files are available throughout the HPC system.

What’s in a Node?

All of the nodes in an HPC system have the same components as your own laptop or desktop: CPUs (sometimes also called processors or cores), memory (or RAM), and disk space. CPUs are a computer’s tool for actually running programs and calculations. Information about a current task is stored in the computer’s memory. Disk refers to all storage that can be accessed like a file system. This is generally storage that can hold data permanently, i.e. data is still there even if the computer has been restarted. While this storage can be local (a hard drive installed inside of it), it is more common for nodes to connect to a shared, remote fileserver or cluster of servers.

/hpc-intro/Node%20anatomy

Explore Your Computer

Try to find out the number of CPUs and amount of memory available on your personal computer.

Note that, if you’re logged in to the remote computer cluster, you need to log out first. To do so, type Ctrl+d or exit:

[yourUsername@login12 ~]$  exit
[user@laptop ~]$

Solution

There are several ways to do this. Most operating systems have a graphical system monitor, like the Windows Task Manager. More detailed information can be found on the command line:

  • Run system utilities 
    [user@laptop ~]$ nproc --all
[user@laptop ~]$ free -m
  • Read from /proc 
    [user@laptop ~]$ cat /proc/cpuinfo
[user@laptop ~]$ cat /proc/meminfo
  • Run system monitor 
    [user@laptop ~]$ htop

Explore the Login Node

Now compare the resources of your computer with those of the login node.

Solution

[user@laptop ~]$ ssh [email protected]
[yourUsername@login12 ~]$  nproc --all
[yourUsername@login12 ~]$  free -m

You can get more information about the processors using lscpu, and a lot of detail about the memory by reading the file /proc/meminfo:

[yourUsername@login12 ~]$  less /proc/meminfo

You can also explore the available filesystems using df to show disk free space. The -h flag renders the sizes in a human-friendly format, i.e., GB instead of B. The type flag -T shows what kind of filesystem each resource is.

[yourUsername@login12 ~]$  df -Th

Different results from df

  • The local filesystems (ext, tmp, xfs, zfs) will depend on whether you’re on the same login node (or compute node, later on).
  • Networked filesystems (beegfs, cifs, gpfs, nfs, pvfs) will be similar – but may include yourUsername, depending on how it is mounted.

Shared Filesystems

This is an important point to remember: files saved on one node (computer) are often available everywhere on the cluster!

Explore a Worker Node

Finally, let’s look at the resources available on the worker nodes where your jobs will actually run. Try running this command to see the name, CPUs and memory available on the worker nodes (the instructors will give you the ID of the compute node to use):

[yourUsername@login12 ~]$  qhost -h node-d00a-001

Compare Your Computer, the Login Node and the Compute Node

Compare your laptop’s number of processors and memory with the numbers you see on the cluster login node and compute node. What implications do you think the differences might have on running your research work on the different systems and nodes?

Solution

Compute nodes are usually built with processors that have higher core-counts than the login node or personal computers in order to support highly parallel tasks. Compute nodes usually also have substantially more memory (RAM) installed than a personal computer. More cores tends to help jobs that depend on some work that is easy to perform in parallel, and more, faster memory is key for large or complex numerical tasks.

Differences Between Nodes

Many HPC clusters have a variety of nodes optimized for particular workloads. Some nodes may have larger amount of memory, or specialized resources such as Graphics Processing Units (GPUs or “video cards”).

With all of this in mind, we will now cover how to talk to the cluster’s scheduler, and use it to start running our scripts and programs!

Key Points

  • An HPC system is a set of networked machines.

  • HPC systems typically provide login nodes and a set of compute nodes.

  • The resources found on independent (worker) nodes can vary in volume and type (amount of RAM, processor architecture, availability of network mounted filesystems, etc.).

  • Files saved on shared storage are available on all nodes.

  • The login node is a shared machine: be considerate of other users.

Scheduler Fundamentals

Overview

Teaching: 45 min
Exercises: 30 min
Questions

  • What is a scheduler and why does a cluster need one?

  • How do I launch a program to run on a compute node in the cluster?

  • How do I capture the output of a program that is run on a node in the cluster?

Objectives

  • Submit a simple script to the cluster.

  • Monitor the execution of jobs using command line tools.

  • Inspect the output and error files of your jobs.

  • Find the right place to put large datasets on the cluster.

Job Scheduler

An HPC system might have thousands of nodes and thousands of users. How do we decide who gets what and when? How do we ensure that a task is run with the resources it needs? This job is handled by a special piece of software called the scheduler. On an HPC system, the scheduler manages which jobs run where and when.

The following illustration compares these tasks of a job scheduler to a waiter in a restaurant. If you can relate to an instance where you had to wait for a while in a queue to get in to a popular restaurant, then you may now understand why sometimes your job do not start instantly as in your laptop.

/hpc-intro/Compare%20a%20job%20scheduler%20to%20a%20waiter%20in%20a%20restaurant

The scheduler used in this lesson is SGE. Although SGE is not used everywhere, running jobs is quite similar regardless of what software is being used. The exact syntax might change, but the concepts remain the same.

Running a Batch Job

The most basic use of the scheduler is to run a command non-interactively. Any command (or series of commands) that you want to run on the cluster is called a job, and the process of using a scheduler to run the job is called batch job submission.

In this case, the job we want to run is a shell script – essentially a text file containing a list of UNIX commands to be executed in a sequential manner. Our shell script will have three parts:

[yourUsername@login12 ~]$  nano example-job.sh

#!/bin/bash -l

echo -n "This script is running on "
hostname

Creating Our Test Job

Run the script. Does it execute on the cluster or just our login node?

Solution

[yourUsername@login12 ~]$  bash example-job.sh

This script is running on login12.myriad.ucl.ac.uk

This script ran on the login node, but we want to take advantage of the compute nodes: we need the scheduler to queue up example-job.sh to run on a compute node.

To submit this task to the scheduler, we use the qsub command. This creates a job which will run the script when dispatched to a compute node which the queuing system has identified as being available to perform the work.

[yourUsername@login12 ~]$  qsub example-job.sh

Your job 36855 ("example-job.sh") has been submitted

And that’s all we need to do to submit a job. Our work is done – now the scheduler takes over and tries to run the job for us. While the job is waiting to run, it goes into a list of jobs called the queue. To check on our job’s status, we check the queue using the command qstat -u yourUsername.

[yourUsername@login12 ~]$  qstat -u yourUsername

job-ID  prior   name       user         state submit/start at     queue
-------------------------------------------------------------------------------
3979883 3.50000 example-jo yourUser     r     06/25/2020 11:36:30 Arya@node-b00

We can see all the details of our job, most importantly that it is in the r or running state. Sometimes our jobs might need to wait in a queue (w or waiting) or have an error (E).

Where’s the Output?

On the login node, this script printed output to the terminal – but now, when qstat shows the job has finished, nothing was printed to the terminal.

Cluster job output is typically redirected to a file in the directory you launched it from. Use ls to find and read the file.

Customising a Job

The job we just ran used all of the scheduler’s default options. In a real-world scenario, that’s probably not what we want. The default options represent a reasonable minimum. Chances are, we will need more cores, more memory, more time, among other special considerations. To get access to these resources we must customize our job script.

Comments in UNIX shell scripts (denoted by #) are typically ignored, but there are exceptions. For instance the special #! comment at the beginning of scripts specifies what program should be used to run it (you’ll typically see #!/usr/bin/env bash). Schedulers like SGE also have a special comment used to denote special scheduler-specific options. Though these comments differ from scheduler to scheduler, SGE’s special comment is #$. Anything following the #$ comment is interpreted as an instruction to the scheduler.

Let’s illustrate this by example. By default, a job’s name is the name of the script, but the -N option can be used to change the name of a job. Add an option to the script:

[yourUsername@login12 ~]$  cat example-job.sh

#!/bin/bash -l
#$ -N hello-world

echo -n "This script is running on "
hostname

Submit the job and monitor its status:

[yourUsername@login12 ~]$  qsub example-job.sh
[yourUsername@login12 ~]$  qstat -u yourUsername

job-ID  prior   name       user         state submit/start at      slots
------------------------------------------------------------------------
38191   0.00000 new_name   yourUser     qw    06/25/2020 13:25:26  1

Fantastic, we’ve successfully changed the name of our job!

Resource Requests

What about more important changes, such as the number of cores and memory for our jobs? One thing that is absolutely critical when working on an HPC system is specifying the resources required to run a job. This allows the scheduler to find the right time and place to schedule our job. If you do not specify requirements (such as the amount of time you need), you will likely be stuck with your site’s default resources, which is probably not what you want.

The following are several key resource requests:

Note that just requesting these resources does not make your job run faster, nor does it necessarily mean that you will consume all of these resources. It only means that these are made available to you. Your job may end up using less memory, or less time, or fewer nodes than you have requested, and it will still run.

It’s best if your requests accurately reflect your job’s requirements. We’ll talk more about how to make sure that you’re using resources effectively in a later episode of this lesson.

Submitting Resource Requests

Modify our hostname script so that it runs for a minute, then submit a job for it on the cluster.

Solution

[yourUsername@login12 ~]$  cat example-job.sh

#!/bin/bash -l
#$ -l h_rt= 00:01 # timeout in HH:MM

echo -n "This script is running on "
sleep 20 # time in seconds
hostname

[yourUsername@login12 ~]$  qsub example-job.sh

Why are the SGE runtime and sleep time not identical?

Job environment variables

When SGE runs a job, it sets a number of environment variables for the job. One of these will let us check what directory our job script was submitted from. The SGE_O_WORKDIR variable is set to the directory from which our job was submitted.

Using the SGE_O_WORKDIR variable, modify your job so that it prints out the location from which the job was submitted.

Solution

[yourUsername@login12 ~]$  nano example-job.sh
[yourUsername@login12 ~]$  cat example-job.sh

#!/bin/bash -l
#SGE -l 00:00:30

echo -n "This script is running on "
hostname

echo "This job was launched in the following directory:"
echo ${SGE_O_WORKDIR}

Resource requests are typically binding. If you exceed them, your job will be killed. Let’s use wall time as an example. We will request 1 minute of wall time, and attempt to run a job for two minutes.

[yourUsername@login12 ~]$  cat example-job.sh

#!/bin/bash -l
#$ -N long_job
#$ -l h_rt= 00:01 # timeout in HH:MM

echo "This script is running on ... "
sleep 240 # time in seconds
hostname

Submit the job and wait for it to finish. Once it is has finished, check the log file.

[yourUsername@login12 ~]$  qsub example-job.sh
[yourUsername@login12 ~]$  qstat -u yourUsername

[yourUsername@login12 ~]$  cat long_job.o*

This script is running on:
node-d00a-007.myriad.ucl.ac.uk

Our job was killed for exceeding the amount of resources it requested. Although this appears harsh, this is actually a feature. Strict adherence to resource requests allows the scheduler to find the best possible place for your jobs. Even more importantly, it ensures that another user cannot use more resources than they’ve been given. If another user messes up and accidentally attempts to use all of the cores or memory on a node, SGE will either restrain their job to the requested resources or kill the job outright. Other jobs on the node will be unaffected. This means that one user cannot mess up the experience of others, the only jobs affected by a mistake in scheduling will be their own.

Cancelling a Job

Sometimes we’ll make a mistake and need to cancel a job. This can be done with the qdel command. Let’s submit a job and then cancel it using its job number (remember to change the walltime so that it runs long enough for you to cancel it before it is killed!).

[yourUsername@login12 ~]$  qsub example-job.sh
[yourUsername@login12 ~]$  qstat -u yourUsername

Your job 38759 ("example-job.sh") has been submitted

job-ID  prior   name       user         state submit/start at      slots
------------------------------------------------------------------------
38759   0.00000 example-jo yourUser     qw    06/25/2020 14:27:46  1

Now cancel the job with its job number (printed in your terminal). A clean return of your command prompt indicates that the request to cancel the job was successful.

[yourUsername@login12 ~]$  qdel 38759
# It might take a minute for the job to disappear from the queue...
[yourUsername@login12 ~]$  qstat -u yourUsername

# ...(no output from qstat when there are no jobs to display)...

Cancelling multiple jobs

We can also cancel all of our jobs at once using the -u option. This will delete all jobs for a specific user (in this case us). Note that you can only delete your own jobs.

Try submitting multiple jobs and then cancelling them all with ` -u yourUsername`.

Solution

First, submit a trio of jobs:

[yourUsername@login12 ~]$  qsub example-job.sh
[yourUsername@login12 ~]$  qsub example-job.sh
[yourUsername@login12 ~]$  qsub example-job.sh

Then, cancel them all:

[yourUsername@login12 ~]$  qdel -u yourUsername

Other Types of Jobs

Up to this point, we’ve focused on running jobs in batch mode. SGE also provides the ability to start an interactive session.

There are very frequently tasks that need to be done interactively. Creating an entire job script might be overkill, but the amount of resources required is too much for a login node to handle. A good example of this might be building a genome index for alignment with a tool like HISAT2. Fortunately, we can run these types of tasks as a one-off with qrsh.

For an interactive session, you reserve some compute nodes via the scheduler and then are logged in live, just like on the login nodes. These can be used for live visualisation, software debugging, or to work up a script to run your program without having to submit each attempt separately to the queue and wait for it to complete.

[yourUsername@login12 ~]$  qrsh -l mem=512M,h_rt=2:00:00

All qsub options are supported like regular job submission with the difference that with qrsh they must be given at the command line, and not with any job script. Once a node is allocated to you, you should be presented with a bash prompt. Note that the prompt will likely change to reflect your new location, in this case the worker node we are logged on. You can also verify this with hostname.

When you are done with the interactive job, type exit to quit your session.

Key Points

  • The scheduler handles how compute resources are shared between users.

  • A job is just a shell script.

  • Request slightly more resources than you will need.

Accessing software via Modules

Overview

Teaching: 30 min
Exercises: 15 min
Questions

  • How do we load and unload software packages?

Objectives

  • Load and use a software package.

  • Explain how the shell environment changes when the module mechanism loads or unloads packages.

On a high-performance computing system, it is seldom the case that the software we want to use is available when we log in. It is installed, but we will need to “load” it before it can run.

Before we start using individual software packages, however, we should understand the reasoning behind this approach. The three biggest factors are:

Software incompatibility is a major headache for programmers. Sometimes the presence (or absence) of a software package will break others that depend on it. Two of the most famous examples are Python 2 and 3 and C compiler versions. Python 3 famously provides a python command that conflicts with that provided by Python 2. Software compiled against a newer version of the C libraries and then used when they are not present will result in a nasty 'GLIBCXX_3.4.20' not found error, for instance.

Software versioning is another common issue. A team might depend on a certain package version for their research project - if the software version was to change (for instance, if a package was updated), it might affect their results. Having access to multiple software versions allow a set of researchers to prevent software versioning issues from affecting their results.

Dependencies are where a particular software package (or even a particular version) depends on having access to another software package (or even a particular version of another software package). For example, the VASP materials science software may depend on having a particular version of the FFTW (Fastest Fourier Transform in the West) software library available for it to work.

Environment Modules

Environment modules are the solution to these problems. A module is a self-contained description of a software package – it contains the settings required to run a software package and, usually, encodes required dependencies on other software packages.

There are a number of different environment module implementations commonly used on HPC systems: the two most common are TCL modules and Lmod. Both of these use similar syntax and the concepts are the same so learning to use one will allow you to use whichever is installed on the system you are using. In both implementations the module command is used to interact with environment modules. An additional subcommand is usually added to the command to specify what you want to do. For a list of subcommands you can use module -h or module help. As for all commands, you can access the full help on the man pages with man module.

On login you may start out with a default set of modules loaded or you may start out with an empty environment; this depends on the setup of the system you are using.

Listing Available Modules

To see available software modules, use module avail:

[yourUsername@login12 ~]$  module avail

--------------------- /shared/ucl/apps/modulefiles/core -----------------------
gerun             ops-tools/1.1.0   screen/4.2.1      userscripts/1.3.0
lm-utils/1.0      ops-tools/2.0.0   userscripts/1.0.0 userscripts/1.4.0
mrxvt/0.5.4       rcps-core/1.0.0   userscripts/1.1.0
ops-tools/1.0.0   rlwrap/0.43       userscripts/1.2.0

----------------- /shared/ucl/apps/modulefiles/applications -------------------
abaqus/2017
adf/2014.10
afni/20151030
afni/20181011
amber/14/mpi/intel-2015-update2
amber/14/openmp/intel-2015-update2
amber/14/serial/intel-2015-update2
amber/16/mpi/gnu-4.9.2

[output truncated]

Listing Currently Loaded Modules

You can use the module list command to see which modules you currently have loaded in your environment. If you have no modules loaded, you will see a message telling you so

[yourUsername@login12 ~]$  module list

No Modulefiles Currently Loaded.

Loading and Unloading Software

To load a software module, use module load. In this example we will use Python 3.

Initially, Python 3 is not loaded. We can test this by using the which command. which looks for programs the same way that Bash does, so we can use it to tell us where a particular piece of software is stored.

[yourUsername@login12 ~]$  which python3

/usr/bin/which: no python3 in (
/shared/ucl/apps/python/3.6.3/gnu-4.9.2/bin:
/shared/ucl/apps/cluster-bin:
/shared/ucl/apps/cluster-scripts:
/shared/ucl/apps/mrxvt/0.5.4/bin:
/shared/ucl/apps/tmux/2.2/gnu-4.9.2/bin:
/shared/ucl/apps/emacs/24.5/gnu-4.9.2/bin:
/shared/ucl/apps/giflib/5.1.1/gnu-4.9.2/bin:
/shared/ucl/apps/dos2unix/7.3/gnu-4.9.2/bin:
/shared/ucl/apps/nano/2.4.2/gnu-4.9.2/bin:
/shared/ucl/apps/apr-util/1.5.4/bin:
/shared/ucl/apps/apr/1.5.2/bin:
/shared/ucl/apps/git/2.19.1/gnu-4.9.2/bin:
/shared/ucl/apps/flex/2.5.39/gnu-4.9.2/bin:
/shared/ucl/apps/cmake/3.13.3/gnu-4.9.2/bin:
/shared/ucl/apps/gcc/4.9.2/bin:/opt/sge/bin:
/opt/sge/bin/lx-amd64:/usr/local/bin:/usr/bin:
/usr/local/sbin:/usr/sbin:/opt/ibutils/bin)

We can load the python3 command with module load:

[yourUsername@login12 ~]$  module load python
[yourUsername@login12 ~]$  which python3

/shared/ucl/apps/python/3.6.3/gnu-4.9.2/bin/python3

So, what just happened?

To understand the output, first we need to understand the nature of the $PATH environment variable. $PATH is a special environment variable that controls where a UNIX system looks for software. Specifically $PATH is a list of directories (separated by :) that the OS searches through for a command before giving up and telling us it can’t find it. As with all environment variables we can print it out using echo.

[yourUsername@login12 ~]$  echo $PATH

/shared/ucl/apps/python/3.6.3/gnu-4.9.2/bin:/shared/ucl/apps/intel-mpi/ucl-wrapper/bin:/shared/ucl/apps/intel/2018.Update3/impi/2018.3.222/intel64/bin:/shared/ucl/apps/intel/2018.Update3/debugger_2018/gdb/intel64_mic/bin:/shared/ucl/apps/intel/2018.Update3/compilers_and_libraries_2018.3.222/linux/mpi/intel64/bin:/shared/ucl/apps/intel/2018.Update3/compilers_and_libraries_2018.3.222/linux/bin/intel64:/shared/ucl/apps/cluster-bin:/shared/ucl/apps/cluster-scripts:/shared/ucl/apps/mrxvt/0.5.4/bin:/shared/ucl/apps/tmux/2.2/gnu-4.9.2/bin:/shared/ucl/apps/emacs/24.5/gnu-4.9.2/bin:/shared/ucl/apps/giflib/5.1.1/gnu-4.9.2/bin:/shared/ucl/apps/dos2unix/7.3/gnu-4.9.2/bin:/shared/ucl/apps/nano/2.4.2/gnu-4.9.2/bin:/shared/ucl/apps/apr-util/1.5.4/bin:/shared/ucl/apps/apr/1.5.2/bin:/shared/ucl/apps/git/2.19.1/gnu-4.9.2/bin:/shared/ucl/apps/flex/2.5.39/gnu-4.9.2/bin:/shared/ucl/apps/cmake/3.13.3/gnu-4.9.2/bin:/shared/ucl/apps/gcc/4.9.2/bin:/opt/sge/bin:/opt/sge/bin/lx-amd64:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin

You’ll notice a similarity to the output of the which command. In this case, there’s only one difference: the different directory at the beginning. When we ran the module load command, it added a directory to the beginning of our $PATH. Let’s examine what’s there:

[yourUsername@login12 ~]$  ls /shared/ucl/apps/python/3.6.3/gnu-4.9.2/bin

[output truncated]

conda-convert           glib-compile-schemas     lconvert           rst2man.py
conda-develop           glib-genmarshal          libpng16-config    tiff2rgba
conda-env               glib-gettextize          libpng-config      tiffcmp
conda-index             glib-mkenums             patchelf           tiffcp
conda-inspect           gobject-query            python             tiffcrop
conda-metapackage       gresource                python3            tiffdither
conda-render            hb-view                  python3.6          tiffdump
conda-server            jupyter-run              python3.6-config   tiffinfo
conda-skeleton          jupyter-serverextension  python3.6m         tiffmedian
elbadmin                jupyter-troubleshoot     python3.6m-config  tiffset
gio-querymodules        jupyter-trust            python3-config
glacier                 kill_instance            pyuic5
glib-compile-resources  launch_instance          pyvenv

[output truncated]

Taking this to its conclusion, module load will add software to your $PATH. It “loads” software. A special note on this - depending on which version of the module program that is installed at your site, module load will also load required software dependencies.

To demonstrate, let’s load the ansys module and then use the module list command to show which modules we currently have loaded in our environment. (ANSYS is an engineering simulation product.)

[yourUsername@login12 ~]$  module load ansys

ansys/2019.r3(73):ERROR:151: Module 'ansys/2019.r3' depends on one of the
module(s) 'giflib/5.1.1'
ansys/2019.r3(73):ERROR:102: Tcl command execution failed:
prereq   giflib/5.1.1

This shows that the default ansys module will not run because it first needs giflib/5.1.1 to be loaded. Some HPC systems will automatically load dependencies like this, but at the time of writing (June 2020) UCL’s Myriad does not.

Let’s load the giflib module:

[yourUsername@login12 ~]$  module load giflib

giflib/5.1.1(18):ERROR:151: Module 'giflib/5.1.1' depends on one of the
module(s) 'gcc-libs/4.9.2'
giflib/5.1.1(18):ERROR:102: Tcl command execution failed: prereq gcc-libs

Here, we see that the giflib module itself also has a dependency, gcc-libs. So we have to load that first, then load giflib, and then finally load ansys.

[yourUsername@login12 ~]$  module load gcc-libs/4.9.2
[yourUsername@login12 ~]$  module load giflib/5.1.1
[yourUsername@login12 ~]$  module load ansys

~/Scratch/.config is configured
...
...
~/.mw doesn't exist - creating

If you now use the module list command, you should see these three modules included in the list.

To unload a specific module, e.g. ansys, run the command module unload ansys. (On some systems, this will also unload the modules it depends on. Currently this is not the case with Myriad.)

If we wanted to unload everything at once (all modules), we could run module purge (unloads everything).

[yourUsername@login12 ~]$  module purge
[yourUsername@login12 ~]$  module list

No Modulefiles Currently Loaded.

Note that this module loading process happens principally through the manipulation of environment variables like $PATH. There is usually little or no data transfer involved.

The module loading process manipulates other special environment variables as well, including variables that influence where the system looks for software libraries, and sometimes variables which tell commercial software packages where to find license servers.

The module command also restores these shell environment variables to their previous state when a module is unloaded.

Software Versioning

So far, we’ve learned how to load and unload software packages. This is very useful. However, we have not yet addressed the issue of software versioning. At some point or other, you will run into issues where only one particular version of some software will be suitable. Perhaps a key bugfix only happened in a certain version, or version X broke compatibility with a file format you use. In either of these example cases, it helps to be very specific about what software is loaded.

Let’s examine the output of module avail more closely.

[yourUsername@login12 ~]$  module avail

--------------------- /shared/ucl/apps/modulefiles/core -----------------------
gerun             ops-tools/1.1.0   screen/4.2.1      userscripts/1.3.0
lm-utils/1.0      ops-tools/2.0.0   userscripts/1.0.0 userscripts/1.4.0
mrxvt/0.5.4       rcps-core/1.0.0   userscripts/1.1.0
ops-tools/1.0.0   rlwrap/0.43       userscripts/1.2.0

----------------- /shared/ucl/apps/modulefiles/applications -------------------
abaqus/2017
adf/2014.10
afni/20151030
afni/20181011
amber/14/mpi/intel-2015-update2
amber/14/openmp/intel-2015-update2
amber/14/serial/intel-2015-update2
amber/16/mpi/gnu-4.9.2

[output truncated]

To be more specific, you can specify the particular software you want. e.g.,

[yourUsername@login12 ~]$  module avail stata

~~~~~~~~~~~~~ /shared/ucl/apps/modulefiles/applications ~~~~~~~~~~~~
stata/14 stata/15

Let’s take a closer look at the matlab module. Matlab is a widely-used piece of software which uses matrix multiplication. As we shall see, there are different versions available, and we want to make sure the one we use is the correct one for our purposes.

Let’s see which versions we have access to.

[yourUsername@login12 ~]$  module avail matlab

~~~~~~~~~~~~~ /shared/ucl/apps/modulefiles/applications ~~~~~~~~~~~~
matlab/full/r2015b/8.6 matlab/full/r2017a/9.2 matlab/full/r2018b/9.5
matlab/full/r2016b/9.1 matlab/full/r2018a/9.4 matlab/full/r2019b/9.7

In this case, there are seven different versions. How do we load each copy, and which copy is the default?

Sometimes, on some systems, a module might have a (default) next to it. This indicates that it is the default (i.e. which would be loaded if we type module load matlab). In this case, we don’t see this, so we will have to load matlab and see what we get.

[yourUsername@login12 ~]$  module load matlab

matlab/full/r2019b/9.7(99):ERROR:151: Module 'matlab/full/r2019b/9.7'
depends on one of the module(s) 'gcc-libs/4.9.2'
matlab/full/r2019b/9.7(99):ERROR:102: Tcl command execution failed:
prereq gcc-libs

Here, we see that the default version of Matlab on the system is r2019b/9.7, which in this case is the most recent version. However, you should not assume that the default version is necessarily the latest.

As we saw in the earlier example, there are one or more dependencies.

Suppose we decide to load an earlier version of Matlab, e.g. r2017a/9.2.

[yourUsername@login12 ~]$  module purge
[yourUsername@login12 ~]$  module list

No Modulefiles Currently Loaded.

[yourUsername@login12 ~]$  module load matlab/full/r2017a/9.2

matlab/full/r2017a/9.2(96):ERROR:151: Module 'matlab/full/r2017a/9.2' depends
on one of the module(s) 'gcc-libs/4.9.2'
matlab/full/r2017a/9.2(96):ERROR:102: Tcl command execution failed:
prereq gcc-libs

[yourUsername@login12 ~]$  module load gcc-libs/4.9.2
[yourUsername@login12 ~]$  module load matlab/full/r2017a/9.2

matlab/full/r2017a/9.2(97):ERROR:151: Module 'matlab/full/r2017a/9.2' depends
on one of the module(s) 'xorg-utils/X11R7.7'
matlab/full/r2017a/9.2(97):ERROR:102: Tcl command execution failed:
prereq xorg-utils/X11R7.7

[yourUsername@login12 ~]$  module load xorg-utils/X11R7.7
[yourUsername@login12 ~]$  module load matlab/full/r2017a/9.2

~/.matlab is a symbolic link pointing to /home/yourUsername/Scratch/.matlab

Matlab setup complete type matlab to start Matlab.

[yourUsername@login12 ~]$  matlab -nodisplay -nosplash -nodesktop

                                               < M A T L A B (R) >
                                     Copyright 1984-2017 The MathWorks, Inc.
                                      R2017a (9.2.0.556344) 64-bit (glnxa64)
                                                  March 27, 2017

To get started, type one of these: helpwin, helpdesk, or demo.
For product information, visit www.mathworks.com.

>> quit

Note that you cannot load two different versions of the same software at once. Currently, we have loaded matlab/full/r2017a/9.2. Let’s try also loading matlab/full/r2015b/8.6:

[yourUsername@login12 ~]$  module load matlab/full/r2015b/8.6

matlab/full/r2015b/8.6(108):ERROR:150: Module 'matlab/full/r2015b/8.6'
conflicts with the currently loaded module(s) 'matlab/full/r2017a/9.2'
matlab/full/r2015b/8.6(108):ERROR:102: Tcl command execution failed:
conflict matlab

As we can see, we get an error message about conflicts. If we do indeed wish to load version r2015b/8.6, we can say

[yourUsername@login12 ~]$  module unload matlab/full/r2017a/9.2
[yourUsername@login12 ~]$  module load matlab/full/r2015b/8.6

or, in one step:

[yourUsername@login12 ~]$  module swap matlab matlab/full/r2015b/8.6

Check that this module has been loaded:

[yourUsername@login12 ~]$  module list

Currently Loaded Modulefiles:
  1) gcc-libs/4.9.2        9) gerun           17) userscripts/1.4.0
     ...                      ...                 ...
  6) apr-util/1.5.4       14) emacs/24.5      22) xorg-utils/X11R7.7
  7) subversion/1.8.13    15) tmux/2.2        23) matlab/full/r2015b/8.6
  8) screen/4.2.1         16) mrxvt/0.5.4

Using Software Modules in Scripts

Create a job that is able to run python3 --version. Remember, no software is loaded by default! Running a job is just like logging on to the system (you should not assume a module loaded on the login node is loaded on a compute node).

Solution

[yourUsername@login12 ~]$  nano python-module.sh
[yourUsername@login12 ~]$  cat python-module.sh

#!/bin/bash -l

module load python3

python3 --version

[yourUsername@login12 ~]$  qsub python-module.sh

Key Points

  • Load software with module load softwareName.

  • Unload software with module unload

  • The module system handles software versioning and package conflicts for you automatically.

Transferring files with remote computers

Overview

Teaching: 15 min
Exercises: 15 min
Questions

  • How do I transfer files to (and from) the cluster?

Objectives

  • Transfer files to and from a computing cluster.

Performing work on a remote computer is not very useful if we cannot get files to or from the cluster. There are several options for transferring data between computing resources using CLI and GUI utilities, a few of which we will cover.

Download Files From the Internet

One of the most straightforward ways to download files is to use either curl or wget. One of these is usually installed in most Linux shells, on Mac OS terminal and in GitBash. Any file that can be downloaded in your web browser through a direct link can be downloaded using curl -O or wget. This is a quick way to download datasets or source code.

The syntax for these commands is: curl -O https://some/link/to/a/file and wget https://some/link/to/a/file. Try it out by downloading some material we’ll use later on, from a terminal on your local machine.

[user@laptop ~]$ curl -O http://github-pages.arc.ucl.ac.uk/hpc-intro/files/hpc-intro-data.tar.gz

or

[user@laptop ~]$ wget http://github-pages.arc.ucl.ac.uk/hpc-intro/files/hpc-intro-data.tar.gz

tar.gz?

This is an archive file format, just like .zip, commonly used and supported by default on Linux, which is the operating system the majority of HPC cluster machines run. You may also see the extension .tgz, which is exactly the same. We’ll talk more about “tarballs,” since “tar-dot-g-z” is a mouthful, later on.

Transferring Single Files and Folders With scp

To copy a single file to or from the cluster, we can use scp (“secure copy”). The syntax can be a little complex for new users, but we’ll break it down. The scp command is a relative of the ssh command we used to access the system, and can use the same public-key authentication mechanism.

To upload to another computer:

[user@laptop ~]$ scp path/to/local/file.txt [email protected]:/path/on/Myriad

To download from another computer:

[user@laptop ~]$ scp [email protected]:/path/on/Myriad/file.txt path/to/local/

Note that everything after the : is relative to our home directory on the remote computer. We can leave it at that if we don’t care where the file goes.

[user@laptop ~]$ scp local-file.txt [email protected]:

Upload a File

Copy the file you just downloaded from the Internet to your home directory on Myriad.

Solution

[user@laptop ~]$ scp hpc-intro-data.tar.gz [email protected]:~/

Most computer clusters are protected from the open internet by a firewall. This means that the curl command will fail, as an address outside the firewall is unreachable from the inside. To get around this, run the curl or wget command from your local machine to download the file, then use the scp command to upload it to the cluster.

Why Not Download on Myriad Directly?

Try downloading the file directly. Note that it may well fail, and that’s OK!

Commands

[user@laptop ~]$ ssh [email protected]
[yourUsername@login12 ~]$  curl -O http://github-pages.arc.ucl.ac.uk/hpc-intro/files/hpc-intro-data.tar.gz
or
[yourUsername@login12 ~]$  wget http://github-pages.arc.ucl.ac.uk/hpc-intro/files/hpc-intro-data.tar.gz

Did it work? If not, what does the terminal output tell you about what happened?

To copy a whole directory, we add the -r flag, for “recursive”: copy the item specified, and every item below it, and every item below those… until it reaches the bottom of the directory tree rooted at the folder name you provided.

[user@laptop ~]$ scp -r some-local-folder [email protected]:target-directory/

Caution

For a large directory – either in size or number of files – copying with -r can take a long time to complete.

What’s in a /?

When using scp, you may have noticed that a : always follows the remote computer name; sometimes a / follows that, and sometimes not, and sometimes there’s a final /. On Linux computers, / is the root directory, the location where the entire filesystem (and others attached to it) is anchored. A path starting with a / is called absolute, since there can be nothing above the root /. A path that does not start with / is called relative, since it is not anchored to the root.

If you want to upload a file to a location inside your home directory – which is often the case – then you don’t need a leading /. After the :, start writing the sequence of folders that lead to the final storage location for the file or, as mentioned above, provide nothing if your home directory is the destination.

A trailing slash on the source directory is optional, and has no effect for scp -r, but is important in other commands, like rsync.

A Note on rsync

As you gain experience with transferring files, you may find the scp command limiting. The rsync utility provides advanced features for file transfer and is typically faster compared to both scp and sftp (see below). It is especially useful for transferring large and/or many files and creating synced backup folders.

The syntax is similar to scp. To transfer to another computer with commonly used options:

[user@laptop ~]$ rsync -avzP path/to/local/file.txt [email protected]:directory/path/on/Myriad/

The options are:

  • a (archive) to preserve file timestamps and permissions among other things
  • v (verbose) to get verbose output to help monitor the transfer
  • z (compression) to compress the file during transit to reduce size and transfer time
  • P (partial/progress) to preserve partially transferred files in case of an interruption and also displays the progress of the transfer.

To recursively copy a directory, we can use the same options:

[user@laptop ~]$ rsync -avzP path/to/local/dir [email protected]:directory/path/on/Myriad/

As written, this will place the local directory and its contents under the specified directory on the remote system. If a trailing slash is added to the source, a new directory corresponding to the transferred directory (‘dir’ in the example) will not be created, and the contents of the source directory will be copied directly into the destination directory. Omitting or adding a trailing slash on the destination makes no difference.

The a (archive) option implies recursion.

To download a file, we simply change the source and destination:

[user@laptop ~]$ rsync -avzP [email protected]:path/on/Myriad/file.txt path/to/local/

All file transfers using the above methods use SSH to encrypt data sent through the network. So, if you can connect via SSH, you will be able to transfer files. By default, SSH uses network port 22. If a custom SSH port is in use, you will have to specify it using the appropriate flag, often -p, -P, or --port. Check --help or the man page if you’re unsure.

Change the Rsync Port

Say we have to connect rsync through port 768 instead of 22. How would we modify this command?

[user@laptop ~]$ rsync test.txt [email protected]:

Solution

[user@laptop ~]$ rsync --help | grep port
     --port=PORT             specify double-colon alternate port number
See http://rsync.samba.org/ for updates, bug reports, and answers
[user@laptop ~]$ rsync --port=768 test.txt [email protected]:

Transferring Files Interactively with FileZilla

FileZilla is a cross-platform client for downloading and uploading files to and from a remote computer. It is absolutely fool-proof and always works quite well. It uses the sftp protocol. You can read more about using the sftp protocol in the command line in the lesson discussion.

Download and install the FileZilla client from https://filezilla-project.org. After installing and opening the program, you should end up with a window with a file browser of your local system on the left hand side of the screen. When you connect to the cluster, your cluster files will appear on the right hand side.

To connect to the cluster, we’ll just need to enter our credentials at the top of the screen:

Leave the password blank there and in any popups to have FileZilla use your existing SSH keys that are loaded in your SSH agent.

Hit “Quickconnect” to connect. You should see your remote files appear on the right hand side of the screen. You can drag-and-drop files between the left (local) and right (remote) sides of the screen to transfer files.

Connecting Remotely Via an SSH Tunnel

If you are remote and not using a VPN, you will need to set up an SSH tunnel using your terminal.

[user@laptop ~]$ ssh -L 3333:myriad.rc.ucl.ac.uk:22 [email protected] -N

This connects the local port 3333 to the cluster’s SSH port, via the remote gateway. The -N option tells it not to execute any remote commands. Leave this running in the terminal and in FileZilla, set

  • Host: sftp://localhost
  • User: Your cluster username
  • Password: Your cluster password (leave blank to use your SSH keys)
  • Port: 3333

Hit “Quickconnect” to connect.

Finally, if you need to move large files (typically larger than a gigabyte) from one remote computer to another remote computer, SSH in to the computer hosting the files and use scp or rsync to transfer over to the other. This will be more efficient than using FileZilla (or related applications) that would copy from the source to your local machine, then to the destination machine.

Archiving Files

One of the biggest challenges we often face when transferring data between remote HPC systems is that of large numbers of files. There is an overhead to transferring each individual file and when we are transferring large numbers of files these overheads combine to slow down our transfers to a large degree.

The solution to this problem is to archive multiple files into smaller numbers of larger files before we transfer the data to improve our transfer efficiency. Sometimes we will combine archiving with compression to reduce the amount of data we have to transfer and so speed up the transfer.

The most common archiving command you will use on a (Linux) HPC cluster is tar. tar can be used to combine files into a single archive file and, optionally, compress it.

Let’s start with the file we downloaded from the lesson site, hpc-lesson-data.tar.gz. The “gz” part stands for gzip, which is a compression library. Reading this file name, it appears somebody took a folder named “hpc-lesson-data,” wrapped up all its contents in a single file with tar, then compressed that archive with gzip to save space. Let’s check using tar with the -t flag, which prints the “table of contents” without unpacking the file, specified by -f <filename>, on the remote computer. Note that you can concatenate the two flags, instead of writing -t -f separately.

[user@laptop ~]$ ssh [email protected]
[yourUsername@login12 ~]$  tar -tf hpc-lesson-data.tar.gz
hpc-intro-data/
hpc-intro-data/north-pacific-gyre/
hpc-intro-data/north-pacific-gyre/NENE01971Z.txt
hpc-intro-data/north-pacific-gyre/goostats
hpc-intro-data/north-pacific-gyre/goodiff
hpc-intro-data/north-pacific-gyre/NENE02040B.txt
hpc-intro-data/north-pacific-gyre/NENE01978B.txt
hpc-intro-data/north-pacific-gyre/NENE02043B.txt
hpc-intro-data/north-pacific-gyre/NENE02018B.txt
hpc-intro-data/north-pacific-gyre/NENE01843A.txt
hpc-intro-data/north-pacific-gyre/NENE01978A.txt
hpc-intro-data/north-pacific-gyre/NENE01751B.txt
hpc-intro-data/north-pacific-gyre/NENE01736A.txt
hpc-intro-data/north-pacific-gyre/NENE01812A.txt
hpc-intro-data/north-pacific-gyre/NENE02043A.txt
hpc-intro-data/north-pacific-gyre/NENE01729B.txt
hpc-intro-data/north-pacific-gyre/NENE02040A.txt
hpc-intro-data/north-pacific-gyre/NENE01843B.txt
hpc-intro-data/north-pacific-gyre/NENE01751A.txt
hpc-intro-data/north-pacific-gyre/NENE01729A.txt
hpc-intro-data/north-pacific-gyre/NENE02040Z.txt

This shows a folder containing another folder, which contains a bunch of files. If you’ve taken The Carpentries’ Shell lesson recently, these might look familiar. Let’s see about that compression, using du for “disk usage”.

[yourUsername@login12 ~]$  du -sh hpc-lesson-data.tar.gz
36K     hpc-intro-data.tar.gz

Files Occupy at Least One “Block”

If the filesystem block size is larger than 36 KB, you’ll see a larger number: files cannot be smaller than one block.

Now let’s unpack the archive. We’ll run tar with a few common flags:

When it’s done, check the directory size with du and compare.

Extract the Archive

Using the four flags above, unpack the lesson data using tar. Then, check the size of the whole unpacked directory using du.

Hint: tar lets you concatenate flags.

Commands

[yourUsername@login12 ~]$  tar -xvzf hpc-lesson-data.tar.gz

hpc-intro-data/
hpc-intro-data/north-pacific-gyre/
hpc-intro-data/north-pacific-gyre/NENE01971Z.txt
hpc-intro-data/north-pacific-gyre/goostats
hpc-intro-data/north-pacific-gyre/goodiff
hpc-intro-data/north-pacific-gyre/NENE02040B.txt
hpc-intro-data/north-pacific-gyre/NENE01978B.txt
hpc-intro-data/north-pacific-gyre/NENE02043B.txt
hpc-intro-data/north-pacific-gyre/NENE02018B.txt
hpc-intro-data/north-pacific-gyre/NENE01843A.txt
hpc-intro-data/north-pacific-gyre/NENE01978A.txt
hpc-intro-data/north-pacific-gyre/NENE01751B.txt
hpc-intro-data/north-pacific-gyre/NENE01736A.txt
hpc-intro-data/north-pacific-gyre/NENE01812A.txt
hpc-intro-data/north-pacific-gyre/NENE02043A.txt
hpc-intro-data/north-pacific-gyre/NENE01729B.txt
hpc-intro-data/north-pacific-gyre/NENE02040A.txt
hpc-intro-data/north-pacific-gyre/NENE01843B.txt
hpc-intro-data/north-pacific-gyre/NENE01751A.txt
hpc-intro-data/north-pacific-gyre/NENE01729A.txt
hpc-intro-data/north-pacific-gyre/NENE02040Z.txt

Note that we did not type out -x -v -z -f, thanks to the flag concatenation, though the command works identically either way.

[yourUsername@login12 ~]$  du -sh hpc-lesson-data
144K    hpc-intro-data

Was the Data Compressed?

Text files compress nicely: the “tarball” is one-quarter the total size of the raw data!

If you want to reverse the process – compressing raw data instead of extracting it – set a c flag instead of x, set the archive filename, then provide a directory to compress:

[user@laptop ~]$ tar -cvzf compressed_data.tar.gz hpc-intro-data

Working with Windows

When you transfer text files to from a Windows system to a Unix system (Mac, Linux, BSD, Solaris, etc.) this can cause problems. Windows encodes its files slightly different than Unix, and adds an extra character to every line.

On a Unix system, every line in a file ends with a \n (newline). On Windows, every line in a file ends with a \r\n (carriage return + newline). This causes problems sometimes.

Though most modern programming languages and software handles this correctly, in some rare instances, you may run into an issue. The solution is to convert a file from Windows to Unix encoding with the dos2unix command.

You can identify if a file has Windows line endings with cat -A filename. A file with Windows line endings will have ^M$ at the end of every line. A file with Unix line endings will have $ at the end of a line.

To convert the file, just run dos2unix filename. (Conversely, to convert back to Windows format, you can run unix2dos filename.)

Key Points

  • wget and curl -O download a file from the internet.

  • scp and rsync transfer files to and from your computer.

  • You can use an SFTP client like FileZilla to transfer files through a GUI.

Running a parallel job

Overview

Teaching: 30 min
Exercises: 60 min
Questions

  • How do we execute a task in parallel?

  • What benefits arise from parallel execution?

  • What are the limits of gains from execution in parallel?

Objectives

  • Construct a program that can execute in parallel.

  • Prepare a job submission script for the parallel executable.

  • Launch jobs with parallel execution.

  • Record and summarize the timing and accuracy of jobs.

  • Describe the relationship between job parallelism and performance.

We now have the tools we need to run a multi-processor job. This is a very important aspect of HPC systems, as parallelism is one of the primary tools we have to improve the performance of computational tasks.

Our example implements a stochastic algorithm for estimating the value of π, the ratio of the circumference to the diameter of a circle. The program generates a large number of random points on a 1×1 square centered on (½,½), and checks how many of these points fall inside the unit circle. On average, π/4 of the randomly-selected points should fall in the circle, so π can be estimated from 4f, where f is the observed fraction of points that fall in the circle. Because each sample is independent, this algorithm is easily implemented in parallel.

/hpc-intro/Algorithm%20for%20computing%20pi%20through%20random%20sampling

A Serial Solution to the Problem

We start from a Python script using concepts taught in Software Carpentry’s Programming with Python workshops. We want to allow the user to specify how many random points should be used to calculate π through a command-line parameter. This script will only use a single CPU for its entire run, so it’s classified as a serial process.

Let’s write a Python program, pi.py, to estimate π for us. Start by importing the numpy module for calculating the results, and the sys module to process command-line parameters:

import numpy as np
import sys

We define a Python function inside_circle that accepts a single parameter for the number of random points used to calculate π. See Programming with Python: Creating Functions for a review of Python functions. It randomly samples points with both x and y on the half-open interval [0, 1). It then computes their distances from the origin (i.e., radii), and returns how many of those distances were less than or equal to 1.0. All of this is done using vectors of double-precision (64-bit) floating-point values.

def inside_circle(total_count):
    x = np.random.uniform(size=total_count)
    y = np.random.uniform(size=total_count)
    radii = np.sqrt(x * x + y * y)
    count = len(radii[np.where(radii<=1.0)])
    return count

Next, we create a main function to call the inside_circle function and calculate π from its returned result. See Programming with Python: Command-Line Programs for a review of main functions and parsing command-line parameters.

def main():
    n_samples = int(sys.argv[1])
    counts = inside_circle(n_samples)
    my_pi = 4.0 * counts / n_samples
    print(my_pi)

if __name__ == '__main__':
    main()

If we run the Python script locally with a command-line parameter, as in python pi-serial.py 1024, we should see the script print its estimate of π:

[user@laptop ~]$ python pi-serial.py 1024
3.10546875

Random Number Generation

In the preceding code, random numbers are conveniently generated using the built-in capabilities of NumPy. In general, random-number generation is difficult to do well, it’s easy to accidentally introduce correlations into the generated sequence.

  • Discuss why generating high quality random numbers might be difficult.
  • Is the quality of random numbers generated sufficient for estimating π in this implementation?

Solution

  • Computers are deterministic and produce pseudo random numbers using an algorithm. The choice of algorithm and its parameters determines how random the generated numbers are. Pseudo random number generation algorithms usually produce a sequence numbers taking the previous output as an input for generating the next number. At some point the sequence of pseudo random numbers will repeat, so care is required to make sure the repetition period is long and that the generated numbers have statistical properties similar to those of true random numbers.
  • Yes.

Measuring Performance of the Serial Solution

The stochastic method used to estimate π should converge on the true value as the number of random points increases. But as the number of points increases, creating the variables x, y, and radii requires more time and more memory. Eventually, the memory required may exceed what’s available on our local laptop or desktop, or the time required may be too long to meet a deadline. So we’d like to take some measurements of how much memory and time the script requires, and later take the same measurements after creating a parallel version of the script to see the benefits of parallelizing the calculations required.

Estimating Memory Requirements

Since the largest variables in the script are x, y, and radii, each containing n_samples points, we’ll modify the script to report their total memory required. Each point in x, y, or radii is stored as a NumPy float64, we can use NumPy’s dtype function to calculate the size of a float64.

Replace the print(my_pi) line with the following:

size_of_float = np.dtype(np.float64).itemsize
memory_required = 3 * n_samples * size_of_float / (1024**3)
print("Pi: {}, memory: {} GiB".format(my_pi, memory_required))

The first line calculates the bytes of memory required for a single 64-bit floating point number using the dtype function. The second line estimates the total amount of memory required to store three variables containing n_samples float64 values, converting the value into units of gibibytes. The third line prints both the estimate of π and the estimated amount of memory used by the script.

The updated Python script is:

import numpy as np
import sys

def inside_circle(total_count):
    x = np.random.uniform(size=total_count)
    y = np.random.uniform(size=total_count)
    radii = np.sqrt(x * x + y * y)
    count = len(radii[np.where(radii<=1.0)])
    return count

def main():
    n_samples = int(sys.argv[1])
    counts = inside_circle(n_samples)
    my_pi = 4.0 * counts / n_samples
    size_of_float = np.dtype(np.float64).itemsize
    memory_required = 3 * n_samples * size_of_float / (1024**3)
    print("Pi: {}, memory: {} GiB".format(my_pi, memory_required))

if __name__ == '__main__':
    main()

Run the script again with a few different values for the number of samples, and see how the memory required changes:

[user@laptop ~]$ python pi-serial.py 1000
Pi: 3.144, memory: 2.2351741790771484e-05 GiB
[user@laptop ~]$ python pi-serial.py 2000
Pi: 3.18, memory: 4.470348358154297e-05 GiB
[user@laptop ~]$ python pi-serial.py 1000000
Pi: 3.140944, memory: 0.022351741790771484 GiB
[user@laptop ~]$ python pi-serial.py 100000000
Pi: 3.14182724, memory: 2.2351741790771484 GiB

Here we can see that the estimated amount of memory required scales linearly with the number of samples used. In practice, there is some memory required for other parts of the script, but the x, y, and radii variables are by far the largest influence on the total amount of memory required.

Estimating Calculation Time

Most of the calculations required to estimate π are in the inside_circle function:

  1. Generating n_samples random values for x and y.
  2. Calculating n_samples values of radii from x and y.
  3. Counting how many values in radii are under 1.0.

There’s also one multiplication operation and one division operation required to convert the counts value to the final estimate of π in the main function.

A simple way to measure the calculation time is to use Python’s datetime module to store the computer’s current date and time before and after the calculations, and calculate the difference between those times.

To add the time measurement to the script, add the following line below the import sys line:

import datetime

Then, add the following line immediately above the line calculating counts:

start_time = datetime.datetime.now()

Add the following two lines immediately below the line calculating counts:

end_time = datetime.datetime.now()
elapsed_time = (end_time - start_time).total_seconds()

And finally, modify the print statement with the following:

print("Pi: {}, memory: {} GiB, time: {} s".format(my_pi, memory_required,
                                                  elapsed_time))

The final Python script for the serial solution is:

import numpy as np
import sys
import datetime

def inside_circle(total_count):
    x = np.random.uniform(size=total_count)
    y = np.random.uniform(size=total_count)
    radii = np.sqrt(x * x + y * y)
    count = len(radii[np.where(radii<=1.0)])
    return count

def main():
    n_samples = int(sys.argv[1])
    start_time = datetime.datetime.now()
    counts = inside_circle(n_samples)
    my_pi = 4.0 * counts / n_samples
    end_time = datetime.datetime.now()
    elapsed_time = (end_time - start_time).total_seconds()
    size_of_float = np.dtype(np.float64).itemsize
    memory_required = 3 * n_samples * size_of_float / (1024**3)
    print("Pi: {}, memory: {} GiB, time: {} s".format(my_pi, memory_required,
                                                      elapsed_time))

if __name__ == '__main__':
    main()

Run the script again with a few different values for the number of samples, and see how the solution time changes:

[user@laptop ~]$ python pi-serial.py 1000000
Pi: 3.139612, memory: 0.022351741790771484 GiB, time: 0.034872 s
[user@laptop ~]$ python pi-serial.py 10000000
Pi: 3.1425492, memory: 0.22351741790771484 GiB, time: 0.351212 s
[user@laptop ~]$ python pi-serial.py 100000000
Pi: 3.14146608, memory: 2.2351741790771484 GiB, time: 3.735195 s

Here we can see that the amount of time required scales approximately linearly with the number of samples used. There could be some variation in additional runs of the script with the same number of samples, since the elapsed time is affected by other programs running on the computer at the same time. But if the script is the most computationally-intensive process running at the time, its calculations are the largest influence on the elapsed time.

Now that we’ve developed our initial script to estimate π, we can see that as we increase the number of samples:

  1. The estimate of π tends to become more accurate.
  2. The amount of memory required scales approximately linearly.
  3. The amount of time to calculate scales approximately linearly.

In general, achieving a better estimate of π requires a greater number of points. Take a closer look at inside_circle: should we expect to get high accuracy on a single machine?

Probably not. The function allocates three arrays of size N equal to the number of points belonging to this process. Using 64-bit floating point numbers, the memory footprint of these arrays can get quite large. Each 100,000,000 points sampled consumes 2.24 GiB of memory. Sampling 400,000,000 points consumes 8.94 GiB of memory, and if your machine has less RAM than that, it will grind to a halt. If you have 16 GiB installed, you won’t quite make it to 750,000,000 points.

Running the Serial Job on a Compute Node

Create a submission file, requesting one task on a single node and enough memory to prevent the job from running out of memory:

[yourUsername@login12 ~]$  nano serial-pi.sh
[yourUsername@login12 ~]$  cat serial-pi.sh

#!/bin/bash -l
#$ -N serial-pi
#$ -l mem=3G
#$ -l h_rt=00:30:00
#$ -cwd

# Load the computing environment we need
module load python3

# Execute the task
python pi.py 100000000

Then submit your job. We will use the batch file to set the options, rather than the command line.

[yourUsername@login12 ~]$  qsub serial-pi.sh

As before, use the status commands to check when your job runs. Use ls to locate the output file, and examine it. Is it what you expected?

Modify the job script to increase both the number of samples and the amount of memory requested (perhaps by a factor of 2, then by a factor of 10), and resubmit the job each time.

Even with sufficient memory for necessary variables, a script could require enormous amounts of time to calculate on a single CPU. To reduce the amount of time required, we need to modify the script to use multiple CPUs for the calculations. In the largest problem scales, we could use multiple CPUs in multiple compute nodes, distributing the memory requirements across all the nodes used to calculate the solution.

Running the Parallel Job

We will run an example that uses the Message Passing Interface (MPI) for parallelism – this is a common tool on HPC systems.

What is MPI?

The Message Passing Interface is a set of tools which allow multiple parallel jobs to communicate with each other. Typically, a single executable is run multiple times, possibly on different machines, and the MPI tools are used to inform each instance of the executable about how many instances there are, which instance it is. MPI also provides tools to allow communication and coordination between instances. An MPI instance typically has its own copy of all the local variables.

While MPI jobs can generally be run as stand-alone executables, in order for them to run in parallel they must use an MPI run-time system, which is a specific implementation of the MPI standard. To do this, they should be started via a command such as mpiexec (or mpirun, or srun, etc. depending on the MPI run-time you need to use), which will ensure that the appropriate run-time support for parallelism is included.

MPI Runtime Arguments

On their own, commands such as mpiexec can take many arguments specifying how many machines will participate in the execution, and you might need these if you would like to run an MPI program on your laptop (for example). In the context of a queuing system, however, it is frequently the case that we do not need to specify this information as the MPI run-time will have been configured to obtain it from the queuing system, by examining the environment variables set when the job is launched.

What Changes Are Needed for an MPI Version of the π Calculator?

First, we need to import the MPI object from the Python module mpi4py by adding an from mpi4py import MPI line immediately below the import datetime line.

Second, we need to modify the “main” function to perform the overhead and accounting work required to:

  • subdivide the total number of points to be sampled,
  • partition the total workload among the various parallel processors available,
  • have each parallel process report the results of its workload back to the “rank 0” process, which does the final calculations and prints out the result.

The modifications to the serial script demonstrate four important concepts:

  • COMM_WORLD: the default MPI Communicator, providing a channel for all the processes involved in this mpiexec to exchange information with one another.
  • Scatter: A collective operation in which an array of data on one MPI rank is divided up, with separate portions being sent out to the partner ranks. Each partner rank receives data from the matching index of the host array.
  • Gather: The inverse of scatter. One rank populates a local array, with the array element at each index assigned the value provided by the corresponding partner rank – including the host’s own value.
  • Conditional Output: since every rank is running the same code, the partitioning, the final calculations, and the print statement are wrapped in a conditional so that only one rank performs these operations.

We add the lines:

comm = MPI.COMM_WORLD
cpus = comm.Get_size()
rank = comm.Get_rank()

immediately before the n_samples line to set up the MPI environment for each process.

We replace the start_time and counts lines with the lines:

if rank == 0:
  start_time = datetime.datetime.now()
  partitions = [ int(n_samples / cpus) ] * cpus
  for a in range(n_samples - sum(partitions)):
    partitions[a] += 1
  counts = [ int(0) ] * cpus
else:
  partitions = None
  counts = None

This ensures that only the rank 0 process measures times and coordinates the work to be distributed to all the ranks, while the other ranks get placeholder values for the partitions and counts variables.

Immediately below these lines, let’s

by adding the following three lines:

partition_item = comm.scatter(partitions, root=0)
count_item = inside_circle(partition_item)
counts = comm.gather(count_item, root=0)

Illustrations of these steps are shown below.

Setup the MPI environment and initialize local variables – including the vector containing the number of points to generate on each parallel processor:

/hpc-intro/MPI%20initialize

Distribute the number of points from the originating vector to all the parallel processors:

/hpc-intro/MPI%20scatter

Perform the computation in parallel:

/hpc-intro/MPI%20compute

Retrieve counts from all the parallel processes:

/hpc-intro/MPI%20gather

Print out the report:

/hpc-intro/MPI%20finalize

Finally, we’ll ensure the my_pi through print lines only run on rank 0. Otherwise, every parallel processor will print its local value, and the report will become hopelessly garbled:

if rank == 0:
   my_pi = 4.0 * sum(counts) / sum(partitions)
   end_time = datetime.datetime.now()
   elapsed_time = (end_time - start_time).total_seconds()
   size_of_float = np.dtype(np.float64).itemsize
   memory_required = 3 * sum(partitions) * size_of_float / (1024**3)
   print("Pi: {}, memory: {} GiB, time: {} s".format(my_pi, memory_required,
                                                            elapsed_time))

A fully commented version of the final MPI parallel python code is available here.

Our purpose here is to exercise the parallel workflow of the cluster, not to optimize the program to minimize its memory footprint. Rather than push our local machines to the breaking point (or, worse, the login node), let’s give it to a cluster node with more resources.

Create a submission file, requesting more than one task on a single node:

[yourUsername@login12 ~]$  nano parallel-pi.sh
[yourUsername@login12 ~]$  cat parallel-pi.sh

#!/bin/bash -l
#$ -N parallel-pi
#$ -l mem=3G
#$ -l h_rt=00:30:00
#$ -pe mpi 4
#$ -cwd

# Load the computing environment we need
module load python3
module load compilers/gnu/4.9.2
module load numactl/2.0.12  
module load binutils/2.29.1/gnu-4.9.2 
module load ucx/1.9.0/gnu-4.9.2
module load openmpi/4.1.1/gnu-4.9.2
module load mpi4py/3.1.4

# Execute the task
gerun python3 pi.py 100000000

Then submit your job. We will use the batch file to set the options, rather than the command line.

[yourUsername@login12 ~]$  qsub parallel-pi.sh

As before, use the status commands to check when your job runs. Use ls to locate the output file, and examine it. Is it what you expected?

Modify the job script to increase both the number of samples and the amount of memory requested (perhaps by a factor of 2, then by a factor of 10), and resubmit the job each time. You can also increase the number of CPUs.

How Much Does MPI Improve Performance?

In theory, by dividing up the π calculations among n MPI processes, we should see run times reduce by a factor of n. In practice, some time is required to start the additional MPI processes, for the MPI processes to communicate and coordinate, and some types of calculations may only be able to run effectively on a single CPU.

Additionally, if the MPI processes operate on different physical CPUs in the computer, or across multiple compute nodes, additional time is required for communication compared to all processes operating on a single CPU.

Amdahl’s Law is one way of predicting improvements in execution time for a fixed parallel workload. If a workload needs 20 hours to complete on a single core, and one hour of that time is spent on tasks that cannot be parallelized, only the remaining 19 hours could be parallelized. Even if an infinite number of cores were used for the parallel parts of the workload, the total run time cannot be less than one hour.

In practice, it’s common to evaluate the parallelism of an MPI program by

The speedup factor S is calculated as the single-CPU execution time divided by the multi-CPU execution time. For a laptop with 8 cores, the graph of speedup factor versus number of cores used shows relatively consistent improvement when using 2, 4, or 8 cores, but using additional cores shows a diminishing return.

/hpc-intro/MPI%20speedup%20factors%20on%20an%208-core%20laptop

For a set of HPC nodes containing 28 cores each, the graph of speedup factor versus number of cores shows consistent improvements up through three nodes and 84 cores, but worse performance when adding a fourth node with an additional 28 cores. This is due to the amount of communication and coordination required among the MPI processes requiring more time than is gained by reducing the amount of work each MPI process has to complete. This communication overhead is not included in Amdahl’s Law.

/hpc-intro/MPI%20speedup%20factors%20on%20an%208-core%20laptop

In practice, MPI speedup factors are influenced by:

In an HPC environment, we try to reduce the execution time for all types of jobs, and MPI is an extremely common way to combine dozens, hundreds, or thousands of CPUs into solving a single problem. To learn more about parallelization, see the parallel novice lesson lesson.

Key Points

  • Parallel programming allows applications to take advantage of parallel hardware; serial code will not ‘just work.’

  • Distributed memory parallelism is a common case, using the Message Passing Interface (MPI).

  • The queuing system facilitates executing parallel tasks.

  • Performance improvements from parallel execution do not scale linearly.

Using resources effectively

Overview

Teaching: 10 min
Exercises: 20 min
Questions

  • How can I review past jobs?

  • How can I use this knowledge to create a more accurate submission script?

Objectives

  • Look up job statistics.

  • Make more accurate resource requests in job scripts based on data describing past performance.

We’ve touched on all the skills you need to interact with an HPC cluster: logging in over SSH, loading software modules, submitting parallel jobs, and finding the output. Let’s learn about estimating resource usage and why it might matter.

Estimating Required Resources Using the Scheduler

Although we covered requesting resources from the scheduler earlier with the π code, how do we know what type of resources the software will need in the first place, and its demand for each? In general, unless the software documentation or user testimonials provide some idea, we won’t know how much memory or compute time a program will need.

Read the Documentation

Most HPC facilities maintain documentation as a wiki, a website, or a document sent along when you register for an account. Take a look at these resources, and search for the software you plan to use: somebody might have written up guidance for getting the most out of it.

A convenient way of figuring out the resources required for a job to run successfully is to submit a test job, and then ask the scheduler about its impact using jobhist. You can use this knowledge to set up the next job with a closer estimate of its load on the system. A good general rule is to ask the scheduler for 20% to 30% more time and memory than you expect the job to need. This ensures that minor fluctuations in run time or memory use will not result in your job being cancelled by the scheduler. Keep in mind that if you ask for too much, your job may not run even though enough resources are available, because the scheduler will be waiting for other people’s jobs to finish and free up the resources needed to match what you asked for.

Stats

Since we already submitted pi.py to run on the cluster, we can query the scheduler to see how long our job took and what resources were used. We will use jobhist to get statistics about parallel-pi.sh.

[yourUsername@login12 ~]$  jobhist

 FSTIME   | FETIME   | HOSTNAME | JOB NO | TASK NO | STATUS |  JOB NAME
----------+----------+----------+--------+---------+--------+-----------
 15:37:56 | 15:37:58 | f00a-001 |   1965 |       0 |      0 | Serial_Job

This shows all the jobs we ran recently (note that there are multiple entries per job). To get info about a specific job, we change command slightly.

[yourUsername@login12 ~]$  jobhist -j 1965

It will show a lot of info, in fact, every single piece of info collected on your job by the scheduler. It may be useful to redirect this information to less to make it easier to view (use the left and right arrow keys to scroll through fields).

[yourUsername@login12 ~]$  jobhist -j

Discussion

This view can help compare the amount of time requested and actually used, duration of residence in the queue before launching, and memory footprint on the compute node(s).

How accurate were our estimates?

Improving Resource Requests

From the job history, we see that pi.py jobs finished executing in at most a few minutes, once dispatched. The time estimate we provided in the job script was far too long! This makes it harder for the queuing system to accurately estimate when resources will become free for other jobs. Practically, this means that the queuing system waits to dispatch our pi.py job until the full requested time slot opens, instead of “sneaking it in” a much shorter window where the job could actually finish. Specifying the expected runtime in the submission script more accurately will help alleviate cluster congestion and may get your job dispatched earlier.

Narrow the Time Estimate

Edit parallel_pi.sh to set a better time estimate. How close can you get?

Hint: use -l h_rt=.

Solution

The following line tells SGE that our job should finish within 2 minutes:

#$ -l h_rt==00:02:00

Key Points

  • Accurate job scripts help the queuing system efficiently allocate shared resources.

Using shared resources responsibly

Overview

Teaching: 15 min
Exercises: 5 min
Questions

  • How can I be a responsible user?

  • How can I protect my data?

  • How can I best get large amounts of data off an HPC system?

Objectives

  • Describe how the actions of a single user can affect the experience of others on a shared system.

  • Discuss the behaviour of a considerate shared system citizen.

  • Explain the importance of backing up critical data.

  • Describe the challenges with transferring large amounts of data off HPC systems.

  • Convert many files to a single archive file using tar.

One of the major differences between using remote HPC resources and your own system (e.g. your laptop) is that remote resources are shared. How many users the resource is shared between at any one time varies from system to system but it is unlikely you will ever be the only user logged into or using such a system.

The widespread usage of scheduling systems where users submit jobs on HPC resources is a natural outcome of the shared nature of these resources. There are other things you, as an upstanding member of the community, need to consider.

Be Kind to the Login Nodes

The login node is often busy managing all of the logged in users, creating and editing files and compiling software. If the machine runs out of memory or processing capacity, it will become very slow and unusable for everyone. While the machine is meant to be used, be sure to do so responsibly – in ways that will not adversely impact other users’ experience.

Login nodes are always the right place to launch jobs. Cluster policies vary, but they may also be used for proving out workflows, and in some cases, may host advanced cluster-specific debugging or development tools. The cluster may have modules that need to be loaded, possibly in a certain order, and paths or library versions that differ from your laptop, and doing an interactive test run on the head node is a quick and reliable way to discover and fix these issues.

Login Nodes Are a Shared Resource

Remember, the login node is shared with all other users and your actions could cause issues for other people. Think carefully about the potential implications of issuing commands that may use large amounts of resource.

Unsure? Ask your friendly systems administrator (“sysadmin”) if the thing you’re contemplating is suitable for the login node, or if there’s another mechanism to get it done safely.

You can always use the commands top and ps ux to list the processes that are running on the login node along with the amount of CPU and memory they are using. If this check reveals that the login node is somewhat idle, you can safely use it for your non-routine processing task. If something goes wrong – the process takes too long, or doesn’t respond – you can use the kill command along with the PID to terminate the process.

Login Node Etiquette

Which of these commands would be a routine task to run on the login node?

  1. python physics_sim.py
  2. make
  3. create_directories.sh
  4. molecular_dynamics_2
  5. tar -xzf R-3.3.0.tar.gz

Solution

Building software, creating directories, and unpacking software are common and acceptable > tasks for the login node: options #2 (make), #3 (mkdir), and #5 (tar) are probably OK. Note that script names do not always reflect their contents: before launching #3, please less create_directories.sh and make sure it’s not a Trojan horse.

Running resource-intensive applications is frowned upon. Unless you are sure it will not affect other users, do not run jobs like #1 (python) or #4 (custom MD code). If you’re unsure, ask your friendly sysadmin for advice.

If you experience performance issues with a login node you should report it to the system staff (usually via the helpdesk) for them to investigate.

Test Before Scaling

Remember that you are generally charged for usage on shared systems. A simple mistake in a job script can end up costing a large amount of resource budget. Imagine a job script with a mistake that makes it sit doing nothing for 24 hours on 1000 cores or one where you have requested 2000 cores by mistake and only use 100 of them! This problem can be compounded when people write scripts that automate job submission (for example, when running the same calculation or analysis over lots of different parameters or files). When this happens it hurts both you (as you waste lots of charged resource) and other users (who are blocked from accessing the idle compute nodes). On very busy resources you may wait many days in a queue for your job to fail within 10 seconds of starting due to a trivial typo in the job script. This is extremely frustrating!

Most systems provide dedicated resources for testing that have short wait times to help you avoid this issue.

Test Job Submission Scripts That Use Large Amounts of Resources

Before submitting a large run of jobs, submit one as a test first to make sure everything works as expected.

Before submitting a very large or very long job submit a short truncated test to ensure that the job starts as expected.

Have a Backup Plan

Although many HPC systems keep backups, it does not always cover all the file systems available and may only be for disaster recovery purposes (i.e. for restoring the whole file system if lost rather than an individual file or directory you have deleted by mistake). Protecting critical data from corruption or deletion is primarily your responsibility: keep your own backup copies.

Version control systems (such as Git) often have free, cloud-based offerings (e.g., GitHub and GitLab) that are generally used for storing source code. Even if you are not writing your own programs, these can be very useful for storing job scripts, analysis scripts and small input files.

If you are building software, you may have a large amount of source code that you compile to build your executable. Since this data can generally be recovered by re-downloading the code, or re-running the checkout operation from the source code repository, this data is also less critical to protect.

For larger amounts of data, especially important results from your runs, which may be irreplaceable, you should make sure you have a robust system in place for taking copies of data off the HPC system wherever possible to backed-up storage. Tools such as rsync can be very useful for this.

Your access to the shared HPC system will generally be time-limited so you should ensure you have a plan for transferring your data off the system before your access finishes. The time required to transfer large amounts of data should not be underestimated and you should ensure you have planned for this early enough (ideally, before you even start using the system for your research).

In all these cases, the helpdesk of the system you are using should be able to provide useful guidance on your options for data transfer for the volumes of data you will be using.

Your Data Is Your Responsibility

Make sure you understand what the backup policy is on the file systems on the system you are using and what implications this has for your work if you lose your data on the system. Plan your backups of critical data and how you will transfer data off the system throughout the project.

Transferring Data

As mentioned above, many users run into the challenge of transferring large amounts of data off HPC systems at some point (this is more often in transferring data off than onto systems but the advice below applies in either case). Data transfer speed may be limited by many different factors so the best data transfer mechanism to use depends on the type of data being transferred and where the data is going.

The components between your data’s source and destination have varying levels of performance, and in particular, may have different capabilities with respect to bandwidth and latency.

Bandwidth is generally the raw amount of data per unit time a device is capable of transmitting or receiving. It’s a common and generally well-understood metric.

Latency is a bit more subtle. For data transfers, it may be thought of as the amount of time it takes to get data out of storage and into a transmittable form. Latency issues are the reason it’s advisable to execute data transfers by moving a small number of large files, rather than the converse.

Some of the key components and their associated issues are:

As mentioned above, if you have related data that consists of a large number of small files it is strongly recommended to pack the files into a larger archive file for long term storage and transfer. A single large file makes more efficient use of the file system and is easier to move, copy and transfer because significantly fewer metadata operations are required. Archive files can be created using tools like tar and zip. We have already met tar when we talked about data transfer earlier.

/hpc-intro/Schematic%20of%20network%20bandwidth
Schematic diagram of bandwidth and latency for disk and network I/O. Each of the components on the figure is connected by a blue line of width proportional to the interface bandwidth. The small mazes at the link points illustrate the latency of the link, with more tortuous mazes indicating higher latency.

Consider the Best Way to Transfer Data

If you are transferring large amounts of data you will need to think about what may affect your transfer performance. It is always useful to run some tests that you can use to extrapolate how long it will take to transfer your data.

Say you have a “data” folder containing 10,000 or so files, a healthy mix of small and large ASCII and binary data. Which of the following would be the best way to transfer them to Myriad?

  1. [user@laptop ~]$ scp -r data [email protected]:~/

  2. [user@laptop ~]$ rsync -ra data [email protected]:~/

  3. [user@laptop ~]$ rsync -raz data [email protected]:~/

  4. [user@laptop ~]$ tar -cvf data.tar data
[user@laptop ~]$ rsync -raz data.tar [email protected]:~/

  5. [user@laptop ~]$ tar -cvzf data.tar.gz data
[user@laptop ~]$ rsync -ra data.tar.gz [email protected]:~/

Solution

  1. scp will recursively copy the directory. This works, but without compression.
  2. rsync -ra works like scp -r, but preserves file information like creation times. This is marginally better.
  3. rsync -raz adds compression, which will save some bandwidth. If you have a strong CPU at both ends of the line, and you’re on a slow network, this is a good choice.
  4. This command first uses tar to merge everything into a single file, then rsync -z to transfer it with compression. With this large number of files, metadata overhead can hamper your transfer, so this is a good idea.
  5. This command uses tar -z to compress the archive, then rsync to transfer it. This may perform similarly to #4, but in most cases (for large datasets), it’s the best combination of high throughput and low latency (making the most of your time and network connection).

Key Points

  • Be careful how you use the login node.

  • Your data on the system is your responsibility.

  • Plan and test large data transfers.

  • It is often best to convert many files to a single archive file before transferring.