HPC

Basic Usage

Last Updated: 08 August 2022

What to Expect For This Session?
● At the end of this session, you should at least have the knowledge to:
○ Access the HPC environment
○ Managing files within the HPC environment
○ Identify the components of the HPC environment
○ Understand various type of parameters for job submission
○ Submit and run a calculation in the HPC
○ Check your analysis output from successful job
What is HPC?
Why do you need to use HPC?
● Computation need more memory than available on your computer.
● Same program needs to be run multiple times with different inputs or
parameters.
● Your program can utilise parallelism to achieve speed up in calculation.

● Need access to a powerful GPU accelerator.

What skills are needed to use HPC?
● Basic Linux knowledge:
○ Directory navigation
○ Files editing
○ Commands execution
● Knowledge on the application you wish to use.
● Optional, but good to have:
○ Basic knowledge about computer architecture
○ Shell Scripting knowledge, for job automation
User Accounts
● If you do not have a HPC account yet, please request a new account at
dicc.um.edu.my.
● You will be prompted to change your password the first time you login into the
HPC Login Node.
● Users are responsible for their own account safety. Account sharing are not
allowed.
○ Higher risk of compromised user accounts.
○ Higher chance of disturbing data from another users sharing the same account.
○ Account owner will be held responsible no matter who caused issues on the
account.
HPC Tests and Account Limits
● All users are required to pass the HPC test every 6 months in order to ensure
they have the proper skill set and knowledge to utilise HPC resources
properly.
● Account that did not pass the test will be limited in terms of resource usage.

Limited Account Full Account

4 CPU/Cores 450 CPU/Cores

16 GB Memory 2 TB Memory
activate
1 GPUs Unlimited GPUs

1 hour Walltime 7 Days Walltime

Accessing HPC
VPN Connection
● VPN connection is required before able to connect to
any of the HPC resources.
● Refers to VPN connection page in confluence for more
information.
● Basically, two key items are required:
○ OpenVPN Profile
○ OpenVPN Client
● Once connected, only traffic to any DICC domain will
be going through the VPN tunnel.
Verifying VPN Connection
● Visit https://umhpc.dicc.um.edu.my
for web based HPC portal.
● If the page load, VPN connection
is established successfully.
Failed Login Attempt
● Account login will be blocked for 10 minutes after 6 failed login attempts.
● IP will be blocked for 1 hour if login attempt failed for 5 times within 10 minutes.
Required Softwares
● The following softwares have been tested in our HPC environment.

Operating System SSH File Transfer

Windows OS PuTTY WinSCP
MobaXTerm MobaXTerm

Linux terminal scp command

Mac OS terminal scp command

iTerm2
Login into HPC Cluster
Windows OS Linux & Mac OS

Use PuTTY or MobaXTerm: Use ssh command in your terminal:

● Connection Type: SSH
● Host Name: umhpc.dicc.um.edu.my $ ssh username@umhpc.dicc.um.edu.my
● Port: 22
Login Node (umhpc)
● umhpc is the gateway to the HPC which is shared by all the HPC users.
● umhpc has only limited amount of CPUs and Memory.
● Users will do most of the stuff here:
○ Transferring and managing files
○ Submitting jobs
○ Checking outputs or errors
○ Monitor job resources usage
Things to avoid in Login Node
● Execute CPU or Memory intensive code or scripts
○ Executing CPU or memory intensive code can potentially lead to
performance degradation for all the users using the HPC.
○ Login Node can potentially crash if all users are to run their code in it.
● Compile your program or code
○ Login Node is a virtualized instance, the system architecture will be
different from the compute nodes used to execute the computation.
○ Most applications libraries are only installed in the compute nodes.
HPC Storage
Home Directory
● General Purpose Storage:
○ Personal storage, not high performance!
○ Basic Network File System (NFS)
○ Persistent
○ Nightly data backup to remote site.
○ Quota limit per user: 100GB
○ Total Size: 59TB
○ /home/user/<username>
Lustre Directory
● High Performance Storage:
○ High performance computing storage
○ Lustre Filesystem
○ Not Persistent, no data backup
○ Support group/project directory
○ Files not being accessed for over 60 days will be erased.
○ No quota limit
○ Total Size: 231TB
○ /lustre/user/<username>
Group/Project Directory
● If your group need access to shared group directory, contact the administrator
through service desk for more details.
● Shared group directory can be located under:
○ /lustre/project/<username>
Lustre Storage Cleanup
● Make sure to backup the files you need from /lustre to /home.
● Files located in /lustre directory that have not been accessed for more than
60 days will be erased.
● We will not be able to recover any erased files.
Checking Storage Usage
NFS (/home) Lustre (/lustre)

To check account usage and quota To check usage information:

information:
$ lfs quota -h -u <user>
$ quota -s /lustre

Example,

$ lfs quota -h -u george

/lustre
Transferring Data
Windows OS Linux & Mac OS

Use WinSCP: Use scp command:

● File Protocol: SCP
● Host Name: umhpc.dicc.um.edu.my $ scp [-r] source destination
● Port Number: 22
For example:

● Transfer file to the HPC Storage:

$ scp /path/to/filename
username@umhpc.dicc.um.edu.my:/home/username

● Transfer directory to the HPC Storage:

$ scp -r /path/to/directory
username@umhpc.dicc.um.edu.my:/home/username
/home Vs /lustre
Home Directory Lustre Directory
File System NFS Lustre

Total Capacity 59TB 231TB

Directory /home /lustre

Storage type Persistent Not Persistent

Quota 100GB per user -

Backup Daily -

Run Jobs Yes Yes

Group Directory Not supported Supported

HPC Scheduler
Overview
Key Components
● Partitions (Queues)
○ The place your jobs sit in while waiting to be scheduled.
● Jobs
○ Composed of your calculations or computations.
● Compute Nodes
○ The compute servers to execute your calculations.
● Resources
○ Comprised of CPUs, Memory, GPUs and etc.
● Reservations
○ List of reserved resources.
Cluster Information
● Use cluster-info command to get summary of resources in the cluster:
Job Partitions
● Also commonly known as job queue in other HPC system.
● Partitions are currently configured with similar nodes grouped together.
● The following are the available partitions:
○ cpu-opteron (default)
○ cpu-epyc
○ gpu-k10
○ gpu-titan
○ gpu-k40c
○ gpu-v100s
Job Parameters
● Job parameters are used to define the characteristics and settings of your
jobs.
● Job parameters are normally defined in format of --parameters=value or
-param value in the job submission script.
● For example, there are two ways to define a job name:
--job-name=ExampleJobName
-J ExampleJobName
Commonly Used Parameters
--partition, -p Specify the partition to submit to.

--job-name, -J Specify the name of the job.

--nodes, -N Specify amount of nodes required for the job.

--ntasks, -n Specify amount of cpus/cores required for the job.

--output, -o Specify the stdout filename.

--error, -e Specify the stderr filename.

--mem Specify memory required for each node.

--mail-type Specify which notification to receive from the job.

--mail-user Specify the email address to send notification to.

--qos, -q Specify QoS for the job.

--gpus, -G Specify amount of GPU resources needed.

GPU Resources
● To specify the amount of GPUs to be allocated:
--gpus=[gpu_name:]gpu_count
● For example:
--gpus=8
--gpus=k10:8
Email Notification
● The scheduler can send notifications to your email when the job start, finish,
failed or requeued.
--mail-type=ALL
--mail-user=george@um.edu.my
Job Queues
Listing Jobs in the Queue
● All jobs in the queue are visible to any users.
● Use squeue command to retrieve a list of all your jobs currently running or
queueing in the cluster.
Example squeue output
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
47684 cpu-epyc gaussian ystan PD 0:00 1 (Resources)
47688 cpu-epyc gaussian ystan PD 0:00 1 (Priority)
47664 cpu-epyc 3Mecm3TL keathouy PD 0:00 1 (Priority)
47671 cpu-epyc 3Mec4Ni keathouy PD 0:00 1 (Priority)
46918 cpu-epyc 3Mecm3TH keathouy R 2-11:24:19 1 cpu12
47668 cpu-epyc md han R 1-10:40:50 1 cpu13
46877 cpu-epyc 3Mec3TH keathouy R 9-20:52:21 1 cpu15
47665 cpu-epyc 3Mecm3Ni keathouy R 1-23:16:59 1 cpu14
47517 cpu-opter al vanna PD 0:00 1 (Resources)
47674 cpu-opter analysus raziltah PD 0:00 1 (Priority)
47675 cpu-opter analysus raziltah PD 0:00 1 (Priority)
47676 cpu-opter analysus raziltah PD 0:00 1 (Priority)
47677 cpu-opter analysus raziltah PD 0:00 1 (Priority)
46934 cpu-opter 3Mec3TL keathouy R 4-07:45:37 1 cpu07
47642 cpu-opter rid_bri thongmay R 1-15:25:57 1 cpu03
47683 cpu-opter gaussian swong R 1:02:50 1 cpu03
Show only Own Jobs with squeue
$ squeue -u george
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1782 intel 50tra george PD 0:00 1 (Resources)
1781 intel 49ixt george PD 0:00 1 (Resources)
1780 intel 30rea george R 1-07:11:48 1 cpu01
1778 amd 092exots george R 1-15:57:37 1 cpu02
1776 intel 032exots george R 1-18:58:28 2 cpu01,cpu03
Common Job State
Job State Description
PD / PENDING Pending for resource scheduling.

R / RUNNING The job is currently running.

RQ / REQUEUED The job has been requeued.

CG / COMPLETING The job has done execute, and is now completing itself.

S / SUSPENDED The job has been suspended.

Job Priority
Job Priority System
● All jobs have their own priority value.
● SLURM scheduler will always attempt to schedule jobs with higher priority
value.
● There are THREE factors that affect jobs priority:
○ Job Age - The amount of time the job spent in queue.
○ Quality of Service (QoS) - The type of QoS selected.
○ Fairshare - The amount of resources used in the past 14 days.
Priority Weightage
● The priority of any given job at any particular time is equal to:
Job Age Priority + QoS Priority + Fairshare Priority
● Each of the priority has a different weightage in calculation:

Priority Weightage

Job Age 30,000

QoS 10,000

Fairshare 50,000
Factor 1: Job Age
● Newly submitted job are less likely to run first compared to the jobs that have
queued for a very long time.
● Job Age Priority increases as the job spends more time in queue.
● Job Age Priority will reach its maximum value when the job has queue for ONE
month.
● The longer the job wait in queue, the higher the priority.
Factor 2: Quality of Service (QoS)
● Each QoS has its own defined priority value.
● Jobs should be submitted with appropriate QoS based on amount of time
needed.
● Longer jobs might clash with system maintenance schedule. Jobs could be
terminated when system maintenance start.
● Longer jobs are more prone to random hardware or software failure.
Type of Quality of Service
QoS Maximum Walltime Priority Boost
short 1 hour +10,000

normal * 1 day 0

long 7 days 0

* normal is the default QoS if no QoS is defined or selected.

Factor 3: Fairshare
● Currently, all users are assigned an equal amount of ‘raw share’.
● Fairshare Priority is determined based on the the past 14 days usage.
● The more resources you used in the past 14 days, the lower your fairshare
priority.
● Use sshare command to check your fairshare and usage detail.
Example sshare Output
$ sshare -la

Account User RawShares NormShares RawUsage EffectvUsage LevelFS

-------------------- ---------- ---------- ----------- ----------- ------------- -----------
free ben 1 0.010870 0 0.000000 1.2837e+07
free chris 1 0.010870 27829526 0.036179 0.300453
free george 1 0.010870 0 0.000000 inf
free james 1 0.010870 1276977 0.001660 6.547847
free john 1 0.010870 809 0.000001 1.0327e+04
free may 1 0.010870 162827104 0.211681 0.051347

Normalized Overall Effective

Shares Usage FairShares
sprio for Pending Jobs’ Priority
$ sprio
JOBID PARTITION PRIORITY SITE AGE FAIRSHARE QOS
49723 cpu-epyc 7314 0 4974 2340 0
49729 cpu-epyc 10048 0 7708 2340 0
49730 cpu-epyc 10048 0 7708 2340 0
49731 cpu-epyc 10048 0 7708 2340 0
49732 cpu-epyc 10048 0 7708 2340 0
49733 cpu-epyc 10048 0 7708 2340 0
49734 cpu-epyc 10048 0 7708 2340 0
49786 cpu-epyc 9278 0 6938 2340 0
50050 gpu-v100s 2242 0 1817 426 0
50053 gpu-v100s 2219 0 1794 426 0
50121 cpu-epyc 1224 0 374 851 0
Application & Modules
Application & Libraries
● All applications and libraries are installed as modules in each compute nodes.
● Use node-modules command to show all available modules in all partitions.
● The following commands can only be used while you are in compute nodes:
○ Use module avail command to list available modules in that particular node.
○ Use module load command to load the module you want to use.
○ Use module unload command to unload specific loaded modules.
○ Use module purge command to unload all loaded modules.
Missing Application or Modules?
● Personal usage: Install in user /home directory.
● Group usage: Request administrator to install in /app directory.
● Only HPC Administrator has the privileges to install or update system libraries.
● DICC does not provide direct license support or hosting for any application.
Jobs Submission
Jobs Submission Model
Batch Mode Interactive Mode
Use submission script to execute codes or scripts. Enter the nodes to execute codes or scripts.

Jobs remain even if connection lost, unless

Job get terminated if connection lost, or on exit.
cancelled by owner.

Cannot enter input halfway during the execution. Can enter input interactively during the execution.

salloc to allocate resources

sbatch command to submit job
srun to join allocated resources and run calculation

Execute until maximum allowed walltime.

Must go through queue for resources allocation.

When to use Batch Mode
● You should use Batch Mode to run your jobs when:
○ You do not need to input commands during your application execution.
○ The application will take a long time to complete execution.
○ The application need to complete a series of calculations one after another.
○ You experienced frequent disconnection due to unstable network connection from
your location.
● This is the recommended way of running calculations in the HPC.
Batch Mode Requirements
● You will need to prepare a job submission script, which usually comprised of
the following:
○ Various job parameters and settings
○ Application modules to be loaded
○ Commands to execute
○ Inputs for the calculation
○ Outputs for the calculation
Sample Job Submission Script
#!/bin/bash -l

#SBATCH --partition=cpu-opteron
#SBATCH --job-name=first_job
#SBATCH --output=%x.out
#SBATCH --error=%x.err
#SBATCH --nodes=1
#SBATCH --mem=2G
#SBATCH --ntasks=16
#SBATCH --qos=normal
#SBATCH --mail-type=ALL
#SBATCH --mail-user=george@um.edu.my

module load myModule

app -inp input.file -out output.file
Submit a Batch Job to the Cluster
● sbatch with external parameters
$ sbatch --partition=cpu-standard --ntasks=16 --nodes=1
--output=%x.out --error=%x.err --mem=64G --qos=normal
submission_script.sh

OR
● (Recommended) sbatch with parameters within submission script
$ sbatch submission_script.sh
Delete / Cancel Submitted Job
● Use scancel to cancel and remove a batch job from the queue.
$ scancel <job_id>
● Cancelled job cannot be recovered!
When to use Interactive Mode
● You should use Interactive Mode to run your jobs when:
○ You need to input commands during the application execution.
○ You are trying to compile your own application.
○ You need to analyse real time terminal output.
○ You are trying to debug or troubleshoot your calculation.
Starting Interactive Session
● Standard CPU Jobs
$ salloc -n4 --mem=16G
$ srun --jobid=1234 --pty bash -l

● Standard GPU Jobs

$ salloc -p gpu-k10 -n4 --mem=16G --gpus=1
$ srun --jobid=1234 --pty bash -l
Exiting Interactive Session
● Type exit twice in the terminal to leave an active Interactive Session and
relinquish allocated resources.
# First exit to leave an active interactive session
[george@cpu01 ~]$ exit
logout

# Second exit to relinquish allocated resources

[george@umhpc ~]$ exit
exit
salloc: Relinquishing job allocation 1234
[george@umhpc ~]$
Job Submission
Checklist
What to do before submitting a Job?
● Determine compatibility of application
○ What application do you want to run?
○ Can it be supported under Linux, specifically CentOS?
● Determine type of job to run
○ Do you need GUI for your application?
○ Do you interactive prompt during the calculation?
i. If yes, can it be converted to non-interactive task?
What to do before submitting a Job? (2)
● Determine type and amount of resources needed
○ How much CPUs are needed?
○ How much Memory are needed?
○ Do you need GPU to run your application?
○ How much time is needed?
○ How many nodes are needed?
● Identify the partition to submit to
○ Which partition should you submit your jobs to?
What to do before submitting a Job? (3)
● Determine appropriate Quality of Service (QoS)
○ Determine QoS based on time needed or amount of resources needed.
● Determine the required modules
○ Is the required modules available?
i. If no, can you install it by yourself?
● Prepare necessary input files
● Draft submission script or salloc command
○ Include all related and needed parameters
Job Execution
How much Resources are good?
● Different jobs has different requirements on the resources.
● Users should always understand how much their calculations can consume.
● Start small, then scale bigger.
Resource Settings for Starter
CPU Memory QoS Usage

4 16G short Troubleshooting

16 50G normal Standard quick job

16 50G long Standard small job

32 100G long Standard medium job

64 for OPTERON
200G long Standard large job
48 for EPYC
Monitor Job Resources
● Login to DICC OnDemand Portal at https://umhpc.dicc.um.edu.my.
● Go to Jobs > Active Jobs
● 50% CPU Usage is max for EPYC, 100% CPU Usage for Opteron
Resources Allocation & Usage
● Jobs can only access resources allocated to the them.
● Jobs should be stopped if they are no longer needed.
● Allocated resources for jobs that are idle for long period of time will be
released. These idle jobs will be hold and prevented from scheduled.
● Jobs exceeding allocated memory will be TERMINATED automatically by the
system cgroup with Out-of-Memory (OOM) error.
● Jobs that generate more CPU loads than allocated CPU cores will be
TERMINATED by the administrator to avoid service degradation. Warning will
be sent to the user that submitted the problematic jobs.
Checking Output & Error Log
● Output and error logs contain useful information to help determine if your
calculation is successful or not.
● Output and error logs can normally be found under the same directory where
you run sbatch command to submit the submission script.
● The filename for the output and error log can be specified with the following
parameters:
--error=error.log
--output=output.log
Account History
Checking Account History
● Use sacct command to view your past job history.
$ sacct --starttime=2022-01-01 --endtime=2022-01-10
● Additional optional parameters:
○ -l, --long
■ full detail that contain every column of the job history data
○ -p, --parsable
■ a parsable record of the history data
Sample Account History
$ sacct --startime=2010-01-01 --endtime=2010-01-10

JobID JobName Partition Account AllocCPUS State ExitCode

------------ ---------- ---------- ---------- ---------- ---------- --------
94 qe_phonon cpu-stand+ george 32 COMPLETED 0:0
94.batch batch george 4 COMPLETED 0:0
94.extern extern george 32 COMPLETED 0:0
95 qiime2 cpu-stand+ george 16 COMPLETED 0:0
95.batch batch george 4 COMPLETED 0:0
95.extern extern george 16 COMPLETED 0:0
96 mpi_test cpu-stand+ george 64 COMPLETED 0:0
96.batch batch george 4 COMPLETED 0:0
96.extern extern george 64 COMPLETED 0:0
Any Question?
Links in DICC Website
Documentation & Guide https://confluence.dicc.um.edu.my
DICC Service Desk https://jira.dicc.um.edu.my/servicedesk
Thank you