Simple Linux Utility for Resource Management (SLURM)

SLURM is the native scheduler software that runs on COARE’s HPC cluster. Users request for allocation of compute resources through SLURM. It arbitrates contention for resources by managing a queue of pending work.

SLURM Entities

SLURM entities are relevant terminologies used in SLURM, which include the following:

• Frontend

  The frontend is where users log in to access the HPC.  This should not be used for computing.

• Job

  An allocation of resources assigned to a user for a specified amount of time.

• Jobs Step

  Sets of (possibly parallel) tasks within a job.

• Message Passing Interface (MPI)

  A standardized and portable message-passing system designed to exchange information between processes running on a different nodes.

• Modules

  Environment modules enable users to choose the software that they want to use and add these to their environment.

• Node

  A physical, stand-alone computer that handles computing tasks and run jobs; a compute resource managed by SLURM

• Partitions

  Logical set of nodes with the same queue parameters (job size limit, job time limit, users permitted to use it, etc.)

• Quality-of-Service (QOS)

  The set of rules and limitations that apply to a category of job.

• Runtime

  The time required for a job to finish its execution.

• Task

  This is typically used to schedule an MPI process which, in turn, can use several CPUs.

Types of Jobs

The following are the types of jobs that users can run in the HPC:

• Multi-node parallel jobs

  Multi-node parallel jobs use more than one node and require message passing interface (MPI) to communicate between nodes. Jobs usually require computing resource (cores) more than a single node can offer. 

• Single-node parallel jobs

  Single-node parallel jobs use only one node, but multiple cores on that node. These include pthreads, OpenMP, and shared memory MPI.

• Truly-serial jobs

  Truly-serial jobs require only one core on one node.

• Array jobs

  Multiple jobs to be executed with identical parameters.

SLURM Partitions

The COARE’s SLURM currently has four (4) partitions: debug, batch, serial, and GPU.

SLURM Job Limits

SLURM Job Limits are imposed for fair usage of the COARE's resources. Every job submitted by users to SLURM is subject to these job limits to prevent hogging of resources.

The COARE’s policy on SLURM Job Limits is as follows:

• Users can request up to 168 hours (1 week, 7 days) for a single job.
• Users can request up to 240 CPU cores (this can be just one job or allocated to multiple jobs).
• Users can have a total of 30 simultaneous running jobs.

Job limits implemented for the COARE's saliksik cluster are summarized in the table below:

Job Script

A job script is a script that has the parameters needed to run the specific job of the user. Users should specify the requirements of the job before submitting it into the scheduler.

SLURM Parameters

The table below lists some common parameters in a job script:

NOTES: 
- A Job Script must begin with the #!/bin/bash directive on the first line. The subsequent lines begin with the SLURM directive #SBATCH followed by a parameter. 
- For more information on SBATCH parameters, please visit this link.

QOS Parameter

Each SLURM partition has its own QOS:

Job Script Example

Job Management

Users are able to manage their jobs by checking the status of nodes, submitting the job script to the queue, checking the job’s status, or cancelling a job. 

The following will be helpful in managing your jobs:

• Node Status
• Job Submission
• Job Status
• Job Cancellation

Node Status

sinfo - view information about SLURM nodes and partitions

Job Submission

sbatch - submit job script to the queue

Job Status

squeue - view information about jobs located in the SLURM scheduling queue.

scontrol - view SLURM configuration and state

nvidia-smi - to check the “occupancy” or usage of the GPU devices. 

Job Cancellation

scancel - to cancel submitted jobs.

For more information on the best practices in using the COARE HPC's SLURM, click here.