Introduction to SLURM


All DKUCC partitions are IT-owned machines.

					sbatch –p (partition name) --account=(account name)

(in a job script) or

					srun –p (partition name) --account=(account name) --pty bash –i

(interactively) In general, the partition name and account name will be the same for most lab-owned machines.

Common DKUCC Partitions

There are four different DKUCC partitions to which batch jobs and interactive sessions can be directed by all DKUCC users:

  • common for jobs that will run on the DKUCC core nodes (up to 64 GB RAM).
  • common-gpu for jobs that will run on DCKUC GPU nodes.

Note: These partitions do not require an “–account=” flag. (i.e., they accept jobs from any account.) If a partition is not specified, the default partition is the common partition.

Running an interactive job

Reserve a compute node by typing srun –pty bash -i

					tm103@dkucc-login-01 ~ $ srun --pty bash -i   
srun: job 186535 queued and waiting for resources   
srun: job 186535 has been allocated resources   
tm103@dkucc-core-05 ~ $
tm103@dkucc-core-05 ~ $ squeue -u tm103  
186535 common bash tm103 R 0:14 1 dkucc-core-05

I now have an interactive session in the common partition on node dkucc-core-05

SLURM commands

sbatch – Submit a batch job

#SBATCH – Specify job parameters in a job script

squeue – Show lists of jobs

scancel – Delete one or more batch jobs

sinfo – Show info about machines

scontrol – Show cluster configuration information

Use sbatch (all lower case) to submit text file job scripts, e.g.


Use #SBATCH (upper case) in your scripts for scheduler directives, e.g.

					#SBATCH --mem=1G
#SBATCH --output=matlab.out

All SLURM directives can be given on the command line instead of the script.

Slurm memory directives

The default memory request (allocation) is 2 GB RAM. This is a hard limit, always request a little more. To request a total amount of memory for the job, use one of the following: * --mem=<MB> additional memory * --mem=<Gigabyte>G the amount of memory required per node, or * --mem-per-cpu=<MB> the amount of memory per CPU core, for multi-threaded jobs

Note: –mem and –mem-per-cpu are mutually exclusive

Slurm parallel directives

All parallel directives have defaults of 1

-N <number> How many nodes (machines)

-n <number> or --ntasks=<number> How many parallel jobs (“tasks”)

-c <number> or --cpus-per-task=<number>

Use -n and -N for multi-node jobs (e.g. MPI)

Use -c (–cpus-per-task) for multi-threaded jobs

Job script examples

#SBATCH ---output=test.out
hostname # print hostname

This prints the name of the compute node in the file “test.out”

					tm103@dkucc-login-01 ~/slurm $ sbatch
Submitted batch job 186554tm103@dkucc-login-01 ~/slurm $ cat test.out 

Long-form commands example

					#SBATCH --output=slurm.out
#SBATCH --error=slurm.err  
#SBATCH --mem=100 # 100 MB RAM 
#SBATCH --partition=scavenger# 
hostname 1>&2  #prints hostname to the error file

Short-form commands example.

SLURM short commands don’t use “=“ signs

#SBATCH -o slurm.out
#SBATCH -e slurm.err
#SBATCH --mem=4G # 4 GBs RAM 
#SBATCH –p scavenger
hostname 1>&2  #prints hostname to the error file

Multi-threaded (multi-core) example

#SBATCH –J test 
#SBATCH –o test.out
#SBATCH –c 4
#SBATCH –-mem-per-cpu=500 #(500 MB) 
myApplication –n $SLURM_CPUS_PER_TASK

The value of $SLURM_CPUS_PER_TASK is the number after -c. This example starts a single, multi-threaded job that uses 4 CPU cores and 2 GB (4x500MB) of RAM

OpenMPI multicore example

#SBATCH –J openmp-test 
#SBATCH –o slurm.out
#SBATCH –c 4
myOpenMPapp # will run on 4 CPU cores

This sets $OMP_NUM_THREADS to the value of $SLURM_CPUS_PER_TASK

Slurm job arrays

Slurm job arrays are a mechanism for submitting and managing collections of similar jobs using one job script and one application program.

  • Add --array or -a option to the job script
  • Each job task will inherit a SLURM_ARRAY_TASK_ID environment variable with a different integer value
  • Each job array can be up 100,000 job tasks on the DKUCC
  • Job arrays are only supported for batch jobs
  • Job array “tasks” must be independent:

For example, in a job script, add the line #SBATCH --array=1-30 or, alternatively, #SBATCH -a 1-30 to submit 30 job tasks. The job array indices can also be specified on the command line, e.g.

					sbatch -a 1-30

The index values can be continuous, e.g.

					-a 0-31 (32 tasks, numbered from 0,1,2,…,31)

or discontinuous, e.g.

					-a 3,5,7-9,12 (6 tasks, numbers 3,5,7,8,9,12)

It can also be a single job task, e.g.

					-a 7

The discontinous notation is useful for resubmitting specific job tasks that had previously failed. Each job task is assigned the enviromental variable $SLURM_ARRAY_TASK_ID set to it’s index value.

					tm103@dcc-login-01  ~/misc/jobarrays $ cat 
tm103@dcc-login-01  ~/misc/jobarrays $ sbatch -a 1-3 
Submitted batch job 24845830
tm103@dcc-login-01  ~/misc/jobarrays $ ls slurm-24845830*
slurm-24845830_1.out  slurm-24845830_2.out  slurm-24845830_3.out
tm103@dcc-login-01  ~/misc/jobarrays $ cat slurm-24845830*
tm103@dcc-login-01  ~/misc/jobarrays $

Importing Environmental Variables




					numCPUs = str2num(getenv('SLURM_CPUS_PER_TASK'))
taskID = str2num(getenv('SLURM_ARRAY_TASK_ID'))

Processing separate input files

Process an existing file list, e.g. files.txt

readarray -t FILES < files.txt

Dynamically generate a file list from “ls”

export FILES=($(ls -1 myfile*))

Example: Using the taskID as part of the file name and output directory for the case with input file names of the form input1,input2,…,inputN for -a 1-N, e.g.

#SBATCH -e slurm_%A_%a.err
#SBATCH -o slurm_%A_%a.out      
mkdir out_${SLURM_ARRAY_TASK_ID}     
myapp ../input_${SLURM_ARRAY_TASK_ID}.txt

where output directories out1, out2, … are created for input files input1.txt, input2.txt,…

“Unrolling” for loops example

Original “serial” code (Python)

					fibonacci = [0,1,1,2,3,5,8,13,21]
for i in range(len(fibonacci)):

Job array version

					import os
fibonacci = [0,1,1,2,3,5,8,13,21]
#for i in range(len(fibonacci)):

where the for loop is commented-out and each job task is doing a single “iteration”

					tm103@dcc-login-01  ~/misc/jobarrays $ cat 
#SBATCH -e slurm.err
module load Python/2.7.11
tm103@dcc-login-01  ~/misc/jobarrays $ sbatch –a 1-8 
Submitted batch job 24856052
tm103@dcc-login-01  ~/misc/jobarrays $ ls slurm-24856052_*
slurm-24856052_1.out  slurm-24856052_3.out  slurm-24856052_5.out  slurm-24856052_7.out
slurm-24856052_2.out  slurm-24856052_4.out  slurm-24856052_6.out  slurm-24856052_8.out
tm103@dcc-login-01  ~/misc/jobarrays $ cat slurm-24856052*
(1, 1)
(2, 1)
(3, 2)
(4, 3)
(5, 5)
(6, 8)
(7, 13)
(8, 21)
tm103@dcc-login-01  ~/misc/jobarrays $

Running MPI jobs

OpenMPI job script

#SBATCH -o openhello.out
#SBATCH -e slurm.err
#SBATCH -n 20
module load OpenMPI/4.0.5-rhel8
mpirun -n $SLURM_NTASKS openhello

OpenMPI example output

					tm103@dkucc-login-01 ~/misc/slurm/openmpi $ cat openhello.out  
dcc-core-01, rank 0 out of 20 processors  
dcc-core-01, rank 1 out of 20 processors  
dcc-core-01, rank 2 out of 20 processors  
dcc-core-01, rank 3 out of 20 processors  
dcc-core-01, rank 4 out of 20 processors   
dcc-core-03, rank 13 out of 20 processors 
dcc-core-03, rank 14 out of 20 processors 
dcc-core-03, rank 10 out of 20 processors 
dcc-core-03, rank 11 out of 20 processors 
dcc-core-03, rank 12 out of 20 processors 
dcc-core-02, rank 8 out of 20 processors 
dcc-core-02, rank 9 out of 20 processors 
dcc-core-02, rank 5 out of 20 processors

GPU jobs

To run a GPU batch job, add the job script lines

					#SBATCH -p gpu-common 
#SBATCH --gres=gpu:1 
#SBATCH --exclusive

To get an interactive GPU node session, type the command line

srun -p gpu-common --gres=gpu:1 --pty bash -i

tm103@dcc-clogin-02 ~ $ srun -p gpu-common –gres=gpu:1 –pty bash -i

tm103@dcc-gpu-01 ~ $ /usr/local/cuda-7.5/samples/1_Utilities/deviceQuery/deviceQuery … Detected 1 CUDA Capable device(s)

					Device 0: "Tesla A40"
    CUDA Driver Version / Runtime Version          7.5 / 7.5
    CUDA Capability Major/Minor version number:    3.7
    Total amount of global memory:                 11520 MBytes (12079136768 bytes)
    (13) Multiprocessors, (192) CUDA Cores/MP:     2496 CUDA Cores

Request a specific GPU type and GPU device number with

					#SBATCH --gres=gpu:(GPU type):(GPU number)

where “GPU number” was a number from 1 to 4 and “GPU type” was from the list below.


Job dependencies

Submit a job that waits for another job to finish.

					$ sbatch dep1.q
Submitted batch job 666898

Make a note of the assigned job ID of dep1

					$ sbatch --dependency=afterok:666898 dep2.q
Job dep2 will not start until dep1 finishes

Job dependencies with arrays

Wait for specific job array elements

					sbatch --depend=after:123_4 my.job sbatch --depend=afterok:123_4:123_8 my.job2

Wait for entire job array to complete

					sbatch --depend=afterany:123 my.job

Wait for entire job array to complete successfully

					sbatch --depend=afterok:123 my.job

Wait for entire job array to complete and at least one task fails

					sbatch --depend=afternotok:123 my.job

Additional Resources

Table of Contents