Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
en:services:application_services:high_performance_computing:running_jobs_slurm:job_arrays [2020/11/13 09:04] – [Job Arrays] mbodenen:services:application_services:high_performance_computing:running_jobs_slurm:job_arrays [2020/11/13 11:20] (current) – [Job Arrays] mboden
Line 1: Line 1:
 +====== Job Arrays ======
  
 +Job arrays are the preferred way to submit many similar jobs, for instance, if you need to run the same program on a number of input files, or with different settings or run them with a range of parameters.
 +Arrays are created with the ''-a start-finish'' sbatch parameter. E.g. ''sbatch -a 0-19'' will create 20 jobs indexed from 0 to 19. There are different ways to index the arrays, which are described below.
 +
 +The behavior of the jobs can then be tied to Slurm Environment variables, which tell the program, which part of the array they are.
 +
 +===== Job Array Indexing, Stepsize and more =====
 +Slurm supports a number of ways to set up the indexing in job arrays.
 +  * Range: ''-a 0-5''
 +  * Multiple values: ''-a 1,5,12''
 +  * Step size:  ''-a 0-5:2'' (same as ''-a 0,2,4'')
 +  * Combined: ''-a 0-5:2,20'' (same as ''-a 0,2,4,20'')
 +Additionally, you can limit the number of simultaneously running jobs with the ''%x'' parameter in there:\\
 +  * ''-a 0-11%4'' only four jobs at once
 +  * ''-a 0-11%1'' run all jobs sequentially
 +  * ''-a 0-5:2,20%2'' everything combined. Run IDs 0,2,4,20, but only two at a time.
 +You can read everything on array indexing in the [[https://slurm.schedmd.com/sbatch.html#OPT_array|sbatch man page]].
 +
 +===== Slurm Array Environment Variables =====
 +The most used environment variable in Slurm arrays is ''$SLURM_ARRAY_TASK_ID''. This contains the index of the job in the array and is different in every Job of the array. Other variables are:
 +<code>
 +SLURM_ARRAY_TASK_COUNT
 +    Total number of tasks in a job array. 
 +SLURM_ARRAY_TASK_ID
 +    Job array ID (index) number. 
 +SLURM_ARRAY_TASK_MAX
 +    Job array's maximum ID (index) number. 
 +SLURM_ARRAY_TASK_MIN
 +    Job array's minimum ID (index) number. 
 +SLURM_ARRAY_TASK_STEP
 +    Job array's index step size. 
 +SLURM_ARRAY_JOB_ID
 +    Job array's master job ID number. 
 +</code>
 +
 +===== Example Job Array =====
 +This is an example of a job array, creates a job for every file ending in ".inp" in the current workding directory:
 +
 +<code bash>
 +#!/bin/bash
 +#SBATCH -p medium
 +#SBATCH -t 01:00
 +# insert X as the number of .inp files you have -1 (since bash arrays start counting from 0)
 +# ls *.inp | wc -l
 +#SBATCH -a 0-X
 +
 +#for safety reasons
 +shopt -s nullglob
 +#create a bash array with all files
 +arr=(./*.inp)
 +
 +#put your command here. This just runs the fictional "big_computation" program with one of the files as input
 +./big_computation ${arr[$SLURM_ARRAY_TASK_ID]}
 +</code>
 +
 +In this case, you have to get the number of files beforehand (fill in the X). You can also automatically do that by removing the ''#SBATCH -a'' line and adding that information when submitting the job:
 +<code>sbatch -a 0-$(($(ls ./*.inp | wc -l)-1)) jobarray.sh</code>
 +The part in the parenthesis just uses ''ls'' to output all .inp files, counts them with ''wc'' and then subtracts 1, since bash arrays start counting at 0.