Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
en:services:application_services:high_performance_computing:running_jobs_slurm:job_arrays [2020/11/13 09:43] – [Job Array Indexing, Stepsize and more] mboden | en:services:application_services:high_performance_computing:running_jobs_slurm:job_arrays [2020/11/13 11:20] (current) – [Job Arrays] mboden | ||
---|---|---|---|
Line 1: | Line 1: | ||
+ | ====== Job Arrays ====== | ||
+ | Job arrays are the preferred way to submit many similar jobs, for instance, if you need to run the same program on a number of input files, or with different settings or run them with a range of parameters. | ||
+ | Arrays are created with the '' | ||
+ | |||
+ | The behavior of the jobs can then be tied to Slurm Environment variables, which tell the program, which part of the array they are. | ||
+ | |||
+ | ===== Job Array Indexing, Stepsize and more ===== | ||
+ | Slurm supports a number of ways to set up the indexing in job arrays. | ||
+ | * Range: '' | ||
+ | * Multiple values: '' | ||
+ | * Step size: '' | ||
+ | * Combined: '' | ||
+ | Additionally, | ||
+ | * '' | ||
+ | * '' | ||
+ | * '' | ||
+ | You can read everything on array indexing in the [[https:// | ||
+ | |||
+ | ===== Slurm Array Environment Variables ===== | ||
+ | The most used environment variable in Slurm arrays is '' | ||
+ | < | ||
+ | SLURM_ARRAY_TASK_COUNT | ||
+ | Total number of tasks in a job array. | ||
+ | SLURM_ARRAY_TASK_ID | ||
+ | Job array ID (index) number. | ||
+ | SLURM_ARRAY_TASK_MAX | ||
+ | Job array' | ||
+ | SLURM_ARRAY_TASK_MIN | ||
+ | Job array' | ||
+ | SLURM_ARRAY_TASK_STEP | ||
+ | Job array' | ||
+ | SLURM_ARRAY_JOB_ID | ||
+ | Job array' | ||
+ | </ | ||
+ | |||
+ | ===== Example Job Array ===== | ||
+ | This is an example of a job array, creates a job for every file ending in " | ||
+ | |||
+ | <code bash> | ||
+ | #!/bin/bash | ||
+ | #SBATCH -p medium | ||
+ | #SBATCH -t 01:00 | ||
+ | # insert X as the number of .inp files you have -1 (since bash arrays start counting from 0) | ||
+ | # ls *.inp | wc -l | ||
+ | #SBATCH -a 0-X | ||
+ | |||
+ | #for safety reasons | ||
+ | shopt -s nullglob | ||
+ | #create a bash array with all files | ||
+ | arr=(./ | ||
+ | |||
+ | #put your command here. This just runs the fictional " | ||
+ | ./ | ||
+ | </ | ||
+ | |||
+ | In this case, you have to get the number of files beforehand (fill in the X). You can also automatically do that by removing the ''# | ||
+ | < | ||
+ | The part in the parenthesis just uses '' |