Slurm supports dependencies between jobs, allowing users to define simple workflows and pipelines. To create a depenency, you can use the following sbatch flag:
-d, --dependency=<dependency_list>
where dependency_list specifies which jobs to depend on and what kind of dependency it is. This is an example of a simple dependency:
sbatch --dependency=afterok:11536887
This will submit a job, which can only start once the job with the ID 11536887 finished successfully.
A list of possible dependency types can be found in the following table and in the sbatch man-page.
| type | Job runs after… |
|---|---|
| after:job_id | job starts or is cancelled |
| afterany:job_id | Terminated in any way |
| afternotok:job_id | Terminated non-zero |
| afterok:job_id | finished normally |
| aftercorr:job_id | After corresponding task in array |
| singleton | Only one of a kind may run at a time |
Multiple jobs can be specified by appending with a colon as delimiter, e.g. -d after:123:124:125.
Manually creating complex job dependencies on the command line can be cumbersome and error-prone. It is advised to set up scripts to do this task for you. sbatch has the --parsable flag to make this task easier: When using this flag, only the job ID will be returned and it can be used to capture the job ID into a variable.
Capture sbatch output into variable and use it in the next job:
# submit a first job, extract job id jobid=$(sbatch --parsable job1.sbatch) # submit a second job with dependency: starts only if the previous job terminates successfully) jobid=$(sbatch --parsable --dependency=afterok:$jobid job2.sbatch) # submit a third job with dependency: starts only if the previous job terminates successfully) jobid=$(sbatch --parsable --dependency=afterok:$jobid job3.sbatch)
Submit scripts, each with a with dependency on the previous one:
#!/bin/bash ID=$(sbatch --parsable $1) shift for script in "$@"; do ID=$(sbatch --parsable --dependency=afterany:${ID} $script) done
Usage: ./sub_dep.sh scriptA.sh scriptB.sh scriptC.sh scriptD.sh