Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revisionNext revisionBoth sides next revision | ||
en:services:application_services:high_performance_computing:running_jobs_slurm [2019/04/04 15:50] – [Available Partitions] tehlers | en:services:application_services:high_performance_computing:running_jobs_slurm [2023/06/21 10:14] – [GPU selection] skrey | ||
---|---|---|---|
Line 1: | Line 1: | ||
+ | ====== Running Jobs with Slurm====== | ||
+ | In the following the basic concepts will be described. | ||
+ | |||
+ | **Cluster**\\ | ||
+ | A collection of networked computers intended to provide compute capabilities. | ||
+ | |||
+ | **Node**\\ | ||
+ | One of these computers, also called host. | ||
+ | |||
+ | **Frontend**\\ | ||
+ | A special node provided to interact with the cluster via shell commands. gwdu101, gwdu102 and gwdu103 are our frontends. | ||
+ | |||
+ | **Task or (Job-)Slot**\\ | ||
+ | Compute capacity for one process (or " | ||
+ | |||
+ | **Job**\\ | ||
+ | A compute task consisting of one or several parallel processes. | ||
+ | |||
+ | **Batch System**\\ | ||
+ | The management system distributing job processes across job slots. In our case [[https:// | ||
+ | |||
+ | **Serial job**\\ | ||
+ | A job consisting of one process using one job slot. | ||
+ | |||
+ | **SMP job**\\ | ||
+ | A job with shared memory parallelization (often realized with OpenMP), meaning that all processes need access to the memory of the same node. Consequently an SMP job uses several job slots //on the same node//. | ||
+ | |||
+ | **MPI job**\\ | ||
+ | A Job with distributed memory parallelization, | ||
+ | |||
+ | **Partition**\\ | ||
+ | A label to sort jobs by general requirements and intended execution nodes. Formerly called " | ||
+ | |||
+ | ===== The sbatch Command: Submitting Jobs to the Cluster | ||
+ | |||
+ | '' | ||
+ | |||
+ | * What is to be done? (path to your program and required parameters) | ||
+ | * What are the requirements? | ||
+ | |||
+ | Slurm then matches the job’s requirements against the capabilities of available job slots. Once sufficient suitable job slots are found, the job is started. Slurm considers jobs to be started in the order of their priority. | ||
+ | |||
+ | ===== Available Partitions | ||
+ | We currently have two meta partitions, corresponding to broad application profiles: | ||
+ | |||
+ | **medium**\\ | ||
+ | This is our general purpose partition, usable for serial and SMP jobs with up to 24 tasks, but it is especially well suited for large MPI jobs. Up to 1024 cores can be used in a single MPI job, and the maximum runtime is 48 hours. | ||
+ | |||
+ | **fat**\\ | ||
+ | This is the partition for SMP jobs, especially those requiring lots of memory. Serial jobs with very high memory requirements do also belong in this partition. Up to 24 cores and up to 512 GB are available on one host. Maximum runtime is 48 hours.\\ | ||
+ | The nodes of the fat+ partitions are also present in this partition, but will only be used, if they are not needed for bigger jobs submitted to the fat+ partition. | ||
+ | |||
+ | **fat+**\\ | ||
+ | This partition is meant for very memory intensive jobs. These partitions are for jobs that require more than 512 GB RAM on single node. Nodes of fat+ partitions have 1.5 and 2 TB RAM. You are required to have specify your memory needs on job submission to use these nodes (see [[en: | ||
+ | As general advice: Try your jobs on the smaller nodes in the fat partition first and work your way up and don't be afraid to ask for help here. | ||
+ | |||
+ | **gpu** - A partition for nodes containing GPUs. Please refer to [[en: | ||
+ | |||
+ | ==== Runtime limits (QoS) ==== | ||
+ | If the default time limit of 48 hours are not sufficient for your jobs, you can use a " | ||
+ | |||
+ | **long**\\ | ||
+ | Here, the maximum runtime is increased to 120 hours. Job slot availability is limited, though, so expect longer waiting times. Be aware that using this QOS does not automatically increase the timelimit of your job. You still have to set it yourself. | ||
+ | |||
+ | **short**\\ | ||
+ | Here, the maximum runtime is decreased to two hours. In turn the queue has a higher base priority, but it also has limited job slot availability. That means that as long as only few jobs are submitted to the " | ||
+ | |||
+ | ===== How to submit jobs ===== | ||
+ | |||
+ | Slurm supports different ways to submit jobs to the cluster: | ||
+ | Interactively or in batch mode. We generally recommend using the | ||
+ | batch mode. If you need to run a job interactively, | ||
+ | information about that in the [[en: | ||
+ | Batch jobs are submitted to the cluster using the ' | ||
+ | and a jobscript or a command:\\ | ||
+ | < | ||
+ | |||
+ | **sbatch** can take a lot of options to give more information on the | ||
+ | specifics of your job, e.g. where to run it, how long it will take | ||
+ | and how many nodes it needs. We will examine a few of the options in | ||
+ | the following paragraphs. For a full list of commands, refer to the | ||
+ | manual of the command with 'man sbatch' | ||
+ | |||
+ | ==== sbatch/srun options | ||
+ | |||
+ | **< | ||
+ | Specifies the account ' | ||
+ | |||
+ | **< | ||
+ | Specifies in which partition the job should run. Multiple partitions | ||
+ | can be specified in a comma separated list. | ||
+ | |||
+ | **< | ||
+ | Maximum runtime of the job. If this time is exceeded the job is killed. Acceptable < | ||
+ | |||
+ | **< | ||
+ | Submit the job using a special QOS. For example '' | ||
+ | |||
+ | **< | ||
+ | Store the job output in " | ||
+ | |||
+ | **< | ||
+ | Some metainformation about your job will be added to your output file. If you do not want that, you can suppress it with this flag.\\ | ||
+ | |||
+ | **< | ||
+ | < | ||
+ | Receive mails when the jobs start, end or both. There are even more options, refer to the sbatch man-page for more information about mail types. If you have a GWDG-mail-address, | ||
+ | |||
+ | ==== Resource Selection | ||
+ | |||
+ | === CPU Selection === | ||
+ | |||
+ | **< | ||
+ | The number of tasks for this job. The default is one task per node. | ||
+ | |||
+ | **< | ||
+ | The number of cpus per tasks. The default is one cpu per task. | ||
+ | |||
+ | **< | ||
+ | As a rule of thumb, if you run your code on a single node, use -c. For multi-node MPI-jobs, use -n.\\ | ||
+ | |||
+ | **< | ||
+ | Minimum and maximum number of nodes that the job should be executed on. If only one number is specified, it is used as the precise node count.\\ | ||
+ | |||
+ | **< | ||
+ | Number of tasks per node. If -n and < | ||
+ | \\ | ||
+ | === Memory Selection === | ||
+ | |||
+ | By default, your available memory per node is the default memory per task times the number of tasks you have running on that node. You can get the default memory per task by looking at the DefMemPerCPU metric as reported by '' | ||
+ | '' | ||
+ | |||
+ | **< | ||
+ | Required memory per node. The Unit can be one of [K|M|G|T], but defaults to M. If your processes exceed this limit, they will be killed. | ||
+ | |||
+ | **< | ||
+ | Required memory per task instead of node. < | ||
+ | |||
+ | **< | ||
+ | Required memory per gpu instead of node. < | ||
+ | === Example === | ||
+ | |||
+ | ''< | ||
+ | |||
+ | ''< | ||
+ | |||
+ | ''< | ||
+ | |||
+ | |||
+ | ==== The GWDG Scientific Compute Cluster | ||
+ | |||
+ | {{ : | ||
+ | |||
+ | This scheme shows the basic cluster setup at GWDG. The shared /scratch is usually the best choice for temporary data in your jobs, but /scratch is only available at the " | ||
+ | |||
+ | ==== '' | ||
+ | |||
+ | **-C scratch**\\ | ||
+ | The node must have access to shared ''/ | ||
+ | |||
+ | **-C local**\\ | ||
+ | The node must have local SSD storage at ''/ | ||
+ | |||
+ | **-C mdc / -C fas**\\ | ||
+ | The node has to be at that location. It is pretty similar to -C scratch / -C scratch2, since the nodes in the MDC have access to scratch and those at the Fassberg location have access to scratch2. This is mainly for easy compatibility with our old partition naming scheme. | ||
+ | |||
+ | **-C [architecture]**\\ | ||
+ | request a specific CPU architecture. Available Options are: haswell, broadwell, cascadelake. See [[en: | ||
+ | |||
+ | |||
+ | ==== Using Job Scripts | ||
+ | |||
+ | A job script is a shell script with a special comment section: In each line beginning with ''# | ||
+ | |||
+ | < | ||
+ | #!/bin/bash | ||
+ | #SBATCH -p medium | ||
+ | #SBATCH -t 10:00 | ||
+ | #SBATCH -o outfile-%J | ||
+ | |||
+ | / | ||
+ | </ | ||
+ | \\ | ||
+ | Job scripts are submitted by the following command: | ||
+ | |||
+ | < | ||
+ | sbatch <script name> | ||
+ | </ | ||
+ | \\ | ||
+ | ==== Exclusive jobs ==== | ||
+ | |||
+ | An exclusive job does use all of its allocated nodes exclusively, | ||
+ | |||
+ | Do not combine ''< | ||
+ | |||
+ | To submit an exclusive job add ''< | ||
+ | < | ||
+ | sbatch --exclusive -p fat -t 12:00:00 --wrap=" | ||
+ | </ | ||
+ | This allocates either a complete gwda nodes with 256GB, or a complete dfa node with 512GB. | ||
+ | |||
+ | For submitting an OpenMP/MPI hybrid job with a total of 8 MPI processes, spread evenly across 2 nodes, use: | ||
+ | < | ||
+ | export OMP_NUM_THREADS=4 | ||
+ | sbatch --exclusive -p medium -N 2 --ntasks-per-node=4 --wrap=" | ||
+ | </ | ||
+ | (each MPI process creates 4 OpenMP threads in this case). | ||
+ | |||
+ | ==== Disk Space Options | ||
+ | |||
+ | You have the following options for attributing disk space to your jobs: | ||
+ | |||
+ | **/ | ||
+ | This is the local hard disk of the node. It is a fast, SSD based option for storing temporary data, available on '' | ||
+ | A directory is automatically created for each job at ''/ | ||
+ | |||
+ | **/ | ||
+ | This is the shared scratch space, available on '' | ||
+ | A directory is automatically created for each job at ''/ | ||
+ | |||
+ | **/ | ||
+ | This space is the same as scratch described above except it is **ONLY** available on the nodes '' | ||
+ | A directory is automatically created for each job at ''/ | ||
+ | |||
+ | **$HOME**\\ | ||
+ | Your home directory is available everywhere, permanent, and comes with backup. Your attributed disk space can be increased. It is comparably slow, however. You can find more information about the $HOME file system [[en: | ||
+ | |||
+ | ==== Recipe: Using ''/ | ||
+ | |||
+ | This recipe shows how to run Gaussian09 using ''/ | ||
+ | |||
+ | <code bash> | ||
+ | #!/bin/bash | ||
+ | #SBATCH -p fat | ||
+ | #SBATCH -N 1 | ||
+ | #SBATCH -n 24 | ||
+ | #SBATCH -C scratch | ||
+ | #SBATCH -t 1-00:00:00 | ||
+ | |||
+ | #Set up gaussian environment | ||
+ | export g09root="/ | ||
+ | . $g09root/ | ||
+ | |||
+ | #set the $TMP_SCRATCH as the scratch dir for gaussian | ||
+ | export GAUSS_SCRDIR=$TMP_SCRATCH | ||
+ | |||
+ | #run the calculations | ||
+ | g09 myjob.com myjob.log | ||
+ | </ | ||
+ | \\ | ||
+ | |||
+ | Please check out the software manual on how to set the directory for temporary files. Many programs have some flags for it, or read environment variables to determine the location, which you can set in your jobscript. | ||
+ | ==== Using ''/ | ||
+ | If you use scratch space only for storing temporary data, and do not need to access data stored previously, you can request /scratch: | ||
+ | < | ||
+ | #SBATCH -C " | ||
+ | </ | ||
+ | You can just use ''/ | ||
+ | |||
+ | ==== Interactive session on the nodes ==== | ||
+ | |||
+ | As stated before, '' | ||
+ | |||
+ | < | ||
+ | \\ | ||
+ | ''< | ||
+ | < | ||
+ | |||
+ | ==== GPU selection ==== | ||
+ | |||
+ | In order to use a GPU you should submit your job to the '' | ||
+ | < | ||
+ | #SBATCH -p gpu | ||
+ | #SBATCH -n 10 | ||
+ | #SBATCH -G gtx1080:2 | ||
+ | </ | ||
+ | |||
+ | |||
+ | You can also omit the model selection, here is an example of selecting 1 GPU of any available model:< | ||
+ | #SBATCH -p gpu | ||
+ | #SBATCH -n 10 | ||
+ | #SBATCH -G 1 | ||
+ | </ | ||
+ | |||
+ | There are different options to select the number of GPUs, such as '' | ||
+ | |||
+ | Currently we have several generations of NVidia GPUs in the cluster, namely: | ||
+ | |||
+ | < | ||
+ | v100: Nvidia Tesla V100 | ||
+ | rtx5000: Nvidia Quadro RTX5000 | ||
+ | gtx1080 : GeForce GTX 1080 | ||
+ | gtx980 | ||
+ | </ | ||
+ | |||
+ | Most GPUs are optimized for single precision calculations. If you need double precision performance, | ||
+ | < | ||
+ | #SBATCH -p gpu | ||
+ | #SBATCH -G v100:1 | ||
+ | </ | ||
+ | |||
+ | < | ||
+ | |||
+ | ===== Miscellaneous Slurm Commands | ||
+ | |||
+ | While '' | ||
+ | |||
+ | **< | ||
+ | Shows current status of the cluster and queues | ||
+ | |||
+ | **< | ||
+ | Lists current jobs (default: all users). Useful options are: ''< | ||
+ | |||
+ | **< | ||
+ | Full job information. Only available while job is running and short time thereafter. | ||
+ | |||
+ | **< | ||
+ | Expected start time. This is a rough estimate. | ||
+ | |||
+ | **< | ||
+ | Get job Information even after the job has finished.\\ | ||
+ | **Note on '' | ||
+ | The ''< | ||
+ | |||
+ | **< | ||
+ | Cancels jobs. Examples:\\ | ||
+ | ''< | ||
+ | ''< | ||
+ | ''< | ||
+ | |||
+ | Have a look at the respective man pages of these commands to learn more about them! |