Differences
This shows you the differences between two versions of the page.
Next revision | Previous revisionNext revisionBoth sides next revision | ||
en:services:application_services:high_performance_computing:workshops:matlab-on-scc-2023 [2023/01/30 20:32] – created vend | en:services:application_services:high_performance_computing:workshops:matlab-on-scc-2023 [2023/01/30 21:05] – add links vend | ||
---|---|---|---|
Line 4: | Line 4: | ||
===== CONFIGURATION – MATLAB client on the cluster ===== | ===== CONFIGURATION – MATLAB client on the cluster ===== | ||
- | After logging into the cluster, configure MATLAB to run parallel jobs on your cluster by calling the shell script configCluster.sh | + | |
+ | After logging into the cluster, configure MATLAB to run parallel jobs on your cluster by calling the shell script | ||
+ | < | ||
$ module load matlab | $ module load matlab | ||
$ configCluster.sh | $ configCluster.sh | ||
+ | </ | ||
+ | |||
Jobs will now default to the cluster rather than submit to the local machine. | Jobs will now default to the cluster rather than submit to the local machine. | ||
- | INSTALLATION and CONFIGURATION – MATLAB client on the desktop | + | |
- | The SCC MATLAB support package can be found at TBD. | + | ==== INSTALLATION and CONFIGURATION – MATLAB client on the desktop |
+ | |||
+ | The SCC MATLAB support package can be found at {{ : | ||
Download the appropriate archive file and start MATLAB. | Download the appropriate archive file and start MATLAB. | ||
+ | < | ||
>> userpath | >> userpath | ||
+ | </ | ||
+ | |||
Configure MATLAB to run parallel jobs on your cluster by calling configCluster. | Configure MATLAB to run parallel jobs on your cluster by calling configCluster. | ||
+ | |||
+ | < | ||
>> configCluster | >> configCluster | ||
+ | </ | ||
+ | |||
Submission to the remote cluster requires SSH credentials. | Submission to the remote cluster requires SSH credentials. | ||
+ | |||
Jobs will now default to the cluster rather than submit to the local machine. | Jobs will now default to the cluster rather than submit to the local machine. | ||
- | NOTE: If you would like to submit to the local machine then run the following command: | + | |
+ | **NOTE:** If you would like to submit to the local machine then run the following command: | ||
+ | |||
+ | < | ||
>> % Get a handle to the local resources | >> % Get a handle to the local resources | ||
>> c = parcluster(' | >> c = parcluster(' | ||
- | CONFIGURING JOBS | + | </ |
- | Prior to submitting the job, we can specify various parameters to pass to our jobs, such as queue, e-mail, walltime, etc. The following is a partial list of parameters. | + | |
+ | ===== CONFIGURING JOBS ==== | ||
+ | Prior to submitting the job, we can specify various parameters to pass to our jobs, such as queue, e-mail, walltime, etc. The following is a partial list of parameters. | ||
+ | |||
+ | < | ||
>> % Get a handle to the cluster | >> % Get a handle to the cluster | ||
>> c = parcluster; | >> c = parcluster; | ||
Line 48: | Line 69: | ||
>> % Specify the wall time (e.g., 5 hours) | >> % Specify the wall time (e.g., 5 hours) | ||
>> c.AdditionalProperties.WallTime = ' | >> c.AdditionalProperties.WallTime = ' | ||
+ | </ | ||
- | Save changes after modifying AdditionalProperties for the above changes to persist between MATLAB sessions. | + | Save changes after modifying |
+ | < | ||
>> c.saveProfile | >> c.saveProfile | ||
+ | </ | ||
- | To see the values of the current configuration options, display AdditionalProperties. | + | To see the values of the current configuration options, display |
+ | < | ||
>> % To view current properties | >> % To view current properties | ||
>> c.AdditionalProperties | >> c.AdditionalProperties | ||
+ | </ | ||
Unset a value when no longer needed. | Unset a value when no longer needed. | ||
+ | |||
+ | < | ||
>> % Turn off email notifications | >> % Turn off email notifications | ||
>> c.AdditionalProperties.EmailAddress = ''; | >> c.AdditionalProperties.EmailAddress = ''; | ||
>> c.saveProfile | >> c.saveProfile | ||
- | INTERACTIVE JOBS - MATLAB client on the cluster | + | </ |
- | To run an interactive pool job on the cluster, continue to use parpool as you’ve done before. | + | |
+ | ===== INTERACTIVE JOBS - MATLAB client on the cluster | ||
+ | |||
+ | To run an interactive pool job on the cluster, continue to use "parpool" | ||
+ | |||
+ | < | ||
>> % Get a handle to the cluster | >> % Get a handle to the cluster | ||
>> c = parcluster; | >> c = parcluster; | ||
Line 68: | Line 101: | ||
>> % Open a pool of 64 workers on the cluster | >> % Open a pool of 64 workers on the cluster | ||
>> pool = c.parpool(64); | >> pool = c.parpool(64); | ||
+ | </ | ||
Rather than running local on the local machine, the pool can now run across multiple nodes on the cluster. | Rather than running local on the local machine, the pool can now run across multiple nodes on the cluster. | ||
+ | < | ||
>> % Run a parfor over 1000 iterations | >> % Run a parfor over 1000 iterations | ||
>> parfor idx = 1:1000 | >> parfor idx = 1:1000 | ||
a(idx) = … | a(idx) = … | ||
end | end | ||
+ | </ | ||
Once we’re done with the pool, delete it. | Once we’re done with the pool, delete it. | ||
+ | < | ||
>> % Delete the pool | >> % Delete the pool | ||
>> pool.delete | >> pool.delete | ||
- | INDEPENDENT BATCH JOB | + | </ |
+ | |||
+ | ===== INDEPENDENT BATCH JOB ===== | ||
Use the batch command to submit asynchronous jobs to the cluster. | Use the batch command to submit asynchronous jobs to the cluster. | ||
+ | |||
+ | < | ||
>> % Get a handle to the cluster | >> % Get a handle to the cluster | ||
>> c = parcluster; | >> c = parcluster; | ||
Line 96: | Line 138: | ||
>> % Delete the job after results are no longer needed | >> % Delete the job after results are no longer needed | ||
>> job.delete | >> job.delete | ||
+ | </ | ||
- | To retrieve a list of currently running or completed jobs, call parcluster to retrieve the cluster object. | + | To retrieve a list of currently running or completed jobs, call "parcluster" |
+ | |||
+ | < | ||
>> c = parcluster; | >> c = parcluster; | ||
>> jobs = c.Jobs; | >> jobs = c.Jobs; | ||
+ | </ | ||
+ | |||
Once we’ve identified the job we want, we can retrieve the results as we’ve done previously. | Once we’ve identified the job we want, we can retrieve the results as we’ve done previously. | ||
- | fetchOutputs is used to retrieve function output arguments; if calling batch with a script, use load instead. | + | "fetchOutputs" |
To view results of a previously completed job: | To view results of a previously completed job: | ||
+ | |||
+ | < | ||
>> % Get a handle to the job with ID 2 | >> % Get a handle to the job with ID 2 | ||
>> job2 = c.Jobs(2); | >> job2 = c.Jobs(2); | ||
+ | </ | ||
- | NOTE: You can view a list of your jobs, as well as their IDs, using the above c.Jobs command. | + | **NOTE:** You can view a list of your jobs, as well as their IDs, using the above c.Jobs command. |
+ | < | ||
>> % Fetch results for job with ID 2 | >> % Fetch results for job with ID 2 | ||
>> job2.fetchOutputs{: | >> job2.fetchOutputs{: | ||
- | PARALLEL BATCH JOB | + | </ |
- | Users can also submit parallel workflows with the batch command. | + | |
+ | ===== PARALLEL BATCH JOB ===== | ||
+ | |||
+ | Users can also submit parallel workflows with the batch command. | ||
+ | |||
+ | < | ||
function [t, A] = parallel_example(iter) | function [t, A] = parallel_example(iter) | ||
Line 132: | Line 188: | ||
end | end | ||
+ | </ | ||
This time when we use the batch command, to run a parallel job, we’ll also specify a MATLAB Pool. | This time when we use the batch command, to run a parallel job, we’ll also specify a MATLAB Pool. | ||
+ | |||
+ | < | ||
>> % Get a handle to the cluster | >> % Get a handle to the cluster | ||
>> c = parcluster; | >> c = parcluster; | ||
Line 148: | Line 207: | ||
ans = | ans = | ||
8.8872 | 8.8872 | ||
+ | </ | ||
+ | |||
The job ran in 8.89 seconds using four workers. | The job ran in 8.89 seconds using four workers. | ||
+ | |||
We’ll run the same simulation but increase the Pool size. This time, to retrieve the results later, we’ll keep track of the job ID. | We’ll run the same simulation but increase the Pool size. This time, to retrieve the results later, we’ll keep track of the job ID. | ||
- | NOTE: For some applications, | + | |
+ | **NOTE:** For some applications, | ||
+ | |||
+ | < | ||
>> % Get a handle to the cluster | >> % Get a handle to the cluster | ||
>> c = parcluster; | >> c = parcluster; | ||
Line 164: | Line 229: | ||
>> % Clear job from workspace (as though we quit MATLAB) | >> % Clear job from workspace (as though we quit MATLAB) | ||
>> clear job | >> clear job | ||
- | Once we have a handle to the cluster, we’ll call the findJob method to search for the job with the specified job ID. | + | </ |
+ | |||
+ | Once we have a handle to the cluster, we’ll call the "findJob" | ||
+ | |||
+ | < | ||
>> % Get a handle to the cluster | >> % Get a handle to the cluster | ||
>> c = parcluster; | >> c = parcluster; | ||
Line 180: | Line 249: | ||
ans = | ans = | ||
4.7270 | 4.7270 | ||
+ | </ | ||
+ | |||
The job now runs in 4.73 seconds using eight workers. | The job now runs in 4.73 seconds using eight workers. | ||
Alternatively, | Alternatively, | ||
- | DEBUGGING | + | {{ : |
- | If a serial job produces an error, call the getDebugLog method to view the error log file. When submitting independent jobs, with multiple tasks, specify the task number. | + | |
+ | ===== DEBUGGING | ||
+ | |||
+ | If a serial job produces an error, call the "getDebugLog" | ||
+ | |||
+ | < | ||
>> c.getDebugLog(job.Tasks(3)) | >> c.getDebugLog(job.Tasks(3)) | ||
+ | </ | ||
+ | |||
For Pool jobs, only specify the job object. | For Pool jobs, only specify the job object. | ||
+ | < | ||
>> c.getDebugLog(job) | >> c.getDebugLog(job) | ||
+ | </ | ||
When troubleshooting a job, the cluster admin may request the scheduler ID of the job. This can be derived by calling schedID | When troubleshooting a job, the cluster admin may request the scheduler ID of the job. This can be derived by calling schedID | ||
+ | < | ||
>> schedID(job) | >> schedID(job) | ||
ans = | ans = | ||
25539 | 25539 | ||
+ | </ | ||
+ | ===== HELPER FUNCTIONS ===== | ||
+ | ^ Function | ||
+ | | clusterFeatures | ||
+ | | clusterGpuCards | ||
+ | | clusterQueueNames | ||
+ | | disableArchiving | ||
+ | | fixConnection | ||
+ | | willRun | ||
+ | ===== TO LEARN MORE ===== | ||
- | HELPER FUNCTIONS | ||
- | Function | ||
- | Description | ||
- | Desktop-only | ||
- | clusterFeatures | ||
- | List of scheduler features/ | ||
- | |||
- | clusterGpuCards | ||
- | List of cluster GPU cards | ||
- | |||
- | clusterQueueNames | ||
- | List of scheduler queue names | ||
- | |||
- | disableArchiving | ||
- | Modify file archiving to resolve file mirroring issue | ||
- | true | ||
- | fixConnection | ||
- | Reestablish cluster connection | ||
- | true | ||
- | willRun | ||
- | Explain why job is not running | ||
- | |||
- | TO LEARN MORE | ||
To learn more about the MATLAB Parallel Computing Toolbox, check out these resources: | To learn more about the MATLAB Parallel Computing Toolbox, check out these resources: | ||
- | • Parallel Computing Coding Examples | + | * [[https:// |
- | | + | * [[http:// |
- | | + | * [[http:// |
- | | + | * [[http:// |
- | | + | * [[http:// |
- | | + | * [[http:// |