Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
Last revisionBoth sides next revision
en:services:application_services:high_performance_computing:workshops:matlab-on-scc-2023 [2023/01/30 20:38] – some styling and upload of athWorks zip venden:services:application_services:high_performance_computing:workshops:matlab-on-scc-2023 [2023/01/30 21:06] – [INSTALLATION and CONFIGURATION – MATLAB client on the desktop] vend
Line 13: Line 13:
 Jobs will now default to the cluster rather than submit to the local machine. Jobs will now default to the cluster rather than submit to the local machine.
  
-==== INSTALLATION and CONFIGURATION – MATLAB client on the desktop =====+===== INSTALLATION and CONFIGURATION – MATLAB client on the desktop =====
  
 The SCC MATLAB support package can be found at {{ :en:services:application_services:high_performance_computing:workshops:gwdg.nonshared.r2022b.zip |here}}. The SCC MATLAB support package can be found at {{ :en:services:application_services:high_performance_computing:workshops:gwdg.nonshared.r2022b.zip |here}}.
Line 39: Line 39:
  
 ===== CONFIGURING JOBS ==== ===== CONFIGURING JOBS ====
-Prior to submitting the job, we can specify various parameters to pass to our jobs, such as queue, e-mail, walltime, etc.  The following is a partial list of parameters.  See AdditionalProperties for the complete list.  None of these are required. +Prior to submitting the job, we can specify various parameters to pass to our jobs, such as queue, e-mail, walltime, etc.  The following is a partial list of parameters.  See "AdditionalProperties" for the complete list.  None of these are required.  
 + 
 +<code> 
 >> % Get a handle to the cluster >> % Get a handle to the cluster
 >> c = parcluster; >> c = parcluster;
Line 67: Line 69:
 >> % Specify the wall time (e.g., 5 hours) >> % Specify the wall time (e.g., 5 hours)
 >> c.AdditionalProperties.WallTime = '05:00:00'; >> c.AdditionalProperties.WallTime = '05:00:00';
 +</code> 
  
-Save changes after modifying AdditionalProperties for the above changes to persist between MATLAB sessions.+Save changes after modifying "AdditionalProperties" for the above changes to persist between MATLAB sessions. 
 +<code>
 >> c.saveProfile >> c.saveProfile
 +</code>
  
-To see the values of the current configuration options, display AdditionalProperties.+To see the values of the current configuration options, display "AdditionalProperties".
  
 +<code>
 >> % To view current properties >> % To view current properties
 >> c.AdditionalProperties >> c.AdditionalProperties
 +</code>
  
 Unset a value when no longer needed. Unset a value when no longer needed.
 +
 +<code>
 >> % Turn off email notifications  >> % Turn off email notifications 
 >> c.AdditionalProperties.EmailAddress = ''; >> c.AdditionalProperties.EmailAddress = '';
 >> c.saveProfile >> c.saveProfile
-INTERACTIVE JOBS - MATLAB client on the cluster +</code> 
-To run an interactive pool job on the cluster, continue to use parpool as you’ve done before.+ 
 +===== INTERACTIVE JOBS - MATLAB client on the cluster ===== 
 + 
 +To run an interactive pool job on the cluster, continue to use "parpool" as you’ve done before. 
 + 
 +<code>
 >> % Get a handle to the cluster >> % Get a handle to the cluster
 >> c = parcluster; >> c = parcluster;
Line 87: Line 101:
 >> % Open a pool of 64 workers on the cluster >> % Open a pool of 64 workers on the cluster
 >> pool = c.parpool(64); >> pool = c.parpool(64);
 +</code>
  
 Rather than running local on the local machine, the pool can now run across multiple nodes on the cluster. Rather than running local on the local machine, the pool can now run across multiple nodes on the cluster.
  
 +<code>
 >> % Run a parfor over 1000 iterations >> % Run a parfor over 1000 iterations
 >> parfor idx = 1:1000 >> parfor idx = 1:1000
       a(idx) = …       a(idx) = …
    end    end
 +</code>
  
 Once we’re done with the pool, delete it. Once we’re done with the pool, delete it.
  
 +<code>
 >> % Delete the pool >> % Delete the pool
 >> pool.delete >> pool.delete
-INDEPENDENT BATCH JOB+</code> 
 + 
 +===== INDEPENDENT BATCH JOB ===== 
 Use the batch command to submit asynchronous jobs to the cluster.  The batch command will return a job object which is used to access the output of the submitted job.  See the MATLAB documentation for more help on batch. Use the batch command to submit asynchronous jobs to the cluster.  The batch command will return a job object which is used to access the output of the submitted job.  See the MATLAB documentation for more help on batch.
 +
 +<code>
 >> % Get a handle to the cluster >> % Get a handle to the cluster
 >> c = parcluster; >> c = parcluster;
Line 115: Line 138:
 >> % Delete the job after results are no longer needed >> % Delete the job after results are no longer needed
 >> job.delete >> job.delete
 +</code>
  
-To retrieve a list of currently running or completed jobs, call parcluster to retrieve the cluster object.  The cluster object stores an array of jobs that were run, are running, or are queued to run.  This allows us to fetch the results of completed jobs.  Retrieve and view the list of jobs as shown below.+To retrieve a list of currently running or completed jobs, call "parcluster" to retrieve the cluster object.  The cluster object stores an array of jobs that were run, are running, or are queued to run.  This allows us to fetch the results of completed jobs.  Retrieve and view the list of jobs as shown below. 
 + 
 +<code>
 >> c = parcluster; >> c = parcluster;
 >> jobs = c.Jobs; >> jobs = c.Jobs;
 +</code>
 +
 Once we’ve identified the job we want, we can retrieve the results as we’ve done previously.  Once we’ve identified the job we want, we can retrieve the results as we’ve done previously. 
-fetchOutputs is used to retrieve function output arguments; if calling batch with a script, use load instead.   Data that has been written to files on the cluster needs be retrieved directly from the file system (e.g. via ftp).+"fetchOutputs" is used to retrieve function output arguments; if calling batch with a script, use load instead.   Data that has been written to files on the cluster needs be retrieved directly from the file system (e.g. via ftp).
 To view results of a previously completed job: To view results of a previously completed job:
 +
 +<code>
 >> % Get a handle to the job with ID 2 >> % Get a handle to the job with ID 2
 >> job2 = c.Jobs(2); >> job2 = c.Jobs(2);
 +</code>
  
-NOTE: You can view a list of your jobs, as well as their IDs, using the above c.Jobs command.  +**NOTE:** You can view a list of your jobs, as well as their IDs, using the above c.Jobs command.   
 +<code>
 >> % Fetch results for job with ID 2 >> % Fetch results for job with ID 2
 >> job2.fetchOutputs{:} >> job2.fetchOutputs{:}
-PARALLEL BATCH JOB +</code> 
-Users can also submit parallel workflows with the batch command.  Let’s use the following example for a parallel job, which is saved as parallel_example.m.   + 
 +===== PARALLEL BATCH JOB ===== 
 + 
 +Users can also submit parallel workflows with the batch command.  Let’s use the following example for a parallel job, which is saved as "parallel_example.m".    
 + 
 +<code>
 function [t, A] = parallel_example(iter) function [t, A] = parallel_example(iter)
    
Line 151: Line 188:
    
 end end
 +</code>
  
 This time when we use the batch command, to run a parallel job, we’ll also specify a MATLAB Pool.      This time when we use the batch command, to run a parallel job, we’ll also specify a MATLAB Pool.     
 +
 +<code>
 >> % Get a handle to the cluster >> % Get a handle to the cluster
 >> c = parcluster; >> c = parcluster;
Line 167: Line 207:
 ans =  ans = 
  8.8872  8.8872
 +</code>
 +
 The job ran in 8.89 seconds using four workers.  Note that these jobs will always request N+1 CPU cores, since one worker is required to manage the batch job and pool of workers.   For example, a job that needs eight workers will consume nine CPU cores.   The job ran in 8.89 seconds using four workers.  Note that these jobs will always request N+1 CPU cores, since one worker is required to manage the batch job and pool of workers.   For example, a job that needs eight workers will consume nine CPU cores. 
 +
 We’ll run the same simulation but increase the Pool size.  This time, to retrieve the results later, we’ll keep track of the job ID. We’ll run the same simulation but increase the Pool size.  This time, to retrieve the results later, we’ll keep track of the job ID.
-NOTE: For some applications, there will be a diminishing return when allocating too many workers, as the overhead may exceed computation time.    + 
 +**NOTE:** For some applications, there will be a diminishing return when allocating too many workers, as the overhead may exceed computation time. 
 + 
 +<code>    
 >> % Get a handle to the cluster >> % Get a handle to the cluster
 >> c = parcluster; >> c = parcluster;
Line 183: Line 229:
 >> % Clear job from workspace (as though we quit MATLAB) >> % Clear job from workspace (as though we quit MATLAB)
 >> clear job >> clear job
-Once we have a handle to the cluster, we’ll call the findJob method to search for the job with the specified job ID.   +</code> 
 + 
 +Once we have a handle to the cluster, we’ll call the "findJob" method to search for the job with the specified job ID.    
 + 
 +<code>
 >> % Get a handle to the cluster >> % Get a handle to the cluster
 >> c = parcluster; >> c = parcluster;
Line 199: Line 249:
 ans =  ans = 
 4.7270 4.7270
 +</code>
 +
 The job now runs in 4.73 seconds using eight workers.  Run code with different number of workers to determine the ideal number to use. The job now runs in 4.73 seconds using eight workers.  Run code with different number of workers to determine the ideal number to use.
 Alternatively, to retrieve job results via a graphical user interface, use the Job Monitor (Parallel > Monitor Jobs). Alternatively, to retrieve job results via a graphical user interface, use the Job Monitor (Parallel > Monitor Jobs).
  
-DEBUGGING +{{ :en:services:application_services:high_performance_computing:workshops:matlab_gui.png?nolink&400 |}} 
-If a serial job produces an error, call the getDebugLog method to view the error log file.  When submitting independent jobs, with multiple tasks, specify the task number.  + 
 +===== DEBUGGING ===== 
 + 
 +If a serial job produces an error, call the "getDebugLog" method to view the error log file.  When submitting independent jobs, with multiple tasks, specify the task number.   
 + 
 +<code>
 >> c.getDebugLog(job.Tasks(3)) >> c.getDebugLog(job.Tasks(3))
 +</code>
 +
 For Pool jobs, only specify the job object. For Pool jobs, only specify the job object.
 +<code>
 >> c.getDebugLog(job) >> c.getDebugLog(job)
 +</code>
 When troubleshooting a job, the cluster admin may request the scheduler ID of the job.  This can be derived by calling schedID When troubleshooting a job, the cluster admin may request the scheduler ID of the job.  This can be derived by calling schedID
 +<code>
 >> schedID(job) >> schedID(job)
 ans =  ans = 
 25539 25539
 +</code>
  
 +===== HELPER FUNCTIONS =====
 +^ Function           ^ Description                                            | Desktop-only  |
 +| clusterFeatures    | List of scheduler features/constraints                 |               |
 +| clusterGpuCards    | List of cluster GPU cards                              |               |
 +| clusterQueueNames  | List of scheduler queue names                          |               |
 +| disableArchiving   | Modify file archiving to resolve file mirroring issue  | true          |
 +| fixConnection      | Reestablish cluster connection                         | true          |
 +| willRun            | Explain, why job is not running                        |               |
  
 +===== TO LEARN MORE =====
  
-HELPER FUNCTIONS 
-Function 
-Description 
-Desktop-only 
-clusterFeatures 
-List of scheduler features/constraints 
- 
-clusterGpuCards 
-List of cluster GPU cards 
- 
-clusterQueueNames 
-List of scheduler queue names 
- 
-disableArchiving 
-Modify file archiving to resolve file mirroring issue 
-true 
-fixConnection 
-Reestablish cluster connection 
-true 
-willRun 
-Explain why job is not running 
- 
-TO LEARN MORE 
 To learn more about the MATLAB Parallel Computing Toolbox, check out these resources: To learn more about the MATLAB Parallel Computing Toolbox, check out these resources:
-    • Parallel Computing Coding Examples +  * [[https://www.mathworks.com/help/parallel-computing/examples.html|Parallel Computing Coding Examples]] 
-    • Parallel Computing Documentation +  * [[http://www.mathworks.com/help/distcomp/index.html|Parallel Computing Documentation]] 
-    • Parallel Computing Overview +  * [[http://www.mathworks.com/products/parallel-computing/index.html|Parallel Computing Overview]] 
-    • Parallel Computing Tutorials +  * [[http://www.mathworks.com/products/parallel-computing/tutorials.html|Parallel Computing Tutorials]] 
-    • Parallel Computing Videos +  * [[http://www.mathworks.com/products/parallel-computing/videos.html|Parallel Computing Videos]] 
-    • Parallel Computing Webinars+  * [[http://www.mathworks.com/products/parallel-computing/webinars.html|Parallel Computing Webinars]]