Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
en:services:application_services:high_performance_computing:ipython_parallel [2019/02/15 09:51] – akhuziyen:services:application_services:high_performance_computing:ipython_parallel [2025/02/19 16:28] (current) – nboelte
Line 1: Line 1:
 +====== Parallel execution of Jupyter notebooks on the cluster ======
  
 +This documentation is based on the IPython example used in our  [[en:services:application_services:high_performance_computing:singularity|Singularity on the SCC]] documentation page.
 +
 +When submitting that image, please add the -B option of singularity for mounting the /opt folder:
 +
 +<code>
 +srun --pty -p jupyter apptainer shell -B /opt sjupyter.sif
 +</code>
 +
 +To be able to run Slurm commands within the container, additional libraries and directories should be bound into the container:
 +
 +<code>
 +apptainer -B /var/run/munge,/run/munge,/usr/lib64/libmunge.so.2,/usr/lib64/libmunge.so.2.0.0,/etc/profile.d/slurm.sh ...
 +</code>
 +
 +Also you need to add //slurm// user to the container when building the image, with following commands: 
 +<code>
 +echo "slurm:x:450:450::/opt/slurm/slurm:/bin/false" >> /etc/passwd
 +echo "slurm:x:450:" >> /etc/group
 +</code>
 +
 +For using IPython Parallel with the cluster, we need to configure it. These steps are required only once, everything will be kept in your ''%%$HOME/.ipython%%'' directory, even if you destroy the container.
 +
 +To create a new profile and configure it for the compute cluster, run following command:
 +
 +<code>
 +ipython profile create --parallel --profile=myslurm
 +</code>
 +
 +This will create the profile at ''%%$HOME/.ipython/profile_myslurm%%''. Now you need to configure it for Slurm.
 +
 +Add following config lines to the file ''%%$HOME/.ipython/profile_myslurm/ipcluster_config.py%%'':
 +
 +<code>
 +c.IPClusterEngines.engine_launcher_class = 'SlurmEngineSetLauncher'
 +c.IPClusterStart.controller_launcher_class = 'SlurmControllerLauncher'
 +c.SlurmControllerLauncher.batch_template_file ='slurm.controller.template'
 +c.SlurmEngineSetLauncher.batch_template_file = 'slurm.engine.template'
 +</code>
 +and comment out the following parameters:
 +<code>
 +#c.SlurmControllerLauncher.batch_template = "..."
 +#c.SlurmEngineSetLauncher.batch_template = "..."
 +</code>
 +
 +Add the following line to ''%%$HOME/.ipython/profile_myslurm/ipcontroller_config.py%%'':
 +
 +<code>
 +c.HubFactory.ip = '*'
 +</code>
 +
 +IPython Parallel is almost ready to use. For submitting Slurm jobs in a specific queue and with additional parameters, create templates for batch jobs in the directory you want to start the container using the names specified in the configuration file, i.e. ''%%slurm.controller.template%%'' and ''%%slurm.engine.template%%''.
 +
 +slurm.controller.template:
 +
 +<code>
 +#!/bin/bash
 +
 +#SBATCH -p medium
 +#SBATCH -J ipcontroller
 +#SBATCH -o jupyterhub-gwdg/current.ipcontroller.log
 +#SBATCH -n 1 
 +#SBATCH -t 1:00:00
 +
 +module load apptainer
 +apptainer exec sjupyter.sif ipcontroller --profile-dir={profile_dir} --location=$HOSTNAME
 +</code>
 +
 +slurm.engine.template:
 +
 +<code>
 +#!/bin/bash
 +
 +#SBATCH -p medium
 +#SBATCH -J ipengine
 +#SBATCH -n {n}
 +#SBATCH -o jupyterhub-gwdg/current.ipengine.log
 +#SBATCH -t 1:00:00
 +
 +module load apptainer
 +srun apptainer exec sjupyter.sif ipengine --profile-dir={profile_dir}
 +</code>
 +
 +Now you can launch a jupyter instance:
 +
 +<code>
 +jupyter notebook --port <port> --ip 0.0.0.0 --no-browser
 +</code>
 +
 +For ''%%<port>%%'' choose a random unrestricted port number, for example 8769. Tunnel the port from the node to your local PC:
 +
 +<code>
 +ssh -L 0.0.0.0:<port>:0.0.0.0:<port> yourlogin@glogin.hpc.gwdg.de ssh -L 0.0.0.0:<port>:<host>:<port> gwdu101 -N
 +</code>
 +
 +For ''%%<host>%%'' insert the node where the container is running. Open the link in the Jupyter output in your browswer.
 +
 +To start the cluster, use the IPython Clusters tab in the Jupyter interface, select the myslurm profile and amount of processes and click **start**. You will be able to see the engines running with the "squeue -u $USER" command.
 +
 +To test if it is working, simply run following script in the Jupyter notebook:
 +
 +<code>
 +import ipyparallel as ipp
 +c = ipp.Client(profile="myslurm")
 +c.ids
 +c[:].apply_sync(lambda : "Hello, World")
 +</code>