Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
en:services:application_services:jupyter:hpc [2020/07/02 19:13] – akhuziy | en:services:application_services:jupyter:hpc [2023/12/07 11:43] (current) – [Running your own Singularity Container] skrey | ||
---|---|---|---|
Line 1: | Line 1: | ||
+ | ====== Jupyter-Hub on the SCC ====== | ||
+ | GWDG also offers [[https:// | ||
+ | **Note** that for using Jupyter-Hub on the Scientific Compute Cluster (SCC) you need a GWDG account which is activated for the use of the compute cluster. Information on how to activate your account can be found [[en: | ||
+ | |||
+ | ===== How to use Jupyter-Hub on the SCC ===== | ||
+ | Jupyter-Hub on the SCC can be used identically to the [[en: | ||
+ | - GWDG HPC | ||
+ | - GWDG HPC with IPython Parallel | ||
+ | - GWDG HPC with Own Container | ||
+ | |||
+ | If you just need a Jupyter notebook, then select the 1st option. If you want to use [[https:// | ||
+ | |||
+ | IPython Parallel allows to increase computational resources and spawn compute workers on any nodes of the HPC cluster (not in the interactive queue) | ||
+ | |||
+ | A Jupyter notebook and IPython Parallel workers run as normal jobs in the cluster. | ||
+ | |||
+ | There are several options you can set on the spawning page to adjust the resourses, like number of cores and amount of memory. | ||
+ | |||
+ | ===== Options ===== | ||
+ | **Job profile**: this option sets which notebook you want to use, normal one, with IPython Parallel or your own. | ||
+ | |||
+ | **Singularity container location**: if you selected your own Container, then you have to provide the full path (or using '' | ||
+ | |||
+ | **Duration**: | ||
+ | |||
+ | **Number of cores**: This option sets the number of cores accessible by the notebook. Note that cores are not for exclusive usage and might be shared if there are more notebooks spawned than available resources. | ||
+ | |||
+ | **Amount of memory**: The option sets the amount of memory accessible by the notebook. Same as the amount of cores, memory might be shared if there are more notebooks than resources. | ||
+ | |||
+ | **Notebook' | ||
+ | |||
+ | ===== Resources ===== | ||
+ | Jupyter notebooks in [[https:// | ||
+ | |||
+ | It means that ~24 CPUs and ~128GB memory of the nodes in the interactive queue are available to use in Jupyter notebook and shared between all users who simultaneously use the same node. Currently there are 4 nodes in the interactive queue and more nodes will be added in case of high demand. | ||
+ | |||
+ | Also you can use /scratch (/scratch2) shared storage, which allows to store large files (terabytes), | ||
+ | |||
+ | <WRAP center round important 90%> | ||
+ | Jupyter notebooks by default start not in the root of your HOME directory but in the folder '' | ||
+ | </ | ||
+ | |||
+ | Currently one session of Jupyter notebook can run maximum **8 hours**, after that it will be killed, but your files will stay intact. | ||
+ | |||
+ | ===== Using IPython Parallel ===== | ||
+ | In order to make use of IPython Parallel, Jupyter should be started with '' | ||
+ | |||
+ | After the Jupyter notebook is launched, you can run engines using " | ||
+ | |||
+ | **Note**, that workers start as normal jobs in the '' | ||
+ | |||
+ | After the engines are up, the spawned cluster of workers can be checked by the following script: | ||
+ | <code python> | ||
+ | import ipyparallel as ipp | ||
+ | c = ipp.Client(profile=" | ||
+ | c.ids | ||
+ | c[: | ||
+ | </ | ||
+ | |||
+ | Workers currently configured to run maximum **1 hour**. If you want to change that, you can edit the submission scripts of workers in '' | ||
+ | |||
+ | ==== Installing additional Python modules ==== | ||
+ | Additional Python modules can be installed via the terminal and the Python package manager " | ||
+ | |||
+ | By default the Internet is not accessible within the Notebook, in order to install or download from Internet, you need to use Proxy by exporting following environment variables: | ||
+ | <code bash> | ||
+ | export http_proxy=" | ||
+ | Afterwards | ||
+ | <code bash> | ||
+ | installs a new module in the home directory. | ||
+ | |||
+ | The installation of large Python modules like " | ||
+ | |||
+ | <code bash> | ||
+ | mkdir -v ~/ | ||
+ | TMPDIR=~/ | ||
+ | </ | ||
+ | |||
+ | You also can use self defined kernels and install conda environments on non-parallel notebook. Please refer to [[en: | ||
+ | |||
+ | ===== Running your own Container ===== | ||
+ | You can build your own Apptainer container with Jupyter notebook and run it. It will allow you to not only be independent of our Jupyter notebooks, but also easily spawn the same notebook in your local environment for development or tests. | ||
+ | | ||
+ | Here is an example of the Apptainer container you might use for your own Jupyter notebook: | ||
+ | | ||
+ | <code bash> | ||
+ | Bootstrap: docker | ||
+ | From: condaforge/ | ||
+ | |||
+ | %post | ||
+ | addgroup --system --gid 300 slurmadmin | ||
+ | adduser --system --uid 300 --gid 300 --home / | ||
+ | --shell /bin/false slurmadmin | ||
+ | | ||
+ | conda install --quiet --yes \ | ||
+ | ' | ||
+ | ' | ||
+ | ' | ||
+ | ' | ||
+ | </ | ||
+ | More example recipes can also be found at [[https:// | ||
+ | | ||
+ | You can extend that notebooks as much as you want. The important thing is that the '' | ||
+ | | ||
+ | The commands involving '' | ||
+ | | ||
+ | Previously, you had to install [[https:// | ||
+ | |||
+ | Starting with the rev/23.12 software release, you can use the '' |