Skip to content

Hydra Cluster: Jupyter Notebook

Introduction

This document walks through the basics of running a Jupyter Notebook on the Hydra cluster.

Creating the environment

We have created a script to make this easier. To create the environment simply run the command hydra-jupyter-create. This will also be run automatically when trying to use the Notebook, so this step could be skipped.

tdb@hydra:~$ hydra-jupyter-create
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: pip in ./hydra-jupyter/lib/python3.10/site-packages (22.0.2)
Collecting pip
  Downloading pip-24.1.2-py3-none-any.whl (1.8 MB)
     ━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 67.6 MB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 22.0.2
    Uninstalling pip-22.0.2:
      Successfully uninstalled pip-22.0.2
Successfully installed pip-24.1.2
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting notebook
  Downloading notebook-7.2.1-py3-none-any.whl.metadata (10 kB)
...
Successfully installed anyio-4.4.0 argon2-cffi-23.1.0 argon2-cffi-bindings-21.2.0 arrow-1.3.0 asttokens-2.4.1 async-lru-2.0.4 attrs-23.2.0 babel-2.15.0 beautifulsoup4-4.12.3 bleach-6.1.0 cffi-1.16.0 charset-normalizer-3.3.2 comm-0.2.2 debugpy-1.8.2 decorator-5.1.1 defusedxml-0.7.1 exceptiongroup-1.2.1 executing-2.0.1 fastjsonschema-2.20.0 fqdn-1.5.1 h11-0.14.0 httpcore-1.0.5 httpx-0.27.0 ipykernel-6.29.5 ipython-8.26.0 isoduration-20.11.0 jedi-0.19.1 json5-0.9.25 jsonschema-4.22.0 jsonschema-specifications-2023.12.1 jupyter-client-8.6.2 jupyter-core-5.7.2 jupyter-events-0.10.0 jupyter-lsp-2.2.5 jupyter-server-2.14.1 jupyter-server-terminals-0.5.3 jupyterlab-4.2.3 jupyterlab-pygments-0.3.0 jupyterlab-server-2.27.2 matplotlib-inline-0.1.7 mistune-3.0.2 nbclient-0.10.0 nbconvert-7.16.4 nbformat-5.10.4 nest-asyncio-1.6.0 notebook-7.2.1 notebook-shim-0.2.4 overrides-7.7.0 packaging-24.1 pandocfilters-1.5.1 parso-0.8.4 prometheus-client-0.20.0 prompt-toolkit-3.0.47 psutil-6.0.0 pure-eval-0.2.2 pycparser-2.22 python-dateutil-2.9.0.post0 python-json-logger-2.0.7 pyzmq-26.0.3 referencing-0.35.1 requests-2.32.3 rfc3339-validator-0.1.4 rfc3986-validator-0.1.1 rpds-py-0.18.1 send2trash-1.8.3 sniffio-1.3.1 soupsieve-2.5 stack-data-0.6.3 terminado-0.18.1 tinycss2-1.3.0 tomli-2.0.1 tornado-6.4.1 traitlets-5.14.3 types-python-dateutil-2.9.0.20240316 typing-extensions-4.12.2 uri-template-1.3.0 webcolors-24.6.0 webencodings-0.5.1 websocket-client-1.8.0

Running a Jupyter Notebook

You'll neeed to decide what resources this notebook needs. For a simple case we'll allocate a single GPU, but you may want to change the GPU type or count, or add additional memory.

We recommend running this with srun, and ensure that you exit the process with Control-C when you're finished.

tdb@hydra:~$ srun -p gpu --gres gpu:1 hydra-jupyter
[I 2024-07-08 12:59:22.281 ServerApp] Extension package jupyter_lsp took 1.0668s to import
[I 2024-07-08 12:59:22.531 ServerApp] Extension package jupyter_server_terminals took 0.2493s to import
[I 2024-07-08 12:59:25.023 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2024-07-08 12:59:25.031 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2024-07-08 12:59:25.037 ServerApp] jupyterlab | extension was successfully linked.
[I 2024-07-08 12:59:25.043 ServerApp] notebook | extension was successfully linked.
[I 2024-07-08 12:59:33.713 ServerApp] notebook_shim | extension was successfully linked.
[I 2024-07-08 12:59:34.334 ServerApp] notebook_shim | extension was successfully loaded.
[I 2024-07-08 12:59:34.337 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2024-07-08 12:59:34.340 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2024-07-08 12:59:34.359 LabApp] JupyterLab extension loaded from /home/cur/tdb/hydra-jupyter/lib/python3.10/site-packages/jupyterlab
[I 2024-07-08 12:59:34.359 LabApp] JupyterLab application directory is /cluster/home/cur/tdb/hydra-jupyter/share/jupyter/lab
[I 2024-07-08 12:59:34.377 LabApp] Extension Manager is 'pypi'.
[I 2024-07-08 12:59:34.425 ServerApp] jupyterlab | extension was successfully loaded.
[I 2024-07-08 12:59:34.428 ServerApp] notebook | extension was successfully loaded.
[I 2024-07-08 12:59:34.429 ServerApp] Serving notebooks from local directory: /cluster/home/cur/tdb
[I 2024-07-08 12:59:34.429 ServerApp] Jupyter Server 2.14.1 is running at:
[I 2024-07-08 12:59:34.429 ServerApp] http://ampere02.hydra.kent.ac.uk:8888/tree?token=alongandsecrettoken
[I 2024-07-08 12:59:34.429 ServerApp]     http://127.0.0.1:8888/tree?token=alongandsecrettoken
[I 2024-07-08 12:59:34.429 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2024-07-08 12:59:34.459 ServerApp]

    To access the server, open this file in a browser:
        file:///cluster/home/cur/tdb/.local/share/jupyter/runtime/jpserver-3048135-open.html
    Or copy and paste one of these URLs:
        http://ampere02.hydra.kent.ac.uk:8888/tree?token=alongandsecrettoken
        http://127.0.0.1:8888/tree?token=alongandsecrettoken
[I 2024-07-08 12:59:34.844 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server

Look for the URL containing .hydra.kent.ac.uk a few lines from then end and copy and paste this whole URL in to your browser. Make sure to copy the token too. You will need the Kent VPN connected if you're off campus.

By default the Notebook will open in whichever directory you ran the command from.

When you're finished please press Control-C to terminate the Notebook. Your files will not be lost, but the Notebook shouldn't be left running. Multiple presses of Control-C may be needed to return to the hydra command prompt.

Adding additional packages

You can add additional packages by running the relevant pip command. The easiest way to do this is within the Notebook interface by launching a console. Alternatively, you can do this from the shell:

tdb@hydra:~$ source ~/hydra-jupyter/bin/activate
(hydra-jupyter) tdb@hydra:~$ pip install some_package_name

Resetting the environment

To start completely from scratch simply remove the hydra-jupyter directory and start again. This won't affect your files, just the installation of Jupyter Notebook. Be very careful typing this command:

tdb@hydra:~$ rm -Rf ~/hydra-jupyter

As usual, please contact us with any queries.