CV4GS - Jupyter Notebook tutorial

In this tutorial, we will cover:

  • Basic Jupyter notebook aspects: cell types, kernel, workflow, shortcuts
  • Markdown syntax
  • Magic commands to expand notebook functionalities
  • Widgets
  • Miscelaneous: converting notebooks, tunneling through SSH

Table of Contents

  • 1  What is a Jupyter Notebook?
    • 1.1  Code cell
    • 1.2  Markdown cell
    • 1.3  Jupyter keyboard shortcuts
  • 2  Markdown syntax
  • 3  IPython Magic commands
  • 4  Excecute system command
  • 5  Run code from a different kernel
  • 6  Widgets
  • 7  Convert notebook
  • 8  Tunneling through SSH

What is a Jupyter Notebook?¶

  • a notebook is made up of cells. There are 3 main types of cells:
    • Code cells
      ⇒ supports "Julia", "Python", or "R" languages (=Julia, Python, R )
      ⇒ press esc+y to turn cell into code cell
    • Markdown cells
      ⇒ uses the "Markdown" language to write text, add images, equations, etc.
      ⇒ press esc+m to turn cell into markdown cell
    • Raw cells (rarely used)
      ⇒ plain text
      ⇒ press esc+r to turn cell into raw cell
  • the notebook kernel is a "computational engine" that executes the code contained in a Notebook. Here we set a "Python 3" to excecute python code. (Kernels for many other languages exist).

Code cell¶

The cell below is a Code cell (default type) with Python code.

Press shift + return to execute it. The result will get rendered beneath the cell.

In [1]:
x = 1
y = 2
print(x+y)
3

Variables are shared between cells. Try executing the cell below:

In [2]:
print(y + 2)
4
Note: Jupyter notebooks are expected to be run from top to bottom, so executing cells out of order can result in errors as variables may need undefined.

Figures will be plotted in the cell:

In [40]:
from matplotlib import pyplot as plt
x = [1,2,3,4,5]
y = [6,7,8,9,10]
plt.plot(x, y)
Out[40]:
[<matplotlib.lines.Line2D at 0x7f5f50fc29a0>]

Markdown cell¶

Markdown is a lightweight markup language* for creating formatted text.

"A markup language is a system for annotating a document in a way that is syntactically distinguishable from the text, meaning when the document is processed for display, the markup language is not shown, and is only used to format the text." (wikipedia)

Examples: HTML, LaTeX, RTF, etc.

Key design goal of Markdown language is readability, ommiting obvious tags and formatting instructions like those used by HTML, RTF, etc.

Important: Press "M" key to convert cell into "Markdown cell"

Jupyter keyboard shortcuts¶

  • Esc & Enter: switch to command or edit mode
  • m => switch cell to markdown type
  • y => switch cell to code type

  • Shift-Enter: execute cell, jump to cell below (or create a new one if last cell is selected)

  • Ctrl-Enter: execute cell, insert new cell below

  • b => add below the current one (Note you have to be in esc mode.)

  • a => add above the current one (Note you have to be in esc mode.)
  • d+d => delete cell (Note you have to be in esc mode.)
  • o => hide/show cell output
  • ctrl+shift+minus: split cell

  • shift+tab: open documentation of a function

  • f: find text in cell/notebook (Note you have to be in esc mode.)
  • alt+c: comment code (if 'comment-uncomment' jupyter extension is enabled)
  • ctrl+\: comment code (american keyboard layout)
  • ctrl+}: comment code (latin american keyboard layout)


  • 0+0 => restart notebook kernel the current one (Note you have to be in esc mode.)
  • ctrl+s => save notebook

Markdown syntax¶

The human-readable syntax allows to format text in many ways: make headings, emphasize text (bolding, italicizing, ...), creating numered/bulleted lists, add links, format mathematical symbols, make tables, etc.

Syntax reference: link

IPython Magic commands¶

Magic commands are special commands which add special functionalities to the jupyter notebook interface.
Below is a selection of some useful commands.

lsmagic¶

List currently available magic functions.
In [37]:
%lsmagic
Out[37]:
Available line magics:
%alias  %alias_magic  %autoawait  %autocall  %automagic  %autosave  %bookmark  %cat  %cd  %clear  %colors  %conda  %config  %connect_info  %cp  %debug  %dhist  %dirs  %doctest_mode  %ed  %edit  %env  %gui  %hist  %history  %killbgscripts  %ldir  %less  %lf  %lk  %ll  %load  %load_ext  %loadpy  %logoff  %logon  %logstart  %logstate  %logstop  %ls  %lsmagic  %lx  %macro  %magic  %man  %matplotlib  %mkdir  %more  %mv  %notebook  %page  %pastebin  %pdb  %pdef  %pdoc  %pfile  %pinfo  %pinfo2  %pip  %popd  %pprint  %precision  %prun  %psearch  %psource  %pushd  %pwd  %pycat  %pylab  %qtconsole  %quickref  %recall  %rehashx  %reload_ext  %rep  %rerun  %reset  %reset_selective  %rm  %rmdir  %run  %save  %sc  %set_env  %store  %sx  %system  %tb  %time  %timeit  %unalias  %unload_ext  %who  %who_ls  %whos  %xdel  %xmode

Available cell magics:
%%!  %%HTML  %%SVG  %%bash  %%capture  %%debug  %%file  %%html  %%javascript  %%js  %%latex  %%markdown  %%perl  %%prun  %%pypy  %%python  %%python2  %%python3  %%ruby  %%script  %%sh  %%svg  %%sx  %%system  %%time  %%timeit  %%writefile

Automagic is ON, % prefix IS NOT needed for line magics.

run file¶

Execute a python script.
In [ ]:
%run <my_script.py>

load file¶

Insert code from an external file into cell.
In [ ]:
%load <my_script.py>

writefile¶

Export the contents of a cell/Show the contents of an external script
In [14]:
%%writefile pythoncode.py 

import numpy
a = 1
b = 2

def add(x,y)
    return x+y

c = add(a, b)
Writing pythoncode.py

time¶

Time the execution of the code. 
In [42]:
%time

import random
for i in range(0, 1000000):
    random.random()
CPU times: user 0 ns, sys: 0 ns, total: 0 ns
Wall time: 8.11 µs

matplotlib display¶

Select how to display matlpotlib graphs.

List the available backends:
Note: not all will work, it depends on your installation

In [48]:
%matplotlib --list
Available matplotlib backends: ['tk', 'gtk', 'gtk3', 'wx', 'qt4', 'qt5', 'qt', 'osx', 'nbagg', 'notebook', 'agg', 'svg', 'pdf', 'ps', 'inline', 'ipympl', 'widget']

Plot static figure embedded in notebook (default setting):

In [6]:
%matplotlib inline
In [7]:
from matplotlib import pyplot as plt
x = [1,2,3,4,5]
y = [6,7,8,9,10]
plt.plot(x, y)
Out[7]:
[<matplotlib.lines.Line2D at 0x7ff98343f2e0>]

Plot interactive figure:

In [2]:
%matplotlib notebook
In [ ]:
from matplotlib import pyplot as plt
x = [1,2,3,4,5]
y = [6,7,8,9,10]
plt.plot(x, y)

autoreload modules¶

Reload modules before executing user code.

%autoreload 0 => Disable automatic reloading.
%autoreload 1 => Reload all modules imported every time before executing the Python code typed.
%autoreload 2 => Reload all modules every time before executing the Python code typed
%aimport      => List modules which are to be automatically imported or not to be imported.
In [ ]:
%load_ext autoreload
%autoreload 2
In [ ]:
from foo import some_function
In [ ]:
# Run function
some_function()
In [ ]:
# Open foo.py in an editor and change some_function to return 43
In [ ]:
# Run function after edit => will reflect the changes and return 43
some_function()

cython¶

Note: the following example requires to have the cython package installed

  • Cython is Python with C data types.
  • Almost any Python code is also valid Cython
  • Cython will compile Python to C (and then to machine code)
  • Easily mix C and Python code
  • Easy writing of wrappers for C code
  • use cdefs and C types to type your code (only allowed at top level indentation)
In [27]:
# --- Python implementation of primes():

def primes():
    result = []
    p = [0] * 1000
    kmax = 1000
    k = 0
    n = 2
    while k < kmax:
        i = 0
        while i < k and n % p[i] != 0:
            i = i + 1
        if i == k:
            p[k] = n
            k = k + 1
            result.append(n)
        n = n + 1
    return result
In [28]:
%timeit primes()
58.4 ms ± 1.13 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [ ]:
# --- Cython implementation of primes():
In [21]:
%load_ext Cython
In [29]:
%%cython

def primes():
    cdef int n, k, i #<--- defines integer variables
    cdef int p[1000] #<--- defines integer array
    result = []

    cdef int kmax = 1000 #<---
    k = 0
    n = 2
    while k < kmax:
        i = 0
        while i < k and n % p[i] != 0:
            i = i + 1
        if i == k:
            p[k] = n
            k = k + 1
            result.append(n)
        n = n + 1
    return result
In [30]:
%timeit primes()
2.16 ms ± 3.85 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Excecute system command¶

Example: list files in working directory

In [ ]:
!ls -la 

Example: test python version, and cython package version

In [ ]:
!python --version
In [ ]:
!conda list | grep cython

Example: install additional package

In [ ]:
!conda install <package_name>
!pip install <package_name>

Run code from a different kernel¶

IPython can run various Kernel in a cell: bash, HTML, ruby, perl, python2, python3, etc.

Example: run bash code

In [6]:
%%bash
for i in {1..5}
do echo "i = $i"
done
i = 1
i = 2
i = 3
i = 4
i = 5
In [23]:
%%python2
for i in xrange(5):
    print i
0
1
2
3
4
In [26]:
%%python3
for i in range(5):
    print(i)
0
1
2
3
4

Widgets¶

See: https://github.com/jupyter-widgets/tutorial

In [12]:
import ipywidgets as widgets

slider¶

In [23]:
slider = widgets.FloatSlider(
    value=7.5,
    min=5.0,
    max=10.0,
    step=0.1,
    description='Input:',
)

slider
FloatSlider(value=7.5, description='Input:', max=10.0, min=5.0)
In [24]:
print(slider.value)
7.5

link widgets¶

EX: set slider, and display value in a text box.

In [22]:
# Create text box to hold slider value
text = widgets.FloatText(description='Value')

# Link slider value and text box value
widgets.link((slider, 'value'), (text, 'value'))

# Put them in a vertical box
widgets.VBox([slider, text])
VBox(children=(FloatSlider(value=8.3, description='Input:', max=10.0, min=5.0), FloatText(value=8.3, descripti…

widget interaction with function¶

In [26]:
def f(x):
    print(x * x)
In [27]:
widgets.interact(f, x=(0, 100));
interactive(children=(IntSlider(value=50, description='x'), Output()), _dom_classes=('widget-interact',))
In [28]:
slider = widgets.FloatSlider(
    value=7.5,
    min=5.0,
    max=10.0,
    step=0.1,
    description='Input:',
)

# Create non-editable text area to display square of value
square_display = widgets.HTML(description="Square: ", value='{}'.format(slider.value**2))

# Create function to update square_display's value when slider changes
def update_square_display(change):
    square_display.value = '{}'.format(change.new**2)
    
slider.observe(update_square_display, names='value')

# Put them in a vertical box
widgets.VBox([slider, square_display])
VBox(children=(FloatSlider(value=7.5, description='Input:', max=10.0, min=5.0), HTML(value='56.25', descriptio…

other Jupyter widget frameworks¶

  • bqplot - 2d plotting library in which everything displayed is a widget
  • ipympl - widget backend for matplotlib graphics
  • pythreejs - low-level 3d graphics library
  • ipyvolume - 3d plotting and volume rendering
  • ipyleaflet_ - interactive maps
  • ipywebrtc - video streaming
  • ipysheet - interactive spreadsheets
  • ipytree - tree for viewing hierarchical material
  • ipycanvas - interactive drawing in a notebook
  • ipyevents - capture mouse/keyboard events on a widget
  • ...

Installation example (from terminal using Anaconda's package manager):

$ conda install -c conda-forge bqplot

Example: binary star simulation dashboard using ipywidgets, pythreejs, and bqplot: https://github.com/JuanCab/AstroInteractives

Convert notebook¶

Notebooks can be converted to a number of formats, using the command from terminal: jupter nbconvert --to <desired format> <notebook.ipynb>

Conversion to slides¶

  1. In notebook:
    => View > Cell toolbar > Slideshow
  2. Select which cells should become slide, using the dropdown menu on the top right of each cell
  3. Create slideshow from notebook running following command in terminal:
    $ jupyter nbconvert --to slides mynotebook.ipynb
    
    => a distinc file called "mynotebook.slides.html" will be generated in the same directory

Connection to remote notebooks¶

1. connection using vscode¶

=> install extension "Remote - SSH"
=> https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh

2. connection without vscode: tunneling through SSH¶

  1. Log into remote computer through SSH
    $ ssh user@ip
    
  2. In remote computer launch Jupyter with the --no-browser option:
    $ jupyter notebook --no-browser --port=8889
    
  3. Open a second SSH connection with the remote computer, using the tunneling port specified above:
    $ ssh -L localhost:8888:localhost:8889 user@ip
    
  4. Open a webrowser from your local computer and open Jupyter
     http://localhost:8888/tree/