sphinx_extensions

Here we document the parts of the code that can be used as sphinx extensions. These sphinx extensions have been created to support the documentation build workflow for this project and related repositories.

Warning

In the future, this module is subject to being factored out from this project since it could benefit the sphinx and jupyter_sphinx community.

notebook_to_jupyter_sphinx

A sphinx extension that converts python Jupyter notebook scripts .rst.py (or .rst.*.py) in the percent format to .rst (.rst.*) files to be executed by sphinx.

The extension purpose is to minimize the required overhead for writing and modifying executable tutorials.

The rationale is to keep things as simple as possible and as easy to debug as possible:

• The code cells are converted into .. jupyter-execute:: rst directives.

• Raw cells are copy-pasted directly, therefore, they should contain rst contents only.

• Cells in markdown format are ignored.

• The generated .rst output files are written to disk for easy inspection. Note that any problems with the rst text will be flagged by sphinx as coming from the output file of this extension. But you are able to insect it to identify the issue (and correct it in the notebook itself!).

Known alternative

An alternative to this extensions is to use nbsphinx in combination with jupytext. It has neat features, e.g. correctly pointing to the .py source file. However, nbsphinx has some limitations and potentially complicated-to-install dependencies (like pandoc). Such limitations include:

• It is not possible to insert notebook cells inside rst directives for example inside a drop-down .. note:: directive.

• Specifying that a raw cell is to be interpreted as rst is tricky and does not seem to be supported in Jupyter Lab.

Usage

1. Create a Jupyter notebook in the percent format with an extra suffix .rst.py, or .rst.*.py (e.g. .rst.txt.py). The extra suffix is necessary in order to collect the files that are to be converted. The percent format allows to keep the scripts compatible with IPyhton, Jupyter and most IDEs.

   Tip

   You can start from the .rst.py percent-formatted Notebook template and sync it with an .ipynb notebook if you wish.

   This is achieved, e.g., with the jupytext extension for Jupyter Lab (pre-installed on recent versions). Open the Jupyter Lab’s Command Palette and start typing “Pair”. The Jupytext commands should show up.

   Tip

   The .rst.*.py extensions, e.g. .rst.txt.py, will preserve its extension(s). This is supported in order to be able to produce rst files that are ignored by sphinx and can be .. include::d in other parts of the project. For example, you might want to keep long code example in a separate directory instead of including everything directly inside a docstring of a class. This makes it also easier to modify examples without having to build the docs in order to test that the examples work.

   The rest of the documentation below applies equally for .rst.py and .rst.*.py, even though the latter is not mentioned explicitly for simplicity.

2. Version control only the .rst.py file. Do not commit the .rst nor the .ipynb files.

   Tip

   To ensure this in a git repository add the following to your .gitignore file:

   *.rst.ipynb
   *.py.rst
   *.py.rst.txt
   

   Tip

   When switching between git branches you might need to clean up all the generated *.rst files. You canuse the following unix commands (or integrate them in the Makefile your project).

   $ find . -iname "*.py.rst" -exec rm -f -i {} +
   $ find . -iname "*.py.rst.txt" -exec rm -f -i {} +
   

   Remove the -i option to remove files without confirmation.

3. Add this extension to your sphinx conf.py file.

   extensions = [
       # ...,
       "quantify_core.sphinx_extensions.notebook_to_jupyter_sphinx",
   ]
   

4. Add the .rst.py file(s) in the same location where you would like the .rst output file(s) to be generated.

5. Add the file(s) to a table of contents as you would usually do for normal .rst file(s). Mind that you do not need to specify the file extension, however, if you do, it must be .rst (and not .rst.py!).

6. Every time the docs are built by sphinx, the .rst file(s) corresponding to all the .rst.py file(s) will be generated under the same directory with the same name. This step will be executed right after sphinx loads its settings from the conf.py file.

   Note

   This extension will not process all .rst.py files but will only write to disk the files that result in different contents compared to the contents of the existing .rst file. Since sphinx is efficient and does not process files that have not changed, this speeds up the development time.

   If you are updating the code that is used in the notebooks you might want to force the rebuild of the .rst files by adding (temporarily) to the conf.py:

   # ...
   notebook_to_jupyter_sphinx_always_rebuild = True
   # ...
   

Code cells configuration magic comment

Sometimes it is necessary to pass some configuration options to this extension in order for it to produce the indented output from code cells. To achieve this a magic comment is used, currently supporting two configuration keys. The configuration is a dictionary that will be parsed as json. In addition a python dictionary with specific name can be defined on the first line of the cell. This can be handy to detect any typos and support IDE autocomplete.

Note

An experienced reader might suggest using the metadata of cells for this task, which is a more “clean” way of storing this information. Nonetheless, it would be more difficult for non-experienced users to understand it and edit the “hidden” metadata of a cell in a notebook environment.

rst_conf = {"indent": "    ", "jupyter_execute_options": [":hide-output:"]}

# ... the rest of the python code in the cell...

OR

# rst-json-conf: {"indent": "    ", "jupyter_execute_options": [":hide-output:"]}

# ... the rest of the python code in the cell...

The "indent" entry specifies the indentation of the .. jupyter-execute:: block produced. You will need this when you intended the block to be included, e.g., inside a .. note::. You might argue that you could just indent the code in the cell instead, which works in, e.g., Jupyter Lab, however the .rst.py file will become an invalid python file, confuse auto formatters and linters, etc..

The "jupyter_execute_options" entry is a list of directive options that will be placed on the line below the .. jupyter-execute::.

The above example will produce the following in the .rst file :

.. jupyter-execute::
    :hide-output:

    # ... the rest of the python code in the cell...

Tip

If you wan to suppress the output of a final line in a notebook cell you could usually use a ;. However, if you use a python auto formatter like black in the repository, it will get removed. To achieve the same effect assign the output of the last line of a cell to the _ variable. E.g., _ = plt.plot(...). You can read more about this python feature here.

Potential enhancements

The extension could be enhanced in a few ways:

• Include the raw rst cells in the notebooks that jupyter_sphinx allows to download.

• Make the “View page source”/”Edit on GitHub/GitLab” point to the .rst.py script instead of the .rst.

• A Jupyter Lab or browser extension for rst code highlighting (see limitation below).

• Support for using markdown cells directly with conversion to .rst using a tool like MYST.

Known limitations

Code highlighting in Jupyter Lab

Unfortunately it seems that it is not possible to make Jupyter Lab highlight the rst code in the (raw) cells of a notebook, which would be useful for this extension. There are some workarounds for Jupyter Notebook involving cell magics but it is not quite worth the effort.

Notebook template

To make use of this extensions you can start from this template.rst.py.

# ---
# jupyter:
#   jupytext:
#     cell_markers: \"\"\"
#     formats: py:percent
#     text_representation:
#       extension: .py
#       format_name: percent
#   kernelspec:
#     display_name: Python 3 (ipykernel)
#     language: python
#     name: python3
# ---

# %% [raw]
"""
The contents of this raw cell will be copy-pasted into the ``.rst`` file.
"""

# %%
# This is a code cell, will be translated into a `.. jupyter-execute::` block.
assert 1+1 == 2

Place it in the desired location, rename it and navigate to its location using file browser in Jupyter Lab. Then right-click the file and under the Open With select Notebook. Note that you need a relatively recent version of Jupyter Lab for this to already be part of the Jupyter Lab interface by default (if not consult the jupytext documentation).

The cell_markers in the header of the template tells jupytext to store the contents of raw notebook cells in the .rst.py files inside blocks that look like this:

# %% [raw]
"""
Raw cell contents
goes here
"""

Instead of the default:

# %% [raw]
# Raw cell contents
# goes here

You can remove that line if you wish to use the default representation.

API

cell_to_rst_str(cell, is_first_cell=False, rst_indent='    ')

Converts a notebook cell dict according to its type (raw or code).

Parameters

• cell (dictdict) – Cell dict object from the notebook file.

• is_first_cell (boolbool (default: False)) – Indicates if it is the first cell in the notebook file. Used to avoid inserting undesired blank lines.

• rst_indent (strstr (default: '    ')) – See make_jupyter_sphinx_block().

Return type

strstr

get_code_indent_and_processed_lines(cell_source_code)

Processes a code cell applying configuration from the magic comment.

Parameters

cell_source_code (strstr) – String containing the code of the cell.

Return type

Tuple[str, List[str]]Tuple[str, List[str]]

make_jupyter_sphinx_block(cell_source_code, rst_indent='    ')

Converts a code cell into rst code under a jupyter-execute directive.

Indentation is applied according to the magic comment.

Note

The contents of the jupyter-execute block require an indentation as well. This one can be set in the conf.py.

E.g., notebook_to_jupyter_sphinx_rst_indent = "    ".

Parameters

• cell_source_code (strstr) – String containing the code of the cell.

• rst_indent (strstr (default: '    ')) – Indentation used to indent the code inside the .. jupyter-execute :: block.

Return type

strstr

make_rst_block(cell_source, prefix='\\n\\n\\n')

Prefixes the raw rst with the prefix.

Parameters

• cell_course – String containing the contents of the raw cell.

• prefix – Prefix to add to cell_source.

Return type

strstr

notebook_to_rst(notebook, rst_indent='    ')

Converts the notebook to an rst string.

Parameters

• notebook (dictdict) – Dict(-like) object of the notebook file.

• rst_indent (strstr (default: '    ')) – See make_jupyter_sphinx_block().

Return type

strstr

notebooks_to_rst(app, config)

Searches for all *.rst.py files and converts them to *.rst files.

The output file will placed in the same directory as the original file.

Parameters

• app – The sphinx app provided by sphinx when calling this function.

• config – The sphinx config provided by sphinx when calling this function.

Return type

NoneNone

setup(app)

Setup the sphinx extension by connecting the converter to one of the events that sphinx emits in the beginning of the docs build execution.