Generating Jupyter/Colab notebooks

The nbgenerate tool converts DevSite HTML and Markdown content and templates into Colab-compatible Jupyter notebooks.

OverView

From the Google Colab site:

Colab notebooks allow you to combine executable code and rich text in a single document, along with images,HTML, LaTeX and more.

In Colab, Python code can be written and executed from a web browser with no additional setup or installation. Markdown content can be interspersed with code cells to allow rich content to be provided in the same document. This makes Colab notebooks a popular solution not only for quantitative research and machine learning but also for demonstrating and experimenting with APIs and concepts like those typically found in developer documentation.

For this reason, content shared in Colab notebooks often overlaps with content published in Google developer documentation. The nbgenerate tool eliminates the need to manually sync DevSite markup with supporting Colab notebooks. The tool converts DevSite HTML and Markdown to the .ipynb file format used by Colab and other Jupyter notebook implementations. This allows DevSite content to be used as a single source of truth for both developer docs and Colab notebooks.

Background and technical details can be found in the original design doc at nbgenerate-dd.

Basic usage

To generate Colab notebooks for pages in your DevSite project, you'll need to add a file called _notebooks.yaml before invoking the nbgenerate command.

Adding _notebooks.yaml to your project

To generate Colab notebooks from pages in a DevSite project, add a file called _notebooks.yaml to the project root (in the same directory as your site's _project.yaml). The file should include one entry for each page being shared as a Colab notebook. The source field of each entry must contain the absolute path within your DevSite tenant to be converted into a Colab notebook. The target field must specify the google3 depot path where generated Colab notebooks will be written.

For example:

- source: /earth-engine/ee-image.html
  target: //depot/google3/third_party/earthengine_community/ipynb/ee-image.ipynb
- source: /earth-engine/guides/image_objects.md
  target: //depot/google3/third_party/earthengine_community/ipynb/guides/image_objects.ipynb

This example uses target paths that are mirrored to GitHub using Copybara so that the generated notebooks can be opened via publicly accessible hyperlinks. See copybara for details on how to sync google3 paths to GitHub. See also Widget examples for examples which add links to generated notebooks in GitHub.

Generating notebooks

Before running nbgenerate for the first time, define an alias in your .bashrc and to your current shell:

alias nbgenerate="/google/bin/releases/devsite/tools/nbgenerate/nbgenerate.par"

You can then run nbgenerate from the command line to regenerate all files listed in _notebooks.yaml:

nbgenerate googledata/devsite/content/en/PROJECT

Specify one or more paths as arguments to nbgenerate to restrict processing to the specified files or paths, for example:

nbgenerate third_party/devsite/developers/en/earth-engine/guides

Directory paths are processed recursively; any files in _notebooks.yaml at or below the specified directory will be processed.

If you receive "Permission denied" when running nbgenerate using the above alias, you may also run the tool with the following command:

blaze run //devsite/tools/nbgenerate -- googledata/devsite/content/en/PROJECT

Filtering content

In cases where some content should only appear either in DevSite pages or in the generated Colab notebook but not both, you can filter content using the is_ipynb variable, which is set to True during the nbgenerate conversion process. To prevent errors when staging and publishing to DevSite, add the following near the top of pages that need to selectively include content:

    
{% if is_ipynb is not defined %}{% set is_ipynb=False %}{% endif %}
     

You can then use Jinja tags to control which content appears in Colab, DevSite, or both. For example:

# Overview

This section will show in both the DevSite page and in the Colab notebook as
a text cell.


{% if is_ipynb %}

This content will only appear in Colab notebooks but not on DevSite pages. This
is useful for providing Colab-specific initialization and setup instructions or
code.
{% endif %}
 


{% if not is_ipynb %}
```js
// This code snippet will only be rendered in DevSite but not in Colab
// notebooks.
console.log('Hello, nbgenerate!');
```
{% endif %}
 

```py
# This snippet will appear in both DevSite and in the Colab notebook as a code
# cell.
print('Hello, nbgenerate!');
```

Widget examples

Sites frequently define their own DevSite widgets to encapsulate common behaviors. Below are some examples of widgets that use the is_ipynb variable, which may be adapted to your site.

Colab notebook buttons widget

Earth Engine developer documentation uses a custom widget to consistently display Run in Colab and View source on GitHub buttons on DevSite pages synced to Colab notebooks in GitHub.

Sample section widget

The sample section widget provides an example of how to set up tabbed sections for code snippets presented in multiple programming languages or environments. The widget takes care of only including Python snippets in Colab notebooks. It also removes markup used to display the tabbed sections when writing the notebook.