The nbgenerate
tool converts DevSite HTML and Markdown content and templates
into Colab-compatible Jupyter notebooks.
OverView
From the Google Colab site:
Colab notebooks allow you to combine executable code and rich text in a single document, along with images,HTML, LaTeX and more.
In Colab, Python code can be written and executed from a web browser with no additional setup or installation. Markdown content can be interspersed with code cells to allow rich content to be provided in the same document. This makes Colab notebooks a popular solution not only for quantitative research and machine learning but also for demonstrating and experimenting with APIs and concepts like those typically found in developer documentation.
For this reason, content shared in Colab notebooks often overlaps with content published in Google developer documentation. The nbgenerate tool eliminates the need to manually sync DevSite markup with supporting Colab notebooks. The tool converts DevSite HTML and Markdown to the .ipynb file format used by Colab and other Jupyter notebook implementations. This allows DevSite content to be used as a single source of truth for both developer docs and Colab notebooks.
Background and technical details can be found in the original design doc at nbgenerate-dd.
Basic usage
To generate Colab notebooks for pages in your DevSite project, you'll need to
add a file called _notebooks.yaml
before invoking the nbgenerate
command.
Adding _notebooks.yaml to your project
To generate Colab notebooks from pages in a DevSite project, add a file called
_notebooks.yaml
to the project root (in the same directory as your site's
_project.yaml
). The file should include one entry for each page being shared
as a Colab notebook. The source
field of each entry must contain the absolute
path within your DevSite tenant to be converted into a Colab notebook. The
target
field must specify the google3 depot path where generated Colab
notebooks will be written.
For example:
- source: /earth-engine/ee-image.html
target: //depot/google3/third_party/earthengine_community/ipynb/ee-image.ipynb
- source: /earth-engine/guides/image_objects.md
target: //depot/google3/third_party/earthengine_community/ipynb/guides/image_objects.ipynb
This example uses target paths that are mirrored to GitHub using Copybara so that the generated notebooks can be opened via publicly accessible hyperlinks. See copybara for details on how to sync google3 paths to GitHub. See also Widget examples for examples which add links to generated notebooks in GitHub.
Generating notebooks
Before running nbgenerate
for the first time, define an alias in your
.bashrc
and to your current shell:
alias nbgenerate="/google/bin/releases/devsite/tools/nbgenerate/nbgenerate.par"
You can then run nbgenerate
from the command line to regenerate all files
listed in _notebooks.yaml
:
nbgenerate googledata/devsite/content/en/PROJECT
Specify one or more paths as arguments to nbgenerate
to restrict processing to
the specified files or paths, for example:
nbgenerate third_party/devsite/developers/en/earth-engine/guides
Directory paths are processed recursively; any files in _notebooks.yaml
at or
below the specified directory will be processed.
If you receive "Permission denied" when running nbgenerate
using the above
alias, you may also run the tool with the following command:
blaze run //devsite/tools/nbgenerate -- googledata/devsite/content/en/PROJECT
Filtering content
In cases where some content should only appear either in DevSite pages or in the
generated Colab notebook but not both, you can filter content using the
is_ipynb
variable, which is set to True
during the nbgenerate
conversion
process. To prevent errors when staging and publishing to DevSite, add the
following near the top of pages that need to selectively include content:
{% if is_ipynb is not defined %}{% set is_ipynb=False %}{% endif %}
You can then use Jinja tags to control which content appears in Colab, DevSite, or both. For example:
# Overview This section will show in both the DevSite page and in the Colab notebook as a text cell. {% if is_ipynb %} This content will only appear in Colab notebooks but not on DevSite pages. This is useful for providing Colab-specific initialization and setup instructions or code. {% endif %} {% if not is_ipynb %} ```js // This code snippet will only be rendered in DevSite but not in Colab // notebooks. console.log('Hello, nbgenerate!'); ``` {% endif %} ```py # This snippet will appear in both DevSite and in the Colab notebook as a code # cell. print('Hello, nbgenerate!'); ```
Widget examples
Sites frequently define their own DevSite widgets to encapsulate common behaviors. Below are some examples of widgets that use the is_ipynb
variable, which may be adapted to your site.
Colab notebook buttons widget
Earth Engine developer documentation uses a custom widget to consistently display Run in Colab and View source on GitHub buttons on DevSite pages synced to Colab notebooks in GitHub.
Sample section widget
The sample section widget provides an example of how to set up tabbed sections for code snippets presented in multiple programming languages or environments. The widget takes care of only including Python snippets in Colab notebooks. It also removes markup used to display the tabbed sections when writing the notebook.