import nbdev.showdoc

The most important function defined in this module is notebooks2script, so you may want to jump to it before scrolling though the rest, which explain the details behind the scenes of the conversion from notebooks to library. The main things to remember are:

• put # export on each cell you want exported
• put # exports on each cell you want exported with the source code shown in the docs
• put # exporti on each cell you want exported without it being added to __all__, and without it showing up in the docs.
• one cell should contain # default_exp followed by the name of the module (with points for submodules and without the py extension) everything should be exported in (if one specific cell needs to be exported in a different module, just indicate it after #export: #export special.module)
• all left members of an equality, functions and classes will be exported and variables that are not private will be put in the __all__ automatically
• to add something to __all__ if it's not picked automatically, write an exported cell with something like #add2all "my_name"

Basic foundations

For bootstrapping nbdev we have a few basic foundations defined in imports, which we test a show here. First, a simple config file class, Config that read the content of your settings.ini file and make it accessible:

Config[source]

Config(cfg_name='settings.ini')

Reading and writing settings.ini

cfg = Config(cfg_name='settings.ini')
test_eq(cfg.lib_name, 'chisel_nbdev')
test_eq(cfg.git_url, "https://github.com/ucsc-vama/chisel_nbdev")
test_eq(cfg.path("lib_path"), Path.cwd().parent/'chisel_nbdev')
test_eq(cfg.path("nbs_path"), Path.cwd())
# test_eq(cfg.path("doc_path"), Path.cwd().parent/'docs')
# test_eq(cfg.custom_sidebar, 'False')

Reading a notebook

What's a notebook?

A jupyter notebook is a json file behind the scenes. We can just read it with the json module, which will return a nested dictionary of dictionaries/lists of dictionaries, but there are some small differences between reading the json and using the tools from nbformat so we'll use this one.

read_nb[source]

read_nb(fname)

Read the notebook in fname.

fname can be a string or a pathlib object.

test_nb = read_nb('test.ipynb')

The root has four keys: cells contains the cells of the notebook, metadata some stuff around the version of python used to execute the notebook, nbformat and nbformat_minor the version of nbformat.

test_nb.keys()

dict_keys(['cells', 'metadata', 'nbformat', 'nbformat_minor'])

test_nb['metadata']

{'kernelspec': {'display_name': 'Scala', 'language': 'scala', 'name': 'scala'},
 'language_info': {'codemirror_mode': 'text/x-scala',
  'file_extension': '.scala',
  'mimetype': 'text/x-scala',
  'name': 'scala',
  'nbconvert_exporter': 'script',
  'version': '2.12.10'}}

f"{test_nb['nbformat']}.{test_nb['nbformat_minor']}"

'4.4'

The cells key then contains a list of cells. Each one is a new dictionary that contains entries like the type (code or markdown), the source (what is written in the cell) and the output (for code cells).

test_nb['cells'][0]

{'cell_type': 'markdown',
 'metadata': {},
 'source': '# Test Chisel notebook\n> An example that defines a module over multiple notebook cells.'}

is_scala_nb[source]

is_scala_nb(fname)

assert(is_scala_nb('test.ipynb'))
assert(not is_scala_nb('00_export_scala.ipynb'))

Finding patterns

The following functions are used to catch the flags used in the code cells.

check_re[source]

check_re(cell, pat, code_only=True)

Check if cell contains a line with regex pat

pat can be a string or a compiled regex. If code_only=True, this function ignores non-code cells, such as markdown.

cell = test_nb['cells'][2].copy()
# print(cell)
assert check_re(cell, '//export') is not None
assert check_re(cell, re.compile('//export')) is not None
assert check_re(cell, '# bla') is None
cell['cell_type'] = 'markdown'
assert check_re(cell, '//export') is None # don't export markdown
assert check_re(cell, '//export', code_only=False) is not None # unless specified

check_re_multi[source]

check_re_multi(cell, pats, code_only=True)

Check if cell contains a line matching any regex in pats, returning the first match found

cell = test_nb['cells'][2].copy()
cell['source'] = "a b c"
# print(cell)
assert check_re(cell, 'a') is not None
assert check_re(cell, 'd') is None
# show that searching with patterns ['d','b','a'] will match 'b'
# i.e. 'd' is not found and we don't search for 'a'
assert check_re_multi(cell, ['d','b','a']).span() == (2,3)

This function returns a regex object that can be used to find nbdev flags in multiline text

• body regex fragment to match one or more flags,
• n_params number of flag parameters to match and catch (-1 for any number of params; (0,1) for 0 for 1 params),
• comment explains what the compiled regex should do.

is_export[source]

is_export(cell, default)

Check if cell is to be exported and returns the name of the module to export it if provided

is_export returns;

• a tuple of ("module name", "external boolean" (False for an internal export)) if cell is to be exported or
• None if cell will not be exported.

The cells to export are marked with //export///exporti///exports, potentially with a module name where we want it exported. The default module is given in a cell of the form //default_exp bla inside the notebook (usually at the top), though in this function, it needs the be passed (the final script will read the whole notebook to find it).

• a cell marked with //export///exporti///exports will be exported to the default module
• an exported cell marked with special.module appended will be exported in special.module (located in lib_name/special/module.py)
• a cell marked with //export will have its signature added to the documentation
• a cell marked with //exports will additionally have its source code added to the documentation
• a cell marked with //exporti will not show up in the documentation, and will also not be added to __all__.

is_non_def_export[source]

is_non_def_export(cell)

Check if cell is to be exported to special (non-defualt) module returns the name of the module to export

cell = test_nb['cells'][2].copy()
cell['source'] = "// export ModA"
test_eq(is_non_def_export(cell), ('ModA', True))

cell = test_nb['cells'][2].copy()
test_eq(is_export(cell, 'export'), ('export', True))
cell['source'] = "// exports"
test_eq(is_export(cell, 'export'), ('export', True))
cell['source'] = "// exporti"
test_eq(is_export(cell, 'export'), ('export', False))
cell['source'] = "// export mod"
test_eq(is_export(cell, 'export'), ('mod', True))

find_default_export[source]

find_default_export(cells)

Find in cells the default export module.

find_non_default_exports[source]

find_non_default_exports(cells)

Find in cells all non default export modules: //export my_mod_name

test_eq(find_non_default_exports(test_nb['cells']), ['NewScript', 'NewScript2'])

Stops at the first cell containing // default_exp (if there are several) and returns the value behind. Returns None if there are no cell with that code.

test_eq(find_default_export(test_nb['cells']), 'test')
assert find_default_export(test_nb['cells'][3:]) is None

Listing all exported objects

Until now the above code has been verbatim to the 00_export.ipynb from nbdev minus changes to regexes swapping '#' to '//'. Now Scala syntax parsing is required as well as a target build infrastructure (sbt, mill, or Ammonite scripts).

For documentation we need to extract out the names of our classes, objects, methods, etc including:

• def
• object
• class | abstract class |
• case class
• trait
• sealed trait
• abstract
• package
• override
• private?
• implicit
• protected
• import
• extends
• final

export_names[source]

export_names(code)

Find the names of objects, functions or classes defined in code that are exported.

This function finds all of the function/class/object names.

test_eq(export_names("def my_func(x: Int): Unit = {\n\tprint(x)\n}\n class MyClass(){}"), ["my_func", "MyClass"])

Create the library

Saving an index

To be able to build back a correspondence between functions and the notebooks they are defined in, we need to store an index. It's done in the private module _nbdev inside your library, and the following function are used to define it.

reset_nbdev_module[source]

reset_nbdev_module()

Create a skeleton for _nbdev

get_nbdev_module[source]

get_nbdev_module()

Reads _nbdev

save_nbdev_module[source]

save_nbdev_module(mod)

Save mod inside _nbdev

Create the modules

return_type tells us if the tuple returned will contain lists of lines or strings with line breaks.

We treat the first comment line as a flag

split_flags_and_code[source]

split_flags_and_code(cell, return_type=list)

Splits the source of a cell into 2 parts and returns (flags, code)

def _test_split_flags_and_code(expected_flags, expected_code):
    cell = nbformat.v4.new_code_cell('\n'.join(expected_flags + expected_code))
    test_eq((expected_flags, expected_code), split_flags_and_code(cell))
    expected=('\n'.join(expected_flags), '\n'.join(expected_code))
    test_eq(expected, split_flags_and_code(cell, str))
    
_test_split_flags_and_code([
    '//export'],
    ['// TODO: write this function',
    'def func(x) = ???'])

create_mod_file[source]

create_mod_file(fname, nb_path, bare=False)

Create a module file for fname.

A new module filename is created each time a notebook has a cell marked with #default_exp. In your collection of notebooks, you should only have one notebook that creates a given module since they are re-created each time you do a library build (to ensure the library is clean). Note that any file you create manually will never be overwritten (unless it has the same name as one of the modules defined in a #default_exp cell) so you are responsible to clean up those yourself.

fname is the notebook that contained the #default_exp cell.

create_mod_files[source]

create_mod_files(files, to_dict=False, bare=False)

Create mod files for default exports found in files

Create module files for all #default_export flags found in files and return a list containing the names of modules created.

Note: The number if modules returned will be less that the number of files passed in if files do not #default_export.

By creating all module files before calling _notebook2script, the order of execution no longer matters - so you can now export to a notebook that is run "later".

You might still have problems when

• converting a subset of notebooks or
• exporting to a module that does not have a #default_export yet

in which case _notebook2script will print warnings like;

Warning: Exporting to "core.py" but this module is not part of this build

If you see a warning like this

• and the module file (e.g. "core.py") does not exist, you'll see a FileNotFoundError
• if the module file exists, the exported cell will be written - even if the exported cell is already in the module file

update_baseurl[source]

update_baseurl()

Add or update baseurl in _config.yml for the docs

nbglob[source]

nbglob(fname=None, recursive=False)

Find all notebooks in a directory given a glob. Ignores hidden directories and filenames starting with _

assert not nbglob(recursive=True).filter(lambda x: '.ipynb_checkpoints' in str(x))

scala_notebook2script[source]

scala_notebook2script(fname=None, silent=False, to_dict=False, bare=False, recursive=False)

Convert notebooks matching fname to modules

Finds cells starting with #export and puts them into the appropriate module. If fname is not specified, this will convert all notebook not beginning with an underscore in the nb_folder defined in setting.ini. Otherwise fname can be a single filename or a glob expression.

silent makes the command not print any statement and to_dict is used internally to convert the library to a dictionary.