Skip to content

Instantly share code, notes, and snippets.

@brabect1

brabect1/python_apidoc_tools.rst

Last active November 24, 2024 10:46
Show Gist options
  • Save brabect1/ff8f466772099c450fb9381eaaab8588 to your computer and use it in GitHub Desktop.
Save brabect1/ff8f466772099c450fb9381eaaab8588 to your computer and use it in GitHub Desktop.
Download ZIP
#python #api #doc
Raw
python_apidoc_tools.rst

API Doc Generators for Python Code

Contents

Generators

API doc generators are tools that turn code docstrings into documentation, usually in HTML format for cross-linkage.

pydoc

pydoc is a standard Python library used for its help() method. Displays an in-console text documentation, or can run a local webserver and display it as HTML. It supports writing HTML for offline documentation.

HTML output is, though, not too pretty. Nonetheless, it does the job better than nothing until you turn to more capable generators.

# in-console text documentation
# (redirect the output to a file for offline text documentation)
python3 -m pydoc <module-name>

# render HTML doc in local webserver at `http://localhost:1234/`
python -m pydoc -p 1234 <module-name>

# write offline HTML
# (the output `<module-name>.html` is generated in the current directory)
python3 -m pydoc -w <module-name>

pdoc3

Python3 pdoc module distributed through pdoc3 pypi package. It is a fork of an earlier/older pdoc package. Install by pip3 install pdoc3. Requires Python3.7+.

Depends on markdown package as it uses Markdown syntax as its primary text input and output. Supports some RST directives (admonitions, image::, todo::, math:: and few others).

Generates text (Markdown) and HTML. PDFs are generated by converting from Markdown.

Supports Google and numpy docstring styles.

Users can customize the default HTML/CSS template.

# generate text/Markdown documentation
pdoc <module-name> > pdoc-text.md

# generate HTML documentation
pdoc -o pdoc-html -f --html <module-name>

# generate PDF convertible Markdown output
# (use `pandoc` or any other format converter supporting Markdown to PDF)
pdoc -f --pdf <module-name> > pdoc-pdf.md

doxygen

Python is among doxygen supported other languages. It may be specifically helpful for multi-language projects. Generates the usual doxygen-looking documentation in HTML and other supported formats.

As usual with doxygen, the API generation starts with a configuration Doxyfile that configures the doxygen tool and tells it what code (files, directories) to process. This may be somewhat inconvenient compared to other API doc generators but that is the philosophy of this tool, which primarily targets larger, structured projects.

doxygen <path-to-doxyfile>

Sphinx

Sphinx is a generic documentation generator built on reStructuredText (RST) markup/syntax. Specific extensions exist to let Sphinx act as am API doc generator.

Similar to doxygen, Sphinx targets larger projects and uses configuration files to tell it what and how. Keep in mind, though, that Sphinx has never been intended as an API generator and its use for that purpose is unintuitive, at least. The key is to understand that in Sphinx, everything starts from a .rst document. Hence wanting to generate API doc for a python code requires to create a .rst document and through Sphinx's specific directives point to Python code elements (modules, classes, etc.). Those directives would trigger Sphinx plugins, like autodoc, which would in run-time extract docstring information from the identified elements.

For the examples of individual methods, assume the following directory structure (you may take a sample mymodule.py code from here):

+-- doc/
|
+-- example/
    +-- mymodule.py

As Sphinx acts on RST markup, make yout docstrings use valid RST markup. You would need to add the Napoleon extension to support numpy or Google styles. That is, add to conf.py of the sphinx project:

extensions = [
    ...
    'sphinx.ext.napoleon',]

# Napoleon settings
napoleon_google_docstring = True
napoleon_numpy_docstring = True
napoleon_include_init_with_doc = True
napoleon_include_private_with_doc = True
napoleon_include_special_with_doc = True
napoleon_use_admonition_for_examples = False
napoleon_use_admonition_for_notes = False
napoleon_use_admonition_for_references = False
napoleon_use_ivar = False
napoleon_use_param = True
napoleon_use_rtype = True
...

autodoc

All sphinx-based API generators do require setting up a sphinx project and extracting docstrings into RST files that sphinx will then turn into documentation. Using autodoc relies on manual call of sphinx-autodoc to create such RST files.

Steps to generate API doc with autodoc:

  1. Create new Sphinx configuration in doc/:

    cd doc && sphinx-quickstart

  2. Enable autodoc in doc/source/conf.py (assuming you configured with separate source and build folders):

    extensions = [
    'sphinx.ext.autodoc',
]

  3. Generate RST's from python code:

    cd doc && PYTHONPATH=/your/path/to/.../example/ sphinx-autodoc -f -o source/ ../example

tree doc/source
>> doc/source/
>> ├── _static
>> ├── _templates
>> ├── conf.py
>> ├── index.rst
>> ├── modules.rst
>> └── mymodule.rst

  4. Add reference to modules.rst to toctree in index.rst:

    .. toctree::
   :maxdepth: 2
   :caption: Contents:

   modules

  5. Build the documentation:

    cd doc && PYTHONPATH=/your/path/to/.../example/ make html
>> Running Sphinx v8.1.3
>> loading translations [en]... done
>> making output directory... done
>> building [mo]: targets for 0 po files that are out of date
>> writing output...
>> building [html]: targets for 3 source files that are out of date
>> updating environment: [new config] 3 added, 0 changed, 0 removed
>> reading sources... [100%] mymodule
>> looking for now-outdated files... none found
>> pickling environment... done
>> checking consistency... done
>> preparing documents... done
>> copying assets...
>> copying static files...
>> Writing evaluated template result to /home/mint/python-doc/autodoc-test/build/html/_static/language_data.js
>> Writing evaluated template result to /home/mint/python-doc/autodoc-test/build/html/_static/documentation_options.js
>> Writing evaluated template result to /home/mint/python-doc/autodoc-test/build/html/_static/basic.css
>> Writing evaluated template result to /home/mint/python-doc/autodoc-test/build/html/_static/alabaster.css
>> copying static files: done
>> copying extra files...
>> copying extra files: done
>> copying assets: done
>> writing output... [100%] mymodule
>> generating indices... genindex py-modindex done
>> writing additional pages... search done
>> dumping search index in English (code: en)... done
>> dumping object inventory... done
>> build succeeded.
>>
>> The HTML pages are in build/html.

For a more complete example see e.g. https://github.com/cimarieta/sphinx-autodoc-example/tree/master

autosummary

The concept of autosummary is to create autodoc .rst stubs that then, when bulding the documentation with sphinx, would process the Python code docstrings. Autosummary extension adds the autosummary:: directive that generates the stubs, either automatically during the build process, or explicitly using the sphinx-autogen utility.

The steps to generate API doc with autosummary:

  1. Create new Sphinx configuration in doc/:

    cd doc && sphinx-quickstart

  2. Enable autodoc and autosummary in doc/source/conf.py (assuming you configured with separate source and build folders):

    extensions = [
    'sphinx.ext.autodoc',
    'sphinx.ext.inheritance_diagram',
    'sphinx.ext.autosummary'
]

  3. Create a RST file that will identify Python modules to be subjected to autosummary processing, e.g. doc/source/modules.rst. The file would look like this (the key is the autosummary directive that will then be processed by sphinx-autogen):

    Modules
=======

.. autosummary::
   :toctree: modules

   mymodule

  4. Run sphinx-autogen:

    cd doc && PYTHONPATH=/your/path/to/.../example/ sphinx-autogen source/modules.rst
>> [autosummary] generating autosummary for: source/modules.rst
>> [autosummary] generating autosummary for: /home/mint/python-doc/autosummary-test/source/modules/mymodule.rst

cd doc && cat source/modules/mymodule.rst
>> mymodule
>> ========
>>
>> .. automodule:: mymodule
>>
>>    .. rubric:: Module Attributes
>>
>>    .. autosummary::
>>
>>       A_CONSTANT
>>       YET_ANOTHER
>>
>>    .. rubric:: Functions
>>
>>    .. autosummary::
>>
>>       a_function
>>
>>    .. rubric:: Classes
>>
>>    .. autosummary::
>>
>>       AnotherClass
>>       MyClass
>>
>>    .. rubric:: Exceptions
>>
>>    .. autosummary::
>>
>>       MyException

    *Note: This and the preceding steps are only if using sphinx-autogen manually (and included here for illustration of using this option). It would be ok to add the autosummary:: directive into index.rst and build the sphinx documentation directly.

  5. Build the documentation:

    cd doc && PYTHONPATH=/your/path/to/.../example/ make html
>> Running Sphinx v8.1.3
>> loading translations [en]... done
>> making output directory... done
>> [autosummary] generating autosummary for: index.rst, modules.rst, modules/mymodule.rst
>> building [mo]: targets for 0 po files that are out of date
>> writing output...
>> building [html]: targets for 3 source files that are out of date
>> updating environment: [new config] 3 added, 0 changed, 0 removed
>> reading sources... [100%] modules/mymodule
>> looking for now-outdated files... none found
>> pickling environment... done
>> checking consistency... /home/mint/python-doc/autosummary-test/source/modules.rst: WARNING: document isn't included in any toctree
>> /home/mint/python-doc/autosummary-test/source/modules/mymodule.rst: document is referenced in multiple toctrees: ['index', 'modules'], selecting: modules <- modules/mymodule
>> done
>> preparing documents... done
>> copying assets...
>> copying static files...
>> Writing evaluated template result to /home/mint/python-doc/autosummary-test/build/html/_static/language_data.js
>> Writing evaluated template result to /home/mint/python-doc/autosummary-test/build/html/_static/documentation_options.js
>> Writing evaluated template result to /home/mint/python-doc/autosummary-test/build/html/_static/basic.css
>> Writing evaluated template result to /home/mint/python-doc/autosummary-test/build/html/_static/alabaster.css
>> copying static files: done
>> copying extra files...
>> copying extra files: done
>> copying assets: done
>> writing output... [100%] modules/mymodule
>> generating indices... genindex py-modindex done
>> writing additional pages... search done
>> dumping search index in English (code: en)... done
>> dumping object inventory... done
>> build succeeded, 1 warning.
>>
>> The HTML pages are in build/html.

autosummary:: yields generating API overview tables, which list Python element names and their brief summary. If wanting to generate the full API documentation, use the automodule:: directive. It is, of course, possible to combine the two together. Here is an example of a modified index.rst:

.. toctree::
   :maxdepth: 2
   :caption: Contents:

.. automodule:: mymodule
    :members:
    :imported-members:

autoapi

autoapi is another, full automation over autodoc and the best approximation to a Sphinx-based API generator. It still has downside that module authors must manually identify the code elements to be extracted into API doc in the __all__ (preferred) or __api__ lists. For example:

__all__ = [
    'A_CONSTANT',
    'YET_ANOTHER',
    'a_function',
    'MyClass',
    'AnotherClass',
    'MyException'
]

The steps to generate API doc with autoapi:

  1. Create new Sphinx configuration in doc/:

    cd doc && sphinx-quickstart

  2. Enable autodoc and autoapi in doc/source/conf.py (assuming you configured with separate source and build folders):

    extensions = [
    'sphinx.ext.autodoc',
    'sphinx.ext.inheritance_diagram',
    'autoapi.sphinx'
]

  3. Enumerate modules to be processed for API doc extraction in the autoapi_modules dictionary in doc/source/conf.py (see autoapi doc for more details about that dictionary):

    autoapi_modules = {'mymodule': None}

  4. Add reference to RST files to be generated from the identified modules to toctree in index.rst. Note that the RST files do not exist at this point but will be created by autoapi from autoapi_modules.

    .. toctree::
   :maxdepth: 2
   :caption: Contents:

   mymodule/mymodule

  5. Build the documentation:

    cd doc && PYTHONPATH=/your/path/to/.../example/ make html
>> Running Sphinx v8.1.3
>> loading translations [en]... done
>> loading pickled environment... done
>> building [mo]: targets for 0 po files that are out of date
>> writing output...
>> building [html]: targets for 1 source files that are out of date
>> updating environment: 0 added, 1 changed, 0 removed
>> reading sources... [100%] mymodule/mymodule
>> looking for now-outdated files... none found
>> pickling environment... done
>> checking consistency... done
>> preparing documents... done
>> copying assets...
>> copying static files...
>> Writing evaluated template result to /home/mint/python-doc/autoapi-test/build/html/_static/language_data.js
>> Writing evaluated template result to /home/mint/python-doc/autoapi-test/build/html/_static/documentation_options.js
>> Writing evaluated template result to /home/mint/python-doc/autoapi-test/build/html/_static/basic.css
>> Writing evaluated template result to /home/mint/python-doc/autoapi-test/build/html/_static/alabaster.css
>> copying static files: done
>> copying extra files...
>> copying extra files: done
>> copying assets: done
>> writing output... [100%] mymodule/mymodule
>> generating indices... genindex py-modindex done
>> writing additional pages... search done
>> dumping search index in English (code: en)... done
>> dumping object inventory... done
>> build succeeded.
>>
>> The HTML pages are in build/html.

pydoctor

Install through pip3 install pydoctor. Supports various docstring styles, incl. plain text and epytext. Generates good-looking HTML.

There are some neat features, such as automated publishing with Github Action or ReadTheDocs.

For more details see pydoctor's ReadTheDocs.

pydoctor \
  --project-name=my_project \
  --project-version=20.7.2 \
  --html-output=docs/api \
  --docformat=plaintext \
  --intersphinx=https://docs.python.org/3/objects.inv \
  ./my_module.py

docstings

Docstring is simply the API description along to the code. Most programming language can only put docstrings into comments. Python has specific docstrings elements represented by regular strings. PEP 257 covers Python docstring conventions.

Docstrings style represents a syntax to mark or attach specific meaning to parts of a docstring. Typical example is identifying parameters/arguments, data types, return values, exceptions, code snippets, cross-references, TODO's, etc.

Style Examples

  • Google style:

    def x_intercept(m, b):
    """
    Return the x intercept of the line M{y=m*x+b}.  The *x intercept*
    of a line is the point at which it crosses the x axis (``y=0``).

    This function can be used in conjuction with `z_transform` to
    find an arbitrary function's zeros.

    Args:
        m (number): The slope of the line.
        b (number): The y intercept of the line.  The *y intercept* of a
                    line is the point at which it crosses the y axis (``x=0``).

    Returns:
        number: The x intercept of the line ``y=m*x+b``.
    """
    return -b/m
...

  • numpy style:

    ...

  • Sphinx style:

    def x_intercept(m, b):
    """
    Return the x intercept of the line M{y=m*x+b}.  The *x intercept*
    of a line is the point at which it crosses the x axis (:math:`y=0`).

    This function can be used in conjuction with `z_transform` to
    find an arbitrary function's zeros.

    :param m: The slope of the line.
    :type  m: number
    :param b: The y intercept of the line.  The *y intercept* of a
              line is the point at which it crosses the y axis (:math:`x=0`).
    :type  b: number
    :return:  the x intercept of the line :math:`y=m*x+b`.
    :rtype:   number
    """
    return -b/m

  • epytext:

    def x_intercept(m, b):
    """
    Return the x intercept of the line M{y=m*x+b}.  The X{x intercept}
    of a line is the point at which it crosses the x axis (M{y=0}).

    This function can be used in conjuction with L{z_transform} to
    find an arbitrary function's zeros.

    @type  m: number
    @param m: The slope of the line.
    @type  b: number
    @param b: The y intercept of the line.  The X{y intercept} of a
              line is the point at which it crosses the y axis (M{x=0}).
    @rtype:   number
    @return:  the x intercept of the line M{y=m*x+b}.
    """
    return -b/m

  • doxygen (note that doxygen can also support Markdown syntax):

    def x_intercept(m, b):
    """
    Return the x intercept of the line \f$y=m \times x + b\f$.  The <em>x intercept</em>
    of a line is the point at which it crosses the x axis (\f$y=0\f$).

    This function can be used in conjuction with `z_transform` to
    find an arbitrary function's zeros.

    @param m  The slope of the line.
    @param b  The y intercept of the line.  The <em>y intercept</em> of a
              line is the point at which it crosses the y axis (\f$x=0\f$).

    @return   the x intercept of the line \f$y=m*x+b\f$.
    """
    return -b/m

misc

  • document with help of ChatGPT https://realpython.com/document-python-code-with-chatgpt/

  • documenting variables: Docstrings do not natively support variables. pdoc3 supports two mechanisms, PEP-224 and #: comment comment syntax:

    module_variable = 1
"""PEP 224 docstring for module_variable."""

class C:
    #: Documentation comment for class_variable
    #: spanning over three lines.
    class_variable = 2  #: Assignment line is included.

    def __init__(self):
        #: Instance variable's doc-comment
        self.variable = 3
        """But note, PEP 224 docstrings take precedence."""
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment