Documentation

Generation

The documentation of CSO that you are reading right now is generated from CSO source files. To (re)generate it, run in the main directory:

make docu

The documentation files are formatted for use with ‘Sphinx’; see:

The formatting of documentation is done using ‘reStructuredText’, see for example:

The ‘Sphinx’ python package should be available to be able to (re)generate the documentation. If not available on your system yet, see below for installation instructions.

Source files

The main files of the documentation are located in:

./doc/source/

Parts of the documentation are included in the various Python modules and scripts. The documentation is configured in such a way that these are automatically incorporated.

The source tree of the documentation files is:

Installation of Sphinx

To check if Sphinx is already installed, try to run the quick-start script:

sphinx-quickstart --version

sphinx-quickstart 2.4.0

Try if it is possible to download and install Sphinx using one of the standard installation commands.

  • For an Anaconda distribution, try:

    conda install -c anaconda sphinx
    
  • For other distributions, try:

    pip install sphinx
    
  • To build Sphinx from source, download the latest version from the Sphinx homepage, for example:

    Sphinx-3.0.3.tar.gz
    

    To let the package be installed at a local destination, define the following environment variable:

    export PYTHON_PREFIX="${HOME}/opt/Python-3.7"
    

    Build and install in the user-defined location using:

    cd Sphinx-3.0.3
    python setup.py install --user
    

    Extend the search path with:

    export PATH="${PYTHON_PREFIX}/bin:${PATH}"
    

Configuration

The source of the documentation is located in (subdirectories of):

doc/

In this directory, the Sphinx quick start command has been called:

sphinx-quickstart \
  --sep \
  --project='CSO' \
  --author='Arjo Segers' \
  --ext-autodoc \
  --ext-intersphinx \
  --ext-mathjax \
  --no-batchfile

The following entities have been created now:

  • source directory to hold the documenation source files; initially the following files are created:

    • conf.py : configuration file;

    • index.rst : source of the main page of the documentation;

  • build directory to hold the generated documenation;

  • Makefile : make commands.

In the ./doc/source/conf.py file created in this way, the following changes were made manually:

  • the location of the python modules was added to the search path;

  • added options for autodoc entries;

  • added options for intersphinx entries;

  • the html theme was changed.

The initial documentation could be created using:

(cd doc; make html)

How to add a new module to the documentation?

Example taken from introduction of the ‘cso_s5p’ module.

  • Create new module file:

    touch  py/cso_s5p.py
    
  • Create a documentation file specific for the new module:

    doc/source/pymod-cso_s5p.rst
    

    which only refers to the documentation included in the module:

    .. automodule:: cso_s5p
    
  • Add a reference to the new module in the Python modules page in the file pymods.rst:

    .. toctree::
       :maxdepth: 1
    
       pymod-cso_dataspace
       pymod-cso_s5p              <---
       :
    
  • Add a reference to the module documentation to the top-level table-of-content in index.rst:

    .. toctree::
       :maxdepth: 2
    
       tutorial
       cso_s5p                    <---
       pymods
       documentation
    

Online publication with Gitlab Pages

The repository is hosted on Gitlab server. This has also the option to automatically generate the documenation and store the resulting html files on a web server associated with the repository, which are called the Gitlab Pages.

The following steps describe the steps that were taken.

Create a GitLab Runner

A ‘GitLab Runner’ is a ‘Docker’ container that will perform the actual jobs of, in this case, generating and uploading the documentation. This should run on a local server that could be contacted by the Gitlab server.

When a ‘GitLab Runner’ is already available, one could go to the next item.

For this project a GitLab Runner has been created using the following steps.

  1. Install ‘Docker’ on the local server. The desktop version is available from: www.docker.com/products/docker-desktop.

  2. Start Docker Desktop. This starts a docker server in the background that enables you to run Docker containers.

  3. Open a terminal.

  4. Download the gitlab-runner image from gitlab.com, install it and start it:

    docker run -d \
        --name gitlab-runner \
        --restart always \
        -v /srv/gitlab-runner/config:/etc/gitlab-runner \
        -v /var/run/docker.sock:/var/run/docker.sock gitlab/gitlab-runner:latest
    

    Basically, this is a customized ‘Docker’ image with a standard Linux distribution and the necessary tools for running GitLab CI/CD pipelines. Every time you commit and push changes to the GitLab repository, the container (i.e., the running instance of the Gitlab image) will execute the pipelines (provided your Docker server is running and the runner is registered in GitLab, see below).

  5. Check it in your Docker Desktop GUI, click on ‘Containers’ in the left menu and you’ll see gitlab-runner listed with a green dot in front of it, showing that the container is running. If you click on the name gitlab-runner, you’ll see the logs of the runner. Since you do not have a configuration yet, there will be error messages like this:

    2025-01-21 14:26:23 ERROR: Failed to load config stat /etc/gitlab-runner/config.toml: no such file or directory  builds=0 max_builds=1
    

    This will be remedied in the next step.

Registration of the runner

The runner needs to be registrated for the GitLab Project, or for its group.

The following steps were taken to registrate the runner for the “CAMS” group holding this project.

  1. Open the Gitlab homepage of the group.

  2. In the side bar, select Settings > CI/CD.

  3. Expand the Runners section.

  4. Check if a already a runner has been registrated. If no runner was registrated yet, use the following steps.

  1. In the side bar, open Build > Runners.

  2. Enable Run untagged jobs.

  3. Click on [Create runner].

  4. Copy the command that is shown, which looks like:

    gitlab-runner register  --url https://ci.tno.nl/gitlab  --token abcdEFGHIjklmnOPQRstuvw
    
  5. Login on the server where the gitlab-runner Docker immage was installed and open a terminal.

  6. Paste the copied command as part of the following command:

    docker  exec \
         --interactive  \
         --tty \
         gitlab-runner \
         gitlab-runner register  --url https://ci.tno.nl/gitlab  --token abcdEFGHIjklmnOPQRstuvw
    

    Provide the following information when asked for:

    • For the GitLab instance URL, choose: https://ci.tno.nl/gitlab

    • For the name, use the default

    • For executor, use: docker

    • For image, use: ubuntu:latest

Create a CI/CD pipeline

The actions to be performed to create the documentation are:

cd doc
make html

This should now be done when code is changed using a ‘pipeline’.

  1. Goto main page of the GitLab project.

  2. Open the [+] drop-down, select create file, and create a file named:

    .gitlab-ci.yml
    

    The content should look like:

    # The Docker image that will be used to build your app
    image: python:3.11
    
    # Functions that should be executed before the build script is run
    before_script:
      - pip install numba
      - pip install sphinx
    
    stages:
      - deploy-pages
    
    pages:
      stage: deploy-pages
      rules:
        - if: '$CI_COMMIT_BRANCH == "master"'
          when: always
      script:
        - cd doc
        - make html
        - cd ../..
        - mkdir public
        - cp -r doc/build/html/* public
      artifacts:
        paths:
          - public
    

    Save and commit the file.

    This tells to create a pipeline to:

    • setup the correct python environment to generate the documentation;

    • create the documentation (in a clone of the current repository);

    • create a directory named public and to copy the documents into there;

    • tell that the end result is located in the just filled directory public.

    This is only done for changes in the master branch, as otherwise the documentation from a development branch will be shown online.

  3. When the above instruction file was committed, the pipeline is started immediately as this is a change of the source code.

    In the side bare open Build > Jobs for a list of jobs that were run. The latest job is probably is probably still running; when finished, click on the status label to open the details. Based on the messages, edit the yml or the source files that might need corrections.

  4. If the pipeline was finished correctly, the documentation should have been copied to a web server.

    Goto the main page of the GitLab project, and in the right side bar click on the Gitlab Pages link to see the result: