Documentation¶
Generation¶
The documentation of CSO that you are reading right now is generated from CSO source files. To (re)generate it, run in the main directory:
make docu
The documentation files are formatted for use with ‘Sphinx’; see:
The formatting of documentation is done using ‘reStructuredText’, see for example:
The ‘Sphinx’ python package should be available to be able to (re)generate the documentation. If not available on your system yet, see below for installation instructions.
Source files¶
The main files of the documentation are located in:
./doc/source/
Parts of the documentation are included in the various Python modules and scripts. The documentation is configured in such a way that these are automatically incorporated.
The source tree of the documentation files is:
Installation of Sphinx¶
To check if Sphinx is already installed, try to run the quick-start script:
sphinx-quickstart --version
sphinx-quickstart 2.4.0
Try if it is possible to download and install Sphinx using one of the standard installation commands.
For an Anaconda distribution, try:
conda install -c anaconda sphinx
For other distributions, try:
pip install sphinx
To build Sphinx from source, download the latest version from the Sphinx homepage, for example:
Sphinx-3.0.3.tar.gz
To let the package be installed at a local destination, define the following environment variable:
export PYTHON_PREFIX="${HOME}/opt/Python-3.7"
Build and install in the user-defined location using:
cd Sphinx-3.0.3 python setup.py install --user
Extend the search path with:
export PATH="${PYTHON_PREFIX}/bin:${PATH}"
Configuration¶
The source of the documentation is located in (subdirectories of):
doc/
In this directory, the Sphinx quick start command has been called:
sphinx-quickstart \
--sep \
--project='CSO' \
--author='Arjo Segers' \
--ext-autodoc \
--ext-intersphinx \
--ext-mathjax \
--no-batchfile
The following entities have been created now:
source
directory to hold the documenation source files; initially the following files are created:build
directory to hold the generated documenation;Makefile
: make commands.
In the ./doc/source/conf.py
file created in this way,
the following changes were made manually:
the location of the python modules was added to the search path;
added options for autodoc entries;
added options for intersphinx entries;
the html theme was changed.
The initial documentation could be created using:
(cd doc; make html)
How to add a new module to the documentation?¶
Example taken from introduction of the ‘cso_s5p’ module.
Create new module file:
touch py/cso_s5p.py
Create a documentation file specific for the new module:
doc/source/pymod-cso_s5p.rst
which only refers to the documentation included in the module:
.. automodule:: cso_s5p
Add a reference to the new module in the Python modules page in the file pymods.rst:
.. toctree:: :maxdepth: 1 pymod-cso_dataspace pymod-cso_s5p <--- :
Add a reference to the module documentation to the top-level table-of-content in index.rst:
.. toctree:: :maxdepth: 2 tutorial cso_s5p <--- pymods documentation
Online publication with Gitlab Pages¶
The repository is hosted on Gitlab server. This has also the option to automatically generate the documenation and store the resulting html files on a web server associated with the repository, which are called the Gitlab Pages.
The following steps describe the steps that were taken.
Create a GitLab Runner¶
A ‘GitLab Runner’ is a ‘Docker’ container that will perform the actual jobs of, in this case, generating and uploading the documentation. This should run on a local server that could be contacted by the Gitlab server.
When a ‘GitLab Runner’ is already available, one could go to the next item.
For this project a GitLab Runner has been created using the following steps.
Install ‘Docker’ on the local server. The desktop version is available from: www.docker.com/products/docker-desktop.
Start Docker Desktop. This starts a docker server in the background that enables you to run Docker containers.
Open a terminal.
Download the
gitlab-runner
image fromgitlab.com
, install it and start it:docker run -d \ --name gitlab-runner \ --restart always \ -v /srv/gitlab-runner/config:/etc/gitlab-runner \ -v /var/run/docker.sock:/var/run/docker.sock gitlab/gitlab-runner:latest
Basically, this is a customized ‘Docker’ image with a standard Linux distribution and the necessary tools for running GitLab CI/CD pipelines. Every time you commit and push changes to the GitLab repository, the container (i.e., the running instance of the Gitlab image) will execute the pipelines (provided your Docker server is running and the runner is registered in GitLab, see below).
Check it in your Docker Desktop GUI, click on ‘Containers’ in the left menu and you’ll see gitlab-runner listed with a green dot in front of it, showing that the container is running. If you click on the name gitlab-runner, you’ll see the logs of the runner. Since you do not have a configuration yet, there will be error messages like this:
2025-01-21 14:26:23 ERROR: Failed to load config stat /etc/gitlab-runner/config.toml: no such file or directory builds=0 max_builds=1
This will be remedied in the next step.
Registration of the runner¶
The runner needs to be registrated for the GitLab Project, or for its group.
The following steps were taken to registrate the runner for the “CAMS” group holding this project.
Open the Gitlab homepage of the group.
In the side bar, select
Settings > CI/CD
.Expand the
Runners
section.Check if a already a runner has been registrated. If no runner was registrated yet, use the following steps.
In the side bar, open
Build > Runners
.Enable
Run untagged jobs
.Click on
[Create runner]
.Copy the command that is shown, which looks like:
gitlab-runner register --url https://ci.tno.nl/gitlab --token abcdEFGHIjklmnOPQRstuvwLogin on the server where the
gitlab-runner
Docker immage was installed and open a terminal.Paste the copied command as part of the following command:
docker exec \ --interactive \ --tty \ gitlab-runner \ gitlab-runner register --url https://ci.tno.nl/gitlab --token abcdEFGHIjklmnOPQRstuvwProvide the following information when asked for:
For the GitLab instance URL, choose:
https://ci.tno.nl/gitlab
For the name, use the default
For executor, use:
docker
For image, use:
ubuntu:latest
Create a CI/CD pipeline¶
The actions to be performed to create the documentation are:
cd doc
make html
This should now be done when code is changed using a ‘pipeline’.
Goto main page of the GitLab project.
Open the
[+]
drop-down, selectcreate file
, and create a file named:.gitlab-ci.yml
The content should look like:
# The Docker image that will be used to build your app image: python:3.11 # Functions that should be executed before the build script is run before_script: - pip install numba - pip install sphinx stages: - deploy-pages pages: stage: deploy-pages rules: - if: '$CI_COMMIT_BRANCH == "master"' when: always script: - cd doc - make html - cd ../.. - mkdir public - cp -r doc/build/html/* public artifacts: paths: - public
Save and commit the file.
This tells to create a pipeline to:
setup the correct python environment to generate the documentation;
create the documentation (in a clone of the current repository);
create a directory named
public
and to copy the documents into there;tell that the end result is located in the just filled directory
public
.
This is only done for changes in the master branch, as otherwise the documentation from a development branch will be shown online.
When the above instruction file was committed, the pipeline is started immediately as this is a change of the source code.
In the side bare open
Build > Jobs
for a list of jobs that were run. The latest job is probably is probably still running; when finished, click on the status label to open the details. Based on the messages, edit theyml
or the source files that might need corrections.If the pipeline was finished correctly, the documentation should have been copied to a web server.
Goto the main page of the GitLab project, and in the right side bar click on the
Gitlab Pages
link to see the result: