.. Documentation description.

.. Label between '.. _' and ':' ; use :ref:`text <label>` for reference
.. _documentation:


*************
Documentation
*************


Generation
==========

The documentation of CSO that you are reading right now is generated
from CSO source files.
To (re)generate it, run in the main directory::

  make docu

The documentation files are formatted for use with 'Sphinx'; see:

* `Sphinx homepage <http://sphinx-doc.org/index.html>`_

The formatting of documentation is done using 'reStructuredText', see for example:

* `reStructuredText Primer <https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html>`_
* `Rst <https://docutils.sourceforge.io/docs/ref/rst/>`_

The 'Sphinx' python package should be available to be able to (re)generate the documentation.
If not available on your system yet, see below for installation instructions.


Source files
============

The main files of the documentation are located in::

  ./doc/source/

Parts of the documentation are included in the various Python modules and scripts.
The documentation is configured in such a way that these are automatically incorporated.

The source tree of the documentation files is:

  * `index.rst <../../source/index.rst>`_
  * `tutorial.rst <../../source/tutorial.rst>`_
  * `pymod-cso.rst <../../source/pymod-cso.rst>`_
  * `pymod-cso_catalogue.rst <../../source/pymod-cso_catalobue.rst>`_
  * `pymod-cso_constants.rst <../../source/pymod-cso_constants.rst>`_
  * `pymod-cso_file.rst <../../source/pymod-cso_file.rst>`_
  * `pymod-cso_gridded.rst <../../source/pymod-cso_gridded.rst>`_
  * `pymod-cso_colocate.rst <../../source/pymod-cso_colocate.rst>`_
  * `pymod-cso_plot.rst <../../source/pymod-cso_plot.rst>`_
  * `pymod-cso_s5p.rst <../../source/pymod-cso_s5p.rst>`_
  * `pymod-cso_dataspace.rst <../../source/pymod-cso_dataspace.rst>`_
  * `pymods.rst <../../source/pymods.rst>`_
  * `documentation.rst <../../source/documentation.rst>`_


Installation of Sphinx
======================

To check if Sphinx is already installed, try to run the quick-start script::

  sphinx-quickstart --version
  
  sphinx-quickstart 2.4.0

Try if it is possible to download and install Sphinx using 
one of the standard installation commands.

* For an `Anaconda <https://www.anaconda.com/>`_ distribution, try::

    conda install -c anaconda sphinx
    
* For other distributions, try::

    pip install sphinx

* To build Sphinx from source, download the latest version from the
  `Sphinx homepage <http://sphinx-doc.org/index.html>`_, for example::

    Sphinx-3.0.3.tar.gz

  To let the package be installed at a local destination, 
  define the following environment variable::

    export PYTHON_PREFIX="${HOME}/opt/Python-3.7"

  Build and install in the user-defined location using::

    cd Sphinx-3.0.3
    python setup.py install --user

  Extend the search path with::

    export PATH="${PYTHON_PREFIX}/bin:${PATH}"


Configuration
=============

The source of the documentation is located in (subdirectories of)::

  doc/

In this directory, the Sphinx quick start command has been called::

    sphinx-quickstart \
      --sep \
      --project='CSO' \
      --author='Arjo Segers' \
      --ext-autodoc \
      --ext-intersphinx \
      --ext-mathjax \
      --no-batchfile

The following entities have been created now:

* ``source`` directory to hold the documenation source files;
  initially the following files are created:
  
  * `conf.py <../../source/conf.py>`_ : configuration file;
  * `index.rst <../../source/index.rst>`_ : source of the main page of the documentation;

* ``build`` directory to hold the generated documenation;

* ``Makefile`` : make commands.

In the ``./doc/source/conf.py`` file created in this way,
the following changes were made manually:

* the location of the python modules was added to the search path;
* added options for `autodoc` entries;
* added options for `intersphinx` entries;
* the html theme was changed.

The initial documentation could be created using::

    (cd doc; make html)


How to add a new module to the documentation?
=============================================

Example taken from introduction of the 'cso_s5p' module.

* Create new module file::

    touch  py/cso_s5p.py
      
* Create a documentation file specific for the new module::

    doc/source/pymod-cso_s5p.rst

  which only refers to the documentation included in the module::

    .. automodule:: cso_s5p

* Add a reference to the new module in the :ref:`modules-and-classes` page
  in the file `pymods.rst <../../source/pymods.rst>`_::

    .. toctree::
       :maxdepth: 1

       pymod-cso_dataspace
       pymod-cso_s5p              <---
       :


* Add a reference to the module documentation to the top-level 
  table-of-content in `index.rst <../../source/index.rst>`_::

    .. toctree::
       :maxdepth: 2

       tutorial
       cso_s5p                    <---
       pymods
       documentation


Online publication with Gitlab Pages
====================================

The repository is hosted on `Gitlab <https://about.gitlab.com/>`_ server.
This has also the option to automatically generate the documenation and store 
the resulting html files on a web server associated with the repository, which are called the *Gitlab Pages*.

The following sections describe the steps that were taken.

Create a GitLab Runner
----------------------

A '*GitLab Runner*' is a '*Docker*' container that will perform the actual jobs of, 
in this case, generating and uploading the documentation.
This should run on a local server that could be contacted by the Gitlab server.

When a '*GitLab Runner*' is already available, one could go to the next item.

For this project a GitLab Runner has been created using the following steps.

1. Install '*Docker*' on the local server. The desktop version is available from: `www.docker.com/products/docker-desktop <https://www.docker.com/products/docker-desktop/>`_.
2. Start Docker Desktop. This starts a docker server in the background that enables you to run Docker containers.
3. Open a terminal.
4. Download the ``gitlab-runner`` image from ``gitlab.com``, install it and start it::

     docker run -d \
         --name gitlab-runner \
         --restart always \
         -v /srv/gitlab-runner/config:/etc/gitlab-runner \
         -v /var/run/docker.sock:/var/run/docker.sock gitlab/gitlab-runner:latest

   Basically, this is a customized '*Docker*' image with a standard Linux distribution and the necessary
   tools for running GitLab CI/CD pipelines. Every time you commit and push changes to the GitLab
   repository, the container (i.e., the running instance of the Gitlab image) will execute the pipelines
   (provided your Docker server is running and the runner is registered in GitLab, see below).
    
5. Check it in your Docker Desktop GUI, click on '*Containers*' in the
   left menu and you'll see gitlab-runner listed with a green dot in front of it, showing that the
   container is running. If you click on the name gitlab-runner, you'll see the logs of the runner.
   Since you do not have a configuration yet, there will be error messages like this::

      2025-01-21 14:26:23 ERROR: Failed to load config stat /etc/gitlab-runner/config.toml: no such file or directory  builds=0 max_builds=1 

   This will be remedied in the next step.


Registration of the runner
--------------------------

The runner needs to be registrated for the GitLab Project, or for its group.

The following steps were taken to registrate the runner for the "CAMS" group holding this project.

1. Open the Gitlab homepage of the group.
2. In the side bar, select ``Settings > CI/CD``.
3. Expand the ``Runners`` section.
4. Check if a already a runner has been registrated. If no runner was registrated yet, use the following steps.

  1. In the side bar, open ``Build > Runners``.
  2. Enable ``Run untagged jobs``.
  3. Click on ``[Create runner]``.
  4. Copy the command that is shown, which looks like::
  
      gitlab-runner register  --url https://ci.tno.nl/gitlab  --token abcdEFGHIjklmnOPQRstuvw
      
  5. Login on the server where the ``gitlab-runner`` Docker immage was installed and open a terminal.
  6. Paste the copied command as part of the following command::
  
      docker  exec \
           --interactive  \
           --tty \
           gitlab-runner \
           gitlab-runner register  --url https://ci.tno.nl/gitlab  --token abcdEFGHIjklmnOPQRstuvw

     Provide the following information when asked for:
     
     * For the GitLab instance URL, choose: ``https://ci.tno.nl/gitlab``
     * For the name, use the default
     * For executor, use: ``docker``
     * For image, use: ``ubuntu:latest``


Create a CI/CD pipeline
-----------------------

The actions to be performed to create the documentation are::

        cd doc
        make html

This should now be done when code is changed using a '*pipeline*'.

1. Goto main page of the GitLab project.
2. Open the ``[+]`` drop-down, select ``create file``, and create a file named::
     
     .gitlab-ci.yml

   The content should look like:

   .. sourcecode:: yaml

      # The Docker image that will be used to build your app
      image: python:3.11

      # Functions that should be executed before the build script is run
      before_script:
        - pip install numba
        - pip install sphinx

      stages:
        - deploy-pages

      pages:
        stage: deploy-pages
        rules:
          - if: '$CI_COMMIT_BRANCH == "master"'
            when: always
        script:
          - cd doc
          - make html
          - cd ..
          - mkdir public
          - cp -r doc/build/html/* public
        artifacts:
          paths:
            - public

   Save and commit the file.

   This tells to create a pipeline to:
   
   * setup the correct python environment to generate the documentation;
   * create the documentation (in a clone of the current repository);
   * create a directory named ``public`` and to copy the documents into there;
   * tell that the end result is located in the just filled directory ``public``.
   
   This is only done for changes in the master branch, as otherwise the documentation from a development branch
   will be shown online.

3. When the above instruction file was committed, the pipeline is started immediately as this is a change of the source code.

   In the side bare open ``Build > Jobs`` for a list of jobs that were run.
   The latest job is probably is probably still running; when finished, click on the status label to open the details.
   Based on the messages, edit the ``yml`` or the source files that might need corrections.

4. If the pipeline was finished correctly, the documentation should have been copied to a web server.

   Goto the main page of the GitLab project, and in the right side bar click on the ``Gitlab Pages`` link to see the result:
   
       `cams.ci.tno.nl/cso <https://cams.ci.tno.nl/cso>`_




