
.. Label between '.. _' and ':' ; use :ref:`text <label>` for reference
.. _s5p-co-processing:

******************************
Sentinel-5p CO data processing
******************************

This chapter describes the tasks performed for processing Sentinel-5p CO data.


Product description
===================

The product guides can be found at:

* `SentiWiki / Sentinel-5P <https://sentiwiki.copernicus.eu/web/s5p-products>`_

  * ``L2__CO___``, ``PUM-CO`` Product User Manual


Notes:

* From processor version v2.4.0 onwards, the units of the averaging kernel changed from ``m`` to a more standard unit-less repesentation.
  When processing older data versions, the conversion into CSO format (which was using unit-less representation already) is different.
  
.. xxxxxxxxxxxxxxxxxxxxxxxxx

  Features:

  * The retrieval product is a column density (mol/m2), which will be treated by CSO as a profile
    with :math:`n_r=1` layers:

    .. math::
        \mathbf{y}_r

  * The simulation of a retrieval product from a model state does not require an apriori profile,
    and is denoted with:

    .. math::
        \mathbf{y}_s ~=~ \mathbf{A}\ \mathbf{H}\mathbf{x}

    where:

    * :math:`\mathbf{y}_s` is the simulated retrieval (mol/m2) defined on :math:`n_r=1` layers;
    * :math:`\mathbf{A}^{trop}` is the *tropospheric* averaging kernel matrix with shape :math:`(n_r,n_a)`;
      with :math:`n_a` the number of *a priori* layers;
    * :math:`\mathbf{x}` is the atmospheric state, which consists of a 3D array of CO concentrations;
    * :math:`\mathbf{H}` extracts a simulated profile from the state using horizontal and vertical interpolation;
      the result should be defined on the :math:`n_a` *a priori* layers
      and have the units of the retrieval product (mol/m2).

    In case :math:`\mathbf{x}` is the true atmoshperic state, the retrieval error is quantified
    by the *retrieval error covariance* :math:`\mathbf{R}` (in this scalar product a variance):

    .. math::
        \mathbf{y}_s ~-~ \mathbf{A}^{trop}\ \mathbf{H}\mathbf{x}^{true}  ~\sim~ \mathcal{N}\left(\mathbf{o},\mathbf{R}^{trop}\right)

  * The retrieval status and quality is indicated by the ``qa_value``. 
    The recommended minimum is 0.5, this excludes cloudy scenes and other problematic retrievals.

.. Label between '.. _' and ':' ; use :ref:`text <label>` for reference


CSO processing
==============

*(See* :ref:`tutorial` *chapter for introduction to CSO scripts and configuration)*

An example configuration of the CSO processing of the S5p/CO data is available via
the following settings:

* `config/Copernicus/cso.rc <../../../config/Copernicus/cso.rc>`_

  Top-level settings that configure the job-tree with various sub-tasks.
  This is a generic file that could be used for multiple S5 products, 
  edit it to select the CO processing.
   
* `config/Copernicus/cso-user-settings.rc <../../../config/Copernicus/cso-user-settings.rc>`_

  User-specific settings such as the work directory.
  
* `config/Copernicus/cso-s5p-co.rc <../../../config/Copernicus/cso-s5p-co.rc>`_
  
  Specific settings for CO product.

Start the job-tree using::

  ./bin/cso  config/Copernicus/cso.rc
  
Selected sub-steps in the processing are described below.



.. _s5p-co-inquire:

Inquire Sentinel-5p/CO archive
================================

S5p/CO observations are available from the
`Copernicus DataSpace <https://dataspace.copernicus.eu/>`_;
see the :ref:`cso-dataspace` module for a detailed description.

Data is available for different processing streams, each identified by a 4-character key:

* ``NRTI`` : `Near real time`, available with a day after observation;
* ``OFFL`` : `Offline`, available within weeks after observations;
* ``RPRO`` : re-processing of all previously made observations;

The portals provide data files created with the same retrieval algorithm, 
but with different processor versions.
It is therefore necessary to first inquire both archives to see which data is available where,
and what the version numbers are.

The :py:class:`CSO_DataSpace_Inquire <cso_dataspace.CSO_DataSpace_Inquire>` class is available to inquire the
*Copernicus DataSpace*. The settings used by this class allow selection
on for example time range and intersection area. 
The result is a csv file which with columns for keywords such orbit number and processor version,
as well as the filename of the data and the url that should be used to actually download the data::

    orbit;start_time;end_time;processing;collection;processor_version;filename;href
    23665;2022-05-08 09:44:59;2022-05-08 11:26:29;RPRO;03;020400;S5P_RPRO_L2__CO_____20220508T094459_20220508T112629_23665_03_020400_20230112T000125.nc;https://zipper.dataspace.copernicus.eu/odata/v1/Products('afbaef9f-5295-4d47-be43-f74270e07b4d')/$value
    23666;2022-05-08 11:26:29;2022-05-08 13:07:59;RPRO;03;020400;S5P_RPRO_L2__CO_____20220508T112629_20220508T130759_23666_03_020400_20230112T000126.nc;https://zipper.dataspace.copernicus.eu/odata/v1/Products('3beada57-7602-482e-bde4-7a10758327e3')/$value
    :

See the section on *File name convention* in the *Product User Manual* for the meaning of all 
parts of the filename.

To visualize what is available from the various portals, the
:py:class:`cso_inquire.CSO_Inquire_Plot` could be used to create an overview figure:

.. figure:: figs/CO/Copernicus_S5p_CO.png
   :scale: 50 %
   :align: center
   :alt: Example overview of available CO processings.

The jobtree configuration to inquire the portals and create the overview figure could look like::

    ! single step:
    cso.s5p.co.inquire.class                      :  utopya.UtopyaJobStep
    ! two tasks:
    cso.s5p.co.inquire.tasks                      :  table-dataspace plot

    !~ inquire files available on DataSpace:
    cso.s5p.co.inquire.table-dataspace.class      :  cso.CSO_DataSpace_Inquire
    cso.s5p.co.inquire.table-dataspace.args       :  '${PWD}/config/Copernicus/cso-s5p-co.rc', \
                                                         rcbase='cso.s5p.co.inquire-table-dataspace'

    !~ create plot of available versions:
    cso.s5p.co.inquire.plot.class                 :  cso.CSO_Inquire_Plot
    cso.s5p.co.inquire.plot.args                  :  '${PWD}/config/Copernicus/cso-s5p-co.rc', \
                                                          rcbase='cso.s5p.co.inquire-plot'



.. xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

  .. Label between '.. _' and ':' ; use :ref:`text <label>` for reference
  .. _s5p-co-convert:

  Conversion to CSO format
  ========================

  The '``cso.s5p.co.convert``' task creates netCDF files with selected pixels,
  for example only those within some region or cloud free pixels.
  The selection criteria are defined in the settings, and added
  to the '``history``' attribute of the created files as reminder.

  The work is done by the :py:class:`.CSO_S5p_Convert` class,
  which is initialized using the settings file::

    ! task initialization:
    cso.s5p.co.convert.class     :  cso.CSO_S5p_Convert
    cso.s5p.co.convert.args      :  '${PWD}/config/Copernicus/cso-s5p-co.rc', rcbase='cso.s5p.co.convert'

  See the class documentation for the general configuration,
  below some specific choices are described.
  The example is based on the S5p CO file from which the header is available in:

  * `doc/samples/S5P_RPRO_L2__CO_____20180601T002121_20180601T020251_03272_03_020400_20220902T150723.txt <../../samples/S5P_RPRO_L2__CO_____20180601T002121_20180601T020251_03272_03_020400_20220902T150723.txt>`_


  Pixel selection
  ---------------

  The :py:class:`.CSO_S5p_Convert` class calls the :py:meth:`.S5p_File.SelectPixels` method
  to create a pixel selection mask for the input file.
  The selection is done using one or more filters.
  First provide a list of filter names::

    cso.s5p.co.convert.filters   :  lons lats valid quality sza error_ratio ground_pixel

  Then provide for each filter the the input variable to be used for testing,
  as a path name in the input file.
  The next settings is the type of filter to be used, see the :py:meth:`.S5p_File.SelectPixels` for supported types,
  and the other settings required by the type.
  The following is an example of a selection on longitude::

    cso.s5p.co.convert.filter.lons.var                :  Geolocation Fields/Longitude
    cso.s5p.co.convert.filter.lons.type               :  minmax
    cso.s5p.co.convert.filter.lons.minmax             :  -30.0 45.0
    cso.s5p.co.convert.filter.lons.units              :  degrees_east

  Extension to the product guide
  ------------------------------

  It is mentioned in the PUM that the quality flag and assesment are incomplete:
  Several additions quality filters were based on Vigouroux et al., from which we quote:

    Several diagnostic variables are provided together with the measurements. 
    Quality assurance (QA) values are defined to perform a quick selection of the observations. 
    QA > 0.5 filters out most observations presenting an error flag or a solar zenith angle (SZA)
    larger than 70\ :sup:`o`, a cloud radiance fraction larger than 0.6 at 340 nm, or an air mass 
    factor smaller than 0.1. The product Readme file reports that, in the current version, 
    the QA values are not always correctly set over snow and ice regions or above an SZA of 75\ :sub:`o`. 
    They also need tobe further checked over cloudy scenes. 
    In the forthcoming S5P version 2, QA values will be refined and will exclude data with a 
    surface albedo larger than 0.2 and a snow or ice warning as well as remaining SZAs larger than 
    75\ :sup:`o`.

  .. figure:: figs/CO/Example_validation_co.png
     :scale: 85 %
     :align: center
     :alt: Validation results from Vigouroux et al., 2020, https://amt.copernicus.org/articles/13/3751/2020/amt-13-3751-2020.pdf


  Variable specification
  ----------------------

  The target file is created as an :py:class:`.CSO_S5p_File` object.
  It's :py:meth:`AddSelection <.CSO_S5p_File.AddSelection>` method is called with the input object as argument,
  and this will copy the selected pixels for variables specified in the settings.

  The variable specification starts with a list with variable names to be 
  created in the target file::

    cso.s5p.co.convert.output.vars     : longitude longitude_bounds \
                                         latitude latitude_bounds \
                                         track_longitude track_longitude_bounds \
                                         track_latitude  track_latitude_bounds \
                                         time \
                                         pressure  qa_value \
                                         vcd vcd_errvar \
                                         cloud_fraction \                           
                                         ground_pixel \
                                         cloud_pressure_crb \
                                         solar_zenith_angle \
                                         amf_troposphere \
                                         quality \
                                         kernel

  For each variable settings should be specified that describe the shape of the variable
  and how it should be filled from the input.
  See the :py:meth:`AddSelection <.CSO_S5p_File.AddSelection>` description for all options,
  here we show some examples.

  The ``longitude`` and ``latitude`` variables are copied almost directly out of the source files,
  the only change that is applied is the selection of pixels.
  All original attributes are copied, except for the ``bound`` attribite since that would
  give warnings from the CF-compliance checker::

    cso.s5p.co.convert.output.var.longitude.dims                   :   pixel
    cso.s5p.co.convert.output.var.longitude.from                   :   PRODUCT/longitude
    cso.s5p.co.convert.output.var.longitude.attrs                  :   { 'bounds' : None }

    cso.s5p.co.convert.output.var.latitude.dims                    :   pixel
    cso.s5p.co.convert.output.var.latitude.from                    :   PRODUCT/latitude
    cso.s5p.co.convert.output.var.latitude.attrs                   :   { 'bounds' : None }

  The pixel boundaries are necessary to know the exact footprint of a pixel,
  which is for example used when averaging over a grid or simulation from a model.
  These are available in the input files, but without a ``units`` attribute as these
  are implied by the pixel center coordinate; the conversion therefore requires that
  units are defined explicitly.
  For the ``longitude_bounds`` a special processing is needed for pixels crossing the dateline,
  as the original data simply uses longitudes modulo 360 degrees::

    ! corner longitudes; no units in file:
    cso.s5p.co.convert.output.var.longitude_bounds.dims            :   pixel corner
    cso.s5p.co.convert.output.var.longitude_bounds.from            :   PRODUCT/SUPPORT_DATA/GEOLOCATIONS/longitude_bounds
    cso.s5p.co.convert.output.var.longitude_bounds.units           :   degrees_east
    ! ensure that near dateline the corners form a convex region around center
    ! (with some points outside [-180,+180] if necessary)
    cso.s5p.co.convert.output.var.longitude_bounds.special         :   longitude_bounds

    ! corner latitudes, no units in file:
    cso.s5p.co.convert.output.var.latitude_bounds.dims             :   pixel corner
    cso.s5p.co.convert.output.var.latitude_bounds.from             :   PRODUCT/SUPPORT_DATA/GEOLOCATIONS/latitude_bounds
    cso.s5p.co.convert.output.var.latitude_bounds.units            :   degrees_north

  Also the locations of the pixels in the original track are copied,
  since these are useful when creating plots. These cannot be copied directly but require special processing::

    cso.s5p.co.convert.output.var.track_longitude.dims             :   track_scan track_pixel
    cso.s5p.co.convert.output.var.track_longitude.special          :   track_longitude
    cso.s5p.co.convert.output.var.track_longitude.from             :   PRODUCT/longitude
    cso.s5p.co.convert.output.var.track_longitude.attrs            :   { 'bounds' : None }

    cso.s5p.co.convert.output.var.track_latitude.dims              :   track_scan track_pixel
    cso.s5p.co.convert.output.var.track_latitude.special           :   track_latitude
    cso.s5p.co.convert.output.var.track_latitude.from              :   PRODUCT/latitude
    cso.s5p.co.convert.output.var.track_latitude.attrs             :   { 'bounds' : None }

  The observattion times are constructed from time steps relative to a reference time;
  this requires special processing too::

    cso.s5p.co.convert.output.var.time.dims                        :   pixel
    cso.s5p.co.convert.output.var.time.special                     :   time-delta
    cso.s5p.co.convert.output.var.time.tref                        :   PRODUCT/time
    cso.s5p.co.convert.output.var.time.dt                          :   PRODUCT/delta_time

  The observed vertical column density could be copied directly.
  The target shape is ``(pixel,retr)`` where ``retr`` is the number of layers in the retrieval product (1 in this case)::

    ! vertical column density:
    cso.s5p.co.convert.output.var.vcd.dims                         :   pixel retr
    cso.s5p.co.convert.output.var.vcd.from                         :   PRODUCT/formaldehyde_tropospheric_vertical_column

  In the converted files, the retrieval error is always expressed as a (co)variance matrix,
  to facilitate (future) conversion of profile products.
  In this example, it is filled from the square of the error standard deviation::

    ! error variance in vertical column density (after application of kernel),
    ! fill with square sums of random and systematic errors
    ! use dims with different names to avoid that cf-checker complains:
    cso.s5p.co.convert.output.var.vcd_errvar.dims                  :   pixel retr retr0
    cso.s5p.co.convert.output.var.vcd_errvar.special               :   square_sum
    cso.s5p.co.convert.output.var.vcd_errvar.from                  :   PRODUCT/formaldehyde_tropospheric_vertical_column_precision
    cso.s5p.co.convert.output.var.vcd_errvar.from2                 :   PRODUCT/SUPPORT_DATA/DETAILED_RESULTS/formaldehyde_tropospheric_vertical_column_kernel_trueness

    The averaging kernel is applied on atmospheric layers, defined by pressure levels.
    In this product the pressure levels are defined using hybride-sigma-pressure coordinates,
    and this requires special processing::

    ! Convert from hybride coefficient bounds in (2,nlev) aray to 3D half level pressure:
    cso.s5p.co.convert.output.var.pressure.dims                    :   pixel layeri
    cso.s5p.co.convert.output.var.pressure.special                 :   hybounds_to_pressure_co
    cso.s5p.co.convert.output.var.pressure.sp                      :   PRODUCT/SUPPORT_DATA/INPUT_DATA/surface_pressure
    cso.s5p.co.convert.output.var.pressure.hyab                    :   PRODUCT/SUPPORT_DATA/INPUT_DATA/tm5_constant_a
    cso.s5p.co.convert.output.var.pressure.hybb                    :   PRODUCT/SUPPORT_DATA/INPUT_DATA/tm5_constant_b
    cso.s5p.co.convert.output.var.pressure.units                   :   Pa

    Averaging kernels are converted to matrices with shape ``(layer,retr)``.

    ! description:
    cso.s5p.co.convert.output.var.kernel.dims                      :   pixel layer retr
    cso.s5p.co.convert.output.var.kernel.from                      :   PRODUCT/SUPPORT_DATA/DETAILED_RESULTS/averaging_kernel
    Other variables can be copied directly::

    ! quality flag:
    cso.s5p.co.convert.output.var.qa_value.dims                   :   pixel
    cso.s5p.co.convert.output.var.qa_value.from                   :   PRODUCT/qa_value
    !~ skip some attributes, cf-checker complains ...
    cso.s5p.co.convert.output.var.qa_value.attrs                  :   { 'valid_min' : None, 'valid_max' : None }

    ! cloud property:
    cso.s5p.co.convert.output.var.cloud_fraction.from             :   PRODUCT/SUPPORT_DATA/INPUT_DATA/cloud_fraction_crb
    cso.s5p.co.convert.output.var.cloud_fraction.units            :   1
    cso.s5p.co.convert.output.var.cloud_fraction.dims         :   pixel

    cso.s5p.co.convert.output.var.solar_zenith_angle.from         :   PRODUCT/SUPPORT_DATA/GEOLOCATIONS/solar_zenith_angle
    cso.s5p.co.convert.output.var.solar_zenith_angle.units        :   degree
    cso.s5p.co.convert.output.var.solar_zenith_angle.dims         :   pixel

    cso.s5p.co.convert.output.var.cloud_pressure_crb.from         :   PRODUCT/SUPPORT_DATA/INPUT_DATA/cloud_pressure_crb
    cso.s5p.co.convert.output.var.cloud_pressure_crb.units        :   Pa
    cso.s5p.co.convert.output.var.cloud_pressure_crb.dims         :   pixel

    cso.s5p.co.convert.output.var.amf_troposphere.from            :   PRODUCT/SUPPORT_DATA/DETAILED_RESULTS/formaldehyde_tropospheric_air_mass_factor
    cso.s5p.co.convert.output.var.amf_troposphere.units           :   1
    cso.s5p.co.convert.output.var.amf_troposphere.dims         :   pixels


  Output files
  ------------

  The name of the target files should be specified with a directory and filename;
  the later could include a template for the orbit number::

      ! output directory and filename:
      ! - times are taken from mid of selection, rounded to hours
      ! - use '%{orbit}' for orbit number
      cso.s5p.co.convert.output.dir          :  /Scratch/CSO/S5p/RPRO/CO/Europe/%Y/%m
      cso.s5p.co.convert.output.filename     :  S5p_RPRO_CO_%{orbit}.nc

  A flag is read to decide if existing files should be renewed or kept::

      cso.s5p.co.convert.renew                  :  True     

  The target file is created as an :py:class:`.CSO_S5p_File` object.
  It's :py:meth:`AddSelection <.CSO_S5p_File.AddSelection>` method is called with the input object as argument,
  and this will copy the selected pixels for variables specified in the settings.
  The :py:meth:`Write <.CSO_File.Write>` method creates the file.

  Global attributes for the target file should be specified with::

      ! global attributes:
      cso.s5p.co.convert.output.attrs               :  format Conventions author institution email
      !
      cso.s5p.co.convert.output.attr.format         :  1.0
      cso.s5p.co.convert.output.attr.Conventions    :  CF-1.7
      cso.s5p.co.convert.output.attr.author         :  Your Name
      cso.s5p.co.convert.output.attr.institution    :  CSO
      cso.s5p.co.convert.output.attr.email          :  Your.Name@cso.org

  The conversion also creates (or updates) a *listing* file with the names of the created files
  (relative to the *listing* file), and the time range of pixels in the file::

      ! csv file that will hold records per file with:
      ! - timerange of pixels in file
      ! - orbit number
      cso.s5p.co.convert.output.listing.file        :  /Scratch/CSO/S5p/listing-CO-Europe.csv

  This file will be used by the observation operator to selects orbits with pixels valid for 
  a desired time range.
  The *listing* is a csv file that looks something like::

      filename                     ;start_time                   ;end_time                     ;orbit
      2018/06/S5p_RPRO_CO_03272.nc;2018-06-01T01:32:46.673000000;2018-06-01T01:36:12.948000000;03272
      2018/06/S5p_RPRO_CO_03273.nc;2018-06-01T03:12:53.649000000;2018-06-01T03:17:43.082000000;03273
      2018/06/S5p_RPRO_CO_03274.nc;2018-06-01T04:52:43.586000000;2018-06-01T04:59:12.377000000;03274
      :



  .. Label between '.. _' and ':' ; use :ref:`text <label>` for reference
  .. _s5p-co-catalogue:

  Catalogue
  =========

  The :py:class:`CSO_Catalogue <.cso_catalogue.CSO_Catalogue>` class could be used
  to create a catalogue of images for the converted files.
  Configuration could look like::

      ! catalogue creation task:
      cso.s5p.co.catalogue.task.figs.class  :  cso.CSO_Catalogue
      cso.s5p.co.catalogue.task.figs.args   :  '${PWD}/config/Copernicus/cso-s5p-co.rc', \
                                                  rcbase='cso.s5p.co.catalogue'

  The configuration describes where to find a *listing* file with orbits, 
  which variables should be plot, the colorbar properties, etc.
  See :py:class:`CSO_Catalogue <.cso_s5p.CSO_Catalogue>` class description for how
  the settings in general look like.

  The class creates figures for a list of variables::

    ! variables to be plotted:
    cso.s5p.co.catalogue.vars                    :  vcd vcd_errvar qa_value \
                                                        cloud_fraction cloud_radiance_fraction

  By default the catalogue creator simply creates a map with the value of the a variable on the track.
  Optionally settings could be used to specifiy a different unit, or the value range for the colorbar::

    ! convert units:
    cso.tutorial.catalogue.var.vcd.units          :  1e15 mlc/cm2
    ! style:
    cso.tutorial.catalogue.var.vcd.vmin           :   0.0
    cso.tutorial.catalogue.var.vcd.vmax           :  10.0

  Figures are saved to files with the basename of the original orbit file and the plotted variable::

      /Scratch/CSO/catalogue/2018/06/01/S5p_RPRO_CO_03278__vcd.png
                                        S5p_RPRO_CO_03278__qa_value.png
                                        :

  .. figure:: figs/CO/S5p_RPRO_CO_03278__vcd.png 
     :scale: 50 %
     :align: center
     :alt: S5p co columns

  To search for interesting features in the data, 
  the :py:class:`Indexer <utopya_index.Indexer>` class could be used to create index pages.
  Configuration could look like::

      ! index creation task:
      cso.s5p.co.catalogue.task.index.class     :  utopya.Indexer
      cso.s5p.co.catalogue.task.index.args      :  '${PWD}/config/Copernicus/cso-s5p-co.rc', \
                                                     rcbase='cso.s5p.co.catalogue-index'

  When succesful, the index creator displays an url that could be loaded in a browser::

      Browse to:
        file:///Scratch/CSO/catalogue/index.html

  .. figure:: figs/CO/CSO_CO_catalogue.png
     :scale: 50 %
     :align: center
     :alt: Index for S5p CO columns



  Configuration of observation operator
  =====================================

  The *observation operator* described in chapter ':ref:`obsoper`' requires settings from
  an rcfile.

  First specify the (relative) location of the *listing* file with orbit file names and time ranges::

      ! template for listing with converted files:
      <rcbase>.listing           : ../S5p/RPRO/CO/CAMS/listing.csv

  The operator should read variables from the data files that are needed to simulate a retrieval
  from the model arrays.
  This includes for example the pressures that define the *a priori* layers, the averaging kernel,
  and for this product, the airmass factor and tropopause level.
  Specify a list of names for these variables::

    ! data variables:
    tutorial.S5p.co.dvars             :  hp yr vr A M nla

  Example settings::

    ! half-level pressures:
    !~ dimensions, copied from data file:
    tutorial.S5p.co.dvar.hp.dims      :  layeri
    !~ source variable:
    tutorial.S5p.co.dvar.hp.source    :  pressure

    ! retrieval: 
    !~ dimensions, copied from data file:
    tutorial.S5p.co.dvar.yr.dims      :  retr
    !~ source variable:
    tutorial.S5p.co.dvar.yr.source    :  vcd

    ! retrieval error covariance: 
    !~ dimensions, copied from data file:
    tutorial.S5p.co.dvar.vr.dims      :  retr retr
    !~ source variable:
    tutorial.S5p.co.dvar.vr.source    :  vcd_errvar

    ! kernel:
    !~ dimensions, copied from data file:
    tutorial.S5p.co.dvar.A.dims       :  retr layer
    !~ source variable:
    tutorial.S5p.co.dvar.A.source     :  kernel_trop

    ! number of apriori layers in retrieval layer:
    !~ dimensions, copied from data file:
    tutorial.S5p.co.dvar.nla.dims     :  retr
    !~ source variable:
    tutorial.S5p.co.dvar.nla.source   :  nla

  For the simulated values, also define a list of variable names that should be created::

    ! state varaiables to be put out from model:
    tutorial.S5p.co.vars                         :  mod_conc mod_hp mod_tcc mod_cc hx ys shx

  Example settings::

    ! model concentration profile:
    !~ model layer dimension:
    tutorial.S5p.co.var.mod_conc.dims            :  model_layer
    !~ standard attributes:
    tutorial.S5p.co.var.mod_conc.attrs           :  long_name units
    tutorial.S5p.co.var.mod_conc.attr.long_name  :  model CO concentrations
    tutorial.S5p.co.var.mod_conc.attr.units      :  ppb

    ! model hpentration profile:
    !~ model layer interfaces:
    tutorial.S5p.co.var.mod_hp.dims              :  model_layeri
    !~ standard attributes:
    tutorial.S5p.co.var.mod_hp.attrs             :  long_name units
    tutorial.S5p.co.var.mod_hp.attr.long_name    :  model pressure at layer interfaces
    tutorial.S5p.co.var.mod_hp.attr.units        :  Pa

    ! total cloud cover:
    !~ no extra dimensions:
    tutorial.S5p.co.var.mod_tcc.dims             :  
    !~ standard attributes:
    tutorial.S5p.co.var.mod_tcc.attrs            :  long_name units
    tutorial.S5p.co.var.mod_tcc.attr.long_name   :  total cloud cover
    tutorial.S5p.co.var.mod_tcc.attr.units       :  1

    ! cloud cover profiles:
    !~ model layer dimension:
    tutorial.S5p.co.var.mod_cc.dims              :  model_layer
    !~ standard attributes:
    tutorial.S5p.co.var.mod_cc.attrs             :  long_name units
    tutorial.S5p.co.var.mod_cc.attr.long_name    :  cloud cover
    tutorial.S5p.co.var.mod_cc.attr.units        :  1

    ! model concentrations at apriori layers:
    !~ apriori layers:
    tutorial.S5p.co.var.hx.dims                  :  layer
    !~ how computed:
    tutorial.S5p.co.var.hx.formula               :  LayerAverage( hp, mod_hp, mod_conc )
    tutorial.S5p.co.var.hx.formula_terms         :  hp: hp mod_hp: mod_hp mod_conc: mod_conc
    !~ standard attributes:
    tutorial.S5p.co.var.hx.attrs                 :  long_name units
    tutorial.S5p.co.var.hx.attr.long_name        :  model simulations at apriori layers
    tutorial.S5p.co.var.hx.attr.units            :  mol m-2

    ! simulated retrievals
    !~ retrieval layers:
    tutorial.S5p.co.var.ys.dims                  :  retr
    !~ how computed:
    tutorial.S5p.co.var.ys.formula               :  A x
    tutorial.S5p.co.var.ys.formula_terms         :  A: A x: hx
    !~ standard attributes:
    tutorial.S5p.co.var.ys.attrs                 :  long_name units
    tutorial.S5p.co.var.ys.attr.long_name        :  simulated retrieval
    tutorial.S5p.co.var.ys.attr.units            :  mol m-2

    ! partial columns as sum over apriori layers
    !~ retrieval layers:
    tutorial.S5p.co.var.shx.dims                 :  retr
    !~ how computed:
    tutorial.S5p.co.var.shx.formula              :  PartialColumns( nla, x )
    tutorial.S5p.co.var.shx.formula_terms        :  nla: nla x: hx
    !~ standard attributes:
    tutorial.S5p.co.var.shx.attrs                :  long_name units
    tutorial.S5p.co.var.shx.attr.long_name       :  tropospheric column in local model
    tutorial.S5p.co.var.shx.attr.units           :  mol m-2


  Sim-Catalogue
  =============

  The :py:class:`CSO_Catalogue <.cso_catalogue.CSO_Catalogue>` class could be used
  to create a catalogue of images for the converted files.
  Configuration could look like::

      ! catalogue creation task:
      cso.s5p.TRACER.sim-catalogue.task.class          :  cso.CSO_SimCatalogue
      cso.s5p.TRACER.sim-catalogue.task.args           :  '${PWD}/config/Copernicus/cso-s5p-TRACER.rc', \
                                                        rcbase='cso.s5p.TRACER.sim-catalogue'

  The configuration describes where to find a *listing* file with orbits, 
  which variables should be plot, the colorbar properties, etc.
  See :py:class:`CSO_SimCatalogue <.cso_s5p.CSO_SimCatalogue>` class description for how
  the settings in general look like.

  The class creates figures for a list of variables::

    ! variables to be plotted:
    cso.s5p.co.catalogue.vars                    :  yr ys

  By default the catalogue creator simply creates a map with the value of the a variable on the track.
  Optionally settings could be used to specifiy a different unit, or the value range for the colorbar::

      ! variable:
      cso.s5p.co.sim-catalogue.var.yr.source          :  data:vcd
      ! convert units:
      cso.s5p.co.sim-catalogue.var.yr.units           :  1e15 mlc/cm2
      ! style:
      cso.s5p.co.sim-catalogue.var.yr.vmin            :   0.0
      cso.s5p.co.sim-catalogue.var.yr.vmax            :  50.0

      ! variable:
      cso.s5p.co.sim-catalogue.var.ys.source          :  state:y
      ! convert units:
      cso.s5p.co.sim-catalogue.var.ys.units           :  1e15 mlc/cm2
      ! style:
      cso.s5p.co.sim-catalogue.var.ys.vmin            :   0.0
      cso.s5p.co.sim-catalogue.var.ys.vmax            :  50.0

  Figures are saved to files with the basename of the original orbit file and the plotted variable::

       file://${my.run.base}/cso-catalogue/CO//2018/06/01/S5p_RPRO_CO_20180601_1200_yr.png
                                                            S5p_RPRO_CO_20180601_1200_ys.png


  To search for interesting features in the data, 
  the :py:class:`Indexer <utopya_index.Indexer>` class could be used to create index pages.
  Configuration could look like::

      ! index creation task:
      cso.s5p.co.catalogue.task.index.class     :  utopya.Indexer
      cso.s5p.co.catalogue.task.index.args      :  '${PWD}/config/Copernicus/cso-s5p-co.rc', \
                                                     rcbase='cso.s5p.co.catalogue-index'

  When succesful, the index creator displays an url that could be loaded in a browser::

      Browse to:
        file://${my.run.base}/cso-catalogue/CO/index__20180601.html

  .. figure:: figs/CO/CSO_CO_sim-catalogue.png
     :scale: 50 %
     :align: center
     :alt: Index for Simulated and S5p CO columns



