
.. Label between '.. _' and ':' ; use :ref:`text <label>` for reference
.. _s5p-o3-processing:

******************************
Sentinel-5p O3 data processing
******************************

This chapter describes the tasks performed for processing Sentinel-5p O\ :sub:`3` data.


.. _s5p-o3-product:

Product description
===================

The product guides can be found at:

* `SentiWiki / Sentinel-5P <https://sentiwiki.copernicus.eu/web/s5p-products>`_

There are several ozone products available

* ``L2__O3____`` : *ozone total column* product.

  See the `PUM-O3 <https://sentiwiki.copernicus.eu/__attachments/1673595/S5P-L2-DLR-PUM-400A%20-%20Sentinel-5P%20Level%202%20Product%20User%20Manual%20Ozone%20Total%20Column%202022%20-%202.4.pdf>`_ Product User Manual.

  * The primary retrieval is vertical column density (mol/m\ :sup:`2`) on :math:`n_r=1` layers:

    .. math::
        y_r

  * The simulation of a retrieval product from a model state requires an apriori column,
    and should be computed from:

    .. math::
        y_s ~=~ \sum \mathbf{y}_a\ +\ \mathbf{A}\ \left(\ \mathbf{V}\mathbf{G}\ \mathbf{x}\ -\ \mathbf{y}_a\ \right)

    where:

    * :math:`y_s` is the simulated retrieval (mol/m\ :sup:`2`) on :math:`n_r=1` layers;
    * :math:`\mathbf{y}_a` is the *a priori* vertical column density (mol/m\ :sup:`2`) defined on :math:`n_a=14` layers;
    * :math:`\mathbf{A}` is the averaging kernel matrix with shape :math:`(n_r,n_a)`;
    * :math:`\mathbf{x}` is the atmospheric state, which probably consists of a 3D array of O\ :sub:`3` concentrations;
    * operators :math:`\mathbf{G}` and :math:`\mathbf{V}` together compute a simulated profile 
      at the :math:`n_a` *a priori* layers from the state, using horizontal (:math:`\mathbf{G}`)
      and vertical (:math:`\mathbf{V}`) mappings;
      units should be the same as the retrieval product (mol/m\ :sup:`2`).

  * In case :math:`\mathbf{x}^{true}` is the true atmoshperic state, the retrieval error is quantified
     by the *retrieval error covariance* :math:`\mathbf{R}`:

    .. math::
        \mathbf{y}_s ~-~ \left[ \mathbf{y}_a\ +\ \mathbf{A}\ \left(\ \mathbf{V}\mathbf{G}\ \mathbf{x}^{true}\ -\ \mathbf{y}_a\ \right)\right] ~\sim~ \mathcal{N}\left(\mathbf{o},\mathbf{R}\right)

    The product does not provide a full matrix :math:`\mathbf{B}` but only a *precission profile*;
    here we define :math:`\mathbf{R}` to be diagonal with the squares of the *precission* as diagonal elements.

* ``L2__O3__PR``  : *ozone profile* product.

  See the `PUM-O3_PR <https://sentiwiki.copernicus.eu/__attachments/1673595/S5P-KNMI-L2-0020-MA%20-%20Sentinel-5P%20Level%202%20Product%20User%20Manual%20Ozone%20profiles%202023%20-%202.6.0.pdf>`_ Product User Manual.

  * The primary retrieval is an ozone concentration (mol/m\ :sup:`3`) profile at :math:`n_r=33` layers:

    .. math::
        \mathbf{y}_r

    In addition, the product supplies derived products:

    * on 6 sub-columns defined by static altitudes 0-6, 6-12, 12-18, 18-24, 24-32, and 32-82 km;
    * a tropospheric column (probably between ``surface_pressure`` and ``pressure_at_tropopause``);
    * a total column product.

    As there are no averaging kernels supplied for the (sub)columns, only the full profiles will be used here.

  * The simulation of a retrieval product from a model state requires an apriori profile,
    and should be computed from:

    .. math::
        \mathbf{y}_s ~=~ \mathbf{y}_a\ +\ \mathbf{A}\ \left(\ \mathbf{V}\mathbf{G}\ \mathbf{x}\ -\ \mathbf{y}_a\ \right)

    where:

    * :math:`\mathbf{y}_s` is the simulated retrieval (mol/m\ :sup:`3`) defined on :math:`n_r` layers;
    * :math:`\mathbf{y}_a` is the *a priori* retrieval (mol/m\ :sup:`3`) defined on :math:`n_a=33` layers;
    * :math:`\mathbf{A}` is the averaging kernel matrix with shape :math:`(n_r,n_a)`;
    * :math:`\mathbf{x}` is the atmospheric state, which probably consists of a 3D array of O\ :sub:`3` concentrations;
    * operators :math:`\mathbf{G}` and :math:`\mathbf{V}` together compute a simulated profile 
      at the :math:`n_a` *a priori* layers from the state, using horizontal (:math:`\mathbf{G}`)
      and vertical (:math:`\mathbf{V}`) mappings;
      units should be the same as the retrieval product (mol/m\ :sup:`3`).

  * In case :math:`\mathbf{x}^{true}` is the true atmoshperic state, the retrieval error is quantified
     by the *retrieval error covariance* :math:`\mathbf{R}`:

    .. math::
        \mathbf{y}_s ~-~ \left[ \mathbf{y}_a\ +\ \mathbf{A}\ \left(\ \mathbf{V}\mathbf{G}\ \mathbf{x}^{true}\ -\ \mathbf{y}_a\ \right)\right] ~\sim~ \mathcal{N}\left(\mathbf{o},\mathbf{R}\right)

    The product does not provide a full matrix :math:`\mathbf{B}` but only a *precission profile*;
    here we define :math:`\mathbf{R}` to be diagonal with the squares of the *precission* as diagonal elements.

  * Recomended pixel selection:

    * The retrieval status and quality is indicated by the ``qa_value``. 
      The recommended minimum is 0.80, this excludes cloudy scenes and other problematic retrievals.
    * In addition, it is recommended to disregard pixels that match the following criterion:   

      .. math::
          \sum_{i=1}^{n_r} \left| y_r(i) - y_a(i) \right|  ~>~  \mbox{1.66e-5}  ~~~~~~\mbox{[mol/m$^3$]}

      *NOTE: older versions of the *Product User Manual* show upper-entiers* (:math:`\lceil\cdot\rceil`)*
      around the difference between retrieval and apriori, this should have been absolute values as used here.*

* ``L2__O3_TCL`` : *ozone tropospheric column* product.
      Level3 product, tropics band, 3-days running average on 1.0 x 0.5 (lon x lat) grid?
      Not used here.
      


CSO processing
==============

*(See* :ref:`tutorial` *chapter for introduction to CSO scripts and configuration)*

An example configuration of the CSO processing of the S5p/O\ :sub:`3` data is available via
the following settings:

* `config/Copernicus/cso.rc <../../../config/Copernicus/cso.rc>`_

  Top-level settings that configure the job-tree with various sub-tasks.
  This is a generic file that could be used for multiple S5 products, 
  edit it to select the O\ :sub:`3` processing.

* `config/Copernicus/cso-user-settings.rc <../../../config/Copernicus/cso-user-settings.rc>`_

  User-specific settings such as the work directory.

* `config/Copernicus/cso-s5p-o3.rc <../../../config/Copernicus/cso-s5p-o3.rc>`_

  Specific settings for O\ :sub:`3` product.

Start the job-tree using::

  ./bin/cso  config/Copernicus/cso.rc

Selected sub-steps in the processing are described below.


.. Label between '.. _' and ':' ; use :ref:`text <label>` for reference
.. _s5p-o3-inquire:

Inquire Sentinel-5p/O3-profile archives
=======================================

S5p/O3-profile observations are available from:

* `Copernicus DataSpace <https://dataspace.copernicus.eu/>`_;
  see the :ref:`cso-dataspace` module for a detailed description.

Data is available for different processing streams, each identified by a 4-character key:

* ``NRTI`` : `Near real time`, available with a day after observation;
* ``OFFL`` : `Offline`, available within weeks after observations;
* ``RPRO`` : re-processing of all previously made observations.

The :py:class:`CSO_DataSpace_Inquire <cso_dataspace.CSO_DataSpace_Inquire>` class is available to inquire the
*Copernicus DataSpace*. The settings used by this class allow selection
on for example time range and intersection area. 
The result is a csv file which with columns for keywords such as orbit number and processor version,
as well as the filename of the data and the url that should be used to actually download the data::

    orbit;start_time;end_time;processing;collection;processor_version;filename;href
    02818;2018-04-30 00:19:50;2018-04-30 02:01:20;RPRO;03;020400;S5P_RPRO_L2__O3__PR_20180430T001950_20180430T020120_02818_03_020400_20230222T152833.nc;https://zipper.dataspace.copernicus.eu/odata/v1/Products(5192222a-1311-42a2-b557-3f507a26defe)/$value
    02819;2018-04-30 02:01:20;2018-04-30 03:42:50;RPRO;03;020400;S5P_RPRO_L2__O3__PR_20180430T020120_20180430T034250_02819_03_020400_20230222T152835.nc;https://zipper.dataspace.copernicus.eu/odata/v1/Products(8dd92be6-e866-46db-9247-1d34408334bc)/$value
    :

See the section on *File name convention* in the *Product User Manual* for the meaning of all 
parts of the filename.

To visualize the available data, the
:py:class:`CSO_Inquire_Plot <cso_inquire.CSO_Inquire_Plot>` could be used to create an overview figure:

.. figure:: figs/O3-PR/Copernicus_S5p_O3-PR.png
   :scale: 50 %
   :align: center
   :alt: Overview of available O\ :sub:`3` processings.

The jobtree configuration to inquire the portals and create the overview figure could look like::

    ! single step:
    cso.s5p.o3.inquire.class                      :  utopya.UtopyaJobStep
    ! two tasks:
    cso.s5p.o3.inquire.tasks                      :  table-dataspace plot

    !~ inquire files available on DataSpace:
    cso.s5p.o3.inquire.table-dataspace.class      :  cso.CSO_DataSpace_Inquire
    cso.s5p.o3.inquire.table-dataspace.args       :  '${PWD}/config/Copernicus/cso-s5p-o3.rc', \
                                                         rcbase='cso.s5p.o3.inquire-table-dataspace'

    !~ create plot of available versions:
    cso.s5p.o3.inquire.plot.class                 :  cso.CSO_Inquire_Plot
    cso.s5p.o3.inquire.plot.args                  :  '${PWD}/config/Copernicus/cso-s5p-o3.rc', \
                                                          rcbase='cso.s5p.o3.inquire-plot'




.. Label between '.. _' and ':' ; use :ref:`text <label>` for reference
.. _s5p-o3-convert:

Conversion to CSO format
========================

The '``cso.s5p.o3.convert``' task converts orbit files downloaded from a portal into a CSO format.

Files are downloaded from a portal if not present locally yet; eventually they are also removed
after conversion to avoid that the portal is completely mirrored.

To save storage, only selected pixels are included in the converted files,
for example only those within some region or cloud free pixels.
The selection criteria are defined in the settings, and added
to the '``history``' attribute of the created files as reminder.

The work is done by the :py:class:`.CSO_S5p_Convert` class,
which is initialized using the settings file::

  ! task initialization:
  cso.s5p.o3.convert.class     :  cso.CSO_S5p_Convert
  cso.s5p.o3.convert.args      :  '${PWD}/config/Copernicus/cso-s5p-o3.rc', rcbase='cso.s5p.o3.convert'

See the class documentation for the general configuration,
below some specific choices are described.
The example is based on the S5p O\ :sub:`3`-profile file from which the header is available in:

* `doc/samples/S5P_RPRO_L2__O3__PR_20180601T002121_20180601T020251_03272_03_020400_20230224T110239.txt <../../samples/S5P_RPRO_L2__O3__PR_20180601T002121_20180601T020251_03272_03_020400_20230224T110239.txt>`_



.. TBD

    Orbit file selection
    --------------------

    Based on the inquiry the download and conversion could be limitted to files created with the most recent processor versions.

    For the S5P files a useful property is also the *collection number*, a 2-digit number that defines a collection of files
    (or actually processor versions) that together form a contineous series. The *collection number* is extracted from the filename,
    and stored as a column of the listing file.

    The following setting is used to select specific files from the archive based on the properities stored
    in the listing file::

        ! Provide ';' seperated list of to decide if a particular orbit file should be processed.
        ! If more than one file is available for a particular orbit (from "OFFL" and "RPRO" processing),
        ! the file with the first match will be used.
        ! The expressions should include templates '%{header}' for the column values.
        ! Example to select files from collection '03', preferably from processing 'RPRO' but otherwise from 'OFFL':
        !   (%{collection} == '03') and (%{processing} == 'RPRO') ; \
        !   (%{collection} == '03') and (%{processing} == 'OFFL')
        !
        cso.s5p.o3.convert.selection                     :  (%{collection} == '03') and (%{processing} == 'RPRO') ; \
                                                             (%{collection} == '03') and (%{processing} == 'OFFL')


    Pixel selection
    ---------------

    The :py:class:`.CSO_S5p_Convert` class calls the :py:meth:`.S5p_File.SelectPixels` method
    to create a pixel selection mask for the input file.
    The selection is done using one or more filters.
    First provide a list of filter names::

      cso.s5p.o3.convert.filters   :  lons lats valid quality cloud_fraction

    Then provide for each filter the the input variable to be used for testing,
    as a path name in the input file.
    The next settings is the type of filter to be used, see the :py:meth:`.S5p_File.SelectPixels` for supported types,
    and the other settings required by the type.
    The following is an example of a selection on longitude::

      cso.s5p.o3.convert.filter.lons.var                :  Geolocation Fields/Longitude
      cso.s5p.o3.convert.filter.lons.type               :  minmax
      cso.s5p.o3.convert.filter.lons.minmax             :  -30.0 45.0
      cso.s5p.o3.convert.filter.lons.units              :  degrees_east


    Variable specification
    ----------------------

    The target file is created as an :py:class:`.CSO_S5p_File` object.
    It's :py:meth:`AddSelection <.CSO_S5p_File.AddSelection>` method is called with the input object as argument,
    and this will copy the selected pixels for variables specified in the settings.

    The variable specification starts with a list with variable names to be 
    created in the target file::

      cso.s5p.o3.convert.output.vars   :  longitude longitude_bounds \
                                           latitude latitude_bounds \
                                           track_longitude track_longitude_bounds \
                                           track_latitude  track_latitude_bounds \
                                           time \
                                           mconc mconc_errvar \
                                           pressure kernel_trop amf amf_trop l_tp nla \
                                           qa_value \
                                           cloud_fraction

    For each variable settings should be specified that describe the shape of the variable
    and how it should be filled from the input.
    See the :py:meth:`AddSelection <.CSO_S5p_File.AddSelection>` description for all options,
    here we show some examples.

    The ``longitude`` and ``latitude`` variables are copied almost directly out of the source files,
    the only change that is applied is the selection of pixels.
    All original attributes are copied, except for the ``bound`` attribite since that would
    give warnings from the CF-compliance checker::

      cso.s5p.o3.convert.output.var.longitude.dims                   :   pixel
      cso.s5p.o3.convert.output.var.longitude.from                   :   PRODUCT/longitude
      cso.s5p.o3.convert.output.var.longitude.attrs                  :   { 'bounds' : None }

      cso.s5p.o3.convert.output.var.latitude.dims                    :   pixel
      cso.s5p.o3.convert.output.var.latitude.from                    :   PRODUCT/latitude
      cso.s5p.o3.convert.output.var.latitude.attrs                   :   { 'bounds' : None }

    The pixel boundaries are necessary to know the exact footprint of a pixel,
    which is for example used when averaging over a grid or simulation from a model.
    These are available in the input files, but without a ``units`` attribute as these
    are implied by the pixel center coordinate; the conversion therefore requires that
    units are defined explicitly.
    For the ``longitude_bounds`` a special processing is needed for pixels crossing the dateline,
    as the original data simply uses longitudes modulo 360 degrees::

      ! corner longitudes; no units in file:
      cso.s5p.o3.convert.output.var.longitude_bounds.dims            :   pixel corner
      cso.s5p.o3.convert.output.var.longitude_bounds.from            :   PRODUCT/SUPPORT_DATA/GEOLOCATIONS/longitude_bounds
      cso.s5p.o3.convert.output.var.longitude_bounds.units           :   degrees_east
      ! ensure that near dateline the corners form a convex region around center
      ! (with some points outside [-180,+180] if necessary)
      cso.s5p.o3.convert.output.var.longitude_bounds.special         :   longitude_bounds

      ! corner latitudes, no units in file:
      cso.s5p.o3.convert.output.var.latitude_bounds.dims             :   pixel corner
      cso.s5p.o3.convert.output.var.latitude_bounds.from             :   PRODUCT/SUPPORT_DATA/GEOLOCATIONS/latitude_bounds
      cso.s5p.o3.convert.output.var.latitude_bounds.units            :   degrees_north

    Also the locations of the pixels in the original track are copied,
    since these are useful when creating plots. These cannot be copied directly but require special processing::

      cso.s5p.o3.convert.output.var.track_longitude.dims             :   track_scan track_pixel
      cso.s5p.o3.convert.output.var.track_longitude.special          :   track_longitude
      cso.s5p.o3.convert.output.var.track_longitude.from             :   PRODUCT/longitude
      cso.s5p.o3.convert.output.var.track_longitude.attrs            :   { 'bounds' : None }

      cso.s5p.o3.convert.output.var.track_latitude.dims              :   track_scan track_pixel
      cso.s5p.o3.convert.output.var.track_latitude.special           :   track_latitude
      cso.s5p.o3.convert.output.var.track_latitude.from              :   PRODUCT/latitude
      cso.s5p.o3.convert.output.var.track_latitude.attrs             :   { 'bounds' : None }

    The observattion times are constructed from time steps relative to a reference time;
    this requires special processing too::

      cso.s5p.o3.convert.output.var.time.dims                        :   pixel
      cso.s5p.o3.convert.output.var.time.special                     :   time-delta
      cso.s5p.o3.convert.output.var.time.tref                        :   PRODUCT/time
      cso.s5p.o3.convert.output.var.time.dt                          :   PRODUCT/delta_time

    The observed  mole concentratin could be copied directly.
    The target shape is ``(pixel,retr)`` where ``retr`` is the number of layers in the retrieval product (1 in this case)::

      ! vertical column density:
      cso.s5p.o3.convert.output.var.mconc.dims                         :   pixel retr
      cso.s5p.o3.convert.output.var.mconc.from                         :   PRODUCT/nitrogendioxide_tropospheric_column

    In the converted files, the retrieval error is always expressed as a (co)variance matrix,
    to facilitate (future) conversion of profile products.
    In this example, it is filled from the square of the error standard deviation::

      ! error variance in vertical column density (after application of kernel),
      ! fill with single element 'covariance matrix', from square of standard error:
      ! use dims with different names to avoid that cf-checker complains:
      cso.s5p.o3.convert.output.var.mconc_errvar.dims                  :   pixel retr retr0
      cso.s5p.o3.convert.output.var.mconc_errvar.special               :   square
      cso.s5p.o3.convert.output.var.mconc_errvar.from                  :   PRODUCT/nitrogendioxide_tropospheric_column_precision_kernel
      !~ skip standard name, modifier "standard_error" is not valid anymore:
      cso.s5p.o3.convert.output.var.mconc_errvar.attrs                 :   { 'standard_name' : None }

    The averaging kernel is applied on atmospheric layers, defined by pressure levels.
    In this product the pressure levels are defined using hybride-sigma-pressure coordinates,
    and this requires special processing::

      ! Convert from hybride coefficient bounds in (2,nlev) aray to 3D half level pressure:
      cso.s5p.o3.convert.output.var.pressure.dims                    :   pixel layeri
      cso.s5p.o3.convert.output.var.pressure.special                 :   hybounds_to_pressure
      cso.s5p.o3.convert.output.var.pressure.sp                      :   PRODUCT/SUPPORT_DATA/INPUT_DATA/surface_pressure
      cso.s5p.o3.convert.output.var.pressure.hyab                    :   PRODUCT/tm5_constant_a
      cso.s5p.o3.convert.output.var.pressure.hybb                    :   PRODUCT/tm5_constant_b
      cso.s5p.o3.convert.output.var.pressure.units                   :   Pa

    Averaging kernels are converted to matrices with shape ``(layer,retr)``.
    Here, the averaging kernels of the tropospheric product should be computed using a ratio
    between total and tropospheric airmass factors::

      ! description:
      !   kernel := averaging_kernel * amf/amft
      cso.s5p.o3.convert.output.var.kernel_trop.dims                 :   pixel layer retr
      cso.s5p.o3.convert.output.var.kernel_trop.special              :   kernel_trop
      cso.s5p.o3.convert.output.var.kernel_trop.avk                  :   PRODUCT/averaging_kernel
      cso.s5p.o3.convert.output.var.kernel_trop.amf                  :   PRODUCT/air_mass_factor_total
      cso.s5p.o3.convert.output.var.kernel_trop.amft                 :   PRODUCT/air_mass_factor_troposphere
      cso.s5p.o3.convert.output.var.kernel_trop.troplayer            :   PRODUCT/tm5_tropopause_layer_index
      cso.s5p.o3.convert.output.var.kernel_trop.attrs                :   { 'coordinates' : None, 'ancillary_variables' : None }

    For the airmass factor correction, the airmass factors and the tropopause level are needed::

      ! (total) airmass factor:
      cso.s5p.o3.convert.output.var.amf.dims                         :   pixel retr
      cso.s5p.o3.convert.output.var.amf.from                         :   PRODUCT/air_mass_factor_total
      cso.s5p.o3.convert.output.var.amf.attrs                        :   { 'coordinates' : None, 'ancillary_variables' : None }

      ! tropospheric airmass factor:
      cso.s5p.o3.convert.output.var.amf_trop.dims                    :   pixel retr
      cso.s5p.o3.convert.output.var.amf_trop.from                    :   PRODUCT/air_mass_factor_troposphere
      cso.s5p.o3.convert.output.var.amf_trop.attrs                   :   { 'coordinates' : None, 'ancillary_variables' : None }

      ! number of apriori layers in retrieval layer,
      ! enforce that it is stored as a short integer:
      cso.s5p.o3.convert.output.var.nla.dims                         :   pixel retr
      cso.s5p.o3.convert.output.var.nla.dtype                        :   i2
      cso.s5p.o3.convert.output.var.nla.from                         :   PRODUCT/tm5_tropopause_layer_index
      cso.s5p.o3.convert.output.var.nla.attrs                        :   { 'coordinates' : None, 'ancillary_variables' : None }

    Other variables can be copied directly::

      ! quality flag:
      cso.s5p.o3.convert.output.var.qa_value.dims                   :   pixel
      cso.s5p.o3.convert.output.var.qa_value.from                   :   PRODUCT/qa_value
      !~ skip some attributes, cf-checker complains ...
      cso.s5p.o3.convert.output.var.qa_value.attrs                  :   { 'valid_min' : None, 'valid_max' : None }

      ! cloud property:
      cso.s5p.o3.convert.output.var.cloud_fraction.dims             :   pixel
      cso.s5p.o3.convert.output.var.cloud_fraction.from             :   PRODUCT/SUPPORT_DATA/INPUT_DATA/cloud_fraction_crb
      cso.s5p.o3.convert.output.var.cloud_fraction.attrs            :   { 'coordinates' : None, 'source' : None }

      ! cloud property:
      cso.s5p.o3.convert.output.var.cloud_radiance_fraction.dims    :   pixel
      cso.s5p.o3.convert.output.var.cloud_radiance_fraction.from    :   PRODUCT/SUPPORT_DATA/DETAILED_RESULTS/cloud_radiance_fraction_nitrogendioxide_window
      cso.s5p.o3.convert.output.var.cloud_radiance_fraction.attrs   :   { 'coordinates' : None, 'ancillary_variables' : None }



    .. _s5p-o3-processing-output:

    Output files
    ------------

    The name of the target files should be specified with a directory and filename;
    the later could include a template for the orbit number::

        ! output directory and filename:
        ! - times are taken from mid of selection, rounded to hours
        ! - use '%{orbit}' for orbit number
        cso.s5p.o3.convert.output.filename     :  /Scratch/CSO-data/Europe/S5p/O3/C03/%Y/%m/S5p_O3_%{orbit}.nc

    A flag is read to decide if existing files should be renewed or kept::

        cso.s5p.o3.convert.renew                  :  True     

    The target file is created as an :py:class:`.CSO_S5p_File` object.
    It's :py:meth:`AddSelection <.CSO_S5p_File.AddSelection>` method is called with the input object as argument,
    and this will copy the selected pixels for variables specified in the settings.
    The :py:meth:`Write <.CSO_File.Write>` method creates the file.

    Global attributes for the target file should be specified with::

        ! global attributes:
        cso.s5p.o3.convert.output.attrs               :  format Conventions author institution email
        !
        cso.s5p.o3.convert.output.attr.format         :  1.0
        cso.s5p.o3.convert.output.attr.Conventions    :  CF-1.7
        cso.s5p.o3.convert.output.attr.author         :  Your Name
        cso.s5p.o3.convert.output.attr.institution    :  CSO
        cso.s5p.o3.convert.output.attr.email          :  Your.Name@cso.org


    .. Label between '.. _' and ':' ; use :ref:`text <label>` for reference
    .. _s5p-o3-listing:

    Listing file
    ============

    A *listing* file contains the names of the converted orbit files,
    and the time range of pixels in the file::

        filename                     ;start_time                   ;end_time                     ;orbit
        2018/06/S5p_RPRO_O3_03272.nc;2018-06-01T01:32:46.673000000;2018-06-01T01:36:12.948000000;03272
        2018/06/S5p_RPRO_O3_03273.nc;2018-06-01T03:12:53.649000000;2018-06-01T03:17:43.082000000;03273
        2018/06/S5p_RPRO_O3_03274.nc;2018-06-01T04:52:43.586000000;2018-06-01T04:59:12.377000000;03274
        :

    This file will be used by the observation operator to selects orbits with pixels valid for 
    a desired time range.

    A listing file is for example created using the :py:class:`.CSO_S5p_Listing` class.
    In the settings passed to the class, define the name of the file to be created::

        ! csv file that will hold records per file with:
        ! - timerange of pixels in file
        ! - orbit number
        <rcbase>.file        :   /Scratch/CSO/S5p/listing-O3-Europe.csv

    An existing listing files is not replaced,
    unless the following flag is set::

        ! renew table?
        <rcbase>.renew           :  True

    Orbit files are searched within a timerange::

        <rcbase>.timerange.start        :  2018-06-01 00:00
        <rcbase>.timerange.end          :  2018-06-03 23:59

    Specify filename filters to search for orbit files;
    the patterns are relative to the basedir of the listing file,
    and might contain templates for the time values.
    Multiple patterns could be defined; if for a certain orbit number more than one
    file is found, the first match is used.
    This could be explored to create a listing that combines reprocessed data
    with near-real-time data::

        <rcbase>.patterns            :  CO3/%Y/%m/S5p_*.nc



    .. Label between '.. _' and ':' ; use :ref:`text <label>` for reference
    .. _s5p-o3-catalogue:

    Catalogue
    =========

    The :py:class:`CSO_Catalogue <.cso_catalogue.CSO_Catalogue>` class could be used
    to create a catalogue of images for the converted files.
    Configuration could look like::

        ! catalogue creation task:
        cso.s5p.o3.catalogue.task.figs.class  :  cso.CSO_Catalogue
        cso.s5p.o3.catalogue.task.figs.args   :  '${PWD}/config/Copernicus/cso-s5p-o3.rc', \
                                                    rcbase='cso.s5p.o3.catalogue'

    The configuration describes where to find a *listing* file with orbits, 
    which variables should be plot, the colorbar properties, etc.
    See :py:class:`CSO_Catalogue <.cso_s5p.CSO_Catalogue>` class description for how
    the settings in general look like.

    The class creates figures for a list of variables::

      ! variables to be plotted:
      cso.s5p.o3.catalogue.vars                    :  mconc mconc_errvar qa_value \
                                                          cloud_fraction cloud_radiance_fraction

    By default the catalogue creator simply creates a map with the value of the a variable on the track.
    Optionally settings could be used to specifiy a different unit, or the value range for the colorbar::

      ! convert units:
      cso.s5p.o3.catalogue.var.mconc.units          :  mol/m3
      ! style:
      cso.s5p.o3.catalogue.var.mconc.vmin           :   0.0
      cso.s5p.o3.catalogue.var.mconc.vmax           :  50.0

    Figures are saved to files with the basename of the original orbit file and the plotted variable::

        /Scratch/CSO/catalogue/2018/06/01/S5p_RPRO_O3_03278__mconc.png
                                          S5p_RPRO_O3_03278__qa_value.png
                                          :

    .. figure:: figs/O3/S5p_RPRO_O3_03278__mconc.png 
       :scale: 50 %
       :align: center
       :alt: S5p O\ :sub:`3` columns

    To search for interesting features in the data, 
    the :py:class:`Indexer <utopya_index.Indexer>` class could be used to create index pages.
    Configuration could look like::

        ! index creation task:
        cso.s5p.o3.catalogue.task.index.class     :  utopya.Indexer
        cso.s5p.o3.catalogue.task.index.args      :  '${PWD}/config/Copernicus/cso-s5p-o3.rc', \
                                                       rcbase='cso.s5p.o3.catalogue-index'

    When succesful, the index creator displays an url that could be loaded in a browser::

        Browse to:
          file:///Scratch/CSO/catalogue/index.html

    .. figure:: figs/O3/CSO_O3_catalogue.png
       :scale: 50 %
       :align: center
       :alt: Index for S5p O3 columns



    Configuration of observation operator
    =====================================

    The *observation operator* described in chapter ':ref:`obsoper`' requires settings from
    an rcfile.

    First specify the (relative) location of the *listing* file with orbit file names and time ranges::

        ! template for listing with converted files:
        <rcbase>.listing           : ../S5p/RPRO/O3/CAMS/listing.csv

    The S5p data contains data defined on orbit tracks, this should be read from the files::

        ! also read info on original track (T|F)?
        ! if enabled, this will be stored in the output too:
        <rcbase>.with_track        :  T

    The operator should read variables from the data files that are needed to simulate a retrieval
    from the model arrays.
    This includes for example the pressures that define the *a priori* layers, the averaging kernel,
    and for this product, the airmass factor and tropopause level.
    Specify a list of names for these variables::

      ! data variables:
      tutorial.S5p.o3.dvars             :  hp yr vr A M nla

    Example settings::

      ! half-level pressures:
      !~ dimensions, copied from data file:
      tutorial.S5p.o3.dvar.hp.dims      :  layeri
      !~ source variable:
      tutorial.S5p.o3.dvar.hp.source    :  pressure

      ! retrieval: 
      !~ dimensions, copied from data file:
      tutorial.S5p.o3.dvar.yr.dims      :  retr
      !~ source variable:
      tutorial.S5p.o3.dvar.yr.source    :  mconc

      ! retrieval error covariance: 
      !~ dimensions, copied from data file:
      tutorial.S5p.o3.dvar.vr.dims      :  retr retr
      !~ source variable:
      tutorial.S5p.o3.dvar.vr.source    :  mconc_errvar

      ! kernel:
      !~ dimensions, copied from data file:
      tutorial.S5p.o3.dvar.A.dims       :  retr layer
      !~ source variable:
      tutorial.S5p.o3.dvar.A.source     :  kernel_trop

      ! tropospheric airmass factor
      !~ dimensions, copied from data file:
      tutorial.S5p.o3.dvar.M.dims       :  retr
      !~ source variable:
      tutorial.S5p.o3.dvar.M.source     :  amf_trop

      ! number of apriori layers in retrieval layer:
      !~ dimensions, copied from data file:
      tutorial.S5p.o3.dvar.nla.dims     :  retr
      !~ source variable:
      tutorial.S5p.o3.dvar.nla.source   :  nla

    For the simulated values, also define a list of variable names that should be created::

      ! state varaiables to be put out from model:
      tutorial.S5p.o3.vars                         :  mod_conc mod_hp mod_tcc mod_cc xs ys Sx M_m A_m yr_m ys_m

    Example settings::

      ! model concentration profile:
      !~ model layer dimension:
      tutorial.S5p.o3.var.mod_conc.dims            :  model_layer
      !~ standard attributes:
      tutorial.S5p.o3.var.mod_conc.attrs           :  long_name units
      tutorial.S5p.o3.var.mod_conc.attr.long_name  :  model O3 concentrations
      tutorial.S5p.o3.var.mod_conc.attr.units      :  ppb

      ! model hpentration profile:
      !~ model layer interfaces:
      tutorial.S5p.o3.var.mod_hp.dims              :  model_layeri
      !~ standard attributes:
      tutorial.S5p.o3.var.mod_hp.attrs             :  long_name units
      tutorial.S5p.o3.var.mod_hp.attr.long_name    :  model pressure at layer interfaces
      tutorial.S5p.o3.var.mod_hp.attr.units        :  Pa

      ! total cloud cover:
      !~ no extra dimensions:
      tutorial.S5p.o3.var.mod_tcc.dims             :  
      !~ standard attributes:
      tutorial.S5p.o3.var.mod_tcc.attrs            :  long_name units
      tutorial.S5p.o3.var.mod_tcc.attr.long_name   :  total cloud cover
      tutorial.S5p.o3.var.mod_tcc.attr.units       :  1

      ! cloud cover profiles:
      !~ model layer dimension:
      tutorial.S5p.o3.var.mod_cc.dims              :  model_layer
      !~ standard attributes:
      tutorial.S5p.o3.var.mod_cc.attrs             :  long_name units
      tutorial.S5p.o3.var.mod_cc.attr.long_name    :  cloud cover
      tutorial.S5p.o3.var.mod_cc.attr.units        :  1

      ! model concentrations at apriori layers:
      !~ apriori layers:
      tutorial.S5p.o3.var.xs.dims                  :  layer
      !~ how computed:
      tutorial.S5p.o3.var.xs.formula               :  LayerAverage( hp, mod_hp, mod_conc )
      tutorial.S5p.o3.var.xs.formula_terms         :  hp: hp mod_hp: mod_hp mod_conc: mod_conc
      !~ standard attributes:
      tutorial.S5p.o3.var.xs.attrs                 :  long_name units
      tutorial.S5p.o3.var.xs.attr.long_name        :  model simulations at apriori layers
      tutorial.S5p.o3.var.xs.attr.units            :  mol m-2

      ! simulated retrievals
      !~ retrieval layers:
      tutorial.S5p.o3.var.ys.dims                  :  retr
      !~ how computed:
      tutorial.S5p.o3.var.ys.formula               :  A x
      tutorial.S5p.o3.var.ys.formula_terms         :  A: A x: hx
      !~ standard attributes:
      tutorial.S5p.o3.var.ys.attrs                 :  long_name units
      tutorial.S5p.o3.var.ys.attr.long_name        :  simulated retrieval
      tutorial.S5p.o3.var.ys.attr.units            :  mol m-2

      ! partial columns as sum over apriori layers
      !~ retrieval layers:
      tutorial.S5p.o3.var.Sx.dims                 :  retr
      !~ how computed:
      tutorial.S5p.o3.var.Sx.formula              :  PartialColumns( nla, x )
      tutorial.S5p.o3.var.Sx.formula_terms        :  nla: nla x: hx
      !~ standard attributes:
      tutorial.S5p.o3.var.Sx.attrs                :  long_name units
      tutorial.S5p.o3.var.Sx.attr.long_name       :  tropospheric column in local model
      tutorial.S5p.o3.var.Sx.attr.units           :  mol m-2

      ! airmass factor from local model
      !~ retrieval layers:
      tutorial.S5p.o3.var.M_m.dims                 :  retr
      !~ how computed:
      tutorial.S5p.o3.var.M_m.formula              :  AltAirMassFactor( M, A, x, Sx )
      tutorial.S5p.o3.var.M_m.formula_terms        :  M: M A: A x: hx Sx: shx
      !~ standard attributes:
      tutorial.S5p.o3.var.M_m.attrs                :  long_name units
      tutorial.S5p.o3.var.M_m.attr.long_name       :  airmass factors from local model
      tutorial.S5p.o3.var.M_m.attr.units           :  1

      ! kernel using airmass factor from local model
      !~ retrieval layers times apriori layers
      tutorial.S5p.o3.var.A_m.dims                 :  retr layer
      !~ how computed:
      tutorial.S5p.o3.var.A_m.formula              :  AltKernel( A, M, M_m )
      tutorial.S5p.o3.var.A_m.formula_terms        :  A: A M: M M_m: M_m
      !~ standard attributes:
      tutorial.S5p.o3.var.A_m.attrs                :  long_name units
      tutorial.S5p.o3.var.A_m.attr.long_name       :  averaging kernel using local airmass factors
      tutorial.S5p.o3.var.A_m.attr.units           :  1

      ! retrieval using airmass factor from local model
      !~ retrieval layers:
      tutorial.S5p.o3.var.yr_m.dims                :  retr
      !~ how computed:
      tutorial.S5p.o3.var.yr_m.formula             :  AltRetrieval( y, M, M_m )
      tutorial.S5p.o3.var.yr_m.formula_terms       :  y: yr M: M M_m: M_m
      !~ standard attributes:
      tutorial.S5p.o3.var.yr_m.attrs               :  long_name units
      tutorial.S5p.o3.var.yr_m.attr.long_name      :  retrieval using local airmass factors
      tutorial.S5p.o3.var.yr_m.attr.units          :  mol m-2

      ! simulated retrievals using airmass factor from local model
      !~ retrieval layers:
      tutorial.S5p.o3.var.ys_m.dims                 :  retr
      !~ how computed:
      tutorial.S5p.o3.var.ys_m.formula              :  A x
      tutorial.S5p.o3.var.ys_m.formula_terms        :  A: A_m x: hx
      !~ standard attributes:
      tutorial.S5p.o3.var.ys_m.attrs                :  long_name units
      tutorial.S5p.o3.var.ys_m.attr.long_name       :  simulated retrieval based on local airmass factors
      tutorial.S5p.o3.var.ys_m.attr.units           :  mol m-2


    .. Label between '.. _' and ':' ; use :ref:`text <label>` for reference
    .. _s5p-o3-sim-catalogue:

    Sim-Catalogue
    =============

    The :py:class:`CSO_SimCatalogue <.cso_catalogue.CSO_SimCatalogue>` class could be used
    to create a catalogue of images for the converted files.
    Configuration could look like::

        ! catalogue creation task:
        cso.s5p.o3.sim-catalogue.task.class          :  cso.CSO_SimCatalogue
        cso.s5p.o3.sim-catalogue.task.args           :  '${PWD}/config/Copernicus/cso-s5p-TRACER.rc', \
                                                          rcbase='cso.s5p.o3.sim-catalogue'

    The configuration describes where to find a *listing* file with orbits, 
    which variables should be plot, the colorbar properties, etc.
    See :py:class:`CSO_SimCatalogue <.cso_s5p.CSO_SimCatalogue>` class description for how
    the settings in general look like.

    The class creates figures for a list of variables::

      ! variables to be plotted:
      cso.s5p.o3.catalogue.vars                    :  yr ys

    By default the catalogue creator simply creates a map with the value of the a variable on the track.
    Optionally settings could be used to specifiy a different unit, or the value range for the colorbar::

        ! variable:
        cso.s5p.o3.sim-catalogue.var.yr.source          :  data:mconc
        ! convert units:
        cso.s5p.o3.sim-catalogue.var.yr.units           :  mol/m3
        ! style:
        cso.s5p.o3.sim-catalogue.var.yr.vmin            :   0.0
        cso.s5p.o3.sim-catalogue.var.yr.vmax            :  50.0

        ! variable:
        cso.s5p.o3.sim-catalogue.var.ys.source          :  state:y
        ! convert units:
        cso.s5p.o3.sim-catalogue.var.ys.units           :  mol/m3
        ! style:
        cso.s5p.o3.sim-catalogue.var.ys.vmin            :   0.0
        cso.s5p.o3.sim-catalogue.var.ys.vmax            :  50.0

    Figures are saved to files with the basename of the original orbit file and the plotted variable::

         file://Scratch/cso-catalogue/O3//2018/06/01/S5p_RPRO_O3_20180601_1200_yr.png
                                                      S5p_RPRO_O3_20180601_1200_ys.png


    .. figure:: figs/O3/S5p_RPRO_O3_20180601_1100_ys.png 
       :scale: 50 %
       :align: center
       :alt: S5p O3 columns

    To search for interesting features in the data, 
    the :py:class:`Indexer <utopya_index.Indexer>` class could be used to create index pages.
    Configuration could look like::

        ! index creation task:
        cso.s5p.o3.catalogue.task.index.class     :  utopya.Indexer
        cso.s5p.o3.catalogue.task.index.args      :  '${PWD}/config/Copernicus/cso-s5p-o3.rc', \
                                                       rcbase='cso.s5p.o3.catalogue-index'

    When succesful, the index creator displays an url that could be loaded in a browser::

        Browse to:
          file://Scratch/cso-catalogue/O3/index.html

    .. figure:: figs/O3/CSO_O3_sim-catalogue.png
       :scale: 50 %
       :align: center
       :alt: Index for Simulated and S5p O\ :sub:`3` columns


