# Sentinel-5p O3 data processing

This chapter describes the tasks performed for processing Sentinel-5p O3 data.

## Product description

The product guides can be found at:

- SentiWiki / Sentinel-5P

There are several ozone products available

- `L2__O3____` : ozone total column product.

  See the PUM-O3 Product User Manual.

  - The primary retrieval is vertical column density (mol/m2) on \(n_r=1\) layers:

    \[y_r\]

  - The simulation of a retrieval product from a model state requires an apriori column, and should be computed from:

    \[y_s ~=~ \sum \mathbf{y}_a\ +\ \mathbf{A}\ \left(\ \mathbf{V}\mathbf{G}\ \mathbf{x}\ -\ \mathbf{y}_a\ \right)\]

    where:

    - \(y_s\) is the simulated retrieval (mol/m2) on \(n_r=1\) layers;

    - \(\mathbf{y}_a\) is the a priori vertical column density (mol/m2) defined on \(n_a=14\) layers;

    - \(\mathbf{A}\) is the averaging kernel matrix with shape \((n_r,n_a)\);

    - \(\mathbf{x}\) is the atmospheric state, which probably consists of a 3D array of O3 concentrations;

    - operators \(\mathbf{G}\) and \(\mathbf{V}\) together compute a simulated profile at the \(n_a\) a priori layers from the state, using horizontal (\(\mathbf{G}\)) and vertical (\(\mathbf{V}\)) mappings; units should be the same as the retrieval product (mol/m2).

  - In case \(\mathbf{x}^{true}\) is the true atmoshperic state, the retrieval error is quantified

      by the retrieval error covariance \(\mathbf{R}\):

    \[\mathbf{y}_s ~-~ \left[ \mathbf{y}_a\ +\ \mathbf{A}\ \left(\ \mathbf{V}\mathbf{G}\ \mathbf{x}^{true}\ -\ \mathbf{y}_a\ \right)\right] ~\sim~ \mathcal{N}\left(\mathbf{o},\mathbf{R}\right)\]

    The product does not provide a full matrix \(\mathbf{B}\) but only a precission profile; here we define \(\mathbf{R}\) to be diagonal with the squares of the precission as diagonal elements.

- `L2__O3__PR` : ozone profile product.

  See the PUM-O3_PR Product User Manual.

  - The primary retrieval is an ozone concentration (mol/m3) profile at \(n_r=33\) layers:

    \[\mathbf{y}_r\]

    In addition, the product supplies derived products:

    - on 6 sub-columns defined by static altitudes 0-6, 6-12, 12-18, 18-24, 24-32, and 32-82 km;

    - a tropospheric column (probably between `surface_pressure` and `pressure_at_tropopause`);

    - a total column product.

    As there are no averaging kernels supplied for the (sub)columns, only the full profiles will be used here.

  - The simulation of a retrieval product from a model state requires an apriori profile, and should be computed from:

    \[\mathbf{y}_s ~=~ \mathbf{y}_a\ +\ \mathbf{A}\ \left(\ \mathbf{V}\mathbf{G}\ \mathbf{x}\ -\ \mathbf{y}_a\ \right)\]

    where:

    - \(\mathbf{y}_s\) is the simulated retrieval (mol/m3) defined on \(n_r\) layers;

    - \(\mathbf{y}_a\) is the a priori retrieval (mol/m3) defined on \(n_a=33\) layers;

    - \(\mathbf{A}\) is the averaging kernel matrix with shape \((n_r,n_a)\);

    - \(\mathbf{x}\) is the atmospheric state, which probably consists of a 3D array of O3 concentrations;

    - operators \(\mathbf{G}\) and \(\mathbf{V}\) together compute a simulated profile at the \(n_a\) a priori layers from the state, using horizontal (\(\mathbf{G}\)) and vertical (\(\mathbf{V}\)) mappings; units should be the same as the retrieval product (mol/m3).

  - In case \(\mathbf{x}^{true}\) is the true atmoshperic state, the retrieval error is quantified

      by the retrieval error covariance \(\mathbf{R}\):

    \[\mathbf{y}_s ~-~ \left[ \mathbf{y}_a\ +\ \mathbf{A}\ \left(\ \mathbf{V}\mathbf{G}\ \mathbf{x}^{true}\ -\ \mathbf{y}_a\ \right)\right] ~\sim~ \mathcal{N}\left(\mathbf{o},\mathbf{R}\right)\]

    The product does not provide a full matrix \(\mathbf{B}\) but only a precission profile; here we define \(\mathbf{R}\) to be diagonal with the squares of the precission as diagonal elements.

  - Recomended pixel selection:

    - The retrieval status and quality is indicated by the `qa_value`. The recommended minimum is 0.80, this excludes cloudy scenes and other problematic retrievals.

    - In addition, it is recommended to disregard pixels that match the following criterion:

      \[\sum_{i=1}^{n_r} \left| y_r(i) - y_a(i) \right| ~>~ \mbox{1.66e-5} ~~~~~~\mbox{[mol/m$^3$]}\]

      NOTE: older versions of the *Product User Manual show upper-entiers* (\(\lceil\cdot\rceil\))* around the difference between retrieval and apriori, this should have been absolute values as used here.*

- `L2__O3_TCL` ozone tropospheric column product.

    Level3 product, tropics band, 3-days running average on 1.0 x 0.5 (lon x lat) grid? Not used here.

## CSO processing

(See Tutorial chapter for introduction to CSO scripts and configuration)

An example configuration of the CSO processing of the S5p/O3 data is available via the following settings:

- config/Copernicus/cso.rc

  Top-level settings that configure the job-tree with various sub-tasks. This is a generic file that could be used for multiple S5 products, edit it to select the O3 processing.

- config/Copernicus/cso-user-settings.rc

  User-specific settings such as the work directory.

- config/Copernicus/cso-s5p-o3.rc

  Specific settings for O3 product.

Start the job-tree using:

    ./bin/cso  config/Copernicus/cso.rc

Selected sub-steps in the processing are described below.

## Inquire Sentinel-5p/O3-profile archives

S5p/O3-profile observations are available from:

- Copernicus DataSpace; see the cso_dataspace module module for a detailed description.

Data is available for different processing streams, each identified by a 4-character key:

- `NRTI` : Near real time, available with a day after observation;

- `OFFL` : Offline, available within weeks after observations;

- `RPRO` : re-processing of all previously made observations.

The `CSO_DataSpace_Inquire` class is available to inquire the Copernicus DataSpace. The settings used by this class allow selection on for example time range and intersection area. The result is a csv file which with columns for keywords such as orbit number and processor version, as well as the filename of the data and the url that should be used to actually download the data:

    orbit;start_time;end_time;processing;collection;processor_version;filename;href
    02818;2018-04-30 00:19:50;2018-04-30 02:01:20;RPRO;03;020400;S5P_RPRO_L2__O3__PR_20180430T001950_20180430T020120_02818_03_020400_20230222T152833.nc;https://zipper.dataspace.copernicus.eu/odata/v1/Products(5192222a-1311-42a2-b557-3f507a26defe)/$value
    02819;2018-04-30 02:01:20;2018-04-30 03:42:50;RPRO;03;020400;S5P_RPRO_L2__O3__PR_20180430T020120_20180430T034250_02819_03_020400_20230222T152835.nc;https://zipper.dataspace.copernicus.eu/odata/v1/Products(8dd92be6-e866-46db-9247-1d34408334bc)/$value
    :

See the section on File name convention in the Product User Manual for the meaning of all parts of the filename.

To visualize the available data, the `CSO_Inquire_Plot` could be used to create an overview figure:

The jobtree configuration to inquire the portals and create the overview figure could look like:

    ! single step:
    cso.s5p.o3.inquire.class                      :  utopya.UtopyaJobStep
    ! two tasks:
    cso.s5p.o3.inquire.tasks                      :  table-dataspace plot

    !~ inquire files available on DataSpace:
    cso.s5p.o3.inquire.table-dataspace.class      :  cso.CSO_DataSpace_Inquire
    cso.s5p.o3.inquire.table-dataspace.args       :  '${PWD}/config/Copernicus/cso-s5p-o3.rc', \
                                                         rcbase='cso.s5p.o3.inquire-table-dataspace'

    !~ create plot of available versions:
    cso.s5p.o3.inquire.plot.class                 :  cso.CSO_Inquire_Plot
    cso.s5p.o3.inquire.plot.args                  :  '${PWD}/config/Copernicus/cso-s5p-o3.rc', \
                                                          rcbase='cso.s5p.o3.inquire-plot'

## Conversion to CSO format

The ‘`cso.s5p.o3.convert`’ task converts orbit files downloaded from a portal into a CSO format.

Files are downloaded from a portal if not present locally yet; eventually they are also removed after conversion to avoid that the portal is completely mirrored.

To save storage, only selected pixels are included in the converted files, for example only those within some region or cloud free pixels. The selection criteria are defined in the settings, and added to the ‘`history`’ attribute of the created files as reminder.

The work is done by the `CSO_S5p_Convert` class, which is initialized using the settings file:

    ! task initialization:
    cso.s5p.o3.convert.class     :  cso.CSO_S5p_Convert
    cso.s5p.o3.convert.args      :  '${PWD}/config/Copernicus/cso-s5p-o3.rc', rcbase='cso.s5p.o3.convert'

See the class documentation for the general configuration, below some specific choices are described. The example is based on the S5p O3-profile file from which the header is available in:

- doc/samples/S5P_RPRO_L2__O3__PR_20180601T002121_20180601T020251_03272_03_020400_20230224T110239.txt