`cso_tropomi_superobs` module¶

The cso_tropomi_superobs module provides classes to convert TROPOMI super-observations (level-3 gridded product) into a CSO format.

Class hierchy¶

The classes and are defined according to the following hierchy:

Classes¶

class cso_s5p_superobs.S5p_SuperObs_File(filename)¶

Bases: object

Base class to access data in S5p file.

Example of content:

netcdf s5p_no2_rpro_v1.2.2_superobs_ll_0.5deg_02833 {
dimensions:
        lon = 720 ;
        lat = 360 ;
        layer = 34 ;
        vertices = 2 ;
variables:
        float lon(lon) ;
                lon:standard_name = "longitude" ;
                lon:long_name = "longitude" ;
                lon:units = "degrees_east" ;
        float lat(lat) ;
                lat:standard_name = "latitude" ;
                lat:long_name = "latitude" ;
                lat:units = "degrees_north" ;
        float layer(layer) ;
                layer:standard_name = "layer" ;
                layer:long_name = "layer" ;
                layer:units = "1" ;
        float no2_superobs(lat, lon) ;
                no2_superobs:standard_name = "nitrogen_dioxide_tropospheric_column_superobservation" ;
                no2_superobs:long_name = "NO2 tropospheric column superobservation, TROPOMI sensor" ;
                no2_superobs:units = "1e-6 mol m^-2" ;
                no2_superobs:_FillValue = 9.96921e+36f ;
        float no2_superobs_uncertainty(lat, lon) ;
                no2_superobs_uncertainty:standard_name = "nitrogen_dioxide_tropospheric_column_superobservation_uncertainty" ;
                no2_superobs_uncertainty:long_name = "NO2 tropospheric column superobservation uncertainty, TROPOMI sensor, including representation error" ;
                no2_superobs_uncertainty:units = "1e-6 mol m^-2" ;
                no2_superobs_uncertainty:_FillValue = 9.96921e+36f ;
        float no2_superobs_uncertainty_representation(lat, lon) ;
                no2_superobs_uncertainty_representation:standard_name = "nitrogen_dioxide_tropospheric_column_superobservation_representativity_uncertainty" ;
                no2_superobs_uncertainty_representation:long_name = "NO2 tropospheric column superobservation uncertainty, TROPOMI sensor, only representation error" ;
                no2_superobs_uncertainty_representation:units = "1e-6 mol m^-2" ;
                no2_superobs_uncertainty_representation:_FillValue = 9.96921e+36f ;
        float average_uncertainty(lat, lon) ;
                average_uncertainty:standard_name = "nitrogen_dioxide_tropospheric_column_average_uncertainty" ;
                average_uncertainty:long_name = "NO2 tropospheric column superobservation uncertainty, TROPOMI sensor, average over individual observation" ;
                average_uncertainty:units = "1e-6 mol m^-2" ;
                average_uncertainty:_FillValue = 9.96921e+36f ;
        float surface_pressure(lat, lon) ;
                surface_pressure:standard_name = "surface_pressure" ;
                surface_pressure:long_name = "superobservation surface pressure" ;
                surface_pressure:units = "Pa" ;
                surface_pressure:_FillValue = 9.96921e+36f ;
        float cloud_radiance_fraction(lat, lon) ;
                cloud_radiance_fraction:standard_name = "cloud_radiance_fraction" ;
                cloud_radiance_fraction:long_name = "superobservation cloud radiance fraction" ;
                cloud_radiance_fraction:units = "1" ;
                cloud_radiance_fraction:_FillValue = 9.96921e+36f ;
        float kernel_troposphere(layer, lat, lon) ;
                kernel_troposphere:standard_name = "averaging_kernel_troposphere" ;
                kernel_troposphere:long_name = "NO2 tropospheric column averaging kernel of the superobservation, TROPOMI sensor" ;
                kernel_troposphere:units = "1" ;
                kernel_troposphere:_FillValue = 9.96921e+36f ;
        float covered_area_fraction(lat, lon) ;
                covered_area_fraction:standard_name = "covered_area_fraction" ;
                covered_area_fraction:long_name = "Fraction of the grid box covered by satellite observations" ;
                covered_area_fraction:units = "1" ;
                covered_area_fraction:_FillValue = 9.96921e+36f ;
        float no2_scd_geo_superobs(lat, lon) ;
                no2_scd_geo_superobs:standard_name = "nitrogen_dioxide_slant_amfgeo_superobservation" ;
                no2_scd_geo_superobs:long_name = "NO2 slant column divided by AMFgeo superobservation, TROPOMI sensor" ;
                no2_scd_geo_superobs:units = "1e-6 mol m^-2" ;
                no2_scd_geo_superobs:_FillValue = 9.96921e+36f ;
        float no2_strat_superobs(lat, lon) ;
                no2_strat_superobs:standard_name = "nitrogen_dioxide_stratospheric_column_superobservation" ;
                no2_strat_superobs:long_name = "NO2 stratospheric column superobservation, TROPOMI sensor" ;
                no2_strat_superobs:units = "1e-6 mol m^-2" ;
                no2_strat_superobs:_FillValue = 9.96921e+36f ;
        float amf_trop_superobs(lat, lon) ;
                amf_trop_superobs:standard_name = "tropospheric_air_mass_factor_superobservation" ;
                amf_trop_superobs:long_name = "NO2 tropospheric air mass factor, TROPOMI sensor" ;
                amf_trop_superobs:units = "1" ;
                amf_trop_superobs:_FillValue = 9.96921e+36f ;
        float amf_strat_superobs(lat, lon) ;
                amf_strat_superobs:standard_name = "stratospheric_air_mass_factor_superobservation" ;
                amf_strat_superobs:long_name = "NO2 stratospheric air mass factor, TROPOMI sensor" ;
                amf_strat_superobs:units = "1" ;
                amf_strat_superobs:_FillValue = 9.96921e+36f ;
        float tm5_constant_a(layer, vertices) ;
                tm5_constant_a:standard_name = "tm5_constant_a" ;
                tm5_constant_a:long_name = "TM5 hybrid A coefficient at upper and lower interface levels" ;
                tm5_constant_a:units = "Pa" ;
                tm5_constant_a:_FillValue = 9.96921e+36f ;
        float tm5_constant_b(layer, vertices) ;
                tm5_constant_b:standard_name = "tm5_constant_b" ;
                tm5_constant_b:long_name = "TM5 hybrid B coefficient at upper and lower interface levels" ;
                tm5_constant_b:units = "1" ;
                tm5_constant_b:_FillValue = 9.96921e+36f ;

// global attributes:
                :authors = "Henk Eskes, eskes@knmi.nl" ;
                :institution = "Royal Netherlands Meteorological Institute (KNMI)" ;
                :product_version = "v 1.2.2, reprocessing March 2019, RPRO" ;
                :project = "S5P, phase E2" ;
                :reference = "http://www.tropomi.eu/" ;
                :source = "TROPOMI / Sentinel-5P" ;
                :title = "Nitrogen dioxide (NO2) column data, superobservations, regular lat-lon, 0.5x0.5 degree" ;
                :vertical_column_processor_version = "TM5-MP-Domino3 v3.5.3" ;
                :date_created = "2020-02-24T11:08:45Z" ;
                :time_coverage_start = "2018-05-01T02:03:58Z" ;
                :time_coverage_end = "2018-05-01T02:56:14Z" ;
                :filter_min_QA_value = 0.75f ;
                :filter_max_QA_value = 1.1f ;
                :data_version = "TM3-MP-Domino v3.5.3, reprocessing March 2019, version 1.2.2" ;
                :S5P_NO2_file_name = "S5P_RPRO_L2__NO2____20180501T014123_20180501T032451_02833_01_010202_20190201T165715.nc" ;
                :S5P_NO2_orbit_number = 2833 ;
                :S5P_NO2_history = "2019-02-01 17:26:08 f_s5pops tropnll2dp /mnt/data1/storage_offl_l2/cache_offl_l2/WORKING-565374376/JobOrder.565372716.xml; 2019-02-27 08:50:05 TM5-MP-DOMINO offline" ;
                :S5P_NO2_references = "http://www.tropomi.eu/data-products/nitrogen-dioxide" ;
                :S5P_NO2_processor_version = "1.2.2" ;
                :S5P_NO2_identifier_product_doi = "10.5270/S5P-s4ljg54" ;
                :S5P_NO2_date_created = "2019-02-01T17:17:00Z" ;
                :S5P_NO2_date_modified = "2019-02-27T08:50:05Z" ;
                :superobs_setting_modelStratVcdError = 0.15f ;
                :superobs_setting_minimal_coverage = 0.5f ;
                :superobs_setting_correlation_factor = 0.15f ;
                :superobs_setting_meanVariabilityFactor = 0.5f ;
}

Arguments:

filename : name of input file

GetData(vname, units=None)¶

Extract data array from input file and perform some adhoc fixes:

Eventually insert units attribute if not present yet, this is then taken from the argument.
Convert data to target units if necessary.
Add a long_name attribute if not present yet.

Arguments:

vname : variable in in input

Optional arguments:

units : target units

Return values:

da : :py:xarray.DataArray` object

GetTimeRange()¶: Return timerange (t1,t2) extracted from attributes.

SelectPixels(rcf, rckey, indent='')¶

Apply filters specified in rcfile.

Arguments:

rcf : RcFile object with settings
rckey : basename for rcfile keys, e.g. “s5p” for the example below

Return values:

selected : boolean array with shape of track (time,scan,pixel) which is True if a pixel passed all checks;
history : list of character string describing the selelections.

Example configuration:

! Specifiy a list of filter names.
! For each name, specify the variable which values are used for testing,
! the type test, and eventually some thresholds or other settings for this type.
! The units of the thresholds should match with the units in the variables,
! the expected units have to be defined too.
! The examples below show possible types and their settings.

! filters:
s5p.filters                        :  lons lats albedo valid

! select range of values:
s5p.filter.lons.var                :  PRODUCT/longitude
s5p.filter.lons.type               :  minmax
s5p.filter.lons.minmax             :  -15.0 35.0
s5p.filter.lons.units              :  degrees_east

! select above a minimum:
s5p.filter.lats.var                :  PRODUCT/latitude
s5p.filter.lats.type               :  minmax
s5p.filter.lats.minmax             :  35.0 70.0
s5p.filter.lats.units              :  degrees_north

! select below a maximum:
s5p.filter.albedo.var              :  Data Fields/SurfaceAlbedo
s5p.filter.albedo.type             :  max
s5p.filter.albedo.max              :  0.3
s5p.filter.albedo.units            :  1

! select only values with data (no "_FillValue"):
s5p.filter.valid.var               :  Data Fields/NO2RetrievalTroposphericVerticalColumn
s5p.filter.valid.type              :  valid

class cso_s5p_superobs.CSO_S5p_SuperObs_File(filename=None)¶

Bases: cso_file.CSO_File

Storage for CSO satellite file filled from S5p data.

AddSelection(sfile, selected, rcf, rcbase, indent='')¶

Add selected OMI to satellite extract file.

Arguments:

sfile : S5p_File object
selected : boolean array as provided by S5p_File.SelectPixels() method
rcf : RcFile instance with settings
rcbase : base of rcfile keys

The first setting that is read is a list with variable names to be created in the target file:

<rcbase>.output.vars    :  longitude corner_longitudes \
                           latitude corner_latitudes \
                           vcd  ...

For each variable, a series of settings should be specified that describe how the variable should look like and how to create it.

The first setting is a list of dimension names that define the shape of the variable. Supported dimensions are:

pixel : selected pixels
corner : number of footprint bounds (probably 4)
layer : number of layers in atmospheric profile (layers in kernel)
layeri : number of layer interfaces in atmospheric profile (layer+1)
retr : number of layers in retrieval product (1 for columns)
retr0 : same as retr, used for matrix dimensions (retr,retr0) to avoid repeated dimensions where the cf-checker complains about

For a 1D variable with values per pixel the dimension setting is therefore:

<rcbase>.output.var.longitude.dims    :   pixel

For most variables it is sufficient to provide only the name of the original variable from which the data should be read:

<rcbase>.output.var.longitude.from    :   lon

For some variables a special processing needs to be done. For these variables a key ‘special’ is used which will enable the correct conversion. The following specials are currently implemented:

longitude_to_bounds : longiude bounds interpolated/extrpolated from centers; requires a .from setting
latitude_to_bounds : latiude bounds interpolated/extrpolated from centers; requires a .from setting

hybounds_to_pressure : form pressure from hybride sigma pressure coordinate, where the available hybride coefficients have shape ('layer','vertices'); requires settings:

<rcbase>.output.var.pressure.sp       :   PRODUCT/SUPPORT_DATA/INPUT_DATA/surface_pressure
<rcbase>.output.var.pressure.hyab     :   PRODUCT/tm5_constant_a
<rcbase>.output.var.pressure.hybb     :   PRODUCT/tm5_constant_b
<rcbase>.output.var.pressure.units    :   Pa

kernel_trop_to_nla : 1-based index of lowest layer where tropo-spheric kernel is non-zero. Required settings:
```
<rcbase>.output.var.kernel.kernel_trop  :   kernel_troposphere
```
square : create a variable as the square of the input; requires a .from setting.

Optionally provide target units too. In the (unlikely) case that the original variable has no units attribute, this setting is required to define the (assumed) units. If the provided units are different from the units in the original variable, the data is converted to the provided units:

<rcbase>.output.var.longitude.units   :   degrees_east

Optionally provide a dictionairy with attributes to be added. If the attribute value is None, the attribute is removed if present from the input; this is sometimes needed if the CF compliance checker complains:

!~ skip some attributes, cf-checker complains ...
<rcbase>.output.var.qa_value.attrs         :   { 'valid_min' : None, 'valid_max' : None }

class cso_s5p_superobs.CSO_S5p_SuperObs_Convert(rcfile, rcbase='', env={}, indent='')¶

Bases: utopya_rc.UtopyaRc

Convert TROPOMI Super-Observations to CSO format. During conversion, a variable and pixel selection could be applied.

Arguments:

rcfile, rcbase, env : settings file, prefix for keys, and environment dictionairy

A time range is read to select the files to be converted:

<rcbase>.timerange.start        :  2018-06-01 00:00
<rcbase>.timerange.end          :  2018-06-03 23:59

The files are searched in subdirectories including templates for time fields; also specify filename patterns to be matched:

! input directory:
<rcbase>.files.dir         :  /Scratch/TROPOMI/NO2/superobs/%Y%m_0.5deg/compressed

! filename filters (space seperated list):
<rcbase>.files.filters     :  s5p_no2_rpro_v1.2.2_superobs_ll_0.5deg_*.nc

The following form is expected for the filenames:

s5p_no2_rpro_v1.2.2_superobs_ll_0.5deg_02833.nc

The name of the target files should be specified with a directory and filename; the later could include a template for the orbit number:

! output directory and filename:
! - times are taken from mid of selection, rounded to hours
! - use '%{orbit}' for orbit number
<rcbase>.output.dir             :  /Scratch/CSO-data/S5p/NO2/superobs/%Y/%m
<rcbase>.output.filename        :  S5p_NO2_superobs_%{orbit}.nc

A flag is read to decide if existing files should be renewed or kept:

<rcbase>.renew : True

If an input file should be converted, it is read into a S5p_SuperObs_File object. The SelectPixels method is called to select pixels based on critera defined in the settings. This method returns a history line to desribe the selection.

The target file is created as an CSO_S5p_SuperObs_File object. It’s AddSelection method is called with the input object as argument, and this will copy the selected pixels for variables specified in the settings. The Write method creates the file.

Global attributes for the target file should be specified with:

! global attributes:
<rcbase>.output.attrs               :  format Conventions author institution email
!
<rcbase>.output.attr.format         :  1.0
<rcbase>.output.attr.Conventions    :  CF-1.7
<rcbase>.output.attr.author         :  Your Name
<rcbase>.output.attr.institution    :  CSO
<rcbase>.output.attr.email          :  Your.Name@cso.org

In addition, the global attribtues from the original file are copied too, preceeded by a specified prefix:

! copy global attributes? specify filename filters:
<rcbase>.output.gattrs.filters      :  *
! prefix for new attributes:
<rcbase>.output.gattrs.prefix       :  source__

class cso_s5p_superobs.CSO_S5p_SuperObs_Listing(rcfile, rcbase='', env={}, indent='')¶

Bases: utopya_rc.UtopyaRc

Create listing file for converted orbit files.

A listing file contains the names of the converted orbit files, and the time range of pixels in the file:

filename                     ;start_time                   ;end_time                     ;orbit
2018/06/S5p_RPRO_NO2_03272.nc;2018-06-01T01:32:46.673000000;2018-06-01T01:36:12.948000000;03272
2018/06/S5p_RPRO_NO2_03273.nc;2018-06-01T03:12:53.649000000;2018-06-01T03:17:43.082000000;03273
2018/06/S5p_RPRO_NO2_03274.nc;2018-06-01T04:52:43.586000000;2018-06-01T04:59:12.377000000;03274
:

This file will be used by the observation operator to selects orbits with pixels valid for a desired time range.

In the settings, define the name of the file to be created:

! csv file that will hold records per file with:
! - timerange of pixels in file
! - orbit number
<rcbase>.file        :  /Scratch/CSO/S5p/RPRO/NO2/Europe/listing.csv

An existing listing file is not replaced, unless the following flag is set:

! renew table?
<rcbase>.renew           :  True

Orbit files are searched within a timerange:

<rcbase>.timerange.start        :  2018-06-01 00:00
<rcbase>.timerange.end          :  2018-06-03 23:59

Specify filename filters to search for orbit files; the patterns are relative to the basedir of the listing file, and might contain templates for the time values. Multiple patterns could be defined; if for a certain orbit number more than one file is found, the first match is used. This could be explored to create a listing that combines reprocessed data with near-real-time data:

<rcbase>.patterns            :  RPRO/NO2/Europe/%Y/%m/S5p_*.nc                                         OFFL/NO2/Europe/%Y/%m/S5p_*.nc

Usually the time range is read from the file, but in case the file does not have a time accordinate, then the following flag might be used to force that the time that matches with the filename is used:

! adhoc: use the time for which file is valid as timerange;
! this is used for the synthetic S4 data that have no time record ...
<rcbase>.use_t              :  True

Alternatively, the time range could be read from the global attributes if it is specified there:

! adhoc: use attributes to extract timerange from?
<listing>.use_attrs           :  True
! if True, define names:
<listing>.attr_tr1            :  source__time_coverage_start
<listing>.attr_tr2            :  source__time_coverage_end

The orbit column in the listing is extra, and is read from global attributes; the list of extra columns is defined with:

! extra columns to be added, read from global attributes:
<rcbase>.xcolumns           :  orbit

`cso_tropomi_superobs` module¶

Class hierchy¶

Classes¶

Table of Contents

This Page

cso_tropomi_superobs module¶

Class hierchy¶

Classes¶

`cso_tropomi_superobs` module¶