Updated description of VIIRS processing. (53511d8b) · Commits · CAMS / CSO

config/VIIRS/cso-user-settings.rc

0 → 100644

+112 −0

Original line number	Diff line number	Diff line
		!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
		!!!
		!!! CSO common configuration
		!!!
		!!! Base settings that are used by multiple tasks:
		!!! - time range(s)
		!!! - target domain(s)
		!!!
		!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!


		!-----------------------------------------------------------
		! id's
		!-----------------------------------------------------------

		! file format:
		my.cso.format : 1.0

		! file format convention:
		my.cso.convention : CF-1.7


		!-----------------------------------------------------------
		! domain
		!-----------------------------------------------------------

		!
		! Used for:
		! - orbit selection durning download
		! - pixel selection
		! - map plots
		!

		! region name:
		my.region : CAMS
		! CAMS regional ensemble domain:
		my.region.west : -30.0
		my.region.east : 45.0
		my.region.south : 30.0
		my.region.north : 76.0
		! size of map figures for this region:
		my.region.figsize : (6,6)

		!! region name:
		!my.region : globe
		!! global domain:
		!my.region.west : -180.0
		!my.region.east : 180.0
		!my.region.south : -90.0
		!my.region.north : 90.0
		!! size of map figures for this region:
		!my.region.figsize : (8,6)


		!----------------------------------------------------------
		! timerange
		!----------------------------------------------------------

		! inquire full timerange:
		my.inquire.timerange.start : 2011-10-01 00:00
		my.inquire.timerange.end : 2024-12-31 23:59

		! testing 3 days
		my.timerange.start : 2018-06-01 00:00
		my.timerange.end : 2018-06-03 23:59


		!----------------------------------------------------------
		! user specific settings:
		!----------------------------------------------------------

		! Attributes written to output files.
		my.attr.author : Your Name
		my.attr.institution : CSO
		my.attr.email : Your.Name@cso.org

		! base location for work directories:
		!my.work : /work/${USER}/CSO-Tutorial
		my.work : /Scratch/${USER}/CSO-VIIRS


		!----------------------------------------------------------
		! job step defaults
		!----------------------------------------------------------

		! run jobs in foreground:
		*.script.class : utopya.UtopyaJobScriptForeground

		! dummy value, will be defined when running in virtrual environment:
		VIRTUAL_ENV :
		! running in virutual envionment?
		#if "${VIRTUAL_ENV}" > ""
		! interpretor from virtual enviornment, the system path is set automatically:
		*.shell : ${VIRTUAL_ENV}/bin/python3
		#else
		! search path for python modules:
		*.pypath : ${CSO_PREFIX}/src:${CSO_PREFIX}/src/utopya:${CSO_PREFIX}/src/cso
		#endif

		! work directory for jobs;
		*.workdir : ${my.work}/__NAME2PATH__

		! for new job files, use jobtree settings from this file:
		*.rcfile : ${__filename__}



		!----------------------------------------------------------
		! end
		!----------------------------------------------------------

config/VIIRS/cso-viirs.rc

0 → 100644

+416 −0

Original line number	Diff line number	Diff line
		!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
		!!!
		!!! CSO - CAMS Satellite Operator
		!!!
		!!! Settings for VIIRS processing.
		!!!
		!!! Environment:
		!!!
		!!! MY_PRODUCT : defines the VIIRS product, one of:
		!!! viirs1-aod-db viirs1-aod-dt
		!!! viirs2-aod-db viirs2-aod-dt
		!!!
		!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!


		!-----------------------------------------------------------------------
		! user specific settings:
		!-----------------------------------------------------------------------

		! include user specfic settings:
		#include cso-user-settings.rc


		!-----------------------------------------------------------------------
		! multi-product
		!-----------------------------------------------------------------------

		! instrument selection:
		#if "${MY_PRODUCT}" in ["viirs1-aod-db","viirs1-aod-dt"]
		my.instrument : VIIRS-1
		my.platform : SNPP
		#elif "${MY_PRODUCT}" in ["viirs2-aod-db","viirs2-aod-dt"]
		my.instrument : VIIRS-2
		my.platform : NOAA20
		#else
		#error unsupported product '${MY_PRODUCT}'
		#endif

		! product selection:
		#if "${MY_PRODUCT}" in ["viirs1-aod-db","viirs2-aod-db"]
		!~ "Deep Blue" products:
		my.retrieval : AERDB
		#elif "${MY_PRODUCT}" in ["viirs1-aod-dt","viirs2-aod-dt"]
		!~ "Dark Target" products:
		my.retrieval : AERDT
		#else
		#error unsupported product '${MY_PRODUCT}'
		#endif

		! version selection:
		my.version : v2.0.0

		! local archive:
		my.arch.dir : ${my.work}/EarthData/${my.instrument}/${my.retrieval}_L2


		!======================================================================
		!===
		!=== Inquire
		!===
		!======================================================================


		!-----------------------------------------------------------------------
		! inquire EarthData
		!-----------------------------------------------------------------------

		! Obtain names of all available VIIRS files.
		! Stored as csv with processing type, processor version, filenames, etc.

		! renew table if file already exists?
		cso.inquire-table-earthaccess.renew : True

		! full time range:
		cso.inquire-table-earthaccess.timerange.start : ${my.inquire.timerange.start}
		cso.inquire-table-earthaccess.timerange.end : ${my.inquire.timerange.end}
		!! ... TESTING
		!cso.inquire-table-earthaccess.timerange.start : 2022-01-01 12:00:00
		!cso.inquire-table-earthaccess.timerange.end : 2022-01-01 15:00:00

		!~ dataset name:
		cso.inquire-table-earthaccess.dataset : ${my.retrieval}_L2_VIIRS_${my.platform}

		!~ domain specified as: west,south,east,north
		cso.inquire-table-earthaccess.area : ${my.region.west},${my.region.south},${my.region.east},${my.region.north}

		! csv file that will hold records per file with:
		! - timerange of pixels in file
		! - orbit number
		cso.inquire-table-earthaccess.output.file : ${my.arch.dir}/inquire/inquire.csv



		!-----------------------------------------------------------
		! overview plot of versions
		!-----------------------------------------------------------

		! renew existing plots?
		cso.inquire-plot.renew : True

		! listing files:
		cso.inquire-plot.file : ${cso.inquire-table-earthaccess.output.file}
		!cso.inquire-plot.filedate : 2023-08-07

		! annote:
		cso.inquire-plot.title : VIIRS-AOD

		! output figure, eventually use templates for time values:
		cso.inquire-plot.output.file : ${my.arch.dir}/inquire/inquire.png


		!======================================================================
		!===
		!=== download
		!===
		!======================================================================


		! renew existing files (True\|False) ?
		cso.download.renew : False
		!! .. TESTING ...
		!cso.download.renew : True

		! time range:
		cso.download.timerange.start : ${my.timerange.start}
		cso.download.timerange.end : ${my.timerange.end}

		! listing of available source files crated by "inquire" job:
		cso.download.inquire.file : ${my.arch.dir}/inquire/inquire.csv
		!!~ historic inquire ...
		!cso.download.inquire.filedate : 2025-02-01

		! processor version:
		#if "${my.version}" == "v2.0.0"
		cso.download.processor_version : 020000
		#else
		#error unsupported my.version "${my.version}"
		#endif

		! target directory, includiong time values:
		cso.download.dir : ${my.arch.dir}/${my.version}/%Y/%j



		!-----------------------------------------------------------
		! listing of downloaded files
		!-----------------------------------------------------------


		! renew existing file (True\|False) ?
		cso.download-listing.renew : True

		! create listing of downloaded files;
		! eventully include time templates %Y-%m-%d etc:
		cso.download-listing.file : ${my.arch.dir}/${my.version}/listing.csv

		! filename pattern for files in subdirs:
		cso.download-listing.pattern : AER*.nc



		!======================================================================
		!===
		!=== convert (and download)
		!===
		!======================================================================


		! renew existing files (True\|False) ?
		!cso.convert.renew : True
		! ... TESTING ...
		cso.convert.renew : False

		! time range:
		cso.convert.timerange.start : ${my.timerange.start}
		cso.convert.timerange.end : ${my.timerange.end}


		!~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
		! input files
		!~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

		! listing file with input data:
		cso.convert.input.listing : ${cso.download-listing.file}

		! selection names:
		cso.convert.filters : lons lats valid quality

		! filter settings:
		cso.convert.filter.lons.type : minmax
		cso.convert.filter.lons.minmax : ${my.region.west} ${my.region.east}
		cso.convert.filter.lons.var : Longitude
		cso.convert.filter.lons.units : degrees_east

		! filter settings:
		cso.convert.filter.lats.type : minmax
		cso.convert.filter.lats.minmax : ${my.region.south} ${my.region.north}
		cso.convert.filter.lats.var : Latitude
		cso.convert.filter.lats.units : degrees_north

		! skip pixel with "no data", use the combined land/ocean variable to check:
		cso.convert.filter.valid.var : Aerosol_Optical_Thickness_550_Land_Ocean_Best_Estimate
		cso.convert.filter.valid.type : valid

		! Descriptions in variables:
		! Aerosol_Optical_Thickness_QA_Flag_Land :long_name = "Deep Blue quality assurance flag over land. 0=no retrieval, 1=poor, 2=moderate, 3=good" ;
		! Aerosol_Optical_Thickness_QA_Flag_Ocean:long_name = "SOAR quality assurance flag over water. 0=no retrieval, 1=poor, 3=good" ;
		! compare for both variables, at least one should have a good quality pixel:
		cso.convert.filter.quality.type : min2
		cso.convert.filter.quality.min : 3
		cso.convert.filter.quality.var : Aerosol_Optical_Thickness_QA_Flag_Land
		cso.convert.filter.quality.var2 : Aerosol_Optical_Thickness_QA_Flag_Ocean
		cso.convert.filter.quality.units : 1



		!~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
		! output files
		!~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

		! output directory and filename:
		! - time values taken from start time
		! - templates: %{orbit}
		cso.convert.output.filename : ${my.work}/${MY_PRODUCT}/data/${my.region}/%Y/%m/${MY_PRODUCT}_%{orbit}.nc

		! pack variables on output:
		cso.convert.output.packed : True
		! zlib compression level, 0 for no compression:
		cso.convert.output.complevel : 1

		! global attributes:
		cso.convert.output.attrs : format Conventions \
		author institution email
		!
		cso.convert.output.attr.format : ${my.cso.format}
		cso.convert.output.attr.Conventions : ${my.cso.convention}
		cso.convert.output.attr.author : ${my.attr.author}
		cso.convert.output.attr.institution : ${my.attr.institution}
		cso.convert.output.attr.email : ${my.attr.email}

		!! no need to swap layes:
		!cso.convert.swap_layers : False


		! ~ variables

		!
		! Describe per variable:
		! * .dims : dimensions list:
		! pixel : selected pixels
		! corner : number of footprint bounds (probably 4)
		! layer : number of layers in atmospheric profile (layers in kernel)
		! layeri : number of layer interfaces in atmospheric profile (layer+1)
		! retr : number of layers in retrieval product (1 for columns) ;
		! for error covariance use (retr,retr0) to avoid repeated dimensions
		! track_scan : original scan index in 2D track
		! track_pixel : original ground pixel in 2D track
		! * .specal : keyword to select special processing
		! * None : no special processing (default)
		! * track_longitude : longitudes at centers of original 2D track
		! * track_latitude : latitudes at centers of original 2D track
		! * .units : target units if different from original
		! * .oper : special postprocessing, currently supported:
		! * square : fill variable with squared valued (used to form variance from standard deviation)
		! In case no special processing is needed:
		! * .from : original variable (group path and variable name)
		!

		! which fields to be put out ?
		cso.convert.output.vars : longitude longitude_bounds \
		latitude latitude_bounds \
		track_longitude track_longitude_bounds \
		track_latitude track_latitude_bounds \
		time \
		aot_412nm aot_488nm aot_550nm aot_670nm aot_865nm aot_1240nm aot_1640nm aot_2250nm

		! qa_value \
		! vcd vcd_errvar
		!
		!! vertical column density:
		!cso.convert.output.var.vcd.dims : pixel retr
		!cso.convert.output.var.vcd.from : PRODUCT/nitrogendioxide_tropospheric_column
		!cso.convert.output.var.vcd.attrs : { 'ancillary_variables' : None, 'multiplication_factor_to_convert_to_molecules_percm2' : None }
		!
		!! error variance in vertical column density (after application of kernel),
		!! fill with single element 'covariance matrix', from square of standard error:
		!! use dims with different names to avoid that cf-checker complains:
		!cso.convert.output.var.vcd_errvar.dims : pixel retr retr0
		!cso.convert.output.var.vcd_errvar.special : square
		!cso.convert.output.var.vcd_errvar.from : PRODUCT/nitrogendioxide_tropospheric_column_precision_kernel
		!!~ skip standard name, modifier "standard_error" is not valid anymore:
		!cso.convert.output.var.vcd_errvar.attrs : { 'standard_name' : None, 'multiplication_factor_to_convert_to_molecules_percm2' : None }

		! centers of pixels;
		! use special processing to ensure correction at swath edges;
		! remove range, gives problems when packed ...
		!~ longitudes
		cso.convert.output.var.longitude.dims : pixel
		cso.convert.output.var.longitude.special : longitude
		cso.convert.output.var.longitude.from : Longitude
		cso.convert.output.var.longitude.units : degrees_east
		cso.convert.output.var.longitude.attrs : { 'valid_range' : None }
		!~ latitudes
		cso.convert.output.var.latitude.dims : pixel
		cso.convert.output.var.latitude.special : latitude
		cso.convert.output.var.latitude.from : Latitude
		cso.convert.output.var.latitude.units : degrees_north
		cso.convert.output.var.latitude.attrs : { 'valid_range' : None }

		! corners of pixels, interpolate/extrapolate from centers;
		! use special processing to ensure correction at swath edges;
		! remove range, gives problems when packed ...
		!~ longitudes
		cso.convert.output.var.longitude_bounds.dims : pixel corner
		cso.convert.output.var.longitude_bounds.special : longitude_bounds
		cso.convert.output.var.longitude_bounds.from : Longitude
		cso.convert.output.var.longitude_bounds.units : degrees_east
		cso.convert.output.var.longitude_bounds.attrs : { 'valid_range' : None }
		!~ latitudes
		cso.convert.output.var.latitude_bounds.dims : pixel corner
		cso.convert.output.var.latitude_bounds.special : latitude_bounds
		cso.convert.output.var.latitude_bounds.from : Latitude
		cso.convert.output.var.latitude_bounds.units : degrees_north
		cso.convert.output.var.latitude_bounds.attrs : { 'valid_range' : None }

		! original track:
		! use special processing to ensure correction at swath edges;
		! remove range, gives problems when packed ...
		!~ longitudes
		cso.convert.output.var.track_longitude.dims : track_scan track_pixel
		cso.convert.output.var.track_longitude.special : track_longitude
		cso.convert.output.var.track_longitude.from : Longitude
		cso.convert.output.var.track_longitude.attrs : { 'valid_range' : None }
		!~ latitudes
		cso.convert.output.var.track_latitude.dims : track_scan track_pixel
		cso.convert.output.var.track_latitude.special : track_latitude
		cso.convert.output.var.track_latitude.from : Latitude
		cso.convert.output.var.track_latitude.attrs : { 'valid_range' : None }
		!~ corner lons
		cso.convert.output.var.track_longitude_bounds.dims : track_scan track_pixel corner
		cso.convert.output.var.track_longitude_bounds.special : track_longitude_bounds
		cso.convert.output.var.track_longitude_bounds.from : Longitude
		cso.convert.output.var.track_longitude_bounds.units : degrees_east
		!~ corner lats
		cso.convert.output.var.track_latitude_bounds.dims : track_scan track_pixel corner
		cso.convert.output.var.track_latitude_bounds.special : track_latitude_bounds
		cso.convert.output.var.track_latitude_bounds.from : Latitude
		cso.convert.output.var.track_latitude_bounds.units : degrees_north

		! time based on TAI93 units:! use time-delta array for shape:
		cso.convert.output.var.time.dims : pixel
		cso.convert.output.var.time.special : time
		cso.convert.output.var.time.from : Scan_Start_Time

		! AOT, merged product:
		cso.convert.output.var.aot_550nm.dims : pixel
		cso.convert.output.var.aot_550nm.from : Aerosol_Optical_Thickness_550_Land_Ocean_Best_Estimate
		cso.convert.output.var.aot_550nm.attrs : { 'valid_range' : None }

		! AOT from land band only:
		#for WVL in 412 488 670 865 1240 1640 2250
		cso.convert.output.var.aot_WVLnm.dims : pixel
		cso.convert.output.var.aot_WVLnm.special : wavelength=WVL
		#if WVL in [412]
		cso.convert.output.var.aot_WVLnm.from : Spectral_Aerosol_Optical_Thickness_Land
		cso.convert.output.var.aot_WVLnm.attrs : { 'valid_range' : None, 'long_name' : 'aerosol optical thickness at WVL nm over land' }
		#elif WVL in [488,670]
		cso.convert.output.var.aot_WVLnm.from : Spectral_Aerosol_Optical_Thickness_Land \
		Spectral_Aerosol_Optical_Thickness_Ocean
		cso.convert.output.var.aot_WVLnm.attrs : { 'valid_range' : None, 'long_name' : 'aerosol optical thickness at WVL nm over land and water' }
		#elif WVL in [865,1240,1640,2250]
		cso.convert.output.var.aot_WVLnm.from : Spectral_Aerosol_Optical_Thickness_Ocean
		cso.convert.output.var.aot_WVLnm.attrs : { 'valid_range' : None, 'long_name' : 'aerosol optical thickness at WVL nm over water' }
		#else
		#error unsupported wavelength WVL
		#endif
		#endfor



		!!======================================================================
		!!===
		!!=== listing
		!!===
		!!======================================================================
		!
		!! csv file that will hold records per file with:
		!! - timerange of pixels in file
		!! - orbit number
		!cso.listing.file : ${my.work}/_PRODUCT_/data/${my.region}/${my.selection}__listing.csv
		!
		!! renew table if file already exists?
		!cso.listing.renew : True
		!
		!! time range:
		!cso.listing.timerange.start : ${my.timerange.start}
		!cso.listing.timerange.end : ${my.timerange.end}
		!
		!! filename filters relative to listing file that should be scanned for orbit files;
		!! names could include time templates ;
		!! if same orbit is found in multiple directories, the first found is used;
		!! remove existing table for safety to ensure that this is done correctly ...
		!cso.listing.patterns : ${my.selection}/%Y/%m/_PRODUCT__*.nc
		!
		!! extra columns to be added, read from global attributes:
		!cso.listing.xcolumns : orbit
		!
		!


		!======================================================================
		!===
		!=== end
		!===
		!======================================================================

config/VIIRS/cso.rc

0 → 100644

+182 −0

File added.

Preview size limit exceeded, changes collapsed.

doc/source/figs/VIIRS/viirs1-aod-db_inquire.png

0 → 100644

+12.2 KiB

Loading image diff...

doc/source/gridding.rst

+2 −2

Original line number	Diff line number	Diff line
		@@ -25,7 +25,7 @@ where:
		* :math:`w_p` is the footprint area [m\ :sup:`2`] of pixel :math:`p`
		* :math:`w_{p,k}` is the area [m\ :sup:`2`] of pixel :math:`p` that overlaps with the cell :math:`k`.

		The overlapping area is computed using the :py:meth:`LonLatPolygonCentroids <cso_mapping.LonLatPolygonCentroids>` method.
		The overlapping area is computed using the :py:meth:`LonLatPolygonCentroids <.LonLatPolygonCentroids>` method.
		This fractions a footprint area into a large number of triangles, and returns
		for each triangle the centroid and the area.
		The centroids are collected per grid cell, and sum over the associated triangle area's is
		@@ -80,7 +80,7 @@ that is configured as::
		rcbase='cso.tutorial.gridded-catalogue-index'

		The actual work is done by the :py:class:`.CSO_GriddedCatalogue`
		and :py:class:`utopya.Indexer <utopya_index.Indexer>` classes,
		and :py:class:`utopya.Indexer <.Indexer>` classes,
		see their description for details on the configuration.
		The result could be viewed in a browser: