TNO Intern

Skip to content
Commits on Source (26)
Expand all Hide whitespace changes
Inline Side-by-side
.gitignore
View file @ 8bce8a14
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# object files:
*.o
# Fortran objects
# Fortran modules:
*.mod
*.o
# executables:
*.x
# job files, output:
*.jb
*.out*
*.err*
*.log*
*.msg*
*.csv*
# python:
__pycache__
*.pyc*
# environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
.vscode
CHANGELOG
View file @ 8bce8a14
......@@ -5,11 +5,12 @@ CSO - CAMS Satellite Operator
Version tags
------------
See "History" chapter in documentation for changes.
See "History" chapter in documentation for overview of versions.
Detailled changes per version are listed below.
Changes for v2.1
----------------
v2.1
----
Updated documentation.
doc/source/documentation.rst
......@@ -109,6 +110,10 @@ Support for S5p/Glyoxal product.
doc/samples/S5P_OFFL_L2__CHOCHO___20200601T001519_20200601T015649_13643_01_010000_20210128.txt
doc/source/index.rst
v2.2
----
Support batch job for Sun Grid Engine.
py/utopya_jobscript.py
py/utopya_jobtree.py
......@@ -153,6 +158,15 @@ Added class to create temporal means of gridded averages.
Read Timestamp object from rcfile settings.
py/rc.py
Support index timeseries over years.
py/utopya_index.py
Updated product description of S5p/Glyoxal.
doc/source/s5p-glyoxal.rst
Updated template settings.
config/tutorial/tutorial.rc
Added tools to collect and plot time series of co-located pixel averages.
py/cso_colocate.py
py/cso.py
......@@ -218,8 +232,8 @@ Updated User guide.
doc/source/
Changes for v2.4
----------------
v2.4
----
Added 'black' target to Makefile, including convention for maximum line length.
Reformatted files following new convention.
......@@ -227,8 +241,8 @@ Reformatted files following new convention.
py/*
Changes for v2.5
----------------
v2.5
----
Fixed definition of CSO_PREFIX environment variable.
bin/cso
......@@ -262,3 +276,54 @@ Patch 2.5.1
Fixed bug in S5P orbit selection.
py/cso_s5p.py
v2.6
----
Support SO2-COBRA product from PAL website.
v2.7
----
Fixed averaging over time values when creating super-observations.
py/cso_superobs.py
Improved output of packed variables.
Support files with "sample" coordinate instead of "pixel".
oper/
Use cartopy for map plots.
py/cso_plot.py
py/cso_catalogue.py
Support orbits that overlap with dateline.
py/cso_s5p.py
py/cso_gridded.py
py/cso_superobs.py
v2.7
----
Removed suppored for NO2-superobs, now create these using own classes.
py/cso_s5p_superobs.py
py/cso.py
config/EMEP/emep-tropomi-superobs.rc
doc/source/index.rst
doc/source/pymod-cso_s5p_superobs.rst
doc/source/s5p-no2-superobs.rst
Removed depricated references to S5p-Hub.
doc/source
py/
Removed whitespace from lines for easier comparison with other versions.
py/
Synchronized layout with CSO public version for easier synchronization.
Synchronized change logs in source files between master and development versions.
py/
Makefile
View file @ 8bce8a14
......@@ -14,7 +14,7 @@ help:
# re-format python files,
# use local maximum line-length convention ..
black:
black --line-length=99 py
black --line-length=99 py doc/source/conf.py
# generate documentation:
docu:
......
config/README.txt
View file @ 8bce8a14
......@@ -5,6 +5,6 @@ tutorial/
Setting following tutorial.
Copernicus/
Example configurations for Sentinel-5P data download from Copernicus SciHub.
Example configurations for Sentinel-5P data download from Copernicus DataSpace.
config/tutorial/tutorial.rc
View file @ 8bce8a14
......@@ -22,10 +22,7 @@ cso.tutorial.class : utopya.UtopyaJobTree
! list of sub-elements:
cso.tutorial.elements : inquire \
convert listing \
catalogue \
gridded gridded-catalogue
! sim-catalogue \
! sim-gridded sim-gridded-catalogue
catalogue
! ... preprocessor steps one by one:
!cso.tutorial.elements : inquire
......@@ -37,6 +34,9 @@ cso.tutorial.elements : inquire \
!cso.tutorial.elements : gridded
!cso.tutorial.elements : gridded-catalogue
!
! ... create super-observations
!cso.tutorial.elements : superobs
!
! ... after running the obs.oper:
!cso.tutorial.elements : sim-catalogue
!cso.tutorial.elements : sim-gridded
......@@ -105,6 +105,21 @@ cso.tutorial.sim-catalogue.index.class : utopya.Indexer
cso.tutorial.sim-catalogue.index.args : '${__filename__}', \
rcbase='cso.tutorial.sim-catalogue-index'
!
! * super observations
!
! single step:
cso.tutorial.superobs.class : utopya.UtopyaJobStep
! tasks:
cso.tutorial.superobs.tasks : create listing
!~ create:
cso.tutorial.superobs.create.class : cso.CSO_SuperObs
cso.tutorial.superobs.create.args : '${__filename__}', rcbase='cso.tutorial.superobs'
!~ create:
cso.tutorial.superobs.listing.class : cso.CSO_S5p_Listing
cso.tutorial.superobs.listing.args : '${__filename__}', rcbase='cso.tutorial.superobs-listing'
!
! * gridded fields
!
......@@ -190,16 +205,23 @@ my.cso.convention : CF-1.7
! region name:
my.region : CAMS
! CAMS regional ensemble domain:
my.region.west : -30.0
my.region.east : 45.0
my.region.south : 30.0
my.region.north : 76.0
! size of map figures for this region:
my.region.figsize : (6,6)
! size of map figures for this region,
! default size is (8,6)
my.region.figsize : (8,6)
!! region name:
!my.region : globe
!! global domain:
!my.region.west : -180.0
!my.region.east : 180.0
!my.region.south : -90.0
!my.region.north : 90.0
!! size of map figures for this region:
!my.region.figsize : (8,6)
!----------------------------------------------------------
......@@ -208,8 +230,7 @@ my.region.figsize : (8,6)
! testing 3 days
my.timerange.start : 2018-06-01 00:00
my.timerange.end : 2018-06-01 23:59
!my.timerange.end : 2018-06-03 23:59
my.timerange.end : 2018-06-03 23:59
!----------------------------------------------------------
......@@ -246,7 +267,7 @@ my.work : /Scratch/${USER}/CSO-Tutorial
!======================================================================
!===
!=== Inquire - S-5P Hub
!=== Inquire DataSpace
!===
!======================================================================
......@@ -322,8 +343,11 @@ cso.tutorial.convert.timerange.end : ${my.timerange.end}
! listing of available source files, created by 'inquire' job:
cso.tutorial.convert.inquire.file : ${my.work}/Copernicus/Copernicus_S5p_NO2_dataspace__%Y-%m-%d.csv
!~ historic inquire ...
cso.tutorial.convert.inquire.filedate : 2023-11-06
!!~ historic inquire ...
!cso.tutorial.convert.inquire.filedate : 2024-01-18
! selection keyword:
my.tutorial.selection : C03
! Provide ';' seperated list of to decide if a particular orbit file should be processed.
! If more than one file is available for a particular orbit (from "OFFL" and "RPRO" processing),
......@@ -385,7 +409,7 @@ cso.tutorial.convert.filter.quality.units : 1
! - times are taken from mid of selection, rounded to hours
! - use '%{processing}' for the processing name
! - use '%{orbit}' for orbit number
cso.tutorial.convert.output.filename : ${my.work}/CSO-data/S5p/%{processing}/NO2/${my.region}/%Y/%m/S5p_%{processing}_NO2_%{orbit}.nc
cso.tutorial.convert.output.filename : ${my.work}/CSO-data/${my.region}/S5p/NO2/${my.tutorial.selection}/%Y/%m/S5p_%{processing}_NO2_%{orbit}.nc
! pack variables on output:
cso.tutorial.convert.output.packed : True
......@@ -447,6 +471,7 @@ cso.tutorial.convert.output.vars : longitude longitude_bounds \
cso.tutorial.convert.output.var.longitude.dims : pixel
cso.tutorial.convert.output.var.longitude.from : PRODUCT/longitude
cso.tutorial.convert.output.var.longitude.attrs : { 'bounds' : None }
! center latitudes; remove bounds attribute, no coordinate ...
cso.tutorial.convert.output.var.latitude.dims : pixel
cso.tutorial.convert.output.var.latitude.from : PRODUCT/latitude
......@@ -456,6 +481,10 @@ cso.tutorial.convert.output.var.latitude.attrs : { 'bounds'
cso.tutorial.convert.output.var.longitude_bounds.dims : pixel corner
cso.tutorial.convert.output.var.longitude_bounds.from : PRODUCT/SUPPORT_DATA/GEOLOCATIONS/longitude_bounds
cso.tutorial.convert.output.var.longitude_bounds.units : degrees_east
! ensure that near dateline the corners form a convex region around center
! (with some points outside [-180,+180] if necessary)
cso.tutorial.convert.output.var.longitude_bounds.special : longitude_bounds
! corner latitudes, no units in file:
cso.tutorial.convert.output.var.latitude_bounds.dims : pixel corner
cso.tutorial.convert.output.var.latitude_bounds.from : PRODUCT/SUPPORT_DATA/GEOLOCATIONS/latitude_bounds
......@@ -467,16 +496,19 @@ cso.tutorial.convert.output.var.track_longitude.dims : track_scan
cso.tutorial.convert.output.var.track_longitude.special : track_longitude
cso.tutorial.convert.output.var.track_longitude.from : PRODUCT/longitude
cso.tutorial.convert.output.var.track_longitude.attrs : { 'bounds' : None }
!
!~ center lat; remove bounds attribute, no coordinate ...
cso.tutorial.convert.output.var.track_latitude.dims : track_scan track_pixel
cso.tutorial.convert.output.var.track_latitude.special : track_latitude
cso.tutorial.convert.output.var.track_latitude.from : PRODUCT/latitude
cso.tutorial.convert.output.var.track_latitude.attrs : { 'bounds' : None }
!
!~ corner lons
cso.tutorial.convert.output.var.track_longitude_bounds.dims : track_scan track_pixel corner
cso.tutorial.convert.output.var.track_longitude_bounds.special : track_longitude_bounds
cso.tutorial.convert.output.var.track_longitude_bounds.from : PRODUCT/SUPPORT_DATA/GEOLOCATIONS/longitude_bounds
cso.tutorial.convert.output.var.track_longitude_bounds.units : degrees_east
!
!~ corner lats
cso.tutorial.convert.output.var.track_latitude_bounds.dims : track_scan track_pixel corner
cso.tutorial.convert.output.var.track_latitude_bounds.special : track_latitude_bounds
......@@ -565,7 +597,7 @@ cso.tutorial.convert.output.var.qa_value.attrs : { 'valid_min
! csv file that will hold records per file with:
! - timerange of pixels in file
! - orbit number
cso.tutorial.listing.file : ${my.work}/CSO-data/S5p/listing-NO2-${my.region}.csv
cso.tutorial.listing.file : ${my.work}/CSO-data/${my.region}/S5p/NO2/${my.tutorial.selection}/listing.csv
! renew table if file already exists?
cso.tutorial.listing.renew : True
......@@ -578,8 +610,7 @@ cso.tutorial.listing.timerange.end : ${my.timerange.end}
! names could include time templates ;
! if same orbit is found in multiple directories, the first found is used;
! remove existing table for safety to ensure that this is done correctly ...
cso.tutorial.listing.patterns : RPRO/NO2/${my.region}/%Y/%m/S5p_*.nc \
OFFL/NO2/${my.region}/%Y/%m/S5p_*.nc
cso.tutorial.listing.patterns : %Y/%m/S5p_*.nc
! extra columns to be added, read from global attributes:
cso.tutorial.listing.xcolumns : orbit
......@@ -602,7 +633,7 @@ cso.tutorial.catalogue.timerange.end : ${my.timerange.end}
! - time values
! - %{orbit} : from listing
! - %{varname} for variable
cso.tutorial.catalogue.output.file : ${my.work}/CSO-data-catalogue/S5p/NO2/${my.region}/%Y/%m/%d/S5p_NO2_%{orbit}__%{varname}.png
cso.tutorial.catalogue.output.file : ${my.work}/CSO-data-catalogue/${my.region}/S5p/NO2/${my.tutorial.selection}/%Y/%m/%d/S5p_NO2_%{orbit}__%{varname}.png
! map domain (west east south north)
cso.tutorial.catalogue.domain : ${my.region.west} ${my.region.east} ${my.region.south} ${my.region.north}
......@@ -611,8 +642,8 @@ cso.tutorial.catalogue.domain : ${my.region.west} ${my.reg
cso.tutorial.catalogue.figsize : ${my.region.figsize}
! renew existing files?
cso.tutorial.catalogue.renew : False
!cso.tutorial.catalogue.renew : True
!cso.tutorial.catalogue.renew : False
cso.tutorial.catalogue.renew : True
! variables to be plotted:
cso.tutorial.catalogue.vars : vcd vcd_errvar qa_value \
......@@ -667,7 +698,7 @@ cso.tutorial.catalogue.var.cloud_radiance_fraction.colors : ['blue','cyan',
!-----------------------------------------------------------
! target location:
cso.tutorial.catalogue-index.outdir : ${my.work}/CSO-data-catalogue/S5p/NO2/${my.region}
cso.tutorial.catalogue-index.outdir : ${my.work}/CSO-data-catalogue/${my.region}/S5p/NO2/${my.tutorial.selection}
! title:
cso.tutorial.catalogue-index.header : CSO catalogue
......@@ -819,7 +850,7 @@ cso.tutorial.gridded-catalogue.output.file : ${my.gridded-catalogue.
!cso.tutorial.gridded-catalogue.output.file : ${my.gridded-catalogue.output.dir}/%Y/%m/%d/S5p_NO2_%Y%m%d_aver_gridded_%{var}.png
! figure size (inches), default is (8,6):
cso.tutorial.gridded-catalogue.figsize : (6,6)
cso.tutorial.gridded-catalogue.figsize : ${my.region.figsize}
! variables to be plotted:
cso.tutorial.gridded-catalogue.vars : yr
......@@ -877,6 +908,94 @@ cso.tutorial.gridded-catalogue-index.date.time.var.kwargs : height=300
!======================================================================
!===
!=== super observations
!===
!======================================================================
! time range:
cso.tutorial.superobs.timerange.start : ${my.timerange.start}
cso.tutorial.superobs.timerange.end : ${my.timerange.end}
! renew existing files?
cso.tutorial.superobs.renew : True
!cso.tutorial.superobs.renew : False
!~
! keyword to describe super-observations,
! here gridding to 1.0x1.0
my.supobs.key : sup1x1__qa08
! grid definition:
cso.tutorial.superobs.grid.west : ${my.region.west}
cso.tutorial.superobs.grid.east : ${my.region.east}
cso.tutorial.superobs.grid.south : ${my.region.south}
cso.tutorial.superobs.grid.north : ${my.region.north}
! resolution:
cso.tutorial.superobs.grid.dlon : 1.0
cso.tutorial.superobs.grid.dlat : 1.0
! mapping level from footprints to grid cells:
!!~ center only, one pixel is enough ..
!cso.tutorial.superobs.mapping.levels : 0
!cso.tutorial.superobs.mapping.minimum_number : 0
!~ footprint area
cso.tutorial.superobs.mapping.levels : 3
cso.tutorial.superobs.mapping.minimum_coverage : 0.10
!~
! input listing:
cso.tutorial.superobs.source.listing : ${cso.tutorial.listing.file}
!~
! keywords for filters:
cso.tutorial.superobs.filters : quality
! minimum quality value required:
cso.tutorial.superobs.filter.quality.var : qa_value
cso.tutorial.superobs.filter.quality.type : min
cso.tutorial.superobs.filter.quality.min : 0.8
cso.tutorial.superobs.filter.quality.units : 1
!~
! output directory and filename:
cso.tutorial.superobs.output.file : ${my.work}/CSO-data/${my.region}/S5p/NO2/${my.tutorial.selection}__${my.supobs.key}/%Y/%m/%{root}.nc
! pack floats as short values?
cso.tutorial.superobs.output.packed : False
! zlib compression level (default 1, 0 for no compression):
cso.tutorial.superobs.output.complevel : 1
!~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
! listing
!~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
! renew table if file already exists?
cso.tutorial.superobs-listing.renew : True
! time range:
cso.tutorial.superobs-listing.timerange.start : ${my.timerange.start}
cso.tutorial.superobs-listing.timerange.end : ${my.timerange.end}
! filename filters relative to listing file that should be scanned for orbit files;
! names could include time templates ;
! if same orbit is found in multiple directories, the first found is used;
! remove existing table for safety to ensure that this is done correctly ...
cso.tutorial.superobs-listing.file : ${my.work}/CSO-data/${my.region}/S5p/NO2/${my.tutorial.selection}__${my.supobs.key}/listing.csv
cso.tutorial.superobs-listing.patterns : %Y/%m/S5p_*.nc
! extra columns to be added, read from global attributes:
cso.tutorial.superobs-listing.xcolumns : orbit processing processor_version
!======================================================================
!===
!=== simulation catalogue
......@@ -902,9 +1021,11 @@ cso.tutorial.sim-catalogue.output.file : ${my.work}/CSO-sim-catalogu
! map domain used for simulations (west east south north):
cso.tutorial.sim-catalogue.domain : -10 30 35 65
!!~ globe:
!cso.tutorial.sim-catalogue.domain : -180 180 -90 90
! figure size (inches), default is (8,6):
cso.tutorial.sim-catalogue.figsize : (6,6)
cso.tutorial.sim-catalogue.figsize : ${my.region.figsize}
! renew existing files?
!cso.tutorial.sim-catalogue.renew : False
......@@ -1032,6 +1153,10 @@ my.gridded.region : ${my.region}
!cso.tutorial.sim-gridded.grid.east : ${my.region.east}
!cso.tutorial.sim-gridded.grid.south : ${my.region.south}
!cso.tutorial.sim-gridded.grid.north : ${my.region.north}
!! resolution:
!cso.tutorial.sim-gridded.grid.dlon : 0.1
!cso.tutorial.sim-gridded.grid.dlat : 0.1
!
!~ observation operator tutorial:
cso.tutorial.sim-gridded.grid.west : -10
cso.tutorial.sim-gridded.grid.east : 30
......@@ -1117,7 +1242,7 @@ cso.tutorial.sim-gridded-catalogue.output.file : ${my.sim-gridded.di
!cso.tutorial.sim-gridded-catalogue.output.file : ${my.sim-gridded.dir}/catalogue/%Y/%m/%d/S5p_NO2_%Y%m%d_aver_gridded_%{var}.png
! figure size (inches), default is (8,6):
cso.tutorial.sim-gridded-catalogue.figsize : (6,6)
cso.tutorial.sim-gridded-catalogue.figsize : ${my.region.figsize}
! variables to be plotted:
cso.tutorial.sim-gridded-catalogue.vars : yr ys yr_m ys_m
......
doc/source/conf.py
View file @ 8bce8a14
......@@ -12,17 +12,18 @@
#
import os
import sys
sys.path.insert(0, os.path.abspath(os.path.join(os.pardir,os.pardir,'py')) )
sys.path.insert(0, os.path.abspath(os.path.join(os.pardir, os.pardir, "py")))
# -- Project information -----------------------------------------------------
project = 'CSO'
copyright = '2020-2023, Arjo Segers'
author = 'Arjo Segers'
project = "CSO"
copyright = "2020-2024, Arjo Segers"
author = "Arjo Segers"
# The full version, including alpha/beta/rc tags
release = 'v2.6.1'
release = "v2.7"
# -- General configuration ---------------------------------------------------
......@@ -31,13 +32,13 @@ release = 'v2.6.1'
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.intersphinx',
'sphinx.ext.mathjax',
"sphinx.ext.autodoc",
"sphinx.ext.intersphinx",
"sphinx.ext.mathjax",
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
templates_path = ["_templates"]
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
......@@ -53,12 +54,9 @@ exclude_patterns = []
# * 'show-inheritance' : name of base class;
# * 'members' : members of modules (classes) and classes (methods);
# * 'inherited-members' : inherited members:
autodoc_default_options = {
'members' : True,
'show-inheritance' : True
}
autodoc_default_options = {"members": True, "show-inheritance": True}
# do not sort methods alphabetically, keep source order:
autodoc_member_order = 'bysource'
autodoc_member_order = "bysource"
# ~~ options for intersphinx
......@@ -66,8 +64,8 @@ autodoc_member_order = 'bysource'
# https://www.sphinx-doc.org/en/master/usage/extensions/intersphinx.html
# add links to existing online documentation if present:
docs_url = 'https://docs.python.org/%i.%i' % (sys.version_info.major,sys.version_info.minor)
intersphinx_mapping = { 'python' : (docs_url,None) }
docs_url = "https://docs.python.org/%i.%i" % (sys.version_info.major, sys.version_info.minor)
intersphinx_mapping = {"python": (docs_url, None)}
# -- Options for HTML output -------------------------------------------------
......@@ -75,12 +73,12 @@ intersphinx_mapping = { 'python' : (docs_url,None) }
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'bizstyle'
html_theme = "bizstyle"
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
#html_static_path = ['_static']
# html_static_path = ['_static']
html_static_path = []
......@@ -89,4 +87,4 @@ html_static_path = []
# -- Options for intersphinx extension ---------------------------------------
# Example configuration for intersphinx: refer to the Python standard library.
intersphinx_mapping = {'https://docs.python.org/3/': None}
intersphinx_mapping = {"https://docs.python.org/3/": None}
doc/source/history.rst
View file @ 8bce8a14
......@@ -91,10 +91,16 @@ A summary of the versions and changes.
| Support new Copernicus *DataSpace* portal to download Sentinel data.
| *(2023-11)*
* | *v2.6.1*
* | *v2.6*
| Support SO\ :sub:`2` COBRA product from PAL website.
| *(2024-01)*
* | *v2.7*
| Fixed averaging over time values when creating super-observations.
| Use cartopy for map plots.
| Support orbits that overlap with dateline.
| *(2024-01)*
To be included
==============
......@@ -119,6 +125,8 @@ A wishlist of developments.
* Also output by observation operator could use the same basename.
* Add all S5p filename parts to listingfile, see *Product User Manual*.
* Add offline tool to re-create listing files.
* Scan directories based on filename patterns.
......@@ -135,30 +143,32 @@ A wishlist of developments.
How to create a version tag
===========================
Example for tag "v2.6.1".
Example for tag "v2.7".
1. Ensure that Python files are formatted following convention::
make black
1. Ensure that all changes are committed and pushed::
2. Update the history sections of source files, commit changes.
git status
git commit -m 'Recent updates.' .
git push
3. Update the `CHANGELOG <../../../CHANGELOG>`.
2. Update the source file `doc/source/history.rst <../../source/history.rst>`_ of this chapter.
4. Update the source file `doc/source/history.rst <../../source/history.rst>`_ of this chapter.
3. Change the release tag that will be shown in the documentation in the file
5. Change the release tag that will be shown in the documentation in the file
`doc/source/conf.py <../../source/conf.py>`_::
# The full version, including alpha/beta/rc tags
release = "v2.6.1"
release = "v2.7"
4. Commit these changes::
6. Commit these changes::
git commit -m 'Tagged v2.6.1 .' doc/source/history.rst doc/source/conf.py
git commit -m 'Tagged v2.7' doc/source/history.rst doc/source/conf.py CHANGELOG
5. Tag and push the code to the git repository::
7. Tag and push the code to the git repository::
git tag -a 'v2.6.1' -m 'Tagged v2.6.1'
git push origin 'v2.6.1'
git tag -a 'v2.7' -m 'Tagged v2.7'
git push origin v2.7
doc/source/s5p-chocho.rst
View file @ 8bce8a14
......@@ -148,24 +148,29 @@ The processing stream is identified by a 4-character key:
* ``RPRO`` : re-processing of all previously made observations.
* ``PAL_`` : processed data stored on the *Product Algorithm Laboratory* portal.
The two archives need to be inquired seperately.
* The :py:class:`CSO_SciHub_Listing <cso_scihub.CSO_SciHub_Listing>` class is available to list the content
of the downloaded orbit files.
The result is a csv file which with columns for keywords such orbit number and processor version,
as well as the filename relative to the location of the csv file::
The portals provide data files created with the same retrieval algorithm, but most recent data
(latest processor version) might be available on only one of the portals.
It is therefore necessary to first inquire both archives to see which data is available where,
and what the version numbers are.
The :py:class:`CSO_DataSpace_Inquire <cso_dataspace.CSO_DataSpace_Inquire>` class is available to inquire the
*Copernicus DataSpace*. The settings used by this class allow selection
on for example time range and intersection area.
The result is a csv file which with columns for keywords such as orbit number and processor version,
as well as the filename of the data and the url that should be used to actually download the data::
filename ;start_time ;end_time ;mission;processing;product_id;orbit;collection;processor_version;processing_time
2020/01/01/S5P_OFFL_L2__CHOCHO___20200101T005246_20200101T023416_11487_01_010000_20210128.nc;2020-01-01 00:52:46;2020-01-01 02:34:16;S5P ;OFFL ;L2__CHOCHO;11487;01 ;010000 ;2021-01-28
2020/01/01/S5P_OFFL_L2__CHOCHO___20200101T023416_20200101T041546_11488_01_010000_20210128.nc;2020-01-01 02:34:16;2020-01-01 04:15:46;S5P ;OFFL ;L2__CHOCHO;11488;01 ;010000 ;2021-01-28
:
filename ;start_time ;end_time ;mission;processing;product_id;orbit;collection;processor_version;processing_time
2020/01/01/S5P_OFFL_L2__CHOCHO___20200101T005246_20200101T023416_11487_01_010000_20210128.nc;2020-01-01 00:52:46;2020-01-01 02:34:16;S5P ;OFFL ;L2__CHOCHO;11487;01 ;010000 ;2021-01-28
2020/01/01/S5P_OFFL_L2__CHOCHO___20200101T023416_20200101T041546_11488_01_010000_20210128.nc;2020-01-01 02:34:16;2020-01-01 04:15:46;S5P ;OFFL ;L2__CHOCHO;11488;01 ;010000 ;2021-01-28
:
See the section on *File name convention* in the *Product User Manual* for the meaning of all
parts of the filename.
See the section on *File name convention* in the *Product User Manual* for the meaning of all
parts of the filename.
A similar class :py:class:`CSO_S5p_Download_Listing <cso_s5p.CSO_S5p_Download_Listing>`
class is available to list the content of the downloaded GlyRetro files.
Also this will produce a table file.
* The :py:class:`CSO_PAL_Inquire <cso_pal.CSO_PAL_Inquire>` class is available to list the content of
the PAL archive.
To visualize what is available from the portal, the
:py:class:`CSO_Inquire_Plot <cso_inquire.CSO_Inquire_Plot>` could be used to create an overview figure:
......@@ -183,7 +188,7 @@ The jobtree configuration to inquire the portals and create the overview figure
cso.s5p.chocho.inquire.tasks : table-glyretro table-pal plot
!~ inquire files downloaded from GlyRetro:
cso.s5p.chocho.inquire.table-glyretro.class : cso.CSO_SciHub_Listing
cso.s5p.chocho.inquire.table-glyretro.class : cso.CSO_S5p_Download_Listing
cso.s5p.chocho.inquire.table-glyretro.args : '${PWD}/config/Copernicus/cso-s5p-chocho.rc', \
rcbase='cso.s5p.chocho.inquire-table-glyretro'
!~ inquire files available on PAL:
......@@ -277,7 +282,7 @@ and this will copy the selected pixels for variables specified in the settings.
The variable specification starts with a list with variable names to be
created in the target file::
cso.s5p.chocho.convert.output.vars : longitude longitude_bounds \
cso.s5p.chocho.convert.output.vars : longitude longitude_bounds \
latitude latitude_bounds \
track_longitude track_longitude_bounds \
track_latitude track_latitude_bounds \
......@@ -305,6 +310,27 @@ give warnings from the CF-compliance checker::
cso.s5p.chocho.convert.output.var.latitude.from : PRODUCT/latitude
cso.s5p.chocho.convert.output.var.latitude.attrs : { 'bounds' : None }
The pixel boundaries are necessary to know the exact footprint of a pixel,
which is for example used when averaging over a grid or simulation from a model.
These are available in the input files, but without a ``units`` attribute as these
are implied by the pixel center coordinate; the conversion therefore requires that
units are defined explicitly.
For the ``longitude_bounds`` a special processing is needed for pixels crossing the dateline,
as the original data simply uses longitudes modulo 360 degrees::
! corner longitudes; no units in file:
cso.s5p.chocho.convert.output.var.longitude_bounds.dims : pixel corner
cso.s5p.chocho.convert.output.var.longitude_bounds.from : PRODUCT/SUPPORT_DATA/GEOLOCATIONS/longitude_bounds
cso.s5p.chocho.convert.output.var.longitude_bounds.units : degrees_east
! ensure that near dateline the corners form a convex region around center
! (with some points outside [-180,+180] if necessary)
cso.s5p.chocho.convert.output.var.longitude_bounds.special : longitude_bounds
! corner latitudes, no units in file:
cso.s5p.chocho.convert.output.var.latitude_bounds.dims : pixel corner
cso.s5p.chocho.convert.output.var.latitude_bounds.from : PRODUCT/SUPPORT_DATA/GEOLOCATIONS/latitude_bounds
cso.s5p.chocho.convert.output.var.latitude_bounds.units : degrees_north
Also the locations of the pixels in the original track are copied,
since these are useful when creating plots. These cannot be copied directly but require special processing::
......
doc/source/s5p-co.rst
View file @ 8bce8a14
......@@ -119,8 +119,8 @@ The result is a csv file which with columns for keywords such orbit number and p
as well as the filename of the data and the url that should be used to actually download the data::
orbit;start_time;end_time;processing;collection;processor_version;filename;href
23665;2022-05-08 09:44:59;2022-05-08 11:26:29;RPRO;03;020400;S5P_RPRO_L2__CO_____20220508T094459_20220508T112629_23665_03_020400_20230112T000125.nc;https://s5phub.copernicus.eu/dhus/odata/v1/Products('afbaef9f-5295-4d47-be43-f74270e07b4d')/$value
23666;2022-05-08 11:26:29;2022-05-08 13:07:59;RPRO;03;020400;S5P_RPRO_L2__CO_____20220508T112629_20220508T130759_23666_03_020400_20230112T000126.nc;https://s5phub.copernicus.eu/dhus/odata/v1/Products('3beada57-7602-482e-bde4-7a10758327e3')/$value
23665;2022-05-08 09:44:59;2022-05-08 11:26:29;RPRO;03;020400;S5P_RPRO_L2__CO_____20220508T094459_20220508T112629_23665_03_020400_20230112T000125.nc;https://zipper.dataspace.copernicus.eu/odata/v1/Products('afbaef9f-5295-4d47-be43-f74270e07b4d')/$value
23666;2022-05-08 11:26:29;2022-05-08 13:07:59;RPRO;03;020400;S5P_RPRO_L2__CO_____20220508T112629_20220508T130759_23666_03_020400_20230112T000126.nc;https://zipper.dataspace.copernicus.eu/odata/v1/Products('3beada57-7602-482e-bde4-7a10758327e3')/$value
:
See the section on *File name convention* in the *Product User Manual* for the meaning of all
......@@ -234,7 +234,7 @@ The jobtree configuration to inquire the portals and create the overview figure
The variable specification starts with a list with variable names to be
created in the target file::
cso.s5p.co.convert.output.vars : longitude longitude_bounds \
cso.s5p.co.convert.output.vars : longitude longitude_bounds \
latitude latitude_bounds \
track_longitude track_longitude_bounds \
track_latitude track_latitude_bounds \
......@@ -267,6 +267,27 @@ The jobtree configuration to inquire the portals and create the overview figure
cso.s5p.co.convert.output.var.latitude.from : PRODUCT/latitude
cso.s5p.co.convert.output.var.latitude.attrs : { 'bounds' : None }
The pixel boundaries are necessary to know the exact footprint of a pixel,
which is for example used when averaging over a grid or simulation from a model.
These are available in the input files, but without a ``units`` attribute as these
are implied by the pixel center coordinate; the conversion therefore requires that
units are defined explicitly.
For the ``longitude_bounds`` a special processing is needed for pixels crossing the dateline,
as the original data simply uses longitudes modulo 360 degrees::
! corner longitudes; no units in file:
cso.s5p.co.convert.output.var.longitude_bounds.dims : pixel corner
cso.s5p.co.convert.output.var.longitude_bounds.from : PRODUCT/SUPPORT_DATA/GEOLOCATIONS/longitude_bounds
cso.s5p.co.convert.output.var.longitude_bounds.units : degrees_east
! ensure that near dateline the corners form a convex region around center
! (with some points outside [-180,+180] if necessary)
cso.s5p.co.convert.output.var.longitude_bounds.special : longitude_bounds
! corner latitudes, no units in file:
cso.s5p.co.convert.output.var.latitude_bounds.dims : pixel corner
cso.s5p.co.convert.output.var.latitude_bounds.from : PRODUCT/SUPPORT_DATA/GEOLOCATIONS/latitude_bounds
cso.s5p.co.convert.output.var.latitude_bounds.units : degrees_north
Also the locations of the pixels in the original track are copied,
since these are useful when creating plots. These cannot be copied directly but require special processing::
......
doc/source/s5p-hcho.rst
View file @ 8bce8a14
......@@ -124,8 +124,8 @@ The result is a csv file which with columns for keywords such as orbit number an
as well as the filename of the data and the url that should be used to actually download the data::
orbit;start_time;end_time;processing;collection;processor_version;filename;href
21497;2021-12-06 14:05:54;2021-12-06 15:47:24;OFFL;02;020301;S5P_OFFL_L2__HCHO____20211206T140554_20211206T154724_21497_02_020301_20211208T043331.nc;https://s5phub.copernicus.eu/dhus/odata/v1/Products('d9d33ffa-9fe5-43cc-b5a1-b65c22e874ad')/$value
21852;2021-12-31 14:37:39;2021-12-31 16:19:09;OFFL;02;020301;S5P_OFFL_L2__HCHO____20211231T143739_20211231T161909_21852_02_020301_20220102T064010.nc;https://s5phub.copernicus.eu/dhus/odata/v1/Products('ff5c922c-450c-43db-97e4-f46bdd55ffb2')/$value
21497;2021-12-06 14:05:54;2021-12-06 15:47:24;OFFL;02;020301;S5P_OFFL_L2__HCHO____20211206T140554_20211206T154724_21497_02_020301_20211208T043331.nc;https://zipper.dataspace.copernicus.eu/odata/v1/Products('d9d33ffa-9fe5-43cc-b5a1-b65c22e874ad')/$value
21852;2021-12-31 14:37:39;2021-12-31 16:19:09;OFFL;02;020301;S5P_OFFL_L2__HCHO____20211231T143739_20211231T161909_21852_02_020301_20220102T064010.nc;https://zipper.dataspace.copernicus.eu/odata/v1/Products('ff5c922c-450c-43db-97e4-f46bdd55ffb2')/$value
:
See the section on *File name convention* in the *Product User Manual* for the meaning of all
......@@ -299,6 +299,28 @@ give warnings from the CF-compliance checker::
cso.s5p.hcho.convert.output.var.latitude.from : PRODUCT/latitude
cso.s5p.hcho.convert.output.var.latitude.attrs : { 'bounds' : None }
The pixel boundaries are necessary to know the exact footprint of a pixel,
which is for example used when averaging over a grid or simulation from a model.
These are available in the input files, but without a ``units`` attribute as these
are implied by the pixel center coordinate; the conversion therefore requires that
units are defined explicitly.
For the ``longitude_bounds`` a special processing is needed for pixels crossing the dateline,
as the original data simply uses longitudes modulo 360 degrees::
! corner longitudes; no units in file:
cso.s5p.hcho.convert.output.var.longitude_bounds.dims : pixel corner
cso.s5p.hcho.convert.output.var.longitude_bounds.from : PRODUCT/SUPPORT_DATA/GEOLOCATIONS/longitude_bounds
cso.s5p.hcho.convert.output.var.longitude_bounds.units : degrees_east
! ensure that near dateline the corners form a convex region around center
! (with some points outside [-180,+180] if necessary)
cso.s5p.hcho.convert.output.var.longitude_bounds.special : longitude_bounds
! corner latitudes, no units in file:
cso.s5p.hcho.convert.output.var.latitude_bounds.dims : pixel corner
cso.s5p.hcho.convert.output.var.latitude_bounds.from : PRODUCT/SUPPORT_DATA/GEOLOCATIONS/latitude_bounds
cso.s5p.hcho.convert.output.var.latitude_bounds.units : degrees_north
Also the locations of the pixels in the original track are copied,
since these are useful when creating plots. These cannot be copied directly but require special processing::
......@@ -658,9 +680,9 @@ to create a catalogue of images for the converted files.
Configuration could look like::
! catalogue creation task:
cso.s5p.so2-cobra.sim-catalogue.task.class : cso.CSO_SimCatalogue
cso.s5p.so2-cobra.sim-catalogue.task.args : '${PWD}/config/Copernicus/cso-s5p-TRACER.rc', \
rcbase='cso.s5p.so2-cobra.sim-catalogue'
cso.s5p.hcho.sim-catalogue.task.class : cso.CSO_SimCatalogue
cso.s5p.hcho.sim-catalogue.task.args : '${PWD}/config/Copernicus/cso-s5p-TRACER.rc', \
rcbase='cso.s5p.hcho.sim-catalogue'
The configuration describes where to find a *listing* file with orbits,
which variables should be plot, the colorbar properties, etc.
......@@ -693,8 +715,8 @@ Optionally settings could be used to specifiy a different unit, or the value ran
Figures are saved to files with the basename of the original orbit file and the plotted variable::
file://Scratch/cso-catalogue/SO2/2018/06/01/S5p_RPRO_HCHO_20180601_1200_yr.png
S5p_RPRO_HCHO_20180601_1200_ys.png
file://Scratch/cso-catalogue/HCHO/2018/06/01/S5p_RPRO_HCHO_20180601_1200_yr.png
S5p_RPRO_HCHO_20180601_1200_ys.png
To search for interesting features in the data,
......@@ -717,3 +739,4 @@ When succesful, the index creator displays an url that could be loaded in a brow
:alt: Index for Simulated and S5p HCHO columns
doc/source/s5p-no2.rst
View file @ 8bce8a14
......@@ -243,8 +243,8 @@ The result is a csv file which with columns for keywords such as orbit number an
as well as the filename of the data and the url that should be used to actually download the data::
orbit;start_time;end_time;processing;collection;processor_version;filename;href
21497;2021-12-06 14:05:54;2021-12-06 15:47:24;OFFL;02;020301;S5P_OFFL_L2__NO2____20211206T140554_20211206T154724_21497_02_020301_20211208T043331.nc;https://s5phub.copernicus.eu/dhus/odata/v1/Products('d9d33ffa-9fe5-43cc-b5a1-b65c22e874ad')/$value
21852;2021-12-31 14:37:39;2021-12-31 16:19:09;OFFL;02;020301;S5P_OFFL_L2__NO2____20211231T143739_20211231T161909_21852_02_020301_20220102T064010.nc;https://s5phub.copernicus.eu/dhus/odata/v1/Products('ff5c922c-450c-43db-97e4-f46bdd55ffb2')/$value
21497;2021-12-06 14:05:54;2021-12-06 15:47:24;OFFL;02;020301;S5P_OFFL_L2__NO2____20211206T140554_20211206T154724_21497_02_020301_20211208T043331.nc;https://zipper.dataspace.copernicus.eu/odata/v1/Products('d9d33ffa-9fe5-43cc-b5a1-b65c22e874ad')/$value
21852;2021-12-31 14:37:39;2021-12-31 16:19:09;OFFL;02;020301;S5P_OFFL_L2__NO2____20211231T143739_20211231T161909_21852_02_020301_20220102T064010.nc;https://zipper.dataspace.copernicus.eu/odata/v1/Products('ff5c922c-450c-43db-97e4-f46bdd55ffb2')/$value
:
See the section on *File name convention* in the *Product User Manual* for the meaning of all
......@@ -367,7 +367,7 @@ The variable specification starts with a list with variable names to be
created in the target file::
cso.s5p.no2.convert.output.vars : longitude longitude_bounds \
latitude latitude_bounds \
latitude latitude_bounds \
track_longitude track_longitude_bounds \
track_latitude track_latitude_bounds \
time \
......@@ -394,6 +394,27 @@ give warnings from the CF-compliance checker::
cso.s5p.no2.convert.output.var.latitude.from : PRODUCT/latitude
cso.s5p.no2.convert.output.var.latitude.attrs : { 'bounds' : None }
The pixel boundaries are necessary to know the exact footprint of a pixel,
which is for example used when averaging over a grid or simulation from a model.
These are available in the input files, but without a ``units`` attribute as these
are implied by the pixel center coordinate; the conversion therefore requires that
units are defined explicitly.
For the ``longitude_bounds`` a special processing is needed for pixels crossing the dateline,
as the original data simply uses longitudes modulo 360 degrees::
! corner longitudes; no units in file:
cso.s5p.no2.convert.output.var.longitude_bounds.dims : pixel corner
cso.s5p.no2.convert.output.var.longitude_bounds.from : PRODUCT/SUPPORT_DATA/GEOLOCATIONS/longitude_bounds
cso.s5p.no2.convert.output.var.longitude_bounds.units : degrees_east
! ensure that near dateline the corners form a convex region around center
! (with some points outside [-180,+180] if necessary)
cso.s5p.no2.convert.output.var.longitude_bounds.special : longitude_bounds
! corner latitudes, no units in file:
cso.s5p.no2.convert.output.var.latitude_bounds.dims : pixel corner
cso.s5p.no2.convert.output.var.latitude_bounds.from : PRODUCT/SUPPORT_DATA/GEOLOCATIONS/latitude_bounds
cso.s5p.no2.convert.output.var.latitude_bounds.units : degrees_north
Also the locations of the pixels in the original track are copied,
since these are useful when creating plots. These cannot be copied directly but require special processing::
......
doc/source/s5p-so2.rst
View file @ 8bce8a14
......@@ -135,8 +135,8 @@ The result is a csv file which with columns for keywords such as orbit number an
as well as the filename of the data and the url that should be used to actually download the data::
orbit;start_time;end_time;processing;collection;processor_version;filename;href
21497;2021-12-06 14:05:54;2021-12-06 15:47:24;OFFL;02;020301;S5P_OFFL_L2__SO2____20211206T140554_20211206T154724_21497_02_020301_20211208T043331.nc;https://s5phub.copernicus.eu/dhus/odata/v1/Products('d9d33ffa-9fe5-43cc-b5a1-b65c22e874ad')/$value
21852;2021-12-31 14:37:39;2021-12-31 16:19:09;OFFL;02;020301;S5P_OFFL_L2__SO2____20211231T143739_20211231T161909_21852_02_020301_20220102T064010.nc;https://s5phub.copernicus.eu/dhus/odata/v1/Products('ff5c922c-450c-43db-97e4-f46bdd55ffb2')/$value
21497;2021-12-06 14:05:54;2021-12-06 15:47:24;OFFL;02;020301;S5P_OFFL_L2__SO2____20211206T140554_20211206T154724_21497_02_020301_20211208T043331.nc;https://zipper.dataspace.copernicus.eu/odata/v1/Products('d9d33ffa-9fe5-43cc-b5a1-b65c22e874ad')/$value
21852;2021-12-31 14:37:39;2021-12-31 16:19:09;OFFL;02;020301;S5P_OFFL_L2__SO2____20211231T143739_20211231T161909_21852_02_020301_20220102T064010.nc;https://zipper.dataspace.copernicus.eu/odata/v1/Products('ff5c922c-450c-43db-97e4-f46bdd55ffb2')/$value
:
See the section on *File name convention* in the *Product User Manual* for the meaning of all
......@@ -326,6 +326,28 @@ give warnings from the CF-compliance checker::
cso.s5p.so2.convert.output.var.latitude.dims : pixel
cso.s5p.so2.convert.output.var.latitude.from : PRODUCT/latitude
cso.s5p.so2.convert.output.var.latitude.attrs : { 'bounds' : None }
The pixel boundaries are necessary to know the exact footprint of a pixel,
which is for example used when averaging over a grid or simulation from a model.
These are available in the input files, but without a ``units`` attribute as these
are implied by the pixel center coordinate; the conversion therefore requires that
units are defined explicitly.
For the ``longitude_bounds`` a special processing is needed for pixels crossing the dateline,
as the original data simply uses longitudes modulo 360 degrees::
! corner longitudes; no units in file:
cso.s5p.so2.convert.output.var.longitude_bounds.dims : pixel corner
cso.s5p.so2.convert.output.var.longitude_bounds.from : PRODUCT/SUPPORT_DATA/GEOLOCATIONS/longitude_bounds
cso.s5p.so2.convert.output.var.longitude_bounds.units : degrees_east
! ensure that near dateline the corners form a convex region around center
! (with some points outside [-180,+180] if necessary)
cso.s5p.so2.convert.output.var.longitude_bounds.special : longitude_bounds
! corner latitudes, no units in file:
cso.s5p.so2.convert.output.var.latitude_bounds.dims : pixel corner
cso.s5p.so2.convert.output.var.latitude_bounds.from : PRODUCT/SUPPORT_DATA/GEOLOCATIONS/latitude_bounds
cso.s5p.so2.convert.output.var.latitude_bounds.units : degrees_north
Also the locations of the pixels in the original track are copied,
since these are useful when creating plots. These cannot be copied directly but require special processing::
......
oper/CHANGELOG 0 → 100644
View file @ 8bce8a14
Changes in CSO Observation Operator code
----------------------------------------
v2.6
----
Improved packing of constant variables.
src/cso_ncfile.F90
Do not pack 'time' variables.
src/cso_pixels.F90
Support files with "sample" coordinate instead of "pixel",
and without "corner" data (also implies no "area").
src/cso_sat.F90
Added option to define datatype of variable;
mainly used to define dtype "i1" (integer(1)) or "c" (character)
that are used for ground observation data.
src/cso_sat.F90
src/cso_pixels.F90
src/cso_domains.F90
src/cso_ncfile.F90
src/cso_comm.F90
Return null pointers for corner locations if these are not available.
src/cso_sat.F90
Check on allocation before cleanup.
src/cso_domains.F90
Improved error check when creating directories using Intel compiler.
src/cso_file.F90
Added "GetWeightsCell" methodes to assign points to grid cells.
src/cso_grid.F90
Trap undefined values in kernel application.
src/cso_pixels.F90
Added formula 'TropoPressures' to create variable with half-level pressures
at surface, top of boundary layer, and top of troposhere.
src/cso_pixels.F90
src/cso_tools.F90
v2.7
----
Ensure that pixels crossing the dateline are simulated correctly.
src/cso_tools.F90
src/cso_grid.F90
src/tutorial_oper_S5p.F90
oper/README.txt
View file @ 8bce8a14
......@@ -18,6 +18,8 @@ launcher*
Changes
-------
(see CHANGELOG for detailed overview)
v1.2
- Support initialization with integer communicator for main programs
with older MPI implementations.
......@@ -33,5 +35,9 @@ v1.6
v1.7
- Implemented vertical mapping and kernel convolution as user-defined operators.
v2.6
- Improved support for packing.
v2.7
- Support pixels crossing dateline.