CSO - CAMS Satellite Operator
=============================


Version tags
------------

See "History" chapter in documentation for overview of versions.
Detailled changes per version are listed below.


v2.1
----

Updated documentation.
  doc/source/documentation.rst
  doc/source/s5p-so2.rst

Improved MPI data type selection.
  oper/src/cso_comm.F90
  oper/src/cso_ncfile.F90

Properly count user defined variables.
  oper/src/cso_pixels.F90

Fixed deallocation.
  oper/src/cso_sat.F90

Only setup exchange between domains if needed.
  oper/src/cso_sat.F90

Optional limit output variables to selection.
  oper/src/cso_sat.F90

Trap non-existing or empty listing files when creating catalogue of converted data.
  py/cso_catalogue.py

Updated documentation of special conversions.
  py/cso_s5p.py

Templates for S5p/CO product.
  doc/samples/S5p_OFFL_L2__CO____*.txt
  py/cso_s5p.py

Processing of NO2 superobservations provided by KNMI.
  doc/source/figs/NO2_superobs
  doc/source/index.rst
  doc/source/obsoper.rst
  doc/source/pymod-cso_s5p_superobs.rst
  doc/source/s5p-no2-superobs.rst
  py/cso.py
  py/cso_s5p.py
  py/cso_s5p_superobs.py

Support timeseries with different resolution.
  py/cso_catalogue.py
  py/cso_index.py

Updated documentation.
  doc/source/s5p-no2.rst
  doc/README.md

Read data on root, scatter to processes.
  oper/src/cso_comm.F90
  oper/src/cso_domains.F90
  oper/src/cso_ncfile.F90
  oper/src/cso_pixels.F90
  oper/src/cso_sat.F90

Create gridded averages using mapping functions similar as used in observation operator.
  py/cso_gridded.py
  py/cso_mapping.py
  py/cso_constants.py
  doc/source/gridding.rst
  doc/source/pymod-cso_mapping.rst
  doc/source/pymods.rst
  
Add time dimension to gridded averages to allow temporal averaging.
  py/cso_gridded.py

When mapping footprints to grid cells, split triangles into 2 new triangles 
based on the longest edge 
  py/cso_mapping.py

Retry reading of S5p data, sometimes fails for unknown reason.
  py/cso_s5p.py

Added class to create plot of S5p versions downloaded from SciHub.
  py/cso_scihub.py

Support processing of S5p/NO2 data from PAL.
  py/cso_pal.py
  
Changed S5p conversion class to download files while converting.
Eventually downloaded files are removed after conversion.
  py/cso_s5p.py
  py/cso_scihub.py

Aggregation over regions.
  py/cso_regions.py
  data/country-codes*.csv
  doc/source/pymod-cso_regions.rst
  doc/source/pymods.rst
  doc/source/gridding.rst

Support for S5p/Glyoxal product.
  config/ESA-S5p/cso-s5p-glyox.rc
  doc/source/s5p-glyoxal.rst
  doc/source/figs/GLYOX/
  doc/samples/S5P_OFFL_L2__CHOCHO___20200601T001519_20200601T015649_13643_01_010000_20210128.txt
  doc/source/index.rst


v2.2
----

Support batch job for Sun Grid Engine.
  py/utopya_jobscript.py
  py/utopya_jobtree.py

Improved inquire tasks for SciHub and PAL portals.
  py/cso_scihub.py
  py/cso_pal.py
  py/cso.py
  py/cso_inquire.py
  doc/source/pymod-cso_inquire.rst
  doc/source/pymod-cso_pal.rst

Updated documentations for inquire tasks.
  doc/source/tutorial.rst
  doc/source/s5p-so2.rst
  doc/source/s5p-no2.rst
  doc/source/s5p-hcho.rst
  config/tutorial/tutorial.rc

Support computation of grid cell areas for 2D arrays.
  py/cso_mapping.py

Added class to create super-observations.
  py/cso_superobs.py
  py/cso.py
  doc/source/pymod-cso_superobs.rst
  doc/source/pymods.rst

Specify output file template for catalogue figures rather than output directory.
Support steps over year when plotting series of gridded observations.
  py/cso_catalogue.py

Write files with packing and compression.
  py/cso_file.py
  py/cso_s5p.py
  oper/src/

Added class to create temporal means of gridded averages.
  py/cso_gridded.py
  doc/source/gridding.rst

Read Timestamp object from rcfile settings.
  py/rc.py

Support index timeseries over years.
  py/utopya_index.py

Updated product description of S5p/Glyoxal.
  doc/source/s5p-glyoxal.rst

Updated template settings.
  config/tutorial/tutorial.rc

Added tools to collect and plot time series of co-located pixel averages.
  py/cso_colocate.py
  py/cso.py
  py/cso_mapping.py
  doc/source/index.rst
  doc/source/colocate.rst
  doc/source/pymods.rst
  doc/source/pymod-cso_colocate.rst
  doc/source/documentation.rst

Create output directory if necessary.
  py/cso_inquire.py

Trap errors from plotting coastlines and country borders.
  py/cso_plot.py

Updated user guide.
  doc/source/figs/footprints_nc4_levels_and_area.png
  doc/source/obsoper.rst

Added option to plot specific layer of gridded averages.
Adhoc fix to trap nans in some files.
  py/cso_catalogue.py 

Support packed output for gridded averages.
  py/cso_gridded.py

Updated documentation.
  py/cso.py

Removed depricated DHUS downloads.
  py/cso_scihub.py

Define environment variable CSO_PREFIX.
  bin/cso
  
Use environment variable for installation prefix, use own filename for settings.
    config/tutorial/tutorial.rc


Changes for v2.2
----------------

Option to create gridded fields from orbit files in listing.
  py/cso_gridded.py
  py/cso_file.py
  config/tutorial/tutorial.rc


Changes for v2.3
----------------

Added collection level to inquire overview plots.
  py/cso_inquire.py

Convert selected orbit files based on selection query.
  py/cso_s5p.py

Synchronized template settings for Copernicus S5P products.
  config/Copernicus/

Updated User guide.
  doc/source/


v2.4
----

Added 'black' target to Makefile, including convention for maximum line length.
Reformatted files following new convention.
  Makefile
  py/*


v2.5
----

Fixed definition of CSO_PREFIX environment variable.  
  bin/cso

Updated tutorial.
  config/tutorial/tutorial.rc
  doc/source/tutorial.rst

Define lon/lat axes of gridded output as floats.
  py/cso_gridded.py

Use pandas concat to combine dataframes.
  py/cso_pal.py

Replaced `where` concstructs by loops after memory errors in output filling.
  oper/src/cso_ncfile.F90
  
Check on `pix_all` value to know if pixels are assigned to any of the sub-domains.
  src/tutorial_oper_S5p.F90

Migration from Copernicus SciHub to DataSpace.
  bin/dhusget.sh
  doc/source/*
  py/*
  config/*


Patch 2.5.1
~~~~~~~~~~~

Fixed bug in S5P orbit selection.
  py/cso_s5p.py


v2.6
----

Support SO2-COBRA product from PAL website.


v2.7
----

Fixed averaging over time values when creating super-observations.
  py/cso_superobs.py

Improved output of packed variables.
Support files with "sample" coordinate instead of "pixel".
  oper/

Use cartopy for map plots.
  py/cso_plot.py
  py/cso_catalogue.py

Support orbits that overlap with dateline.
  py/cso_s5p.py
  py/cso_gridded.py
  py/cso_superobs.py


v2.7
----

Removed support for NO2-superobs, now create these using own classes.
  py/cso_s5p_superobs.py
  py/cso.py
  config/EMEP/emep-tropomi-superobs.rc
  doc/source/index.rst
  doc/source/pymod-cso_s5p_superobs.rst
  doc/source/s5p-no2-superobs.rst

Removed depricated references to S5p-Hub.
  doc/source
  py/

Removed whitespace from lines for easier comparison with other versions.
  py/

Synchronized layout with CSO public version for easier synchronization.
Synchronized change logs in source files between master and development versions.
  py/


v2.7.1
~~~~~~

Fixed creation of super-observations on local domain.
  py/cso_superobs.py


v2.7.2
~~~~~~

Updated url of DataSpace OpenSearch API.


v2.8
----

Improved submission of parallel jobs.
  py/utopya_jobtree.py

Added "CSO_S5p_Download" class to only download S5p files without conversion.
	py/cso_s5p.py
	config/Copernicus/cso.rc
	config/Copernicus/cso-s5p-co.rc
	config/Copernicus/cso-s5p-hcho.rc
	config/Copernicus/cso-s5p-no2.rc
	config/Copernicus/cso-s5p-so2-cobra.rc
	config/Copernicus/cso-s5p-so2.rc

Define environment variable 'CSO_RCFILE' for use in settings.
  bin/cso

Updated re-try settings for downloads based on latest experiences.
Trap download errors.
  py/cso_dataspace.py

Support creation of multiple gridded output files.
  py/cso_gridded.py

Updated plotting features.
  py/cso_inquire.py
  py/cso_plot.py

Improved plotting of maps and timeseries per country.
Support box-plot timeseries to show variation over region.
  py/cso_regions.py

Added tools to make working copy of scripts and settings.
  py/utopya_build.py


v2.9
----

Updated links to Sentinel documents.
  doc/surc/s5p-*.rst

Initial support for O3 profiles.
	py/cso_s5p.py
	config/Copernicus/cso.rc
	config/Copernicus/cso-s5p-o3.rc
	doc/source/documentation.rst
	doc/source/index.rst
	doc/source/s5p-o3.rst
	doc/samples/S5P_RPRO_L2__O3__PR_20180601T002121_20180601T020251_03272_03_020400_20230224T110239.txt
	doc/source/figs/O3-PR/Copernicus_S5p_O3-PR.png

v2.9.1
~~~~~~

Increase waiting time after failed download.
  py/cso_dataspace.py 

Show extra info on pixel selection.
  py/cso_s5p.py 

Support colormaps from cmcrameri package.
  py/cso_plot.py

Start jobtree with 'copy' job to install scripts and settings in work directory.
  bin/cso
  config/Copernicus/cso.rc
  doc/source/tutorial.rst

Updated ItemSearch following deprication warning.
  py/cso_pal.py

Minor fixes for processing S5P HCHO and CHOCHO products.
  py/cso_s5p.py


v2.9.2
~~~~~~

Configurations for TROPOMI O3-profile and O3-column data.
	py/cso_s5p.py
	config/Copernicus/cso.rc
	config/Copernicus/cso-s5p-o3-col.rc
	config/Copernicus/cso-s5p-o3-pr.rc
	doc/source/s5p-o3.rst
	doc/samples/S5P_RPRO_L2__O3_____20200101T055716_20200101T073846_11490_03_020401_20221019T171546.txt

v2.9.3
~~~~~~

Updated code and settings for latest PAL downloads.
  py/cso_pal.py
  config/Copernicus/cso-s5p-so2-cobra.rc
  config/Copernicus/cso-s5p-chocho.rc
  doc/source/s5p-chocho.rst

v2.9.4
~~~~~~

Skip test collections "90" etc in inquire plot.
  py/cso_inquire.py
  
Inquire Dataspace per month after change in allowed maximum number of records.
Sort listing files by processing and processor version.
  py/cso_dataspace.py

v2.9.5
~~~~~~

Changed size calculation after deprication warning.
  py/cso_superobs.py
  py/cso_colocate.py

v2.9.6
~~~~~~

Changed insert of new columns to dataframe after efficiency warning.
  py/cso_superobs.py

v2.9.7
~~~~~~

Increased maximum number of records and introduced sort order to avoid that
search request return a different result when repeated.
  py/cso_dataspace.py