.. Label between '.. _' and ':' ; use :ref:`text <label>` for reference
.. _history:

******************
History and future
******************


Versions
========

A summary of the versions and changes.

* | *v1.0*
  | Initial release.
  | *(2020-06)*

* | *v1.1*
  | Support averaging of grid cells over footprints.
  | Added output options for horizontal mapping info, model concentrations at native layers, and simulated profile.
  | *(2020-09)*

* | *v1.2*
  | Updates for retrieval products including apriori profiles.
  | *(2020-10)*

* | *v1.3*
  | Added operator for mapping vertical profiles.
  | *(2020-10)*

* | *v1.4*
  | Create listing files not as part of convert job but in a seperate job.
  | Use weights expressed in area in horizontal mapping.
  | Added tools to create gridded averages.
  | *(2020-11)*

* | *v1.5*
  | Added option to search for orbit files collected in tar files.
  | Fixes for mapping information.
  | *(2020-11)*

* | *v1.6*
  | Put out user defined variables over pixels from a simulation model.
  | *(2021-01)*

* | *v1.7*
  | Introduced formula definitions for operations on pixel data.
  | Support airmass factor corrections as used for NO2 products.
  | *(2021-05)*

* | *v1.8*
  | Replaced default download method for OpenSearch API.
  | Added HCHO basic rc file.
  | Extended support for DLR TROPOMI products.
  | Add adjoint operations and adjoint test.
  | *(2021-06)*

* | *v2.0*
  | Added ``--single`` flag to run script to allow running just one job step.
  | Added class to create listing of files downloaded from ESA/SciHub.
  | Regridding operator using CSO Python routines rather than using mapping information 
    saved from Fortran operator.
  | For mapping between footprints and grid cells, triangles are now splitted into 2 new 
    triangles based on the longest edge.
  | *(2021-10)*

* | *v2.1*
  | Read data on root and scatter to domains, instead of reading by all domains.
  | Support S5p/NO2 level3 observations from KNMI.
  | Store settings in subdirectory 'config'.
  | Postprocessing tools with agregation over regions.
  | Added example for adjoint test to operator code.
  | Added routines to swap operator data to a different domain decomposition.
  | Support S5p/Glyoxal retrievals from GlyRetro.
  | *(2022-08)*

* | *v2.2*
  | Option to grid converted orbit files using listing file.
  | *(2023-08)*

* | *v2.3*
  | Added collection level to inquire overview plots.
  | Convert selected orbit files based on selection query.
  | *(2023-08)*

* | *v2.4*
  | Reformatted using 'black'.
  | *(2023-08)*

* | *v2.5*
  | Support new Copernicus *DataSpace* portal to download Sentinel data.
  | *(2023-11)*

* | *v2.6*
  | Support SO\ :sub:`2` COBRA product from PAL website.
  | *(2024-01)*

* | *v2.7*
  | Fixed averaging over time values when creating super-observations.
  | Use cartopy for map plots.
  | Support orbits that overlap with dateline.
  | *(2024-01)*
  
* | *v2.8*
  | Re-introduced 'Download' class to download a series of S5P files without converting.
  | Updated re-try settings for downloading.
  | Extended support for creation of gridded averages.
  | Extended support for region plots.
  
* | *v2.9*
  | Initial support for S5P O3-profile product.
  
* | *v2.10*
  | Support inquiry of ColHub mirror archive.
  
* | *v2.11*
  | Support inquiry and download from NASA EarthData archive.


To be included
==============

Features available in branches, that should be included in master version.

* Create map plots as a list of pixels, in case track information is not available. *(E.D.)*
  

To be done
==========

A wishlist of developments.

* Regridding following `Kuhlman et al. (2014) <https://amt.copernicus.org/articles/7/451/2014/amt-7-451-2014.pdf>`_.

* Include time range in all filenames?

  * Proposal: use names similar to the original S5p files,  up to the collection number::
  
        S5P_NRTI_L2__NO2____20140101T000000_20140102T000000_00099_01.nc
        
  * Also output by observation operator could use the same basename.
  
* Add offline tool to re-create listing files.

  * Scan directories based on filename patterns.
  * Eventually specify order, thus ``RPRO`` if available, otherwise ``OFFL`` or ``NRTI``.
  
* If the original track is saved, add flags to log which selection criterium was used
  to exclude a pixel.
  
* Store covariances as sparse matrix (diagonal?) if possible.

* Storage of additional observation representation errors by assimilation system.


How to create a version tag
===========================

Example for tag "v2.11".

1. Update file formatting following conventions.

   a. For Python files::

        make black
     
   b. Eventually also cleanup Fortran sources.
   
2. Ensure that all changes in source files are committed.

3. Update change logs:

   a. Update the `CHANGELOG <../../../CHANGELOG>`_.

   b. Update the source file `doc/source/history.rst <../../source/history.rst>`_ of this chapter.

   c. Change the release tag that will be shown in the documentation in the file
      `doc/source/conf.py <../../source/conf.py>`_::

        # The full version, including alpha/beta/rc tags
        release = "v2.11"
    
   d. Commit and push these changes::

       git commit -m 'Defined v2.11' .
       git push

4. Tag and push the code to the git repository::

      git tag -a 'v2.11' -m 'Tagged v2.11'
      git push origin v2.11


  
