Updated User Guide.

#   https://www.sphinx-doc.org/en/master/usage/extensions/intersphinx.html

# add links to existing online documentation if present:

docs_url = f"https://docs.python.org/{sys.version_info.major}"

intersphinx_mapping = {"python": (docs_url, None)}

intersphinx_mapping = {

    "python": (f"https://docs.python.org/{sys.version_info.major}", None),

    "xarray": ("https://docs.xarray.dev/en/stable/", None),

    "pandas": ("https://pandas.pydata.org/docs/", None),

}

# ~~ figures

# enable numbering:

numfig = True

# -- Options for HTML output -------------------------------------------------

for latest processor version:

.. figure:: figs/CrIS/cris2-nh3_inquire.png

   :scale: 50 %

   :width: 90 %

   :align: center

   :alt: Example of availability of CrIS-2 NH3 data.

   Example of availability of CrIS-2 NH3 data.

The jobtree configuration to inquire the portals and create the overview figure looks like::

    ! single step:

The following figure shows an example of the patches available for a particular *archive* file.

.. figure:: figs/CrIS/cris2-nh3_patches.png

   :scale: 70 %

   :width: 90 %

   :align: center

   :alt: Example of geographic patches collected in CrIS NH3 archive file data.

   Example of geographic patches collected in CrIS NH3 archive file data.

.. Example of filesizes:                   year?

   - collection file:     1.2 G            0.4 TB    cris_noaa20_nh3_global_v1_6_4_2023_03_01.tar.gz  

* `CrIS__Combined_NH3_n020_0_n010_0_n010_0_n005_0_20230301.txt <../../samples/CrIS__Combined_NH3_n020_0_n010_0_n010_0_n005_0_20230301.txt>`_

The retrieval product is a *profile* :math:`y_r` that is available in::

Footprints

==========

The footprint of a CrIS observation is eleptic. At nadir direction, the diameter is about 14 km.

The left panel of :numref:`fig:cris-kernel` shows an example of the footprints.

Aaveraging kernels

==================

The retrieval product is a *profile* :math:`x_r` that is available in::

    double xretv(Observations, Layers) ;

        xretv:long_name = "Retrieved Species Concentration " ;

To compare this with a model profile, convolve with *averaging kernel* in log-space:

.. math::

    \ln\ y_s ~=~ \ln\ x_a ~+~ A_{log}\ (\ln x_s ~-~ \ln x_a)

    \ln x_{rs} ~=~ \ln x_a ~+~ A_{\log}\ (\ln x_s ~-~ \ln x_a)

The *averaging kernel* is available from::

with also a detailed description in the ``long_name`` attribute. 

This description is needed for correct interpretation of the duplicate ``Layers`` dimension [#f1]_.

Our intepretation of the ``avk_kernel`` dimensions is that they represent::

The current intepretation of the ``avk_kernel`` dimensions is that they represent::

    double avg_kernel(Observations, Layers, Retrievals) ;

where ``Layers`` and ``Retrievals`` have the same length. With this, the ``long_name`` description

where ``Layers`` and ``Retrievals`` have in this case the same length. With this, the ``long_name`` description

is reformulated to::

    The averaging kernel is the sensitivity of the estimated state to variations in the atmospheric state. 

         avg_kernel[iobs, iLayer, 0:Retrievals ]

    For atmospheric species this is the sensitivity of retrieved ln(vmr) to the true ln(vmr).

:numref:`fig:cris-kernel` shows an example of the averaging kernel as matrix (middle) and for each of the individual

retrieval layers. In this example, the sensitivity of a retrieved layer for ammonia in the atmosphere is rather similar.

The retrieved profile is most senstive to ammonia in layers 4-5 (~800 hPa)

.. _fig:cris-kernel:

.. figure:: figs/CrIS/CrIS_kernel_example.png

   :width: 90 %

   :align: center

   :alt: Example of CrIS NH3 kernel profiles.

   Example of CrIS NH3 kernel matrix (middle) for pixel marked in the left panel.

   The individual profiles for each retrieved layer is shown in the right panel,

   where the dot marks the retrieval layer. The lowest of the 15 layers is not used for this pixel.

For convience, it is also possible to create a *linearized* version of the kernel

(see :py:meth:`linearize_avg_kernel <cso.cso_tools.linearize_avg_kernel>` method).

This could be applied to the mixing-ratio profiles without log-transform:

.. math::

     x_{rs} ~=~  x_a ~+~ A_{\mbox{lin}}\ ( x_s ~-~  x_a)

The degree-of-freedom in the retreived profile is much lower than the number of levels (15),

which is also illustred by rather similar shape of the kernel profiles in :numref:`fig:cris-kernel`.

It is therefore convenient to express the result as total columns.

The square kernel (linear) could therefore be integrated over the retrieval layers, as well as the *apriori* profile,

to form:

.. math::

    y_s ~=~ y_a ~+~ A\ ( x_s ~-~  x_a)

Convert (and download)

======================

The :py:class:`.CSO_CrIS_Convert` class is available extract the required data from downloaded files.

The :py:class:`.CSO_CrIS_Convert` class is available extract and convert data from the original files.

If a data file is not present yet, it will be downloaded and archived.

.. xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

After reading and applying some basic changes (rename duplicate dimensions, fill NaN-values),

the :py:meth:`CrIS_File.SelectPixels <cso.cso_cris.CrIS_File.SelectPixels>` is called to apply  filter selection.

The filters are define dby a list of keywords::

    ! selection names:

    cso.convert.filters                        :  lons lats daytime landfraction pressure valid cloud_flag quality

For each of the keywords, define the selection criterium and the data variable on which it should be applied.

For example for the quality flag, define that only highest quality pixels should be selected::

    .. Label between '.. _' and ':' ; use :ref:`text <label>` for reference

    .. _cris-nh3-download:

    ! Following recomendations, use pixels with minimum quality flag of 5:

    cso.convert.filter.quality.var             :  Quality_Flag

    cso.convert.filter.quality.type            :  min

    cso.convert.filter.quality.min             :  5

    cso.convert.filter.quality.units           :  1

    Download CrIS NH3 data

    ======================

Typical selection criteria are:

* Selection of a longitude/latitude box.

* Use day-time values only.

* Minimum land fraction to exclude pixels over water where thermal background makes it difficult to retrieve ammonia.

* Minimum surface pressure (or actually, pressure in lowest layer with data defined), to exclude pixels over mountain area.

* Strictly cloud-free pixels.

* Highest quality pixels only.

The variables to be created are defined with a list::

    ! which fields to be put out ?

    cso.convert.output.vars    :   longitude latitude pixel_bounds time \

                                    pressure hpressure altitude isfc pressure_sfc landfraction \

                                    tot_col tot_col_meas_errvar tot_col_total_errvar xr yr \

                                    xa ya \

                                    log_kernel lin_kernel A \

                                    quality_flag cloud_flag

For each of these variables define how it should be formed from the input data.

* Most variables could be simply copied from a source variable (only the selected pixels);

  for example, the ``longitude`` varialbe is copied from the ``Longitude`` variable::

      ! description:

      cso.convert.output.var.longitude.dims                  :  pixel

      cso.convert.output.var.longitude.from                  :  Longitude

      cso.convert.output.var.longitude.units                 :  degrees_east

      cso.convert.output.var.longitude.attrs                 :  { 'standard_name' : 'longitude' }

  The ``units`` are the target units, eventually a conversion is applied.

  Extra attributes could be added to describe the output variable.

* Some variables require extra processing, which is defined using a ``special`` keyword.

  For example, for the linearized kernel a special conversion is applied based on the description::

    ! linearization from units "ln(ppmv)/ln(ppmv)" to "ppmv/ppmv":

    !   kernel := averaging_kernel * amf/amft

    cso.convert.output.var.lin_kernel.dims                :   pixel layer retr_profile

    cso.convert.output.var.lin_kernel.special             :   linearize_kernel

    cso.convert.output.var.lin_kernel.avk                 :   avg_kernel

    cso.convert.output.var.lin_kernel.apri                :   xa

    cso.convert.output.var.lin_kernel.units               :   1

    cso.convert.output.var.lin_kernel.attrs               :   { 'long_name' : 'linearized averaging kernel matrix in ppmv/ppmv' }

In the example, in total 3 retrieval products are defined:

    To be able to download data, an account has to be created on `EarthData Login <https://urs.earthdata.nasa.gov/>`_.

    Store the login name and password in the ``~/.netrc`` file in your home directory

    (or in a file defined by the ``NETRC`` environment variable) as::

* The retrieved profile ``xr`` in ppmv with shape ``(retr_profile=15)``;

* a retrieved total column ``tot_col`` as defined in the product;

* a column ``xrc`` with shape ``(retr=1)`` computed from ``xr``; 

  this is equal to `tot_col``, but used to check whether the calculation 

  of a column (incl. unit conversions) is done correctly.

        machine  urs.earthdata.nasa.gov   login your.name@inst.nx  password ********

Also 3 averaging kernels are defined:  

    The :py:class:`.CSO_EarthAccess_Download` could then be used to download the data.

    The files to be downloaded are selected from the inquiry table.

    With the default configuration the downloaded files are stored in sub-directories per year and day-in-the-year,

    similar as used in the filenames::

* the original ``log_kernel`` with shape ``(layer,retr_profile)`` for convolution of log-mixing-ratios;

* a linearized version ``lin_kernel`` with shape ``(layer,retr_profile)``;

* the linearized kernel and column integrated kernel ``A`` with shape ``(layer,retr)`` where ``retr=1``.

        CrIS-1/AERDB_L2/v2.0.0/2018/152/AERDB_L2_CrIS_SNPP.A2018152.0124.002.2023076173157.nc

                                         AERDB_L2_CrIS_SNPP.A2018152.0130.002.2023076173155.nc

                                         AERDB_L2_CrIS_SNPP.A2018152.0306.002.2023076173155.nc

Similar as the original data, the *converted* files collect pixels for a single day::

    ECCC/CrIS2/v1_6_4/xEMEP/2024/01/CrIS2_NH3_20240101.nc

                                    CrIS2_NH3_20240102.nc

                                    :

    The job task for the download task is configured like::

        !~ download data:

        ! single step:

        cso.cris1-nh3.download.class                     :  utopya.UtopyaJobStep

        ! download task:

        cso.cris1-nh3.download.task.class                :  cso.CSO_EarthAccess_Download

        cso.cris1-nh3.download.task.args                 :  '${my.work}/rc/cso-cris.rc', \

                                                                  rcbase='cso.download', \

                                                                  env={ 'MY_PRODUCT' : 'cris1-nh3-db' }

Catalogue of converted data

===========================

The :py:class:`CSO_Catalogue <cso.cso_catalogue.CSO_Catalogue>` could be used to create figures

and index pages of the converted data.

:numref:`fig:cris-catalogue` shows an example of catalgoue showing the retrieved and *a prior* columns,

as well as the quality and cloud flags for a selected region.

.. _fig:cris-catalogue:

.. figure:: figs/CrIS/CrIS_catalogue.png

   :width: 90 %

   :align: center

   :alt: Example of catalogue with CrIS NH3 data.

   Example of catalogue with CrIS NH3 data.

    For operations on the files it is useful to have a *listing file*, a csv file with per record the path to a file

    and the time range of the the observations in the file.

    This could be created using the :py:class:`.CSO_EarthAccess_Download_Listing` class, and is for example the file::

        CrIS-1/AERDB_L2/v2.0.0/listing.csv

Model simulations

=================

    with content::

The :ref:`obsoper` code could be used to simulate the CrIS retrievals directly from a model.

        filename                                                       ;start_time         ;end_time           ;orbit

        2018/152/AERDB_L2_CrIS_SNPP.A2018152.0124.002.2023076173157.nc;2018-06-01 00:00:00;2018-06-02 00:00:00;34159

        2018/152/AERDB_L2_CrIS_SNPP.A2018152.0130.002.2023076173155.nc;2018-06-01 00:00:00;2018-06-02 00:00:00;34159

        2018/152/AERDB_L2_CrIS_SNPP.A2018152.0306.002.2023076173155.nc;2018-06-01 00:00:00;2018-06-02 00:00:00;34160

Example configuration settings are provided in::

* :ref:`oper/tutorial_oper_CrIS.rc <../../../oper/tutorial_oper_CrIS.rc>`

The operator uses the *converted* files as input. 

For each of the input files, the output consists of:

* a *data* file with a selection of the input data, for example the geo-location information, the retrievd values, etc;

* one or more *state* files with simultions; usually there is only one file, but in case of assimilattion or ensemble runs

  more might have been created.

A *listing file* is created with a table of the output files thate were created::

    output/cso_listing_cris-nh3.csv

           2024/01/CrIS2_NH3_20240101_data.nc

                   CrIS2_NH3_20240101_state.nc

                   CrIS2_NH3_20240102_data.nc

                   CrIS2_NH3_20240102_state.nc

                   :

    The job task for the listing task is configured like::

        ! single step:

        cso.cris1-nh3.download-listing.class             :  utopya.UtopyaJobStep

        ! download task:

        cso.cris1-nh3.download-listing.task.class        :  cso.CSO_EarthAccess_Download_Listing

        cso.cris1-nh3.download-listing.task.args         :  '${my.work}/rc/cso-cris.rc', \

                                                                  rcbase='cso.download-listing', \

                                                                  env={ 'MY_PRODUCT' : 'cris1-nh3-db' }

Catalogue of simulated data

===========================

The output from the observation operator could be shown using the :py:class:`CSO_SimCatalogue <cso.cso_catalogue.CSO_SimCatalogue>` 

:numref:`fig:cris-sim-catalogue` shows an example of catalgoue showing the retrieved, simulated, and *a prior* columns.

.. _fig:cris-sim-catalogue:

.. figure:: figs/CrIS/CrIS_sim-catalogue.png

   :width: 90 %

   :align: center

   :alt: Example of catalogue with retrieved and simulated CrIS NH3 data.

   Example of catalogue with retrieved and simulated CrIS NH3 data.

Spatial and temporal averages

=============================

The :ref:`gridding` tools could be used to average the output of the observation operator at a regular grid.

In addition, temporal means could be calculdated.

All data is saved to netcdf files, and classes are available to create catalogues.

:numref:`fig:cris-sims` shows an example of gridded monthly means of CrIS NH3 retrievals and simulations.

Top left shows the retrieval; top right the native model columns (sampled at the pixel locations).

The bottom row shows the simulations using either the log- or linear-kernel.

The example shows that:

* The simulated retrievals could be substantially lower than the full model columns;

  this could be due to the limitted sensitivity of the instrument to ammonia at the surface, 

  where mixing ratio's are actually highest.

* Simulations using the (approximate) linear kernel could be significantly different from simulations

  using the (original) log-based kernel.

.. _fig:cris-sims:

.. figure:: figs/CrIS/CrIS_simulations.png

   :width: 90 %

   :align: center

   :alt: Example of monthly means of CrIS NH3 retrievals and simulations, as well as the model total columns.

   Example of monthly means of CrIS NH3 retrievals and simulated retrievals, as well as the simulated total columns.

.. rubric:: Footnotes

.. [#f1] The use of duplicate dimensions is also probelematic when reading the data file

   into an :py:class:`xarray.Dataset` object. The :py:class:`.CSO_CrIS_Convert` class therefore

   into a :py:class:`xarray.Dataset` object. The :py:class:`.CSO_CrIS_Convert` class therefore

   renames the second dimension from ``Layers`` to ``Layers2``.