TNO Intern

Skip to content
Expand all Show whitespace changes
Inline Side-by-side
doc/source/history.rst
View file @ 69c32484
...@@ -87,6 +87,10 @@ A summary of the versions and changes. ...@@ -87,6 +87,10 @@ A summary of the versions and changes.
| Reformatted using 'black'. | Reformatted using 'black'.
| *(2023-08)* | *(2023-08)*
* | *v2.5*
| Support new Copernicus *DataSpace* portal to download Sentinel data.
| *(2023-11)*
To be included To be included
============== ==============
......
doc/source/pymod-cso_scihub.rst → doc/source/pymod-cso_dataspace.rst
View file @ 69c32484
.. Documentation for module. .. Documentation for module.
.. Import documentation from ".py" file: .. Import documentation from ".py" file:
.. automodule:: cso_scihub .. automodule:: cso_dataspace
doc/source/pymods.rst
View file @ 69c32484
...@@ -27,8 +27,8 @@ Classes used for specific tasks are implemented in the ``cso_*`` modules. ...@@ -27,8 +27,8 @@ Classes used for specific tasks are implemented in the ``cso_*`` modules.
:maxdepth: 1 :maxdepth: 1
pymod-cso_inquire pymod-cso_inquire
pymod-cso_dataspace
pymod-cso_pal pymod-cso_pal
pymod-cso_scihub
pymod-cso_s5p pymod-cso_s5p
pymod-cso_file pymod-cso_file
pymod-cso_gridded pymod-cso_gridded
......
doc/source/s5p-chocho.rst
View file @ 69c32484
...@@ -104,6 +104,36 @@ Note that the official S5p filename formatting rules require exactly 10 characte ...@@ -104,6 +104,36 @@ Note that the official S5p filename formatting rules require exactly 10 characte
in the current product a 12-character key ``L2__CHOCHO__`` is used. in the current product a 12-character key ``L2__CHOCHO__`` is used.
CSO processing
==============
*(See* :ref:`tutorial` *chapter for introduction to CSO scripts and configuration)*
An example configuration of the CSO processing of the S5p/CHOCHO data is available via
the following settings:
* `config/Copernicus/cso.rc <../../../config/Copernicus/cso.rc>`_
Top-level settings that configure the job-tree with various sub-tasks.
This is a generic file that could be used for multiple S5 products,
edit it to select the CHOCHO processing.
* `config/Copernicus/cso-user-settings.rc <../../../config/Copernicus/cso-user-settings.rc>`_
User-specific settings such as the work directory.
* `config/Copernicus/cso-s5p-chocho.rc <../../../config/Copernicus/cso-s5p-chocho.rc>`_
Specific settings for CHOCHO product.
Start the job-tree using::
./bin/cso config/Copernicus/cso.rc
Selected sub-steps in the processing are described below.
Inquire archives Inquire archives
================ ================
......
doc/source/s5p-co.rst
View file @ 69c32484
...@@ -60,14 +60,46 @@ Notes: ...@@ -60,14 +60,46 @@ Notes:
The recommended minimum is 0.5, this excludes cloudy scenes and other problematic retrievals. The recommended minimum is 0.5, this excludes cloudy scenes and other problematic retrievals.
.. Label between '.. _' and ':' ; use :ref:`text <label>` for reference .. Label between '.. _' and ':' ; use :ref:`text <label>` for reference
CSO processing
==============
*(See* :ref:`tutorial` *chapter for introduction to CSO scripts and configuration)*
An example configuration of the CSO processing of the S5p/CO data is available via
the following settings:
* `config/Copernicus/cso.rc <../../../config/Copernicus/cso.rc>`_
Top-level settings that configure the job-tree with various sub-tasks.
This is a generic file that could be used for multiple S5 products,
edit it to select the CO processing.
* `config/Copernicus/cso-user-settings.rc <../../../config/Copernicus/cso-user-settings.rc>`_
User-specific settings such as the work directory.
* `config/Copernicus/cso-s5p-co.rc <../../../config/Copernicus/cso-s5p-co.rc>`_
Specific settings for CO product.
Start the job-tree using::
./bin/cso config/Copernicus/cso.rc
Selected sub-steps in the processing are described below.
.. _s5p-co-inquire: .. _s5p-co-inquire:
Inquire Sentinel-5p/CO archive Inquire Sentinel-5p/CO archive
================================ ================================
S5p/CO observations are available from the S5p/CO observations are available from the
`Copernicus Open Access Hub <https://scihub.copernicus.eu/>`_; `Copernicus DataSpace <https://dataspace.copernicus.eu/>`_;
see the :ref:`cso-scihub` module for a detailed description. see the :ref:`cso-dataspace` module for a detailed description.
Data is available for different processing streams, each identified by a 4-character key: Data is available for different processing streams, each identified by a 4-character key:
...@@ -80,8 +112,8 @@ but with different processor versions. ...@@ -80,8 +112,8 @@ but with different processor versions.
It is therefore necessary to first inquire both archives to see which data is available where, It is therefore necessary to first inquire both archives to see which data is available where,
and what the version numbers are. and what the version numbers are.
The :py:class:`cso_scihub.CSO_SciHub_Inquire` class is available to inquire the The :py:class:`CSO_DataSpace_Inquire <cso_dataspace.CSO_DataSpace_Inquire>` class is available to inquire the
*Copernicus Open Access Hub*. The settings used by this class allow selection *Copernicus DataSpace*. The settings used by this class allow selection
on for example time range and intersection area. on for example time range and intersection area.
The result is a csv file which with columns for keywords such orbit number and processor version, The result is a csv file which with columns for keywords such orbit number and processor version,
as well as the filename of the data and the url that should be used to actually download the data:: as well as the filename of the data and the url that should be used to actually download the data::
...@@ -105,17 +137,19 @@ To visualize what is available from the various portals, the ...@@ -105,17 +137,19 @@ To visualize what is available from the various portals, the
The jobtree configuration to inquire the portals and create the overview figure could look like:: The jobtree configuration to inquire the portals and create the overview figure could look like::
! single step: ! single step:
cso.s5p.co.inquire-scihub.class : utopya.UtopyaJobStep cso.s5p.co.inquire.class : utopya.UtopyaJobStep
! two tasks: ! two tasks:
cso.s5p.co.inquire-scihub.tasks : table cso.s5p.co.inquire.tasks : table-dataspace plot
!~ inquire available files:
cso.s5p.co.inquire-scihub.table.class : cso.CSO_SciHub_Inquire !~ inquire files available on DataSpace:
cso.s5p.co.inquire-scihub.table.args : '${PWD}/config/ESA-S5p/cso-s5p-co.rc', \ cso.s5p.co.inquire.table-dataspace.class : cso.CSO_DataSpace_Inquire
rcbase='cso.s5p.co.inquire-s5phub-table' cso.s5p.co.inquire.table-dataspace.args : '${PWD}/config/Copernicus/cso-s5p-co.rc', \
rcbase='cso.s5p.co.inquire-table-dataspace'
!~ create plot of available versions: !~ create plot of available versions:
cso.s5p.co.inquire-scihub.plot.class : cso.CSO_SciHub_InquirePlot cso.s5p.co.inquire.plot.class : cso.CSO_Inquire_Plot
cso.s5p.co.inquire-scihub.plot.args : '${PWD}/config/ESA-S5p/cso-s5p-co.rc', \ cso.s5p.co.inquire.plot.args : '${PWD}/config/Copernicus/cso-s5p-co.rc', \
rcbase='cso.s5p.co.inquire-s5phub-plot' rcbase='cso.s5p.co.inquire-plot'
......
doc/source/s5p-hcho.rst
View file @ 69c32484
...@@ -65,6 +65,36 @@ References ...@@ -65,6 +65,36 @@ References
| Atmos. Meas. Tech., 13, 3751-3767, `<https://doi.org/10.5194/amt-13-3751-2020>`_, 2020. | Atmos. Meas. Tech., 13, 3751-3767, `<https://doi.org/10.5194/amt-13-3751-2020>`_, 2020.
CSO processing
==============
*(See* :ref:`tutorial` *chapter for introduction to CSO scripts and configuration)*
An example configuration of the CSO processing of the S5p/HCHO data is available via
the following settings:
* `config/Copernicus/cso.rc <../../../config/Copernicus/cso.rc>`_
Top-level settings that configure the job-tree with various sub-tasks.
This is a generic file that could be used for multiple S5 products,
edit it to select the HCHO processing.
* `config/Copernicus/cso-user-settings.rc <../../../config/Copernicus/cso-user-settings.rc>`_
User-specific settings such as the work directory.
* `config/Copernicus/cso-s5p-hcho.rc <../../../config/Copernicus/cso-s5p-hcho.rc>`_
Specific settings for HCHO product.
Start the job-tree using::
./bin/cso config/Copernicus/cso.rc
Selected sub-steps in the processing are described below.
.. Label between '.. _' and ':' ; use :ref:`text <label>` for reference .. Label between '.. _' and ':' ; use :ref:`text <label>` for reference
.. _s5p-hcho-inquire: .. _s5p-hcho-inquire:
...@@ -72,8 +102,8 @@ Inquire Sentinel-5p/HCHO archive ...@@ -72,8 +102,8 @@ Inquire Sentinel-5p/HCHO archive
================================ ================================
S5p/HCHO observations are available from the S5p/HCHO observations are available from the
`Copernicus Open Access Hub <https://scihub.copernicus.eu/>`_; `Copernicus DataSpace <https://dataspace.copernicus.eu/>`_;
see the :ref:`cso-scihub` module for a detailed description. see the :ref:`cso-dataspace` module for a detailed description.
Data is available for different processing streams, each identified by a 4-character key: Data is available for different processing streams, each identified by a 4-character key:
...@@ -86,8 +116,8 @@ but with different processor versions. ...@@ -86,8 +116,8 @@ but with different processor versions.
It is therefore necessary to first inquire both archives to see which data is available where, It is therefore necessary to first inquire both archives to see which data is available where,
and what the version numbers are. and what the version numbers are.
The :py:class:`cso_scihub.CSO_SciHub_Inquire` class is available to inquire the The :py:class:`CSO_DataSpace_Inquire <cso_dataspace.CSO_DataSpace_Inquire>` class is available to inquire the
*Copernicus Open Access Hub*. The settings used by this class allow selection *Copernicus DataSpace*. The settings used by this class allow selection
on for example time range and intersection area. on for example time range and intersection area.
The result is a csv file which with columns for keywords such orbit number and processor version, The result is a csv file which with columns for keywords such orbit number and processor version,
as well as the filename of the data and the url that should be used to actually download the data:: as well as the filename of the data and the url that should be used to actually download the data::
...@@ -111,17 +141,19 @@ To visualize what is available from the various portals, the ...@@ -111,17 +141,19 @@ To visualize what is available from the various portals, the
The jobtree configuration to inquire the portals and create the overview figure could look like:: The jobtree configuration to inquire the portals and create the overview figure could look like::
! single step: ! single step:
cso.s5p.hcho.inquire-scihub.class : utopya.UtopyaJobStep cso.s5p.hcho.inquire.class : utopya.UtopyaJobStep
! two tasks: ! two tasks:
cso.s5p.hcho.inquire-scihub.tasks : table cso.s5p.hcho.inquire.tasks : table-dataspace plot
!~ inquire available files:
cso.s5p.hcho.inquire-scihub.table.class : cso.CSO_SciHub_Inquire !~ inquire files available on DataSpace:
cso.s5p.hcho.inquire-scihub.table.args : '${PWD}/config/ESA-S5p/cso-s5p-hcho.rc', \ cso.s5p.hcho.inquire.table-dataspace.class : cso.CSO_DataSpace_Inquire
rcbase='cso.s5p.hcho.inquire-s5phub-table' cso.s5p.hcho.inquire.table-dataspace.args : '${PWD}/config/Copernicus/cso-s5p-hcho.rc', \
rcbase='cso.s5p.hcho.inquire-table-dataspace'
!~ create plot of available versions: !~ create plot of available versions:
cso.s5p.hcho.inquire-scihub.plot.class : cso.CSO_SciHub_InquirePlot cso.s5p.hcho.inquire.plot.class : cso.CSO_Inquire_Plot
cso.s5p.hcho.inquire-scihub.plot.args : '${PWD}/config/ESA-S5p/cso-s5p-hcho.rc', \ cso.s5p.hcho.inquire.plot.args : '${PWD}/config/Copernicus/cso-s5p-hcho.rc', \
rcbase='cso.s5p.hcho.inquire-s5phub-plot' rcbase='cso.s5p.hcho.inquire-plot'
......
doc/source/s5p-no2.rst
View file @ 69c32484
...@@ -175,6 +175,36 @@ and simulations. ...@@ -175,6 +175,36 @@ and simulations.
CSO processing
==============
*(See* :ref:`tutorial` *chapter for introduction to CSO scripts and configuration)*
An example configuration of the CSO processing of the S5p/NO2 data is available via
the following settings:
* `config/Copernicus/cso.rc <../../../config/Copernicus/cso.rc>`_
Top-level settings that configure the job-tree with various sub-tasks.
This is a generic file that could be used for multiple S5 products,
edit it to select the NO2 processing.
* `config/Copernicus/cso-user-settings.rc <../../../config/Copernicus/cso-user-settings.rc>`_
User-specific settings such as the work directory.
* `config/Copernicus/cso-s5p-no2.rc <../../../config/Copernicus/cso-s5p-no2.rc>`_
Specific settings for NO2 product.
Start the job-tree using::
./bin/cso config/Copernicus/cso.rc
Selected sub-steps in the processing are described below.
.. Label between '.. _' and ':' ; use :ref:`text <label>` for reference .. Label between '.. _' and ':' ; use :ref:`text <label>` for reference
.. _s5p-no2-inquire: .. _s5p-no2-inquire:
...@@ -183,8 +213,8 @@ Inquire Sentinel-5p/NO2 archives ...@@ -183,8 +213,8 @@ Inquire Sentinel-5p/NO2 archives
S5p/NO2 observations from KNMI have been available from at least these sources: S5p/NO2 observations from KNMI have been available from at least these sources:
* `Copernicus Open Access Hub <https://scihub.copernicus.eu/>`_; * `Copernicus DataSpace <https://dataspace.copernicus.eu/>`_;
see the :ref:`cso-scihub` module for a detailed description. see the :ref:`cso-dataspace` module for a detailed description.
*This is the operational version.* *This is the operational version.*
...@@ -206,8 +236,8 @@ The portals provide data files created with the same retrieval algorithm, but mo ...@@ -206,8 +236,8 @@ The portals provide data files created with the same retrieval algorithm, but mo
It is therefore necessary to first inquire both archives to see which data is available where, It is therefore necessary to first inquire both archives to see which data is available where,
and what the version numbers are. and what the version numbers are.
The :py:class:`CSO_SciHub_Inquire <cso_scihub.CSO_SciHub_Inquire>` class is available to inquire the The :py:class:`CSO_DataSpace_Inquire <cso_dataspace.CSO_DataSpace_Inquire>` class is available to inquire the
*Copernicus Open Access Hub*. The settings used by this class allow selection *Copernicus DataSpace*. The settings used by this class allow selection
on for example time range and intersection area. on for example time range and intersection area.
The result is a csv file which with columns for keywords such as orbit number and processor version, The result is a csv file which with columns for keywords such as orbit number and processor version,
as well as the filename of the data and the url that should be used to actually download the data:: as well as the filename of the data and the url that should be used to actually download the data::
...@@ -220,7 +250,7 @@ as well as the filename of the data and the url that should be used to actually ...@@ -220,7 +250,7 @@ as well as the filename of the data and the url that should be used to actually
See the section on *File name convention* in the *Product User Manual* for the meaning of all See the section on *File name convention* in the *Product User Manual* for the meaning of all
parts of the filename. parts of the filename.
A similar class :py:class:`CSO_PAL_Inquire <cso_scihub.CSO_PAL_Inquire>` class is available to list the content A similar class :py:class:`CSO_PAL_Inquire <cso_pal.CSO_PAL_Inquire>` class is available to list the content
of the *Product Algorithm Laboratory* portal. Also this will produce a table file. of the *Product Algorithm Laboratory* portal. Also this will produce a table file.
To visualize what is available from the various portals, the To visualize what is available from the various portals, the
...@@ -234,24 +264,23 @@ To visualize what is available from the various portals, the ...@@ -234,24 +264,23 @@ To visualize what is available from the various portals, the
The jobtree configuration to inquire the portals and create the overview figure could look like:: The jobtree configuration to inquire the portals and create the overview figure could look like::
! single step: ! single step:
cso.s5p.no2.inquire-scihub.class : utopya.UtopyaJobStep cso.s5p.no2.inquire.class : utopya.UtopyaJobStep
! inquire from portals, plot overview: ! two tasks:
cso.s5p.no2.inquire.tasks : scihub pal plot cso.s5p.no2.inquire.tasks : table-dataspace plot
!~ create table with available files on portal:
cso.s5p.no2.inquire.scihub.class : cso.CSO_SciHub_Inquire !~ inquire files available on DataSpace:
cso.s5p.no2.inquire.scihub.args : '${PWD}/config/EMEP/cso-s5p-no2.rc', \ cso.s5p.no2.inquire.table-dataspace.class : cso.CSO_DataSpace_Inquire
rcbase='cso.s5p.no2.inquire-s5phub' cso.s5p.no2.inquire.table-dataspace.args : '${PWD}/config/Copernicus/cso-s5p-no2.rc', \
!~ create table with available files on portal: rcbase='cso.s5p.no2.inquire-table-dataspace'
cso.s5p.no2.inquire.pal.class : cso.CSO_PAL_Inquire
cso.s5p.no2.inquire.pal.args : '${PWD}/config/EMEP/cso-s5p-no2.rc', \
rcbase='cso.s5p.no2.inquire-pal'
!~ create plot of available versions: !~ create plot of available versions:
cso.s5p.no2.inquire.plot.class : cso.CSO_Inquire_Plot cso.s5p.no2.inquire.plot.class : cso.CSO_Inquire_Plot
cso.s5p.no2.inquire.plot.args : '${PWD}/config/EMEP/cso-s5p-no2.rc', \ cso.s5p.no2.inquire.plot.args : '${PWD}/config/Copernicus/cso-s5p-no2.rc', \
rcbase='cso.s5p.no2.inquire-plot' rcbase='cso.s5p.no2.inquire-plot'
.. Label between '.. _' and ':' ; use :ref:`text <label>` for reference .. Label between '.. _' and ':' ; use :ref:`text <label>` for reference
.. _s5p-no2-convert: .. _s5p-no2-convert:
......
doc/source/s5p-so2.rst
View file @ 69c32484
...@@ -77,6 +77,36 @@ Acknowledgements ...@@ -77,6 +77,36 @@ Acknowledgements
We hereby thank D. Griffin and V. Fioletov for their valuable input. We hereby thank D. Griffin and V. Fioletov for their valuable input.
CSO processing
==============
*(See* :ref:`tutorial` *chapter for introduction to CSO scripts and configuration)*
An example configuration of the CSO processing of the S5p/SO2 data is available via
the following settings:
* `config/Copernicus/cso.rc <../../../config/Copernicus/cso.rc>`_
Top-level settings that configure the job-tree with various sub-tasks.
This is a generic file that could be used for multiple S5 products,
edit it to select the SO2 processing.
* `config/Copernicus/cso-user-settings.rc <../../../config/Copernicus/cso-user-settings.rc>`_
User-specific settings such as the work directory.
* `config/Copernicus/cso-s5p-so2.rc <../../../config/Copernicus/cso-s5p-so2.rc>`_
Specific settings for SO2 product.
Start the job-tree using::
./bin/cso config/Copernicus/cso.rc
Selected sub-steps in the processing are described below.
.. Label between '.. _' and ':' ; use :ref:`text <label>` for reference .. Label between '.. _' and ':' ; use :ref:`text <label>` for reference
.. _s5p-so2-inquire: .. _s5p-so2-inquire:
...@@ -84,8 +114,8 @@ Inquire Sentinel-5p/SO2 archive ...@@ -84,8 +114,8 @@ Inquire Sentinel-5p/SO2 archive
=============================== ===============================
S5p/SO2 observations are available from the S5p/SO2 observations are available from the
`Copernicus Open Access Hub <https://scihub.copernicus.eu/>`_; `Copernicus DataSpace <https://dataspace.copernicus.eu/>`_;
see the :ref:`cso-scihub` module for a detailed description. see the :ref:`cso-dataspace` module for a detailed description.
Data is available for different processing streams, each identified by a 4-character key: Data is available for different processing streams, each identified by a 4-character key:
...@@ -93,13 +123,12 @@ Data is available for different processing streams, each identified by a 4-chara ...@@ -93,13 +123,12 @@ Data is available for different processing streams, each identified by a 4-chara
* ``OFFL`` : `Offline`, available within weeks after observations; * ``OFFL`` : `Offline`, available within weeks after observations;
* ``RPRO`` : re-processing of all previously made observations; * ``RPRO`` : re-processing of all previously made observations;
The portals provide data files created with the same retrieval algorithm, The portal provides data files created with different processor versions.
but with different processor versions.
It is therefore necessary to first inquire both archives to see which data is available where, It is therefore necessary to first inquire both archives to see which data is available where,
and what the version numbers are. and what the version numbers are.
The :py:class:`cso_scihub.CSO_SciHub_Inquire` class is available to inquire the The :py:class:`CSO_DataSpace_Inquire <cso_dataspace.CSO_DataSpace_Inquire>` class is available to inquire the
*Copernicus Open Access Hub*. The settings used by this class allow selection *Copernicus DataSpace*. The settings used by this class allow selection
on for example time range and intersection area. on for example time range and intersection area.
The result is a csv file which with columns for keywords such orbit number and processor version, The result is a csv file which with columns for keywords such orbit number and processor version,
as well as the filename of the data and the url that should be used to actually download the data:: as well as the filename of the data and the url that should be used to actually download the data::
...@@ -123,17 +152,19 @@ To visualize what is available from the various portals, the ...@@ -123,17 +152,19 @@ To visualize what is available from the various portals, the
The jobtree configuration to inquire the portals and create the overview figure could look like:: The jobtree configuration to inquire the portals and create the overview figure could look like::
! single step: ! single step:
cso.s5p.so2.inquire-scihub.class : utopya.UtopyaJobStep cso.s5p.so2.inquire.class : utopya.UtopyaJobStep
! two tasks: ! two tasks:
cso.s5p.so2.inquire-scihub.tasks : table cso.s5p.so2.inquire.tasks : table-dataspace plot
!~ inquire available files:
cso.s5p.so2.inquire-scihub.table.class : cso.CSO_SciHub_Inquire !~ inquire files available on DataSpace:
cso.s5p.so2.inquire-scihub.table.args : '${PWD}/config/ESA-S5p/cso-s5p-so2.rc', \ cso.s5p.so2.inquire.table-dataspace.class : cso.CSO_DataSpace_Inquire
rcbase='cso.s5p.so2.inquire-s5phub-table' cso.s5p.so2.inquire.table-dataspace.args : '${PWD}/config/Copernicus/cso-s5p-so2.rc', \
rcbase='cso.s5p.so2.inquire-table-dataspace'
!~ create plot of available versions: !~ create plot of available versions:
cso.s5p.so2.inquire-scihub.plot.class : cso.CSO_SciHub_InquirePlot cso.s5p.so2.inquire.plot.class : cso.CSO_Inquire_Plot
cso.s5p.so2.inquire-scihub.plot.args : '${PWD}/config/ESA-S5p/cso-s5p-so2.rc', \ cso.s5p.so2.inquire.plot.args : '${PWD}/config/Copernicus/cso-s5p-so2.rc', \
rcbase='cso.s5p.so2.inquire-s5phub-plot' rcbase='cso.s5p.so2.inquire-plot'
......
doc/source/tutorial.rst
View file @ 69c32484
...@@ -34,7 +34,7 @@ Job tree ...@@ -34,7 +34,7 @@ Job tree
The configuration defines a series of jobs to be created and started. The configuration defines a series of jobs to be created and started.
For example, the following jobs might be defined to process Sentinel-5p data:: For example, the following jobs might be defined to process Sentinel-5p data::
cso.tutorial.inquire-scihub cso.tutorial.inquire
cso.tutorial.convert cso.tutorial.convert
cso.tutorial.listing cso.tutorial.listing
cso.tutorial.catalogue cso.tutorial.catalogue
...@@ -42,14 +42,14 @@ For example, the following jobs might be defined to process Sentinel-5p data:: ...@@ -42,14 +42,14 @@ For example, the following jobs might be defined to process Sentinel-5p data::
This list is actually defined as tree, using lists in which the elements could be a list too:: This list is actually defined as tree, using lists in which the elements could be a list too::
cso # tree with element "tutorial" cso # tree with element "tutorial"
.tutorial # tree with elements "inquire-scihub", "convert", "listing", and "catalogue" .tutorial # tree with elements "inquire", "convert", "listing", and "catalogue"
.inquire-scihub # step .inquire # step
.convert # step .convert # step
.listing # step .listing # step
.catalogue # step .catalogue # step
For each element in the tree, the configuration file should specify For each element in the tree, the configuration file should specify
the name of a python class that takes care of the job creating.. the name of a python class that takes care of the job creation.
If the element is a *tree*, use the :py:class:`utopya.UtopyaJobTree <utopya_jobtree.UtopyaJobTree>` class, If the element is a *tree*, use the :py:class:`utopya.UtopyaJobTree <utopya_jobtree.UtopyaJobTree>` class,
and add a line to specify the names of the sub-elements. and add a line to specify the names of the sub-elements.
For the main ``cso`` job, this looks like:: For the main ``cso`` job, this looks like::
...@@ -87,7 +87,7 @@ it will only print a message:: ...@@ -87,7 +87,7 @@ it will only print a message::
! dummy task: ! dummy task:
cso.tutorial.inquire.task.class : utopya.UtopyaJobTask cso.tutorial.inquire.task.class : utopya.UtopyaJobTask
cso.tutorial.inquire.task.args : msg='Inquire SciHub ...' cso.tutorial.inquire.task.args : msg='Inquire archive ...'
Running a single job step Running a single job step
...@@ -103,7 +103,7 @@ Job files ...@@ -103,7 +103,7 @@ Job files
The job tree defined above will create a series of job files in a work directory, The job tree defined above will create a series of job files in a work directory,
with each job in a separate sub directory:: with each job in a separate sub directory::
/work/yourname/CSO-Tests/cso/cso.jb /work/yourname/CSO-Tutorial/cso/cso.jb
cso/tutorial/cso.s5p.jb cso/tutorial/cso.s5p.jb
cso/tutorial/inquire/cso.tutorial.inquire.jb cso/tutorial/inquire/cso.tutorial.inquire.jb
...@@ -111,7 +111,7 @@ The location of the work directories is specified in the settings using:: ...@@ -111,7 +111,7 @@ The location of the work directories is specified in the settings using::
! work directory for jobs; ! work directory for jobs;
! here path including subdirectories for job name elements: ! here path including subdirectories for job name elements:
*.workdir : /work/yourname/CSO-Tests/__NAME2PATH__ *.workdir : /work/yourname/CSO-Tutorial/__NAME2PATH__
The default settings create simple job files that will run in foreground, The default settings create simple job files that will run in foreground,
perform a specified task, and submit the next job in the tree. perform a specified task, and submit the next job in the tree.
...@@ -130,15 +130,10 @@ Step 1 - Inquire S5p archive ...@@ -130,15 +130,10 @@ Step 1 - Inquire S5p archive
*(See* :ref:`s5p-no2-inquire` *section for full description of S5p/NO2 inquireries)* *(See* :ref:`s5p-no2-inquire` *section for full description of S5p/NO2 inquireries)*
The ``cso.tutorial.inquire`` job is configured in ``rc/tutorial.rc`` to crawl The ``cso.tutorial.inquire`` job is configured in
the Copernicus SciHub archive for available Sentinel-5p NO2 observations. `config/tutorial/tutorial.rc <../../../config/tutorial/tutorial.rc>`_
to crawl the `Copernicus DataSpace <https://dataspace.copernicus.eu/>`_ archive
.. IMPORTANT:: for available Sentinel-5p NO2 observations.
Access to the `Copernicus Open Access Hub <https://scihub.copernicus.eu/>`_
requires a (not personal) login and password; see also the :ref:`SciHub-OpenHub` section.
Add the following login/password setting to your ``~/.netrc`` file::
machine s5phub.copernicus.eu login s5pguest password s5pguest
To run the inquire job only, limit the element list of the ``cso.tutorial`` job to:: To run the inquire job only, limit the element list of the ``cso.tutorial`` job to::
...@@ -150,28 +145,35 @@ observations:: ...@@ -150,28 +145,35 @@ observations::
! single step: ! single step:
cso.tutorial.inquire.class : utopya.UtopyaJobStep cso.tutorial.inquire.class : utopya.UtopyaJobStep
! two tasks: ! two tasks:
cso.tutorial.inquire.tasks : table-scihub plot cso.tutorial.inquire.tasks : table-dataspace plot
!~ inquire available files:
cso.tutorial.inquire.table-scihub.class : cso.CSO_SciHub_Inquire !~ task: inquire available files:
cso.tutorial.inquire.table-scihub.args : '${PWD}/config/tutorial/tutorial.rc', \ cso.tutorial.inquire.table-dataspace.class : cso.CSO_DataSpace_Inquire
rcbase='cso.tutorial.inquire-table-scihub' cso.tutorial.inquire.table-dataspace.args : '${__filename__}', \
!~ create plot of available versions: rcbase='cso.tutorial.inquire-table-dataspace'
!~ task: create plot of available versions:
cso.tutorial.inquire.plot.class : cso.CSO_Inquire_Plot cso.tutorial.inquire.plot.class : cso.CSO_Inquire_Plot
cso.tutorial.inquire.plot.args : '${PWD}/config/tutorial/tutorial.rc', \ cso.tutorial.inquire.plot.args : '${__filename__}', \
rcbase='cso.tutorial.inquire-plot' rcbase='cso.tutorial.inquire-plot'
The first setting defines that this is single job step that should do some work. The first setting defines that this is single job step that should do some work.
The ``tasks`` list defines keywords for the two tasks to be performed. The ``tasks`` list defines keywords for the two tasks to be performed.
* For the ``table-scihub`` task, define that the :py:class:`CSO_SciHub_Inquire <cso_scihub.CSO_SciHub_Inquire>` class * For the ``table-dataspace`` task, define that the :py:class:`CSO_DataSpace_Inquire <cso_dataspace.CSO_DataSpace_Inquire>` class
should be used to do the work; the class is accessible from the :py:mod:`cso` module should be used to do the work; the class is accessible from the :py:mod:`cso` module
(implemented in``py/cso.py``). (implemented in``py/cso.py``).
The arguments that initialize the class specify the name of an rcfile with settings The first arguments that initialize the class specifies the name of an rcfile with settings;
(``tutorial.rc``) and that the settings start with keywords ``'cso.tutorial.inquire-table-scihub'``. in this case these settings are in the same file (``tutorial.rc``) that defines the job-tree,
and therefore the special keyword ``%{__filename__}`` could be used.
The second argument ``rcbase`` is optional and specifies that the settings start with keywords
``'cso.tutorial.inquire-table-dataspace'``.
* Similar for the ``plot`` task, the settings define that the * Similar for the ``plot`` task, the settings define that the
:py:class:`CSO_Inquire_Plot <cso_scihub.CSO_Inquire_Plot>` class should be used to do the work. :py:class:`CSO_Inquire_Plot <cso_inquire.CSO_Inquire_Plot>` class should be used to do the work.
The tutorial settings will inquire the time range 2018-2023. The tutorial settings will inquire the time range 2018-2023.
...@@ -180,10 +182,10 @@ the work directories. ...@@ -180,10 +182,10 @@ the work directories.
This is specified with a `user defined` keyword, here chosen to start with '``my.``':: This is specified with a `user defined` keyword, here chosen to start with '``my.``'::
! base location for work directories: ! base location for work directories:
my.work : /work/${USER}/CSO-Tests my.work : /work/yourname/CSO-Tutorial
This `user defined` key is for example used to specify the work directory of the This `user defined` key is for example used to specify the work directory of the
``cso.tutorial.inquire-scihub`` (and other) jobs:: ``cso.tutorial.inquire`` (and other) jobs::
! work directory for jobs; ! work directory for jobs;
! here path including subdirectories for job name elements: ! here path including subdirectories for job name elements:
...@@ -192,26 +194,51 @@ This `user defined` key is for example used to specify the work directory of the ...@@ -192,26 +194,51 @@ This `user defined` key is for example used to specify the work directory of the
This tells that the work directory of the job should include the jobname expanded This tells that the work directory of the job should include the jobname expanded
as subdirectories. For this example, the full path becomes:: as subdirectories. For this example, the full path becomes::
/work/yourname/CSO-Tests/cso/tutorial/inquire/ /work/yourname/CSO-Tutorial/cso/tutorial/inquire/
The base of the work directories is also used to specify where the inquired table file should be stored:: The base of the work directories is also used to specify where the inquired table file should be stored::
! output table, date of today: ! output table, date of today:
cso.tutorial.inquire-s5phub-table.output.file : ${my.work}/Copernicus/Copernicus_S5p_NO2_scihub__%Y-%m-%d.csv cso.tutorial.inquire-dataspace-table.output.file : ${my.work}/Copernicus/Copernicus_S5p_NO2_dataspace__%Y-%m-%d.csv
The created table is then for example:: The created table is then for example::
/work/yourname/CSO-Tests/Copernicus/Copernicus_S5p_NO2_scihub__2023-08-09.csv /work/yourname/tCSO-Tutorial/Copernicus/Copernicus_S5p_NO2_dataspace__2023-08-09.csv
If not already done, run the ``cso`` script with the tutorial settings::
./bin/cso config/tutorial/tutorial.rc
It could take a long time time to inquire the full time period!
Be patient, or limit the time range in the settings ...
When the inquiry is finished, check if indeed the table with available orbits is created.
To visualize what is available, the :py:class:`cso_inquire.CSO_DataSpace_InquirePlot` class
should have created an overview figure next to the table file::
To visualize what is available from the various portals, the /work/yourname/tCSO-Tutorial/Copernicus/Copernicus_S5p_NO2_dataspace__2023-08-09.png
:py:class:`cso_scihub.CSO_SciHub_InquirePlot` could be used to create an overview figure.
The figure file looks like: The figure should look like:
.. figure:: figs/NO2/Copernicus_S5p_NO2.png .. figure:: figs/NO2/Copernicus_S5p_NO2.png
:scale: 50 % :scale: 50 %
:align: center :align: center
:alt: Overview of available NO2 processings on SciHub. :alt: Overview of available NO2 processings on DataSpace.
For the same orbit, multiple data files could be available.
A single S5p data file is uniquely identified by:
* processor version ``x.y.z``
* processing:
* ``NRTI`` : *Near Real Time*, processed within hours after observation;
* ``OFFL`` : *Offline* data, processed within a few weeks after observations;
* ``RPRO`` : *Reproduced* data, processed long after observations using latest processor version.
The collection numbers ``01``, ``02``, etc are used to identify a single timeseries
of the entire data set.
Step 2 - Convert to CSO format Step 2 - Convert to CSO format
...@@ -219,9 +246,21 @@ Step 2 - Convert to CSO format ...@@ -219,9 +246,21 @@ Step 2 - Convert to CSO format
*(See* :ref:`s5p-no2-convert` *section for full description of S5p/NO2 conversion)* *(See* :ref:`s5p-no2-convert` *section for full description of S5p/NO2 conversion)*
The ``cso.tutorial.convert`` job is configured to convert the downloaded orbit files into a common format. The ``cso.tutorial.convert`` job is configured to convert the original S5p/NO2 data
into a common format. If the original data is not present yet, it is downloaded.
.. IMPORTANT::
Downloading data from the *Copernicus DataSpace* requires a personal login and password.
Add the login/password setting to your ``~/.netrc`` file::
machine zipper.dataspace.copernicus.eu login Your.Name@institute.org password ***********
See also the :ref:`dataspace-account` section in the descrption of the
:py:mod:`cso_dataspace` module.
The conversion includes filter options to selected only pixels within a certain domain, and with The conversion includes filter options to selected only pixels within a certain domain, and with
some minimum quality flag, etc; this could strongly limit the data volume. some minimum quality flag; this will strongly limit the data volume.
It is not necessary to keep the original data, eventually it could be downloaded again when needed.
To run the conversion job only, limit in ``rc/tutorial.rc`` the element list of the ``cso.tutorial`` job to:: To run the conversion job only, limit in ``rc/tutorial.rc`` the element list of the ``cso.tutorial`` job to::
...@@ -233,17 +272,18 @@ The conversion job is configured with:: ...@@ -233,17 +272,18 @@ The conversion job is configured with::
cso.tutorial.convert.class : utopya.UtopyaJobStep cso.tutorial.convert.class : utopya.UtopyaJobStep
! conversion task: ! conversion task:
cso.tutorial.convert.task.class : cso.CSO_S5p_Convert cso.tutorial.convert.task.class : cso.CSO_S5p_Convert
cso.tutorial.convert.task.args : '${PWD}/config/tutorial/tutorial.rc', \ cso.tutorial.convert.task.args : '${__filename__}', \
rcbase='cso.tutorial.convert' rcbase='cso.tutorial.convert'
The conversion is thus done using the :py:class:`CSO_S5p_Convert <cso_s5p.CSO_S5p_Convert>` class The conversion is thus done using the :py:class:`CSO_S5p_Convert <cso_s5p.CSO_S5p_Convert>` class
that can be accessed from the :py:mod:`cso` module. that can be accessed from the :py:mod:`cso` module.
The arguments that initialize the class specify the name of an rcfile with settings The arguments that initialize the class specify the name of a rcfile with settings
(``tutorial.rc``) and that the settings start with keywords ``'cso.tutorial.convert'``. (in this case the ``tutorial.rc`` that holds the job-tree definition)
and that the settings start with keywords ``'cso.tutorial.convert'``.
The result of the conversion is a set of files holding the selected pixels per orbit:: The result of the conversion is a set of files holding the selected pixels per orbit::
/work/yourname/CSO-Tests/CSO-data/S5p/RPRO/NO2/CAMS/2018/06/S5p_RPRO_NO2_03272.nc /work/yourname/CSO-Tutorial/CSO-data/S5p/RPRO/NO2/CAMS/2018/06/S5p_RPRO_NO2_03272.nc
S5p_RPRO_NO2_03273.nc S5p_RPRO_NO2_03273.nc
: :
...@@ -271,7 +311,7 @@ The following is a demo Python code that creates an S5p/NO2 map:: ...@@ -271,7 +311,7 @@ The following is a demo Python code that creates an S5p/NO2 map::
import cso import cso
# sample file: # sample file:
filename = '/work/yourname/CSO-Tests/CSO-data/S5p/RPRO/NO2/CAMS/2018/06/S5p_RPRO_NO2_03278.nc' filename = '/work/yourname/CSO-Tutorial/CSO-data/S5p/RPRO/NO2/CAMS/2018/06/S5p_RPRO_NO2_03278.nc'
# read: # read:
orb = cso.CSO_File( filename=filename ) orb = cso.CSO_File( filename=filename )
...@@ -327,7 +367,7 @@ This file will be used by the observation operator to selects orbits with pixels ...@@ -327,7 +367,7 @@ This file will be used by the observation operator to selects orbits with pixels
a desired time range. a desired time range.
Also the *catalogue* creator described below uses the listing file to select orbits. Also the *catalogue* creator described below uses the listing file to select orbits.
The configuration of describes the name of the listing file to be created, The configuration of the job specifies the name of the listing file to be created,
and the directories to be scanned for orbit files. and the directories to be scanned for orbit files.
...@@ -337,10 +377,10 @@ Step 5 - Catalogue of figures ...@@ -337,10 +377,10 @@ Step 5 - Catalogue of figures
*(See* :ref:`s5p-no2-catalogue` *section for main description)* *(See* :ref:`s5p-no2-catalogue` *section for main description)*
For a first impression of how the downloaded and converted satellite data looks like, For a first impression of how the downloaded and converted satellite data looks like,
the ``cso.tutorial.catalogue`` job can be used. This will figures out of the converted files, the ``cso.tutorial.catalogue`` job can be used. This will create figures out of the converted files,
in particular maps that show values on the track. in particular maps that show values on the track.
To run the catalogue job only, limit in ``rc/tutorial.rc`` the element list of the ``cso.tutorial`` job to:: To run the catalogue job only, limit in ``tutorial.rc`` the element list of the ``cso.tutorial`` job to::
cso.tutorial.elements : catalogue cso.tutorial.elements : catalogue
...@@ -355,25 +395,26 @@ This is configured using:: ...@@ -355,25 +395,26 @@ This is configured using::
! catalogue creation task: ! catalogue creation task:
cso.tutorial.catalogue.figs.class : cso.CSO_Catalogue cso.tutorial.catalogue.figs.class : cso.CSO_Catalogue
cso.tutorial.catalogue.figs.args : '${PWD}/config/tutorial/tutorial.rc', \ cso.tutorial.catalogue.figs.args : '${__filename__}', \
rcbase='cso.tutorial.catalogue' rcbase='cso.tutorial.catalogue'
! indexer task: ! indexer task:
cso.tutorial.catalogue.index.class : utopya.Indexer cso.tutorial.catalogue.index.class : utopya.Indexer
cso.tutorial.catalogue.index.args : '${PWD}/config/tutorial/tutorial.rc', \ cso.tutorial.catalogue.index.args : '${__filename__}', \
rcbase='cso.tutorial.catalogue-index' rcbase='cso.tutorial.catalogue-index'
The ``figs`` task that creates the figures uses the :py:class:`CSO_Catalogue <.cso_catalogue.CSO_Catalogue>` class The ``figs`` task that creates the figures uses the :py:class:`CSO_Catalogue <.cso_catalogue.CSO_Catalogue>` class
that can be accessed from the :py:mod:`cso` module. that can be accessed from the :py:mod:`cso` module.
The arguments that initialize the class specify the name of an rcfile with settings The arguments that initialize the class specify the name of an rcfile with settings
(``tutorial.rc``) and that the settings start with keywords ``'cso.tutorial.catalogue'``. (in this case the ``tutorial.rc`` that holds the job-tree definition)
and that the settings start with keywords ``'cso.tutorial.catalogue'``.
The configuration describes where to find a *listing* file with orbits, The configuration describes where to find a *listing* file with orbits,
which variables should be plot, the colorbar properties, etc. which variables should be plot, the colorbar properties, etc.
The names of the created figures is composed from the base name of the converted files The names of the created figures are composed from the base name of the converted files
and the variable that is plotted:: and the variable that is plotted::
/work/yourname/CSO-Tests/CSO-data-catalogue/2018/06/01/S5p_RPRO_NO2_03278__vcd.png /work/yourname/CSO-Tutorial/CSO-data-catalogue/2018/06/01/S5p_RPRO_NO2_03278__vcd.png
S5p_RPRO_NO2_03278__qa_value.png S5p_RPRO_NO2_03278__qa_value.png
: :
...@@ -393,7 +434,7 @@ The arguments that initialize the class specify the name of an rcfile with setti ...@@ -393,7 +434,7 @@ The arguments that initialize the class specify the name of an rcfile with setti
When succesful, the index creator displays an url that could be loaded in a browser:: When succesful, the index creator displays an url that could be loaded in a browser::
Browse to: Browse to:
file:///work/yourname/CSO-Tests/CSO-data-catalogue/index.html file:///work/yourname/CSO-Tutorial/CSO-data-catalogue/S5p/NO2/CAMS/index.html
.. figure:: figs/NO2/CSO_NO2_catalogue.png .. figure:: figs/NO2/CSO_NO2_catalogue.png
:scale: 50 % :scale: 50 %
...@@ -430,7 +471,7 @@ For testing, copy the entire sub directory to a work directory; ...@@ -430,7 +471,7 @@ For testing, copy the entire sub directory to a work directory;
configuration assumes a location in the work directory of the pre-processor where configuration assumes a location in the work directory of the pre-processor where
the converted orbit files are:: the converted orbit files are::
cd /work/yourname/CSO-Tests cd /work/yourname/CSO-Tutorial
cp -r ~/CSO/oper CSO-oper cp -r ~/CSO/oper CSO-oper
cd CSO-oper cd CSO-oper
...@@ -569,11 +610,11 @@ The job for this is ``cso.tutorial.catalogue``, which is configured using:: ...@@ -569,11 +610,11 @@ The job for this is ``cso.tutorial.catalogue``, which is configured using::
cso.tutorial.sim-catalogue.tasks : figs index cso.tutorial.sim-catalogue.tasks : figs index
! catalogue creation task: ! catalogue creation task:
cso.tutorial.sim-catalogue.figs.class : cso.CSO_SimCatalogue cso.tutorial.sim-catalogue.figs.class : cso.CSO_SimCatalogue
cso.tutorial.sim-catalogue.figs.args : '${PWD}/config/tutorial/tutorial.rc', \ cso.tutorial.sim-catalogue.figs.args : '${__filename__}', \
rcbase='cso.tutorial.sim-catalogue' rcbase='cso.tutorial.sim-catalogue'
! indexer task: ! indexer task:
cso.tutorial.sim-catalogue.index.class : utopya.Indexer cso.tutorial.sim-catalogue.index.class : utopya.Indexer
cso.tutorial.sim-catalogue.index.args : '${PWD}/config/tutorial/tutorial.rc', \ cso.tutorial.sim-catalogue.index.args : '${__filename__}', \
rcbase='cso.tutorial.sim-catalogue-index' rcbase='cso.tutorial.sim-catalogue-index'
The ``figs`` task that creates the figures is thus done using the The ``figs`` task that creates the figures is thus done using the
...@@ -604,7 +645,7 @@ For these variables it is necessary to define whether they should be read from a ...@@ -604,7 +645,7 @@ For these variables it is necessary to define whether they should be read from a
The names of the created figures is composed from the base name of the converted files The names of the created figures is composed from the base name of the converted files
and the variable that is plotted:: and the variable that is plotted::
/work/yourname/CSO-Tests/CSO-oper/sim-catalogue/2018/06/01/S5p_RPRO_NO2_20180601_1100_yr.png /work/yourname/CSO-Tutorial/CSO-oper/sim-catalogue/2018/06/01/S5p_RPRO_NO2_20180601_1100_yr.png
S5p_RPRO_NO2_20180601_1100_ys.png S5p_RPRO_NO2_20180601_1100_ys.png
: :
...@@ -625,7 +666,7 @@ The arguments that initialize the class specify the name of an rcfile with setti ...@@ -625,7 +666,7 @@ The arguments that initialize the class specify the name of an rcfile with setti
When successful, the index creator displays an url that could be loaded in a browser:: When successful, the index creator displays an url that could be loaded in a browser::
Browse to: Browse to:
file:///work/yourname/CSO-Tests/CSO-oper/sim-catalogue/index.html file:///work/yourname/CSO-Tutorial/CSO-oper/sim-catalogue/index.html
.. figure:: figs/NO2/CSO_NO2_sim-catalogue.png .. figure:: figs/NO2/CSO_NO2_sim-catalogue.png
:scale: 50 % :scale: 50 %
...@@ -733,7 +774,7 @@ The configuration of the gridding job is:: ...@@ -733,7 +774,7 @@ The configuration of the gridding job is::
cso.tutorial.sim-gridded.class : utopya.UtopyaJobStep cso.tutorial.sim-gridded.class : utopya.UtopyaJobStep
! catalogue creation task: ! catalogue creation task:
cso.tutorial.sim-gridded.task.class : cso.CSO_GriddedAverage cso.tutorial.sim-gridded.task.class : cso.CSO_GriddedAverage
cso.tutorial.sim-gridded.task.args : '${PWD}/config/tutorial/tutorial.rc', \ cso.tutorial.sim-gridded.task.args : '${__filename__}', \
rcbase='cso.tutorial.gridded' rcbase='cso.tutorial.gridded'
It is also possible to create a catalogue of the gridded fields. It is also possible to create a catalogue of the gridded fields.
......
oper/src/cso_ncfile.F90
View file @ 69c32484
...@@ -14,6 +14,9 @@ ...@@ -14,6 +14,9 @@
! 2023-08, Arjo Segers ! 2023-08, Arjo Segers
! Replaced `where` constructs by loops after memory errors on some systems. ! Replaced `where` constructs by loops after memory errors on some systems.
! !
! 2023-11, Arjo Segers
! Close files also in 'read' mode ...
!
!############################################################################### !###############################################################################
! !
#define TRACEBACK write (csol,'("in ",a," (",a,", line",i5,")")') rname, __FILE__, __LINE__; call csoErr #define TRACEBACK write (csol,'("in ",a," (",a,", line",i5,")")') rname, __FILE__, __LINE__; call csoErr
...@@ -2530,7 +2533,6 @@ contains ...@@ -2530,7 +2533,6 @@ contains
subroutine NcFile_Done( self, status ) subroutine NcFile_Done( self, status )
use NetCDF , only : NF90_Close use NetCDF , only : NF90_Close
use CSO_Comm, only : csoc use CSO_Comm, only : csoc
! --- in/out --------------------------------- ! --- in/out ---------------------------------
...@@ -2549,12 +2551,13 @@ contains ...@@ -2549,12 +2551,13 @@ contains
! switch: ! switch:
select case ( self%rwmode ) select case ( self%rwmode )
! read, open: ! open:
case ( 'r', 'o' ) case ( 'o' )
! open/close is managed externally,
! nothing to be done ! nothing to be done
! write: ! read, write:
case ( 'w' ) case ( 'r', 'w' )
! written on root... ! written on root...
if ( csoc%root ) then if ( csoc%root ) then
......
py/cso.py
View file @ 69c32484
...@@ -36,7 +36,7 @@ Actual implementations can be found in submodules: ...@@ -36,7 +36,7 @@ Actual implementations can be found in submodules:
pymod-cso_file pymod-cso_file
pymod-cso_inquire pymod-cso_inquire
pymod-cso_scihub pymod-cso_dataspace
pymod-cso_pal pymod-cso_pal
pymod-cso_s5p pymod-cso_s5p
pymod-cso_s5p_superobs pymod-cso_s5p_superobs
...@@ -63,7 +63,7 @@ and are defined according to the following hierchy: ...@@ -63,7 +63,7 @@ and are defined according to the following hierchy:
* :py:class:`.UtopyaRc` * :py:class:`.UtopyaRc`
* :py:class:`.CSO_Inquire_Plot` * :py:class:`.CSO_Inquire_Plot`
* :py:class:`.CSO_SciHub_Inquire` * :py:class:`.CSO_DataSpace_Inquire`
* :py:class:`.CSO_PAL_Inquire` * :py:class:`.CSO_PAL_Inquire`
* :py:class:`.CSO_S5p_Convert` * :py:class:`.CSO_S5p_Convert`
* :py:class:`.CSO_S5p_Listing` * :py:class:`.CSO_S5p_Listing`
...@@ -112,7 +112,7 @@ and are defined according to the following hierchy: ...@@ -112,7 +112,7 @@ and are defined according to the following hierchy:
from cso_file import * from cso_file import *
from cso_inquire import * from cso_inquire import *
from cso_scihub import * from cso_dataspace import *
from cso_pal import * from cso_pal import *
from cso_s5p import * from cso_s5p import *
from cso_s5p_superobs import * from cso_s5p_superobs import *
......
py/cso_dataspace.py 0 → 100644
View file @ 69c32484
This diff is collapsed.
py/cso_file.py
View file @ 69c32484
...@@ -14,6 +14,9 @@ ...@@ -14,6 +14,9 @@
# 2023-08, Arjo Segers # 2023-08, Arjo Segers
# Reformatted using 'black'. # Reformatted using 'black'.
# #
# 2023-11, Arjo Segers
# Added "CheckDir" method.
#
######################################################################## ########################################################################
### ###
...@@ -53,6 +56,34 @@ import logging ...@@ -53,6 +56,34 @@ import logging
######################################################################## ########################################################################
def CheckDir(filename):
"""
Check if ``filename`` has a directory path;
if so, create that directory if it does not exist yet.
"""
# modules:
import os
# directory name, could be empty:
dname = os.path.dirname(filename)
# directory defined?
if len(dname) > 0:
# not present yet?
if not os.path.isdir(dname):
# create including subdirs:
os.makedirs(dname)
# endif # dname present
# endif # dname defined
# enddef CheckDir
# *
def Pack_DataArray(da, dtype="i2"): def Pack_DataArray(da, dtype="i2"):
""" """
......
py/cso_gridded.py
View file @ 69c32484
...@@ -532,8 +532,10 @@ class CSO_GriddedAverage(utopya.UtopyaRc): ...@@ -532,8 +532,10 @@ class CSO_GriddedAverage(utopya.UtopyaRc):
datafiles.sort() datafiles.sort()
# info .. # info ..
logging.info(indent + " found %i file(s) matching: %s" logging.info(
% (len(datafiles), infile_curr) ) indent
+ " found %i file(s) matching: %s" % (len(datafiles), infile_curr)
)
# endif # listing or filenames # endif # listing or filenames
......
py/cso_inquire.py
View file @ 69c32484
...@@ -67,28 +67,29 @@ import utopya ...@@ -67,28 +67,29 @@ import utopya
class CSO_Inquire_Plot(utopya.UtopyaRc): class CSO_Inquire_Plot(utopya.UtopyaRc):
""" """
Create plot of processing version versus time to indicate the available orbits in the SciHub archive. Create plot of data version versus time to indicate the available orbits.
The information on orbits is taken from a csv table created by :py:class:`CSO_SciHub_Inquire` class. The information on orbits is taken from a csv table created by for example
the :py:class:`CSO_DataSpace_Inquire` class.
Specifify the name of the table file in the settings:: Specifify the name of the table file in the settings::
! listing file: ! listing file:
cso.tutorial.inquire-s5phub-plot.file : ${my.work}/Copernicus/Copernicus_S5p_NO2_s5phub_%Y-%m-%d.csv <rcbase>.file : ${my.work}/Copernicus/Copernicus_S5p_NO2_%Y-%m-%d.csv
The date templates are by default filled for the current day. The date templates are by default filled for the current day.
Alternatively, specify an explicit date:: Alternatively, specify an explicit date::
!~ specify dates ("yyyy-mm-dd") to use historic table: !~ specify dates ("yyyy-mm-dd") to use historic table:
cso.tutorial.inquire-s5phub-plot.filedate : 2022-01-28 <rcbase>.filedate : 2022-01-28
The plot could also be created by combining multiple tables; The plot could also be created by combining multiple tables;
use a semi-colon to seperate the file names (and eventually the dates):: use a semi-colon to seperate the file names (and eventually the dates)::
! listing files: ! listing files:
cso.tutorial.inquire-s5phub-plot.file : ${my.work}/Copernicus/Copernicus_S5p_NO2_s5phub_%Y-%m-%d.csv ; \\ <rcbase>.file : ${my.work}/Copernicus/Copernicus_S5p_NO2_%Y-%m-%d.csv ; \\
${my.work}/Copernicus/Copernicus_S5p_NO2_pal_%Y-%m-%d.csv ${my.work}/Copernicus/Copernicus_S5p_NO2_pal_%Y-%m-%d.csv
!~ specify dates ("yyyy-mm-dd") to use historic tables: !~ specify dates ("yyyy-mm-dd") to use historic tables:
!cso.tutorial.inquire-s5phub-plot.filedate : 2022-01-28 ; 2022-01-28 !<rcbase>.filedate : 2022-01-28 ; 2022-01-28
The created plot shows a time line and on the vertical ax the processor versions; The created plot shows a time line and on the vertical ax the processor versions;
a bar indicates when a certain version was used to process orbits: a bar indicates when a certain version was used to process orbits:
...@@ -108,7 +109,7 @@ class CSO_Inquire_Plot(utopya.UtopyaRc): ...@@ -108,7 +109,7 @@ class CSO_Inquire_Plot(utopya.UtopyaRc):
The following flag is used to ensure that the plot is renewed:: The following flag is used to ensure that the plot is renewed::
! renew existing plots? ! renew existing plots?
cso.tutorial.inquire-s5phub-plot.renew : True <rcbase>.renew : True
""" """
......
py/cso_pal.py
View file @ 69c32484
...@@ -107,7 +107,7 @@ class CSO_PAL_Inquire(utopya.UtopyaRc): ...@@ -107,7 +107,7 @@ class CSO_PAL_Inquire(utopya.UtopyaRc):
Name of output csv file:: Name of output csv file::
! output table, date of today: ! output table, date of today:
cso.s5p.no2.inquire-s5phub.output.file : ${my.work}/PAL_S5P_NO2_%Y-%m-%d.csv <rcbase>.output.file : ${my.work}/PAL_S5P_NO2_%Y-%m-%d.csv
Example records:: Example records::
...@@ -238,8 +238,14 @@ class CSO_PAL_Inquire(utopya.UtopyaRc): ...@@ -238,8 +238,14 @@ class CSO_PAL_Inquire(utopya.UtopyaRc):
platform_name, rest = bname.split("_", 1) platform_name, rest = bname.split("_", 1)
processing = rest[0:4] processing = rest[0:4]
product_type = rest[5:15] product_type = rest[5:15]
(start_time,end_time,orbit,collection,\ (
processor_version,production_time) = rest[16:].split("_") start_time,
end_time,
orbit,
collection,
processor_version,
production_time,
) = rest[16:].split("_")
# convert: # convert:
tfmt = "%Y%m%dT%H%M%S" tfmt = "%Y%m%dT%H%M%S"
......
py/cso_plot.py
View file @ 69c32484
...@@ -754,8 +754,9 @@ class ColorbarFigure(Figure): ...@@ -754,8 +754,9 @@ class ColorbarFigure(Figure):
# endif # endif
# get red/green/blue arrays for extensions: # get red/green/blue arrays for extensions:
(red_under, green_under, blue_under) = \ (red_under, green_under, blue_under) = matplotlib.colors.colorConverter.to_rgb(
matplotlib.colors.colorConverter.to_rgb(color_under) color_under
)
red_over, green_over, blue_over = matplotlib.colors.colorConverter.to_rgb(color_over) red_over, green_over, blue_over = matplotlib.colors.colorConverter.to_rgb(color_over)
# initialise color dictionary: # initialise color dictionary:
...@@ -1701,8 +1702,9 @@ def mid2corners(xx): ...@@ -1701,8 +1702,9 @@ def mid2corners(xx):
# * # *
def GetGrid( shp, xx=None, yy=None, x=None, y=None, def GetGrid(
xm=None, ym=None, xxm=None, yym=None, domain=None): shp, xx=None, yy=None, x=None, y=None, xm=None, ym=None, xxm=None, yym=None, domain=None
):
""" """
Return 2D grid arrays with corner points. Return 2D grid arrays with corner points.
......
py/cso_regions.py
View file @ 69c32484
...@@ -2118,8 +2118,10 @@ class CSO_Catalogue_RegionsTimeSeries(cso_catalogue.CSO_CatalogueBase): ...@@ -2118,8 +2118,10 @@ class CSO_Catalogue_RegionsTimeSeries(cso_catalogue.CSO_CatalogueBase):
# store: # store:
if (len(reg_used) == 0) or (reg_code not in reg_used["code"].values): if (len(reg_used) == 0) or (reg_code not in reg_used["code"].values):
reg_used = pandas.concat( reg_used = pandas.concat(
[ reg_used, [
pandas.DataFrame({"code": reg_code, "name": reg_name}), ], reg_used,
pandas.DataFrame({"code": reg_code, "name": reg_name}),
],
ignore_index=True, ignore_index=True,
) )
# endif # endif
...@@ -2463,15 +2465,20 @@ class CSO_Statistics_RegionsTables(utopya.UtopyaRc): ...@@ -2463,15 +2465,20 @@ class CSO_Statistics_RegionsTables(utopya.UtopyaRc):
rbias_label = "(sim-obs)/obs" rbias_label = "(sim-obs)/obs"
# add record: # add record:
df = pandas.concat( df = pandas.concat(
[ df, [
pandas.DataFrame( { df,
pandas.DataFrame(
{
"iso2": [reg_code2], "iso2": [reg_code2],
"iso3": [reg_code], "iso3": [reg_code],
"name": [reg_name], "name": [reg_name],
"time": [tlab], "time": [tlab],
obs_label: [obs], obs_label: [obs],
sim_label: [sim], sim_label: [sim],
rbias_label: [rbias], } ), ], rbias_label: [rbias],
}
),
],
ignore_index=True, ignore_index=True,
) )
......
py/cso_s5p.py
View file @ 69c32484
...@@ -18,6 +18,10 @@ ...@@ -18,6 +18,10 @@
# 2023-08, Arjo Segers # 2023-08, Arjo Segers
# Reformatted using 'black'. # Reformatted using 'black'.
# #
# 2023-09, Arjo Segers
# Fixed bug in definition of listing file dates from rcfile settings.
#
#
######################################################################## ########################################################################
### ###
...@@ -692,6 +696,15 @@ class CSO_S5p_File(cso_file.CSO_File): ...@@ -692,6 +696,15 @@ class CSO_S5p_File(cso_file.CSO_File):
* ``square`` : create a variable as the square of the input; requires a ``.from`` setting. * ``square`` : create a variable as the square of the input; requires a ``.from`` setting.
Optionally swap layers, for example to have profiles in upward direction
(surface to top) rather than downward (top to bottom)::
<rcbase>.output.var.longitude.swap_layers : True
Optionally provide a target data type; by default original data type in the input file is used::
<rcbase>.output.var.longitude.dtype : f4
Optionally provide target units too. Optionally provide target units too.
In the (unlikely) case that the original variable has no ``units`` attribute, In the (unlikely) case that the original variable has no ``units`` attribute,
this setting is required to define the (assumed) units. this setting is required to define the (assumed) units.
...@@ -2030,7 +2043,7 @@ class CSO_S5p_Convert(utopya.UtopyaRc): ...@@ -2030,7 +2043,7 @@ class CSO_S5p_Convert(utopya.UtopyaRc):
<rcbase>.timerange.end : 2018-06-03 23:59 <rcbase>.timerange.end : 2018-06-03 23:59
The input files are searched in a table created by an *inquire* class, The input files are searched in a table created by an *inquire* class,
for example :py:class:`CSO_SciHub_Inquire <cso_scihub.CSO_SciHub_Inquire>` for example :py:class:`CSO_DataSpace_Inquire <cso_dataspace.CSO_DataSpace_Inquire>`
or :py:class:`CSO_PAL_Inquire <cso_pal.CSO_PAL_Inquire>`. or :py:class:`CSO_PAL_Inquire <cso_pal.CSO_PAL_Inquire>`.
These have scanned the archives to examine which processings and versions are available, These have scanned the archives to examine which processings and versions are available,
and stored the result in a csv file. and stored the result in a csv file.
...@@ -2038,7 +2051,7 @@ class CSO_S5p_Convert(utopya.UtopyaRc): ...@@ -2038,7 +2051,7 @@ class CSO_S5p_Convert(utopya.UtopyaRc):
that is taken from another key:: that is taken from another key::
! listing of available source files, ! listing of available source files,
! created by 'inquire-s5phub' job: ! created by for example 'inquire' job:
<rcbase>.inquire.file : /data/Copernicus/S5p/Copernicus_S5P_NO2_%Y-%m-%d.csv <rcbase>.inquire.file : /data/Copernicus/S5p/Copernicus_S5P_NO2_%Y-%m-%d.csv
!! date used in filename, leave empty for today: !! date used in filename, leave empty for today:
!<rcbase>.inquire.filedate : 2022-01-28 !<rcbase>.inquire.filedate : 2022-01-28
...@@ -2053,7 +2066,7 @@ class CSO_S5p_Convert(utopya.UtopyaRc): ...@@ -2053,7 +2066,7 @@ class CSO_S5p_Convert(utopya.UtopyaRc):
! remove downloaded input files after convert? ! remove downloaded input files after convert?
<rcbase>.downloads.cleanup : False <rcbase>.downloads.cleanup : False
The input files keep the same name as used in the SciHub archive, for example:: The input files keep the same name as used in the *DataSpace* archive, for example::
/data/Copernicus/S5P/OFFL/NO2/2018/07/S5P_OFFL_L2__NO2____20180701T005930_20180701T024100_03698_01_010002_20180707T022838.nc /data/Copernicus/S5P/OFFL/NO2/2018/07/S5P_OFFL_L2__NO2____20180701T005930_20180701T024100_03698_01_010002_20180707T022838.nc
start_time end_time orbit start_time end_time orbit
...@@ -2141,10 +2154,11 @@ class CSO_S5p_Convert(utopya.UtopyaRc): ...@@ -2141,10 +2154,11 @@ class CSO_S5p_Convert(utopya.UtopyaRc):
import datetime import datetime
import fnmatch import fnmatch
import pandas import pandas
import numpy
# tools: # tools:
import cso_file import cso_file
import cso_scihub import cso_dataspace
import utopya import utopya
# info ... # info ...
...@@ -2169,7 +2183,7 @@ class CSO_S5p_Convert(utopya.UtopyaRc): ...@@ -2169,7 +2183,7 @@ class CSO_S5p_Convert(utopya.UtopyaRc):
# inquire tables: # inquire tables:
filename__templates = self.GetSetting("inquire.file").split(";") filename__templates = self.GetSetting("inquire.file").split(";")
# time stamp in file? # time stamp in file?
filedates = self.GetSetting("inquire.filedate", default="") filedates = self.GetSetting("inquire.filedate", default="").split(";")
if len(filedates) == 0: if len(filedates) == 0:
filedates = [""] * len(filename__templates) filedates = [""] * len(filename__templates)
elif len(filedates) != len(filename__templates): elif len(filedates) != len(filename__templates):
...@@ -2454,8 +2468,13 @@ class CSO_S5p_Convert(utopya.UtopyaRc): ...@@ -2454,8 +2468,13 @@ class CSO_S5p_Convert(utopya.UtopyaRc):
if not os.path.isfile(input_file): if not os.path.isfile(input_file):
# info .. # info ..
logging.info(" not present yet, download ...") logging.info(" not present yet, download ...")
# check ..
if ("href" not in rec.keys()) or numpy.isnan(rec["href"]):
logging.error(f"cannot download, no 'href' element in record ...")
raise Exception
# endif
# download ... # download ...
cso_scihub.CSO_SciHub_DownloadFile(rec["href"], input_file) cso_dataspace.CSO_DataSpace_DownloadFile(rec["href"], input_file)
# store name: # store name:
downloads.append(input_file) downloads.append(input_file)
# endif # endif
...@@ -2532,7 +2551,7 @@ class CSO_S5p_Convert(utopya.UtopyaRc): ...@@ -2532,7 +2551,7 @@ class CSO_S5p_Convert(utopya.UtopyaRc):
for key in ["orbit", "processing", "processor_version", "collection"]: for key in ["orbit", "processing", "processor_version", "collection"]:
attrs[key] = rec[key] attrs[key] = rec[key]
# endfor # endfor
attrs["orbit_file"] = input_file attrs["orbit_file"] = os.path.basename(input_file)
# write: # write:
csf.Write( csf.Write(
filename=output_filename, filename=output_filename,
...@@ -2813,6 +2832,265 @@ class CSO_S5p_Listing(utopya.UtopyaRc): ...@@ -2813,6 +2832,265 @@ class CSO_S5p_Listing(utopya.UtopyaRc):
# endclass CSO_S5p_Listing # endclass CSO_S5p_Listing
########################################################################
###
### create listing file for downloaded S5P files
###
########################################################################
class CSO_S5p_Download_Listing(utopya.UtopyaRc):
"""
Create *listing* file for files download from S5P data portals.
A *listing* file contains the names of the converted orbit files,
the time range of pixels in the file, and other information extracted from the filenames:
filename ;mission;processing;product_id;start_time ;end_time ;orbit;collection;processor_version;processing_time
RPRO/CH4/2018/04/S5P_RPRO_L2__CH4____20180430T001851_20180430T020219_02818_01_010301_20190513T141133.nc;S5P ;RPRO ;L2__CH4___;2018-04-30T00:18:51;2018-04-30T02:02:19;02818;01 ;010301 ;2019-05-13T14:11:33
RPRO/CH4/2018/04/S5P_RPRO_L2__CH4____20180430T020021_20180430T034349_02819_01_010301_20190513T135953.nc;S5P ;RPRO ;L2__CH4___;2018-04-30T02:00:21;2018-04-30T03:43:49;02819;01 ;010301 ;2019-05-13T13:59:53
:
This file could be used to scan for available versions and how they were produced.
In the settings, define the name of the file to be created::
! csv file that will hold records per file with:
! - timerange of pixels in file
! - orbit number
! time templates are replaced with todays date
<rcbase>.file : /Scratch/Copernicus/S5p/listing-CH4__%Y-%m-%d.csv
An existing listing file is not replaced,
unless the following flag is set::
! renew table?
<rcbase>.renew : True
Orbit files are searched within a timerange::
<rcbase>.timerange.start : 2018-06-01 00:00
<rcbase>.timerange.end : 2018-06-03 23:59
Specify filename filters to search for orbit files;
the patterns are relative to the basedir of the listing file,
and might contain templates for the time values.
Multiple patterns could be defined; if for a certain orbit number more than one
file is found, the first match is used.
This could be explored to create a listing that combines reprocessed data
with near-real-time data::
<rcbase>.patterns : RPRO/CH4/%Y/%m/S5p_*.nc \
OFFL/CH4/%Y/%m/S5p_*.nc
"""
def __init__(self, rcfile, rcbase="", env={}, indent=""):
"""
Convert data.
"""
# modules:
import os
import datetime
import glob
import collections
# tools:
import cso_file
# info ...
logging.info(indent + "")
logging.info(indent + "** create listing file")
logging.info(indent + "")
# init base object:
utopya.UtopyaRc.__init__(self, rcfile=rcfile, rcbase=rcbase, env=env)
# renew output?
renew = self.GetSetting("renew", totype="bool")
# table file to be written:
lst_file = self.GetSetting("file")
# evaluate current time:
lst_file = datetime.datetime.now().strftime(lst_file)
# create?
if (not os.path.isfile(lst_file)) or renew:
# info ..
logging.info(indent + "create %s ..." % lst_file)
# time range:
t1 = self.GetSetting("timerange.start", totype="datetime")
t2 = self.GetSetting("timerange.end", totype="datetime")
# info ...
tfmt = "%Y-%m-%d %H:%M"
logging.info(indent + " timerange: [%s,%s]" % (t1.strftime(tfmt), t2.strftime(tfmt)))
# base directory:
bdir = os.path.dirname(lst_file)
# create?
if len(bdir) > 0:
if not os.path.isdir(bdir):
os.makedirs(bdir)
# endif
# current directory?
if len(bdir) == 0:
bdir = "."
# info ...
logging.info(indent + " base directory: %s ..." % bdir)
# initiallize for (re)creation:
listing = cso_file.CSO_Listing(lst_file, indent=indent + " ")
# info ...
logging.info(indent + " cleanup records if necessary ...")
# remove entries that do not exist anymore:
listing.Cleanup(indent=indent + " ")
# filename pattern templates:
pattern_templates = self.GetSetting("patterns").split()
# collection of scanned patterns:
patterns = []
# loop over days:
t = t1
while t <= t2:
# loop over patterns:
for pattern_template in pattern_templates:
# expand time values:
pattern = t.strftime(pattern_template)
# skip if already scanned ...
if pattern in patterns:
continue
# store:
patterns.append(pattern)
# info ...
logging.info(indent + "scan %s ..." % pattern)
# list relative to basedir:
cwd = os.getcwd()
os.chdir(bdir)
fnames = glob.glob(pattern)
os.chdir(cwd)
# empty ?
if len(fnames) == 0:
logging.info(indent + " empty ..")
continue
# endif
# sort in place:
fnames.sort()
# loop over files:
for fname in fnames:
# absolute path:
filename = os.path.join(bdir, fname)
# already in table?
if fname in listing:
# info ...
logging.info(indent + " keep entry %s ..." % fname)
else:
# info ...
logging.info(indent + " add entry %s ..." % fname)
# Example filename:
# S5P_RPRO_L2__CH4____20180430T001851_20180430T020219_02818_01_010301_20190513T141133.nc
#
# Some products have incorrect product id (should be 10 characters):
# S5P_OFFL_L2__CHOCHO___20200101T005246_20200101T023416_11487_01_010000_20210128.nc
# The extracted product id is then truncated to 10 characters.
#
# basename:
bname, ext = os.path.splitext(os.path.basename(filename))
# extract:
try:
mission, processing, rest = bname.split("_", 2)
if rest.startswith("L2__CHOCHO__"):
product_id = rest[0:10]
(
start_time,
end_time,
orbit,
collection,
processor_version,
prod_time,
) = rest[13:].split("_")
else:
product_id = rest[0:10]
(
start_time,
end_time,
orbit,
collection,
processor_version,
prod_time,
) = rest[11:].split("_")
# endif
except:
logging.error("could not extract filename parts; expected format:")
logging.error(
" S5P_RPRO_L2__CH4____20180430T001851_20180430T020219_02818_01_010301_20190513T141133"
)
logging.error("found:")
logging.error(" %s" % bname)
raise
# endif
# fill data record:
data = collections.OrderedDict()
tfmt = "%Y%m%dT%H%M%S"
data["start_time"] = datetime.datetime.strptime(start_time, tfmt)
data["end_time"] = datetime.datetime.strptime(end_time, tfmt)
data["mission"] = mission
data["processing"] = processing
data["product_id"] = product_id
data["orbit"] = orbit
data["collection"] = collection
data["processor_version"] = processor_version
if len(prod_time) == 8:
data["processing_time"] = datetime.datetime.strptime(
prod_time, "%Y%m%d"
)
else:
data["processing_time"] = datetime.datetime.strptime(
prod_time, tfmt
)
# endif
# update record:
listing.UpdateRecord(fname, data, indent=indent + " ")
# endif # new record?
# endfor # filenames
# endfor # patterns
## testing ...
# break
# next
t = t + datetime.timedelta(0, 3600)
# endwhile
# save:
listing.Close(indent=indent + " ")
else:
# info ..
logging.info(indent + "keep %s ..." % lst_file)
# endif
# info ...
logging.info(indent + "")
logging.info(indent + "** end listing")
logging.info(indent + "")
# enddef __init__
# endclass CSO_S5p_Download_Listing
######################################################################## ########################################################################
### ###
### end ### end
......
py/cso_scihub.py deleted 100644 → 0
View file @ d6b2a4ee
This diff is collapsed.