#
# Changes
#
# 2022-09, Arjo Segers
#   Updated documentation.
#
# 2023-06, Arjo Segers
#   Use "pandas.concat()" instead of "df.append()" to avoid warnings.
#
# 2023-08, Arjo Segers
#   Updated logging messages.
#
# 2023-08, Arjo Segers
#   Reformatted using 'black'.
#

########################################################################
###
### help
###
########################################################################

"""
.. _cso-scihub:

*********************
``cso_scihub`` module
*********************

The :py:mod:`cso_scihub` module provides classes for accessing  data from the 
`Copernicus Open Access Hub <https://scihub.copernicus.eu/>`_.
Below first the data hub itself is described, followed by how the CSO
pre-processor could be used for batch download of a selection.


Copernicus Open Access Hub
==========================

The `Copernicus Open Access Hub <https://scihub.copernicus.eu/>`_ is the official
portal for Copernicus satellite data.

.. figure:: figs/sci-hub.png
   :scale: 50 %
   :align: center
   :alt: Sci Hub main page.
   
   *Home page of Copernicus Open Access Hub* (https://scihub.copernicus.eu/)

Different hubs are provided for different data sets.
Below we describe:

* **Open Hub** Provides access to Sentinel-1,2,3 data.
* **S-5P Pre-Ops** Provides access to (pre-operational) Sentinel-5P data.


.. _SciHub-OpenHub:

Open Hub
--------

The *Open Hub* provides access to Sentinel-1,2,3 data.
Since this is the oldest data, most examples in the User Guide refer to this hub.

To download data it is necessary to register.
On the main page, select:

* `User Guide <https://scihub.copernicus.eu/userguide/>`_, 
  follow instructions on `Self Registration <https://scihub.copernicus.eu/userguide/>`_ .

The user name and password should be stored in your home directory in the ``~/.netrc`` file
(ensure that it has read/write permissions only for you)::

    # access Copernicus Open Access Hub
    machine scihub.copernicus.eu    login **********   password *********


S-5P Pre-Ops Hub
----------------

On the main page, select *'S-5P Pre-Ops'* for the
`Sentinel-5P Pre-Operations Data Hub <https://s5phub.copernicus.eu/dhus/>`_ .

Access is open, the login/password are shown in the login pop-up.
The user name and password should be stored in your home directory in the ``~/.netrc`` file
(ensure that it has read/write permissions only for you)::

    # access Copernicus Open Access Hub
    machine s5phub.copernicus.eu    login s5pguest     password s5pguest

Data can be selected and downloaded interactively.
In the search bar, open the *'Advanced Search'* menu and specify a selection.
The figure below shows an example for Level2 NO2 data.


.. figure:: figs/s5p-hub-advanced-search.png
   :scale: 75 %
   :align: center
   :alt: Advanced Search menu at S-5P Hub.
   
   *Advanced Search menu at S-5P Hub.*

The result of a search is a list of product files.
Each product file contains data from one orbit.
The name of a product file contains the scan period, the orbit number,
and a processing time::

  S5P_OFFL_L2__NO2____20190701T102357_20190701T120527_08882_01_010302_20190707T120219
  |   |    \_prodid_/ \_start-time__/ \__end-time___/ orbit |  |      \__processed__/
  |   stream                                                |  processor
  mission                                                   collection

The search query is shown above the product list, and contains the time 
and product selections; the later could be useful for batch download::

    ( beginPosition:[2019-07-01T00:00:00.000Z TO 2019-07-01T23:59:59.999Z] AND
        endPosition:[2019-07-01T00:00:00.000Z TO 2019-07-01T23:59:59.999Z]     )
    AND ( (    platformname:Sentinel-5  AND 
                producttype:L2__NO2___  AND 
            processinglevel:L2          AND 
             processingmode:Offline         ) )



.. _scihub-opensearch-download:

OpenSearch API
--------------

For batch processing one can use the 
`OpenSearch API <https://scihub.copernicus.eu/userguide/OpenSearchAPI>`_ .
First a search query is send to the server, and the result then contains links that can be used to download 
selected files.


    
.. _scihub-batch-download:

Batch download
--------------

An alternative for batch processing is to use the 
`API Hub <https://scihub.copernicus.eu/twiki/do/view/SciHubWebPortal/APIHubDescription>`_ .
This page also contains a link to the `section of the User Guide <https://scihub.copernicus.eu/userguide/8BatchScripting>`_  
with instructions for batch processing.

From the User Guide one can download the *'dhusget.sh'* script.
This will take care of searching the archive, downloading files, checking if the download
is complete, etc. 
The call to download all NO2 data for a single day that overlaps with a longitude/latitude
box over Europe looks like::

  ./dhusget.sh \\
      -d 'https://s5phub.copernicus.eu/dhus' \\
      -S '2018-07-01T00:00:00.000Z' \\
      -E '2018-07-02T00:00:00.000Z' \\
      -c '-30,30:45,76' \\
      -F 'platformname:Sentinel-5 AND producttype:L2__NO2___ AND processinglevel:L2 AND processingmode:Reprocessing' \\
      -o product \\
      -O /work/Sentinel-5P/TROPOMI/NO2/ \\
      -D

The download script from the API HUb is included in the CSO package,
with a few minor modifications:

  `bin/dhusget.sh <../../../bin/dhusget.sh>`_ 


Batch download via CSO
======================

The CSO download tools use by default the :ref:`scihub-opensearch-download`,
but could also use the :ref:`scihub-batch-download` option.

The :py:class:`.CSO_SciHub_Download` class performs the download using the OpenSearch API.
This class is prefered since:

* existing files could be kept without a new download;
* error messages are cleaner.

See the documentation of the class for the settings to be used.

Alternatively, the :py:class:`.CSO_SciHub_Download_DHuS` class could be used
which will call the ``dhusget.sh`` script to perform the download.
This will give a lot of intermediate log files, and error messages that are not easily interpreted.


Class hierchy
=============

The classes and are defined according to the following hierchy:


* :py:class:`.CSO_SciHub_DownloadFile`
* :py:class:`.UtopyaRc`

  * :py:class:`.CSO_SciHub_Inquire`
  * :py:class:`.CSO_SciHub_DownloadFile`
  * :py:class:`.CSO_SciHub_Download`



Classes
=======


"""


########################################################################
###
### modules
###
########################################################################

# modules:
import logging

# tools:
import utopya


########################################################################
###
### OpenSearch inquire
###
########################################################################


class CSO_SciHub_Inquire(utopya.UtopyaRc):

    """
    Inquire available Sentinel data from the `Copernicus Open Access Hub <https://scihub.copernicus.eu/>`_
    using the OpenSearch API:
    
    * `Copernicus Open Access Hub <https://scihub.copernicus.eu/>`_

      * `User Guide <https://scihub.copernicus.eu/userguide/>`_
  
        * `6 OpenSearch API <https://scihub.copernicus.eu/userguide/OpenSearchAPI>`_
      
    A query is sent to search for products that are available 
    for a certain time and overlap with a specified region. 
    The result is a list with orbit files and instructions on how to download them.
    
    In the settings, specify the url of the hub, which is either the Open Data hub or the S5-P hub::

      ! server url; provide login/password in ~/.netrc:
      <rcbase>.url              :  https://s5phub.copernicus.eu/dhus

    Specify the time range over which files should be downloaded::

      <rcbase>.timerange.start  :  2018-07-01 00:00
      <rcbase>.timerange.end    :  2018-07-01 23:59

    Specify a target area, only orbits with some pixels within the defined box will be downloaded::

      ! target area, leave empty for globe; format:  west,south:east,north
      <rcbase>.area             :  
      !<rcbase>.area             :  -30,30:35,76

    A query is used to select the required data.
    The search box on the hub could be used for inspiration on the format.
    Note that the '``producttype``' should have exactly 10 characters, 
    with the first 3 used for the retrieval level, and the last 6 for the product; 
    empty characters should have an underscore instead::

      ! search query, obtained from interactive download:
      <rcbase>.query            :      platformname:Sentinel-5    AND \\
                                        producttype:L2__NO2___    AND \\
                                    processinglevel:L2

    Name of ouptut csv file::
    
        ! output table, date of today:
        <rcbase>.output.file            :  ${my.work}/Copernicus_S5P_NO2_%Y-%m-%d.csv

    """

    def __init__(self, rcfile, rcbase="", env={}, indent=""):

        """
        Download orbits using OpenSearch API.
        """

        # modules:
        import sys
        import os
        import datetime
        import requests
        import xml.etree.ElementTree
        import pandas

        # info ...
        logging.info(indent + "")
        logging.info(indent + "** Inquire files available on Copernicus Data Hub")
        logging.info(indent + "")

        # init base object:
        utopya.UtopyaRc.__init__(self, rcfile=rcfile, rcbase=rcbase, env=env)

        # domain:
        url = self.GetSetting("url")
        # info ...
        logging.info(indent + "url          : %s" % url)

        # area of interest: west,south:east,north
        area = self.GetSetting("area")
        # defined?
        if len(area) > 0:
            # convert from format for "dhusget.sh":
            #    west,south:east,north
            west, south, east, north = map(float, area.replace(":", " ").replace(",", " ").split())
        else:
            # globe:
            west, south, east, north = -180, -90, 180, 90
        # endif
        # info ...
        logging.info(
            indent + "area         : [%8.2f,%8.2f] x [%8.2f,%8.2f]" % (west, east, south, north)
        )

        # query, probably obtained from interactive download page:
        product = self.GetSetting("query")
        # remove excesive whitespace:
        product = " ".join(product.split())
        # info ...
        logging.info(indent + "product      : %s" % product)

        # target file, might include time templates:
        output_file__template = self.GetSetting("output.file")
        # current time:
        output_file = datetime.datetime.now().strftime(output_file__template)

        # new output table:
        output_df = pandas.DataFrame()

        # time range:
        t1 = self.GetSetting("timerange.start", totype="datetime")
        t2 = self.GetSetting("timerange.end", totype="datetime")
        # info ...
        tfmt = "%Y-%m-%d %H:%M"
        logging.info(indent + "timerange: [%s,%s]" % (t1.strftime(tfmt), t2.strftime(tfmt)))

        # targefile:
        qfile = "query-%i.xml" % os.getpid()

        # timeout of requests in seconds:
        timeout = 60

        # search query could only return a maximum number of records;
        # a 'page' of records is requested using a row offet and the number of rows:
        row0 = 0
        nrow = 100  # maximum allowed is 100 ...

        # time labels:
        tfmt = "%Y-%m-%dT%H:%M:%SZ"
        # time of interest:
        toi = "beginPosition:[%s TO %s]" % (t1.strftime(tfmt), t2.strftime(tfmt))

        # geographic area:
        #   'POLYGON((P1Lon P1Lat, P2Lon P2Lat, ..., PnLon PnLat))'
        area = "POLYGON((%f %f, %f %f, %f %f, %f %f, %f %f))" % (
            west,
            south,
            east,
            south,
            east,
            north,
            west,
            north,
            west,
            south,
        )
        # area of interest:
        aoi = 'footprint:"Intersects(%s)"' % area

        # info ..
        logging.info(indent + "  time of interest: %s" % toi)
        logging.info(
            indent
            + "  area of interest: [%8.2f,%8.2f] x [%8.2f,%8.2f]" % (west, east, south, north)
        )
        logging.info(indent + "  product         : %s" % product)

        # combine:
        query = "%s AND %s AND (%s)" % (toi, product, aoi)

        # init counter:
        ipage = 0

        # loop over pages of query result:
        while True:

            # increase counter:
            ipage += 1
            # info ...
            logging.info(indent + "  page %i (entries %i,..,%i)" % (ipage, row0 + 1, row0 + nrow))

            # send query to search page;
            # result is an xml tree with entries for matching orbit files;
            # a maximum of 'nrow' entries is returned, from a zero-based 'start' index onwards:
            r = requests.get(
                os.path.join(url, "search"),
                params=dict(q=query, start=row0, rows=nrow),
                timeout=timeout,
            )
            # check status, raise error if request failed:
            try:
                r.raise_for_status()
            except requests.exceptions.HTTPError as err:
                msg = str(err)
                logging.error("from query; message received:")
                logging.error("  %s" % msg)
                if msg.startswith("401 Client Error: Unauthorized for url:"):
                    logging.error(
                        'Interpretation: the (username,password) received from your "~/.netrc" file is incorrect.'
                    )
                    logging.error("For the S5p-hub, the file should contain the following entry:")
                    logging.error(
                        "  machine s5phub.copernicus.eu      login s5pguest        password s5pguest"
                    )
                    # get authorization as test:
                    try:
                        username, password = requests.utils.get_netrc_auth(url, raise_errors=True)
                        logging.error('The username extracted from "~/.netrc" for this url is:')
                        logging.error("  %s" % username)
                        logging.error("Maybe this is the default entry?")
                    except Exception as err:
                        logging.error(
                            'The "~/.netrc" file could not be parsed correctly; the error raised from:'
                        )
                        logging.error(
                            '  username,password = requests.utils.get_netrc_auth( "%s", raise_errors=True )'
                            % url
                        )
                        logging.error("is:")
                        logging.error("  %s" % str(err))
                    # endtry
                # endif
            except Exception as err:
                msg = str(err)
                logging.error("from query; message received:")
                logging.error("  %s" % msg)
                sys.exit(1)
            # endtry

            # save:
            with open(qfile, "w") as f:
                f.write(r.text)
            # endwith

            #
            # Example content:
            #
            #    <?xml version="1.0" encoding="utf-8"?>
            #    <feed xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/" xmlns="http://www.w3.org/2005/Atom">
            #      <title>Data Hub Service search results for: beginPosition:[2018-07-01T00:00:00Z TO 2018-07-02T00:00:00Z] AND platformname:Sentinel-5 AND producttype:L2__NO2___ AND processinglevel:L2 AND processingmode:Reprocessing AND (footprint:"Intersects(POLYGON((-30.000000 30.000000, 45.000000 30.000000, 45.000000 76.000000, -30.000000 76.000000, -30.000000 30.000000)))")</title>
            #      ...
            #      <entry>
            #        <title>S5P_RPRO_L2__NO2____20180701T024001_20180701T042329_03699_01_010202_20190211T185158</title>
            #        <link href="https://s5phub.copernicus.eu/dhus/odata/v1/Products('a66d4240-4999-4724-9903-aa2db410bbad')/$value"/>
            #        <str name="filename">S5P_RPRO_L2__NO2____20180701T024001_20180701T042329_03699_01_010202_20190211T185158.nc</str>
            #        <str name="uuid"    >a66d4240-4999-4724-9903-aa2db410bbad</str>
            #        ...
            #      </entry>
            #      ...
            #   </feed>
            #

            # open query result:
            tree = xml.etree.ElementTree.parse(qfile)
            # get root element ("feed"):
            root = tree.getroot()

            # the tag contains an url prefix:
            #   root.tag = "{http://www.w3.org/2005/Atom}feed"
            # prefixed?
            if root.tag.startswith("{"):
                # extract prefix:
                prefix, tag = root.tag[1:].split("}")
            else:
                # no prefix:
                prefix = ""
            # endif
            # create namespace to translate the "atom:" to the prefix:
            ns = {"atom": prefix}

            # find "entry" elements:
            entries = root.findall("atom:entry", namespaces=ns)

            # count:
            nrec = len(entries)
            # info ..
            logging.info(indent + "    number of records found: %i" % nrec)
            # check ..
            if (ipage == 1) and (nrec == 0):
                logging.warning(
                    "      no records found for this day; something wrong in query? continue with next day ..."
                )
                break
            # endif

            # loop over entries:
            for entry in entries:

                # get target filename, this is the content of the "str" element
                # with attribute "name" equal to "filename":
                #        <str name="filename">S5P_RPRO_L2__NO2____20180701T024001_20180701T042329_03699_01_010202_20190211T185158.nc</str>
                filename = entry.find('atom:str[@name="filename"]', ns).text

                # info ...
                logging.info(indent + "    file : %s" % filename)

                #
                # filenames:
                #
                #    S5P_OFFL_L2__NO2____20180701T005930_20180701T024100_03698_01_010002_20180707T022838.nc
                #    plt proc [product-] [starttime....] [endtime......] orbit cl prvers [prodtime.....]
                #
                bname = os.path.basename(filename).replace(".nc", "")
                # split:
                platform_name, processing, rest = bname.split("_", 2)
                product_type = rest[0:10]
                start_time, end_time, orbit, collection, processor_version, production_time = rest[
                    11:
                ].split("_")

                # convert:
                tfmt = "%Y%m%dT%H%M%S"
                ts = datetime.datetime.strptime(start_time, tfmt)
                te = datetime.datetime.strptime(end_time, tfmt)

                # find first "link"  element:
                link = entry.find("atom:link", ns)
                # extract download link:
                href = link.attrib["href"]

                # row:
                rec = {
                    "orbit": [orbit],
                    "start_time": [ts],
                    "end_time": [te],
                    "processing": [processing],
                    "collection": [collection],
                    "processor_version": [processor_version],
                    "filename": [filename],
                    "href": [href],
                }
                # add record:
                output_df = pandas.concat((output_df, pandas.DataFrame(rec)), ignore_index=True)

            # endfor # entries

            # cleanup:
            os.remove(qfile)

            # leave loop over pages?
            if nrec < nrow:
                break
            # increse offset:
            row0 += nrow

        # endwhile # pages

        # info ..
        logging.info("save to: %s ..." % output_file)
        # create directory:
        dirname = os.path.dirname(output_file)
        if len(dirname) > 0:
            if not os.path.isdir(dirname):
                os.makedirs(dirname)
        # endif
        # write:
        output_df.to_csv(output_file, sep=";", index=False)

        # info ...
        logging.info(indent + "")
        logging.info(indent + "** end SciHub inquire")
        logging.info(indent + "")

    # enddef __init__


# endclass CSO_SciHub_Inquire


########################################################################
###
### OpenSearch download
###
########################################################################


class CSO_SciHub_DownloadFile(object):

    """
    Download single file from SciHub.
    
    Arguments:

    * ``href`` : download url: ``https://s5phub.copernicus.eu/dhus/odata/v1/Products('d483baa0-3a61-4985-aa0c-5642a83c9214')/$value``
    * ``output_file`` : target file
    
    Optional arguments:
    
    * ``maxtry`` : number of times to try again if download fails
    * ``timeout`` : delay in seconds between requests
    """

    def __init__(self, href, output_file, maxtry=10, timeout=60, indent=""):

        """
        Download file.
        """

        # modules:
        import os
        import requests

        # retry loop ..
        ntry = 0
        while True:
            # try to download and save:
            try:

                # try to download:
                try:

                    # download:
                    r = requests.get(href, timeout=timeout)
                    # check status, raise error if request failed:
                    r.raise_for_status()

                    # info ..
                    logging.info(indent + "      write to: %s ..." % os.path.dirname(output_file))
                    # target dir:
                    dname = os.path.dirname(output_file)
                    if len(dname) > 0:
                        if not os.path.isdir(dname):
                            os.makedirs(dname)
                    # endif
                    # write to temporary target first ..
                    tmpfile = output_file + ".tmp"
                    # open destination file for binary write:
                    with open(tmpfile, "wb") as fd:
                        # prefered way to write content following:
                        #   https://docs.python-requests.org/en/master/user/quickstart/
                        for chunk in r.iter_content(chunk_size=128):
                            fd.write(chunk)
                        # endfor
                    # endwith
                    # rename:
                    os.rename(tmpfile, output_file)

                except requests.exceptions.HTTPError as err:
                    # info ..
                    msg = str(err)
                    logging.error("exception from download; message received:")
                    logging.error("  %s" % msg)
                    # catch known problem ...
                    if msg.startswith("401 Client Error: Unauthorized for url:"):
                        logging.error(
                            'Interpretation: the (username,password) received from your "~/.netrc" file is incorrect.'
                        )
                        logging.error(
                            "For the S5p-hub, the file should contain the following entry:"
                        )
                        logging.error(
                            "  machine s5phub.copernicus.eu      login s5pguest        password s5pguest"
                        )
                        # try to get authorization as test:
                        try:
                            username, password = requests.utils.get_netrc_auth(
                                url, raise_errors=True
                            )
                            logging.error(
                                'The username extracted from "~/.netrc" for this url is:'
                            )
                            logging.error("  %s" % username)
                            logging.error("Maybe this is the default entry?")
                        except Exception as err:
                            logging.error(
                                'The "~/.netrc" file could not be parsed correctly; the error raised from:'
                            )
                            logging.error(
                                '  username,password = requests.utils.get_netrc_auth( "%s", raise_errors=True )'
                                % url
                            )
                            logging.error("is:")
                            logging.error("  %s" % str(err))
                        # endtry
                        # should be solved first, leave retry loop ...
                        break
                    # endif

                except MemoryError as err:
                    logging.error(
                        "memory error from download; need to increase allocated resources?"
                    )
                    # quit with error:
                    raise

                except Exception as err:
                    # info ..
                    logging.error("from download; message received:")
                    logging.error("  %s" % str(err))
                    # quit with error:
                    raise

                # endtry

            # error from download or save:
            except:
                # increase counter:
                ntry += 1
                # switch:
                if ntry == maxtry:
                    logging.warning(
                        indent + "      exception from download; tried %i times ..." % maxtry
                    )
                    raise Exception
                else:
                    logging.warning(indent + "      exception from download; try again ...")
                    continue  # while-loop
                # endif
            # endtry
            # leave retry loop,
            # either because download was ok,
            # or because maximum number of retries was reached:
            break

        # endwhile # retry

    # enddef __init__


# endclass CSO_SciHub_DownloadFile

# *


class CSO_SciHub_Download(utopya.UtopyaRc):

    """
    Download Sentinel data from the `Copernicus Open Access Hub <https://scihub.copernicus.eu/>`_
    using the OpenSearch API:
    
    * `Copernicus Open Access Hub <https://scihub.copernicus.eu/>`_

      * `User Guide <https://scihub.copernicus.eu/userguide/>`_
  
        * `6 OpenSearch API <https://scihub.copernicus.eu/userguide/OpenSearchAPI>`_
      
    To download orbit files, first a query is sent to search for products that are available 
    for a certain time and overlap with a specified region. 
    The result is a list with orbit files and instructions on how to download them.
    
    In the settings, specify the url of the hub, which is either the Open Data hub or the S5-P hub::

      ! server url; provide login/password in ~/.netrc:
      <rcbase>.url              :  https://s5phub.copernicus.eu/dhus

    Specify the time range over which files should be downloaded::

      <rcbase>.timerange.start  :  2018-07-01 00:00
      <rcbase>.timerange.end    :  2018-07-01 23:59

    Specify a target area, only orbits with some pixels within the defined box will be downloaded::

      ! target area, leave empty for globe; format:  west,south:east,north
      <rcbase>.area             :  
      !<rcbase>.area             :  -30,30:35,76

    A query is used to select the required data.
    The search box on the hub could be used for inspiration on the format.
    Note that the '``producttype``' should have exactly 10 characters, 
    with the first 3 used for the retrieval level, and the last 6 for the product; 
    empty characters should have an underscore instead::

      ! search query, obtained from interactive download:
      <rcbase>.query            :      platformname:Sentinel-5    AND \\
                                        producttype:L2__NO2___    AND \\
                                    processinglevel:L2            AND \\
                                     processingmode:Offline

    The target directory for downloaded file could include templates for time values::

      ! output archive, store per month:
      <rcbase>.output.dir       :  /data/Copernicus/S5P/OFFL/NO2/%Y/%m
      
    Use the following flag to keep files that are already present::
    
      ! renew existing files?
      <rcbase.renew             :  False

    """

    def __init__(self, rcfile, rcbase="", env={}, indent=""):

        """
        Download orbits using OpenSearch API.
        """

        # modules:
        import sys
        import os
        import datetime
        import requests
        import xml.etree.ElementTree

        # info ...
        logging.info(indent + "")
        logging.info(indent + "** Download from Copernicus Data Hub")
        logging.info(indent + "")

        # init base object:
        utopya.UtopyaRc.__init__(self, rcfile=rcfile, rcbase=rcbase, env=env)

        # domain:
        url = self.GetSetting("url")
        # info ...
        logging.info(indent + "url          : %s" % url)

        # area of interest: west,south:east,north
        area = self.GetSetting("area")
        # convert from format for "dhusget.sh":
        #    west,south:east,north
        west, south, east, north = map(float, area.replace(":", " ").replace(",", " ").split())
        # info ...
        logging.info(
            indent + "area         : [%8.2f,%8.2f] x [%8.2f,%8.2f]" % (west, east, south, north)
        )

        # query, probably obtained from interactive download page:
        product = self.GetSetting("query")
        # remove excesive whitespace:
        product = " ".join(product.split())
        # info ...
        logging.info(indent + "product      : %s" % product)

        # output dir, might contain time templates:
        output_dir__template = self.GetSetting("output.dir")

        # renew existing files?
        renew = self.GetSetting("renew", totype="bool")

        # time range:
        t1 = self.GetSetting("timerange.start", totype="datetime")
        t2 = self.GetSetting("timerange.end", totype="datetime")
        # info ...
        tfmt = "%Y-%m-%d %H:%M"
        logging.info(indent + "timerange: [%s,%s]" % (t1.strftime(tfmt), t2.strftime(tfmt)))

        # targefile:
        qfile = "query-%i.xml" % os.getpid()

        # timeout of requests in seconds:
        timeout = 60

        # search query could only return a maximum number of records;
        # a certain 'page' of records is requesing a row offet and number:
        row0 = 0
        nrow = 100  # maximum allowed is 100 ...

        # number of retries:
        maxtry = 10

        # loop over days:
        t = t1
        while t < t2:

            # info ...
            logging.info(indent + "%s ..." % t.strftime("%Y-%m-%d"))

            # start and end time:
            ts = t
            # add 24h:
            tx = t + datetime.timedelta(1)
            # convert to midnight:
            tx24 = datetime.datetime(tx.year, tx.month, tx.day, 0, 0)
            # not after t2 ...
            te = min(tx24, t2)

            # time labels:
            tfmt = "%Y-%m-%dT%H:%M:%SZ"
            # time of interest:
            toi = "beginPosition:[%s TO %s]" % (ts.strftime(tfmt), te.strftime(tfmt))

            # geographic area:
            #   'POLYGON((P1Lon P1Lat, P2Lon P2Lat, ..., PnLon PnLat))'
            area = "POLYGON((%f %f, %f %f, %f %f, %f %f, %f %f))" % (
                west,
                south,
                east,
                south,
                east,
                north,
                west,
                north,
                west,
                south,
            )
            # area of interest:
            aoi = 'footprint:"Intersects(%s)"' % area

            # info ..
            logging.info(indent + "  time of interest: %s" % toi)
            logging.info(
                indent
                + "  area of interest: [%8.2f,%8.2f] x [%8.2f,%8.2f]" % (west, east, south, north)
            )
            logging.info(indent + "  product         : %s" % product)

            # combine:
            query = "%s AND %s AND (%s)" % (toi, product, aoi)

            # init counter:
            ipage = 0

            # current:
            output_dir = t.strftime(output_dir__template)
            # info ..
            logging.info(indent + "  output directory: %s" % output_dir)
            # create?
            if not os.path.isdir(output_dir):
                os.makedirs(output_dir)

            # loop over pages of query result:
            while True:

                # increase counter:
                ipage += 1
                # info ...
                logging.info(
                    indent + "  page %i (entries %i,..,%i)" % (ipage, row0 + 1, row0 + nrow)
                )

                # send query to search page;
                # result is an xml tree with entries for matching orbit files;
                # a maximum of 'nrow' entries is returned, from a zero-based 'start' index onwards:
                r = requests.get(
                    os.path.join(url, "search"),
                    params=dict(q=query, start=row0, rows=nrow),
                    timeout=timeout,
                )
                # check status, raise error if request failed:
                try:
                    r.raise_for_status()
                except requests.exceptions.HTTPError as err:
                    msg = str(err)
                    logging.error("from query; message received:")
                    logging.error("  %s" % msg)
                    if msg.startswith("401 Client Error: Unauthorized for url:"):
                        logging.error(
                            'Interpretation: the (username,password) received from your "~/.netrc" file is incorrect.'
                        )
                        logging.error(
                            "For the S5p-hub, the file should contain the following entry:"
                        )
                        logging.error(
                            "  machine s5phub.copernicus.eu      login s5pguest        password s5pguest"
                        )
                        # get authorization as test:
                        try:
                            username, password = requests.utils.get_netrc_auth(
                                url, raise_errors=True
                            )
                            logging.error(
                                'The username extracted from "~/.netrc" for this url is:'
                            )
                            logging.error("  %s" % username)
                            logging.error("Maybe this is the default entry?")
                        except Exception as err:
                            logging.error(
                                'The "~/.netrc" file could not be parsed correctly; the error raised from:'
                            )
                            logging.error(
                                '  username,password = requests.utils.get_netrc_auth( "%s", raise_errors=True )'
                                % url
                            )
                            logging.error("is:")
                            logging.error("  %s" % str(err))
                        # endtry
                    # endif
                except Exception as err:
                    msg = str(err)
                    logging.error("from query; message received:")
                    logging.error("  %s" % msg)
                    sys.exit(1)
                # endtry

                # save:
                with open(qfile, "w") as f:
                    f.write(r.text)
                # endwith

                #
                # Example content:
                #
                #    <?xml version="1.0" encoding="utf-8"?>
                #    <feed xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/" xmlns="http://www.w3.org/2005/Atom">
                #      <title>Data Hub Service search results for: beginPosition:[2018-07-01T00:00:00Z TO 2018-07-02T00:00:00Z] AND platformname:Sentinel-5 AND producttype:L2__NO2___ AND processinglevel:L2 AND processingmode:Reprocessing AND (footprint:"Intersects(POLYGON((-30.000000 30.000000, 45.000000 30.000000, 45.000000 76.000000, -30.000000 76.000000, -30.000000 30.000000)))")</title>
                #      ...
                #      <entry>
                #        <title>S5P_RPRO_L2__NO2____20180701T024001_20180701T042329_03699_01_010202_20190211T185158</title>
                #        <link href="https://s5phub.copernicus.eu/dhus/odata/v1/Products('a66d4240-4999-4724-9903-aa2db410bbad')/$value"/>
                #        <str name="filename">S5P_RPRO_L2__NO2____20180701T024001_20180701T042329_03699_01_010202_20190211T185158.nc</str>
                #        <str name="uuid"    >a66d4240-4999-4724-9903-aa2db410bbad</str>
                #        ...
                #      </entry>
                #      ...
                #   </feed>
                #

                # open query result:
                tree = xml.etree.ElementTree.parse(qfile)
                # get root element ("feed"):
                root = tree.getroot()

                # the tag contains an url prefix:
                #   root.tag = "{http://www.w3.org/2005/Atom}feed"
                # prefixed?
                if root.tag.startswith("{"):
                    # extract prefix:
                    prefix, tag = root.tag[1:].split("}")
                else:
                    # no prefix:
                    prefix = ""
                # endif
                # create namespace to translate the "atom:" to the prefix:
                ns = {"atom": prefix}

                # find "entry" elements:
                entries = root.findall("atom:entry", namespaces=ns)

                # count:
                nrec = len(entries)
                # info ..
                logging.info(indent + "    number of records found: %i" % nrec)
                # check ..
                if (ipage == 1) and (nrec == 0):
                    logging.warning(
                        "      no records found for this day; something wrong in query? continue with next day ..."
                    )
                    break
                # endif

                # loop over entries:
                for entry in entries:

                    # get target filename, this is the content of the "str" element
                    # with attribute "name" equal to "filename":
                    #        <str name="filename">S5P_RPRO_L2__NO2____20180701T024001_20180701T042329_03699_01_010202_20190211T185158.nc</str>
                    filename = entry.find('atom:str[@name="filename"]', ns).text

                    # full path:
                    output_file = os.path.join(output_dir, filename)

                    # info ...
                    logging.info(indent + "    file : %s" % filename)
                    # (re)new?
                    if (not os.path.isfile(output_file)) or renew:

                        # find first "link"  element:
                        link = entry.find("atom:link", ns)
                        # extract download link:
                        href = link.attrib["href"]
                        # info ..
                        logging.info(indent + "      download: %s" % href)

                        # sinlge file:
                        CSO_SciHub_DownloadFile(
                            href, output_file, maxtry=maxtry, timeout=timeout, indent=indent
                        )

                    else:
                        # info ..
                        logging.info(indent + "      keep in: %s ..." % output_dir)
                    # endif

                    ## testing ...
                    # break

                # endfor # entries

                # cleanup:
                os.remove(qfile)

                # leave loop over pages?
                if nrec < nrow:
                    break
                # increse offset:
                row0 += nrow

            # endwhile # pages

            # start of next day:
            t = te
        # endwhile # day loop

        # info ...
        logging.info(indent + "")
        logging.info(indent + "** end SciHub Download")
        logging.info(indent + "")

    # enddef __init__


# endclass CSO_SciHub_Download


########################################################################
###
### create listing file
###
########################################################################


class CSO_SciHub_Listing(utopya.UtopyaRc):

    """
    Create *listing* file for files download from SciHub or other portals that use equivalent filenames.
    
    A *listing* file contains the names of the converted orbit files,
    the time range of pixels in the file, and other information extracted from the filenames:

        filename                                                                                               ;mission;processing;product_id;start_time         ;end_time           ;orbit;collection;processor_version;processing_time
        RPRO/CH4/2018/04/S5P_RPRO_L2__CH4____20180430T001851_20180430T020219_02818_01_010301_20190513T141133.nc;S5P    ;RPRO      ;L2__CH4___;2018-04-30T00:18:51;2018-04-30T02:02:19;02818;01        ;010301           ;2019-05-13T14:11:33
        RPRO/CH4/2018/04/S5P_RPRO_L2__CH4____20180430T020021_20180430T034349_02819_01_010301_20190513T135953.nc;S5P    ;RPRO      ;L2__CH4___;2018-04-30T02:00:21;2018-04-30T03:43:49;02819;01        ;010301           ;2019-05-13T13:59:53
        :

    This file could be used to scan for available versions and how they were produced.
    
    In the settings, define the name of the file to be created::

        ! csv file that will hold records per file with:
        ! - timerange of pixels in file
        ! - orbit number
        ! time templates are replaced with todays date
        <rcbase>.file        :  /Scratch/Copernicus/S5p/listing-CH4__%Y-%m-%d.csv
        
    An existing listing file is not replaced,
    unless the following flag is set::

        ! renew table?
        <rcbase>.renew           :  True
        
    Orbit files are searched within a timerange::
    
        <rcbase>.timerange.start        :  2018-06-01 00:00
        <rcbase>.timerange.end          :  2018-06-03 23:59

    Specify filename filters to search for orbit files;
    the patterns are relative to the basedir of the listing file,
    and might contain templates for the time values.
    Multiple patterns could be defined; if for a certain orbit number more than one
    file is found, the first match is used.
    This could be explored to create a listing that combines reprocessed data
    with near-real-time data::

        <rcbase>.patterns            :  RPRO/CH4/%Y/%m/S5p_*.nc \
                                        OFFL/CH4/%Y/%m/S5p_*.nc

    """

    def __init__(self, rcfile, rcbase="", env={}, indent=""):

        """
        Convert data.
        """

        # modules:
        import os
        import datetime
        import glob
        import collections

        # tools:
        import cso_file

        # info ...
        logging.info(indent + "")
        logging.info(indent + "** create listing file")
        logging.info(indent + "")

        # init base object:
        utopya.UtopyaRc.__init__(self, rcfile=rcfile, rcbase=rcbase, env=env)

        # renew output?
        renew = self.GetSetting("renew", totype="bool")

        # table file to be written:
        lst_file = self.GetSetting("file")
        # evaluate current time:
        lst_file = datetime.datetime.now().strftime(lst_file)

        # create?
        if (not os.path.isfile(lst_file)) or renew:
            # info ..
            logging.info(indent + "create %s ..." % lst_file)

            # time range:
            t1 = self.GetSetting("timerange.start", totype="datetime")
            t2 = self.GetSetting("timerange.end", totype="datetime")
            # info ...
            tfmt = "%Y-%m-%d %H:%M"
            logging.info(indent + "  timerange: [%s,%s]" % (t1.strftime(tfmt), t2.strftime(tfmt)))

            # base directory:
            bdir = os.path.dirname(lst_file)
            # create?
            if len(bdir) > 0:
                if not os.path.isdir(bdir):
                    os.makedirs(bdir)
            # endif
            # current directory?
            if len(bdir) == 0:
                bdir = "."
            # info ...
            logging.info(indent + "  base directory: %s ..." % bdir)

            # initiallize for (re)creation:
            listing = cso_file.CSO_Listing(lst_file, indent=indent + "    ")

            # info ...
            logging.info(indent + "  cleanup records if necessary ...")
            # remove entries that do not exist anymore:
            listing.Cleanup(indent=indent + "    ")

            # filename pattern templates:
            pattern_templates = self.GetSetting("patterns").split()
            # collection of scanned patterns:
            patterns = []

            # loop over days:
            t = t1
            while t <= t2:

                # loop over patterns:
                for pattern_template in pattern_templates:
                    # expand time values:
                    pattern = t.strftime(pattern_template)
                    # skip if already scanned ...
                    if pattern in patterns:
                        continue
                    # store:
                    patterns.append(pattern)
                    # info ...
                    logging.info(indent + "scan %s ..." % pattern)

                    # list relative to basedir:
                    cwd = os.getcwd()
                    os.chdir(bdir)
                    fnames = glob.glob(pattern)
                    os.chdir(cwd)
                    # empty ?
                    if len(fnames) == 0:
                        logging.info(indent + "  empty ..")
                        continue
                    # endif
                    # sort in place:
                    fnames.sort()
                    # loop over files:
                    for fname in fnames:

                        # absolute path:
                        filename = os.path.join(bdir, fname)
                        # already in table?
                        if fname in listing:
                            # info ...
                            logging.info(indent + "  keep entry %s ..." % fname)
                        else:
                            # info ...
                            logging.info(indent + "  add entry %s ..." % fname)
                            # Example filename:
                            #   S5P_RPRO_L2__CH4____20180430T001851_20180430T020219_02818_01_010301_20190513T141133.nc
                            #
                            # Some products have incorrect product id (should be 10 characters):
                            #   S5P_OFFL_L2__CHOCHO___20200101T005246_20200101T023416_11487_01_010000_20210128.nc
                            # The extracted product id is then truncated to 10 characters.
                            #
                            # basename:
                            bname, ext = os.path.splitext(os.path.basename(filename))
                            # extract:
                            try:
                                mission, processing, rest = bname.split("_", 2)
                                if rest.startswith("L2__CHOCHO__"):
                                    product_id = rest[0:10]
                                    (
                                        start_time,
                                        end_time,
                                        orbit,
                                        collection,
                                        processor_version,
                                        prod_time,
                                    ) = rest[13:].split("_")
                                else:
                                    product_id = rest[0:10]
                                    (
                                        start_time,
                                        end_time,
                                        orbit,
                                        collection,
                                        processor_version,
                                        prod_time,
                                    ) = rest[11:].split("_")
                                # endif
                            except:
                                logging.error("could not extract filename parts; expected format:")
                                logging.error(
                                    "  S5P_RPRO_L2__CH4____20180430T001851_20180430T020219_02818_01_010301_20190513T141133"
                                )
                                logging.error("found:")
                                logging.error("  %s" % bname)
                                raise
                            # endif
                            # fill data record:
                            data = collections.OrderedDict()
                            tfmt = "%Y%m%dT%H%M%S"
                            data["start_time"] = datetime.datetime.strptime(start_time, tfmt)
                            data["end_time"] = datetime.datetime.strptime(end_time, tfmt)
                            data["mission"] = mission
                            data["processing"] = processing
                            data["product_id"] = product_id
                            data["orbit"] = orbit
                            data["collection"] = collection
                            data["processor_version"] = processor_version
                            if len(prod_time) == 8:
                                data["processing_time"] = datetime.datetime.strptime(
                                    prod_time, "%Y%m%d"
                                )
                            else:
                                data["processing_time"] = datetime.datetime.strptime(
                                    prod_time, tfmt
                                )
                            # endif
                            # update record:
                            listing.UpdateRecord(fname, data, indent=indent + "    ")
                        # endif  # new record?

                    # endfor # filenames

                # endfor # patterns

                ## testing ...
                # break

                # next
                t = t + datetime.timedelta(0, 3600)
            # endwhile

            # save:
            listing.Close(indent=indent + "  ")

        else:
            # info ..
            logging.info(indent + "keep %s ..." % lst_file)
        # endif

        # info ...
        logging.info(indent + "")
        logging.info(indent + "** end listing")
        logging.info(indent + "")

    # enddef __init__


# endclass CSO_SciHub_Listing


#
# * plot listing
#


class CSO_SciHub_ListingPlot(utopya.UtopyaRc):

    """
    Create timeseries plot of number of orbits per processor version.
    Information taken from *listing* file created by :py:class:`CSO_SciHub_Listing` class.
    
    """

    def __init__(self, rcfile, rcbase="", env={}, indent=""):

        """
        Convert data.
        """

        # modules:
        import os
        import pandas
        import matplotlib.pyplot as plt
        import matplotlib.dates as mdates

        # info ...
        logging.info(indent + "")
        logging.info(indent + "** create listing plot")
        logging.info(indent + "")

        # init base object:
        utopya.UtopyaRc.__init__(self, rcfile=rcfile, rcbase=rcbase, env=env)

        # renew output?
        renew = self.GetSetting("renew", totype="bool")

        # table file to be written:
        lst_file = self.GetSetting("file")
        # info ..
        logging.info(indent + "listing file: %s" % lst_file)

        # target file:
        fig_file = lst_file.replace(".csv", ".png")
        # create?
        if (not os.path.isfile(fig_file)) or renew:

            # info ..
            logging.info(indent + "create %s ..." % fig_file)

            # frequency:
            # freq = 'MS'  ; freqlabel = 'month'
            freq = "W"
            freqlabel = "week"
            # freq = 'DS'  ; freqlabel = 'day'

            # color list:
            colors = ["purple", "blue", "cyan", "gold", "green", "red", "magenta", "brown"]

            # read:
            df = pandas.read_csv(
                lst_file,
                sep=";",
                index_col="filename",
                dtype="str",
                parse_dates=["start_time", "end_time", "processing_time"],
            )

            # time range:
            t1 = df["start_time"].min()
            t2 = df["start_time"].max()
            # full years, extra space for legends:
            t1 = pandas.Timestamp(year=t1.year, month=1, day=1)
            t2 = max(
                t2 + pandas.Timedelta(180, "days"),
                pandas.Timestamp(year=t2.year + 1, month=1, day=1),
            )

            # streams: 'OFFL', 'RPRO', ..
            streams = df["stream"].unique()
            streams.sort()

            # procesors: '010101', ...
            procs = df["processor"].unique()
            procs.sort()
            # count:
            nproc = len(procs)

            # convert processor labels '010203' to version 'v1.2.3':
            proclabs = {}
            for proc in procs:
                proclabs[proc] = "v%i.%i.%i" % (int(proc[0:2]), int(proc[2:4]), int(proc[4:6]))
            # endfor

            # list of labeled processors:
            labeled = []
            # storage for handles used for legends:
            proch = {}
            streamh = {}

            # new:
            fig = plt.figure(figsize=(12, 4))
            ax = fig.add_axes([0.05, 0.07, 0.92, 0.90])
            # loop:
            for iproc in range(nproc):
                # current:
                proc = procs[iproc]
                color = colors[iproc]
                # loop:
                for stream in streams:
                    # select:
                    df2 = df[(df["stream"] == stream) & (df["processor"] == proc)]
                    # any?
                    if len(df2) > 0:
                        # group by month, count number of orbits:
                        nn = (
                            df2.set_index("start_time")
                            .groupby(pandas.Grouper(freq=freq))["orbit"]
                            .count()
                        )
                        # annote:
                        proclab = proclabs[proc]
                        if proc not in labeled:
                            label = proclab
                            labeled.append(proc)
                        else:
                            label = None
                        # endif
                        # style:
                        if freq in ["MS"]:
                            style = dict(color=color, linestyle="-", marker="o")
                            if stream == "OFFL":
                                style["linestyle"] = "--"
                        else:
                            style = dict(
                                color=color, linestyle="None", marker="o", markerfacecolor="None"
                            )
                            if stream == "RPRO":
                                style["markerfacecolor"] = color
                        # endif
                        # plot non-zero values:
                        p = ax.plot(nn[nn > 0], label=label, **style)
                        # store handle for legends:
                        if proclab not in proch.keys():
                            proch[proclab] = p[0]
                        if stream not in streamh.keys():
                            streamh[stream] = p[0]
                    # endif
                # endfor # streams
            # endfor  # processors
            # time axis:
            ax.set_xlim((t1, t2))
            ax.xaxis.set_minor_locator(mdates.MonthLocator())
            ax.xaxis.set_major_locator(mdates.YearLocator())
            ax.xaxis.set_major_formatter(mdates.DateFormatter("%Y"))
            ax.grid(axis="x")
            # y-axis:
            ax.set_ylabel("# orbits / %s" % freqlabel)
            ax.set_ylim((0, None))
            # annote:
            # plt.legend()
            # legend for streams:
            ax.add_artist(
                plt.legend(list(streamh.values()), list(streamh.keys()), loc="upper right")
            )
            ax.add_artist(plt.legend(list(proch.values()), list(proch.keys()), loc="center right"))

            # save:
            fig.savefig(fig_file)

        else:

            # info ..
            logging.info(indent + "keep  %s ..." % fig_file)

        # endif # renew

        # info ...
        logging.info(indent + "")
        logging.info(indent + "** end listing plot")
        logging.info(indent + "")

    # enddef __init__


# endclass CSO_SciHub_ListingPlot


########################################################################
###
### end
###
########################################################################
