#
# Changes
# 
# 2022-09, Arjo Segers
#   Updated documentation.
#
# 2023-06, Arjo Segers
#   Use "pandas.concat()" instead of "df.append()" to avoid warnings.
#
# 2023-08, Arjo Segers
#   Updated logging messages.
#

########################################################################
###
### help
###
########################################################################

"""
.. _cso-scihub:

*********************
``cso_scihub`` module
*********************

The :py:mod:`cso_scihub` module provides classes for accessing  data from the 
`Copernicus Open Access Hub <https://scihub.copernicus.eu/>`_.
Below first the data hub itself is described, followed by how the CSO
pre-processor could be used for batch download of a selection.


Copernicus Open Access Hub
==========================

The `Copernicus Open Access Hub <https://scihub.copernicus.eu/>`_ is the official
portal for Copernicus satellite data.

.. figure:: figs/sci-hub.png
   :scale: 50 %
   :align: center
   :alt: Sci Hub main page.
   
   *Home page of Copernicus Open Access Hub* (https://scihub.copernicus.eu/)

Different hubs are provided for different data sets.
Below we describe:

* **Open Hub** Provides access to Sentinel-1,2,3 data.
* **S-5P Pre-Ops** Provides access to (pre-operational) Sentinel-5P data.


.. _SciHub-OpenHub:

Open Hub
--------

The *Open Hub* provides access to Sentinel-1,2,3 data.
Since this is the oldest data, most examples in the User Guide refer to this hub.

To download data it is necessary to register.
On the main page, select:

* `User Guide <https://scihub.copernicus.eu/userguide/>`_, 
  follow instructions on `Self Registration <https://scihub.copernicus.eu/userguide/>`_ .

The user name and password should be stored in your home directory in the ``~/.netrc`` file
(ensure that it has read/write permissions only for you)::

    # access Copernicus Open Access Hub
    machine scihub.copernicus.eu    login **********   password *********


S-5P Pre-Ops Hub
----------------

On the main page, select *'S-5P Pre-Ops'* for the
`Sentinel-5P Pre-Operations Data Hub <https://s5phub.copernicus.eu/dhus/>`_ .

Access is open, the login/password are shown in the login pop-up.
The user name and password should be stored in your home directory in the ``~/.netrc`` file
(ensure that it has read/write permissions only for you)::

    # access Copernicus Open Access Hub
    machine s5phub.copernicus.eu    login s5pguest     password s5pguest

Data can be selected and downloaded interactively.
In the search bar, open the *'Advanced Search'* menu and specify a selection.
The figure below shows an example for Level2 NO2 data.


.. figure:: figs/s5p-hub-advanced-search.png
   :scale: 75 %
   :align: center
   :alt: Advanced Search menu at S-5P Hub.
   
   *Advanced Search menu at S-5P Hub.*

The result of a search is a list of product files.
Each product file contains data from one orbit.
The name of a product file contains the scan period, the orbit number,
and a processing time::

  S5P_OFFL_L2__NO2____20190701T102357_20190701T120527_08882_01_010302_20190707T120219
  |   |    \_prodid_/ \_start-time__/ \__end-time___/ orbit |  |      \__processed__/
  |   stream                                                |  processor
  mission                                                   collection

The search query is shown above the product list, and contains the time 
and product selections; the later could be useful for batch download::

    ( beginPosition:[2019-07-01T00:00:00.000Z TO 2019-07-01T23:59:59.999Z] AND
        endPosition:[2019-07-01T00:00:00.000Z TO 2019-07-01T23:59:59.999Z]     )
    AND ( (    platformname:Sentinel-5  AND 
                producttype:L2__NO2___  AND 
            processinglevel:L2          AND 
             processingmode:Offline         ) )



.. _scihub-opensearch-download:

OpenSearch API
--------------

For batch processing one can use the 
`OpenSearch API <https://scihub.copernicus.eu/userguide/OpenSearchAPI>`_ .
First a search query is send to the server, and the result then contains links that can be used to download 
selected files.


    
.. _scihub-batch-download:

Batch download
--------------

An alternative for batch processing is to use the 
`API Hub <https://scihub.copernicus.eu/twiki/do/view/SciHubWebPortal/APIHubDescription>`_ .
This page also contains a link to the `section of the User Guide <https://scihub.copernicus.eu/userguide/8BatchScripting>`_  
with instructions for batch processing.

From the User Guide one can download the *'dhusget.sh'* script.
This will take care of searching the archive, downloading files, checking if the download
is complete, etc. 
The call to download all NO2 data for a single day that overlaps with a longitude/latitude
box over Europe looks like::

  ./dhusget.sh \\
      -d 'https://s5phub.copernicus.eu/dhus' \\
      -S '2018-07-01T00:00:00.000Z' \\
      -E '2018-07-02T00:00:00.000Z' \\
      -c '-30,30:45,76' \\
      -F 'platformname:Sentinel-5 AND producttype:L2__NO2___ AND processinglevel:L2 AND processingmode:Reprocessing' \\
      -o product \\
      -O /work/Sentinel-5P/TROPOMI/NO2/ \\
      -D

The download script from the API HUb is included in the CSO package,
with a few minor modifications:

  `bin/dhusget.sh <../../../bin/dhusget.sh>`_ 


Batch download via CSO
======================

The CSO download tools use by default the :ref:`scihub-opensearch-download`,
but could also use the :ref:`scihub-batch-download` option.

The :py:class:`.CSO_SciHub_Download` class performs the download using the OpenSearch API.
This class is prefered since:

* existing files could be kept without a new download;
* error messages are cleaner.

See the documentation of the class for the settings to be used.

Alternatively, the :py:class:`.CSO_SciHub_Download_DHuS` class could be used
which will call the ``dhusget.sh`` script to perform the download.
This will give a lot of intermediate log files, and error messages that are not easily interpreted.


Class hierchy
=============

The classes and are defined according to the following hierchy:


* :py:class:`.CSO_SciHub_DownloadFile`
* :py:class:`.UtopyaRc`

  * :py:class:`.CSO_SciHub_Inquire`
  * :py:class:`.CSO_SciHub_DownloadFile`
  * :py:class:`.CSO_SciHub_Download`



Classes
=======


"""


########################################################################
###
### modules
###
########################################################################

# modules:
import logging

# tools:
import utopya


########################################################################
###
### OpenSearch inquire
###
########################################################################

class CSO_SciHub_Inquire( utopya.UtopyaRc ) :

    """
    Inquire available Sentinel data from the `Copernicus Open Access Hub <https://scihub.copernicus.eu/>`_
    using the OpenSearch API:
    
    * `Copernicus Open Access Hub <https://scihub.copernicus.eu/>`_

      * `User Guide <https://scihub.copernicus.eu/userguide/>`_
  
        * `6 OpenSearch API <https://scihub.copernicus.eu/userguide/OpenSearchAPI>`_
      
    A query is sent to search for products that are available 
    for a certain time and overlap with a specified region. 
    The result is a list with orbit files and instructions on how to download them.
    
    In the settings, specify the url of the hub, which is either the Open Data hub or the S5-P hub::

      ! server url; provide login/password in ~/.netrc:
      <rcbase>.url              :  https://s5phub.copernicus.eu/dhus

    Specify the time range over which files should be downloaded::

      <rcbase>.timerange.start  :  2018-07-01 00:00
      <rcbase>.timerange.end    :  2018-07-01 23:59

    Specify a target area, only orbits with some pixels within the defined box will be downloaded::

      ! target area, leave empty for globe; format:  west,south:east,north
      <rcbase>.area             :  
      !<rcbase>.area             :  -30,30:35,76

    A query is used to select the required data.
    The search box on the hub could be used for inspiration on the format.
    Note that the '``producttype``' should have exactly 10 characters, 
    with the first 3 used for the retrieval level, and the last 6 for the product; 
    empty characters should have an underscore instead::

      ! search query, obtained from interactive download:
      <rcbase>.query            :      platformname:Sentinel-5    AND \\
                                        producttype:L2__NO2___    AND \\
                                    processinglevel:L2

    Name of ouptut csv file::
    
        ! output table, date of today:
        <rcbase>.output.file            :  ${my.work}/Copernicus_S5P_NO2_%Y-%m-%d.csv

    """
    
    def __init__( self, rcfile, rcbase='', env={}, indent='' ) :
    
        """
        Download orbits using OpenSearch API.
        """
                        
        # modules:
        import sys
        import os
        import datetime
        import requests
        import xml.etree.ElementTree
        import pandas

        # info ...
        logging.info( indent+'' )
        logging.info( indent+'** Inquire files available on Copernicus Data Hub' )
        logging.info( indent+'' )
        
        # init base object:
        utopya.UtopyaRc.__init__( self, rcfile=rcfile, rcbase=rcbase, env=env )
        
        # domain:
        url = self.GetSetting( 'url' )
        # info ...
        logging.info( indent+'url          : %s' % url )
        
        # area of interest: west,south:east,north
        area = self.GetSetting( 'area' )
        # defined?
        if len(area) > 0 :
            # convert from format for "dhusget.sh":
            #    west,south:east,north
            west,south,east,north = map( float, area.replace(':',' ').replace(',',' ').split() )
        else :
            # globe:
            west,south,east,north = -180,-90,180,90
        #endif
        # info ...
        logging.info( indent+'area         : [%8.2f,%8.2f] x [%8.2f,%8.2f]' % (west,east,south,north) )
        
        # query, probably obtained from interactive download page:
        product = self.GetSetting( 'query' )
        # remove excesive whitespace:
        product = ' '.join( product.split() )
        # info ...
        logging.info( indent+'product      : %s' % product )

        # target file, might include time templates:
        output_file__template = self.GetSetting( 'output.file' )
        # current time:
        output_file = datetime.datetime.now().strftime( output_file__template )
        
        # new output table:
        output_df = pandas.DataFrame()

        # time range:
        t1 = self.GetSetting( 'timerange.start', totype='datetime' )
        t2 = self.GetSetting( 'timerange.end'  , totype='datetime' )
        # info ...
        tfmt = '%Y-%m-%d %H:%M'
        logging.info( indent+'timerange: [%s,%s]' % (t1.strftime(tfmt),t2.strftime(tfmt)) )
        
        # targefile:
        qfile = 'query-%i.xml' % os.getpid()

        # timeout of requests in seconds:
        timeout = 60

        # search query could only return a maximum number of records;
        # a 'page' of records is requested using a row offet and the number of rows:
        row0 = 0
        nrow = 100    # maximum allowed is 100 ...
        
        # time labels:
        tfmt = '%Y-%m-%dT%H:%M:%SZ'
        # time of interest:
        toi = 'beginPosition:[%s TO %s]' % (t1.strftime(tfmt),t2.strftime(tfmt))

        # geographic area:
        #   'POLYGON((P1Lon P1Lat, P2Lon P2Lat, ..., PnLon PnLat))'
        area = 'POLYGON((%f %f, %f %f, %f %f, %f %f, %f %f))' % (west,south, east,south, east,north, west,north, west,south)
        # area of interest:
        aoi = 'footprint:"Intersects(%s)"' % area

        # info ..
        logging.info( indent+'  time of interest: %s' % toi )
        logging.info( indent+'  area of interest: [%8.2f,%8.2f] x [%8.2f,%8.2f]' % (west,east,south,north) )
        logging.info( indent+'  product         : %s' % product )

        # combine:
        query = '%s AND %s AND (%s)' % (toi,product,aoi)

        # init counter:
        ipage = 0

        # loop over pages of query result:
        while True :

            # increase counter:
            ipage += 1
            # info ...
            logging.info( indent+'  page %i (entries %i,..,%i)' % (ipage,row0+1,row0+nrow) ) 

            # send query to search page;
            # result is an xml tree with entries for matching orbit files;
            # a maximum of 'nrow' entries is returned, from a zero-based 'start' index onwards:
            r = requests.get( os.path.join(url,'search'), params=dict(q=query,start=row0,rows=nrow), timeout=timeout )
            # check status, raise error if request failed:
            try :
                r.raise_for_status()
            except requests.exceptions.HTTPError as err :
                msg = str(err)
                logging.error( 'from query; message received:' )
                logging.error( '  %s' % msg )
                if msg.startswith('401 Client Error: Unauthorized for url:') :
                    logging.error( 'Interpretation: the (username,password) received from your "~/.netrc" file is incorrect.' )
                    logging.error( 'For the S5p-hub, the file should contain the following entry:' )
                    logging.error( '  machine s5phub.copernicus.eu      login s5pguest        password s5pguest' )
                    # get authorization as test:
                    try :
                        username,password = requests.utils.get_netrc_auth( url, raise_errors=True )
                        logging.error( 'The username extracted from "~/.netrc" for this url is:' )
                        logging.error( '  %s' % username )
                        logging.error( 'Maybe this is the default entry?' )
                    except Exception as err :
                        logging.error( 'The "~/.netrc" file could not be parsed correctly; the error raised from:' )
                        logging.error( '  username,password = requests.utils.get_netrc_auth( "%s", raise_errors=True )' % url )
                        logging.error( 'is:' )
                        logging.error( '  %s' % str(err) )
                    #endtry
                #endif
            except Exception as err :
                msg = str(err)
                logging.error( 'from query; message received:' )
                logging.error( '  %s' % msg )
                sys.exit(1)
            #endtry

            # save:
            with open( qfile, 'w' ) as f :
                f.write( r.text )
            #endwith

            #
            # Example content:
            #
            #    <?xml version="1.0" encoding="utf-8"?>
            #    <feed xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/" xmlns="http://www.w3.org/2005/Atom">
            #      <title>Data Hub Service search results for: beginPosition:[2018-07-01T00:00:00Z TO 2018-07-02T00:00:00Z] AND platformname:Sentinel-5 AND producttype:L2__NO2___ AND processinglevel:L2 AND processingmode:Reprocessing AND (footprint:"Intersects(POLYGON((-30.000000 30.000000, 45.000000 30.000000, 45.000000 76.000000, -30.000000 76.000000, -30.000000 30.000000)))")</title>
            #      ...
            #      <entry>
            #        <title>S5P_RPRO_L2__NO2____20180701T024001_20180701T042329_03699_01_010202_20190211T185158</title>
            #        <link href="https://s5phub.copernicus.eu/dhus/odata/v1/Products('a66d4240-4999-4724-9903-aa2db410bbad')/$value"/>
            #        <str name="filename">S5P_RPRO_L2__NO2____20180701T024001_20180701T042329_03699_01_010202_20190211T185158.nc</str>
            #        <str name="uuid"    >a66d4240-4999-4724-9903-aa2db410bbad</str>
            #        ...
            #      </entry>
            #      ...
            #   </feed>
            #

            # open query result:
            tree = xml.etree.ElementTree.parse( qfile )
            # get root element ("feed"):
            root = tree.getroot()

            # the tag contains an url prefix:
            #   root.tag = "{http://www.w3.org/2005/Atom}feed"
            # prefixed?
            if root.tag.startswith('{') :
                # extract prefix:
                prefix,tag = root.tag[1:].split('}')
            else :
                # no prefix:
                prefix = ''
            #endif
            # create namespace to translate the "atom:" to the prefix:
            ns = { 'atom' : prefix }

            # find "entry" elements:
            entries = root.findall( 'atom:entry', namespaces=ns )

            # count:
            nrec = len(entries)
            # info ..
            logging.info( indent+'    number of records found: %i' % nrec )
            # check ..
            if (ipage == 1) and (nrec == 0) :
                logging.warning( '      no records found for this day; something wrong in query? continue with next day ...' )
                break
            #endif

            # loop over entries:
            for entry in entries :

                # get target filename, this is the content of the "str" element
                # with attribute "name" equal to "filename":
                #        <str name="filename">S5P_RPRO_L2__NO2____20180701T024001_20180701T042329_03699_01_010202_20190211T185158.nc</str>
                filename = entry.find( 'atom:str[@name="filename"]', ns ).text

                # info ...
                logging.info( indent+'    file : %s' % filename )

                #
                # filenames:
                #
                #    S5P_OFFL_L2__NO2____20180701T005930_20180701T024100_03698_01_010002_20180707T022838.nc
                #    plt proc [product-] [starttime....] [endtime......] orbit cl prvers [prodtime.....]
                #
                bname = os.path.basename(filename).replace('.nc','')
                # split:
                platform_name,processing,rest = bname.split('_',2)
                product_type = rest[0:10]
                start_time,end_time,orbit,collection,processor_version,production_time = rest[11:].split('_')

                # convert:
                tfmt = '%Y%m%dT%H%M%S'
                ts = datetime.datetime.strptime(start_time,tfmt)
                te = datetime.datetime.strptime(end_time  ,tfmt)

                # find first "link"  element:
                link = entry.find( 'atom:link', ns )
                # extract download link:
                href = link.attrib['href']

                # row:
                rec = { 'orbit'               : [orbit],
                        'start_time'          : [ts],
                        'end_time'            : [te],
                        'processing'          : [processing],
                        'collection'          : [collection],
                        'processor_version'   : [processor_version],
                        'filename'            : [filename],
                        'href'                : [href] }
                # add record:
                output_df = pandas.concat( (output_df,pandas.DataFrame(rec)), ignore_index=True )

            #endfor # entries

            # cleanup:
            os.remove( qfile )

            # leave loop over pages?
            if nrec < nrow : break
            # increse offset:
            row0 += nrow

        #endwhile # pages
        
        # info ..
        logging.info( 'save to: %s ...' % output_file )
        # create directory:
        dirname = os.path.dirname( output_file )
        if len(dirname) > 0 :
            if not os.path.isdir(dirname) : os.makedirs( dirname )
        #endif
        # write:
        output_df.to_csv( output_file, sep=';', index=False )

        # info ...
        logging.info( indent+'' )
        logging.info( indent+'** end SciHub inquire' )
        logging.info( indent+'' )
        
    #enddef __init__
        
#endclass CSO_SciHub_Inquire




########################################################################
###
### OpenSearch download
###
########################################################################

class CSO_SciHub_DownloadFile( object ) :

    """
    Download single file from SciHub.
    
    Arguments:

    * ``href`` : download url: ``https://s5phub.copernicus.eu/dhus/odata/v1/Products('d483baa0-3a61-4985-aa0c-5642a83c9214')/$value``
    * ``output_file`` : target file
    
    Optional arguments:
    
    * ``maxtry`` : number of times to try again if download fails
    * ``timeout`` : delay in seconds between requests
    """
    
    def __init__( self, href, output_file, maxtry=10, timeout=60, indent='' ) :
    
        """
        Download file.
        """
        
        # modules:
        import os
        import requests

        # retry loop ..
        ntry = 0
        while True :
            # try to download and save:
            try :

                # try to download:
                try :

                    # download:
                    r = requests.get( href, timeout=timeout )
                    # check status, raise error if request failed:
                    r.raise_for_status()

                    # info ..
                    logging.info( indent+'      write to: %s ...' % os.path.dirname(output_file) )
                    # target dir:
                    dname = os.path.dirname( output_file )
                    if len(dname) > 0 :
                        if not os.path.isdir(dname) : os.makedirs( dname )
                    #endif
                    # write to temporary target first ..
                    tmpfile = output_file+'.tmp'
                    # open destination file for binary write:
                    with open(tmpfile,'wb') as fd :
                        # prefered way to write content following:
                        #   https://docs.python-requests.org/en/master/user/quickstart/
                        for chunk in r.iter_content(chunk_size=128) :
                            fd.write(chunk)
                        #endfor
                    #endwith
                    # rename:
                    os.rename( tmpfile, output_file )

                except requests.exceptions.HTTPError as err :
                    # info ..
                    msg = str(err)
                    logging.error( 'exception from download; message received:' )
                    logging.error( '  %s' % msg )
                    # catch known problem ...
                    if msg.startswith('401 Client Error: Unauthorized for url:') :
                        logging.error( 'Interpretation: the (username,password) received from your "~/.netrc" file is incorrect.' )
                        logging.error( 'For the S5p-hub, the file should contain the following entry:' )
                        logging.error( '  machine s5phub.copernicus.eu      login s5pguest        password s5pguest' )
                        # try to get authorization as test:
                        try :
                            username,password = requests.utils.get_netrc_auth( url, raise_errors=True )
                            logging.error( 'The username extracted from "~/.netrc" for this url is:' )
                            logging.error( '  %s' % username )
                            logging.error( 'Maybe this is the default entry?' )
                        except Exception as err :
                            logging.error( 'The "~/.netrc" file could not be parsed correctly; the error raised from:' )
                            logging.error( '  username,password = requests.utils.get_netrc_auth( "%s", raise_errors=True )' % url )
                            logging.error( 'is:' )
                            logging.error( '  %s' % str(err) )
                        #endtry
                        # should be solved first, leave retry loop ...
                        break
                    #endif

                except MemoryError as err :
                    logging.error( 'memory error from download; need to increase allocated resources?' )
                    # quit with error:
                    raise

                except Exception as err :
                    # info ..
                    logging.error( 'from download; message received:' )
                    logging.error( '  %s' % str(err) )
                    # quit with error:
                    raise

                #endtry

            # error from download or save:
            except :
                # increase counter:
                ntry += 1
                # switch:
                if ntry == maxtry :
                    logging.warning( indent+'      exception from download; tried %i times ...' % maxtry )
                    raise Exception
                else :
                    logging.warning( indent+'      exception from download; try again ...' )
                    continue  # while-loop
                #endif
            #endtry
            # leave retry loop,
            # either because download was ok,
            # or because maximum number of retries was reached:
            break

        #endwhile # retry
        
    #enddef __init__
    
#endclass CSO_SciHub_DownloadFile

# *

class CSO_SciHub_Download( utopya.UtopyaRc ) :

    """
    Download Sentinel data from the `Copernicus Open Access Hub <https://scihub.copernicus.eu/>`_
    using the OpenSearch API:
    
    * `Copernicus Open Access Hub <https://scihub.copernicus.eu/>`_

      * `User Guide <https://scihub.copernicus.eu/userguide/>`_
  
        * `6 OpenSearch API <https://scihub.copernicus.eu/userguide/OpenSearchAPI>`_
      
    To download orbit files, first a query is sent to search for products that are available 
    for a certain time and overlap with a specified region. 
    The result is a list with orbit files and instructions on how to download them.
    
    In the settings, specify the url of the hub, which is either the Open Data hub or the S5-P hub::

      ! server url; provide login/password in ~/.netrc:
      <rcbase>.url              :  https://s5phub.copernicus.eu/dhus

    Specify the time range over which files should be downloaded::

      <rcbase>.timerange.start  :  2018-07-01 00:00
      <rcbase>.timerange.end    :  2018-07-01 23:59

    Specify a target area, only orbits with some pixels within the defined box will be downloaded::

      ! target area, leave empty for globe; format:  west,south:east,north
      <rcbase>.area             :  
      !<rcbase>.area             :  -30,30:35,76

    A query is used to select the required data.
    The search box on the hub could be used for inspiration on the format.
    Note that the '``producttype``' should have exactly 10 characters, 
    with the first 3 used for the retrieval level, and the last 6 for the product; 
    empty characters should have an underscore instead::

      ! search query, obtained from interactive download:
      <rcbase>.query            :      platformname:Sentinel-5    AND \\
                                        producttype:L2__NO2___    AND \\
                                    processinglevel:L2            AND \\
                                     processingmode:Offline

    The target directory for downloaded file could include templates for time values::

      ! output archive, store per month:
      <rcbase>.output.dir       :  /data/Copernicus/S5P/OFFL/NO2/%Y/%m
      
    Use the following flag to keep files that are already present::
    
      ! renew existing files?
      <rcbase.renew             :  False

    """
    
    def __init__( self, rcfile, rcbase='', env={}, indent='' ) :
    
        """
        Download orbits using OpenSearch API.
        """
                        
        # modules:
        import sys
        import os
        import datetime
        import requests
        import xml.etree.ElementTree

        # info ...
        logging.info( indent+'' )
        logging.info( indent+'** Download from Copernicus Data Hub' )
        logging.info( indent+'' )
        
        # init base object:
        utopya.UtopyaRc.__init__( self, rcfile=rcfile, rcbase=rcbase, env=env )
        
        # domain:
        url = self.GetSetting( 'url' )
        # info ...
        logging.info( indent+'url          : %s' % url )
        
        # area of interest: west,south:east,north
        area = self.GetSetting( 'area' )
        # convert from format for "dhusget.sh":
        #    west,south:east,north
        west,south,east,north = map( float, area.replace(':',' ').replace(',',' ').split() )
        # info ...
        logging.info( indent+'area         : [%8.2f,%8.2f] x [%8.2f,%8.2f]' % (west,east,south,north) )
        
        # query, probably obtained from interactive download page:
        product = self.GetSetting( 'query' )
        # remove excesive whitespace:
        product = ' '.join( product.split() )
        # info ...
        logging.info( indent+'product      : %s' % product )
        
        # output dir, might contain time templates:
        output_dir__template = self.GetSetting( 'output.dir' )
        
        # renew existing files?
        renew = self.GetSetting( 'renew', totype='bool' )

        # time range:
        t1 = self.GetSetting( 'timerange.start', totype='datetime' )
        t2 = self.GetSetting( 'timerange.end'  , totype='datetime' )
        # info ...
        tfmt = '%Y-%m-%d %H:%M'
        logging.info( indent+'timerange: [%s,%s]' % (t1.strftime(tfmt),t2.strftime(tfmt)) )
        
        # targefile:
        qfile = 'query-%i.xml' % os.getpid()

        # timeout of requests in seconds:
        timeout = 60

        # search query could only return a maximum number of records;
        # a certain 'page' of records is requesing a row offet and number:
        row0 = 0
        nrow = 100    # maximum allowed is 100 ...
        
        # number of retries:
        maxtry = 10

        # loop over days:
        t = t1
        while t < t2 :
    
            # info ...
            logging.info( indent+'%s ...' % t.strftime('%Y-%m-%d') )
                
            # start and end time:
            ts = t
            # add 24h:
            tx = t + datetime.timedelta(1)
            # convert to midnight:
            tx24 = datetime.datetime(tx.year,tx.month,tx.day,0,0)
            # not after t2 ...
            te = min( tx24, t2 )
            
            # time labels:
            tfmt = '%Y-%m-%dT%H:%M:%SZ'
            # time of interest:
            toi = 'beginPosition:[%s TO %s]' % (ts.strftime(tfmt),te.strftime(tfmt))

            # geographic area:
            #   'POLYGON((P1Lon P1Lat, P2Lon P2Lat, ..., PnLon PnLat))'
            area = 'POLYGON((%f %f, %f %f, %f %f, %f %f, %f %f))' % (west,south, east,south, east,north, west,north, west,south)
            # area of interest:
            aoi = 'footprint:"Intersects(%s)"' % area

            # info ..
            logging.info( indent+'  time of interest: %s' % toi )
            logging.info( indent+'  area of interest: [%8.2f,%8.2f] x [%8.2f,%8.2f]' % (west,east,south,north) )
            logging.info( indent+'  product         : %s' % product )

            # combine:
            query = '%s AND %s AND (%s)' % (toi,product,aoi)

            # init counter:
            ipage = 0

            # current:
            output_dir = t.strftime( output_dir__template )
            # info ..
            logging.info( indent+'  output directory: %s' % output_dir )
            # create?
            if not os.path.isdir( output_dir ) : os.makedirs( output_dir )
            
            # loop over pages of query result:
            while True :

                # increase counter:
                ipage += 1
                # info ...
                logging.info( indent+'  page %i (entries %i,..,%i)' % (ipage,row0+1,row0+nrow) ) 

                # send query to search page;
                # result is an xml tree with entries for matching orbit files;
                # a maximum of 'nrow' entries is returned, from a zero-based 'start' index onwards:
                r = requests.get( os.path.join(url,'search'), params=dict(q=query,start=row0,rows=nrow), timeout=timeout )
                # check status, raise error if request failed:
                try :
                    r.raise_for_status()
                except requests.exceptions.HTTPError as err :
                    msg = str(err)
                    logging.error( 'from query; message received:' )
                    logging.error( '  %s' % msg )
                    if msg.startswith('401 Client Error: Unauthorized for url:') :
                        logging.error( 'Interpretation: the (username,password) received from your "~/.netrc" file is incorrect.' )
                        logging.error( 'For the S5p-hub, the file should contain the following entry:' )
                        logging.error( '  machine s5phub.copernicus.eu      login s5pguest        password s5pguest' )
                        # get authorization as test:
                        try :
                            username,password = requests.utils.get_netrc_auth( url, raise_errors=True )
                            logging.error( 'The username extracted from "~/.netrc" for this url is:' )
                            logging.error( '  %s' % username )
                            logging.error( 'Maybe this is the default entry?' )
                        except Exception as err :
                            logging.error( 'The "~/.netrc" file could not be parsed correctly; the error raised from:' )
                            logging.error( '  username,password = requests.utils.get_netrc_auth( "%s", raise_errors=True )' % url )
                            logging.error( 'is:' )
                            logging.error( '  %s' % str(err) )
                        #endtry
                    #endif
                except Exception as err :
                    msg = str(err)
                    logging.error( 'from query; message received:' )
                    logging.error( '  %s' % msg )
                    sys.exit(1)
                #endtry

                # save:
                with open( qfile, 'w' ) as f :
                    f.write( r.text )
                #endwith

                #
                # Example content:
                #
                #    <?xml version="1.0" encoding="utf-8"?>
                #    <feed xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/" xmlns="http://www.w3.org/2005/Atom">
                #      <title>Data Hub Service search results for: beginPosition:[2018-07-01T00:00:00Z TO 2018-07-02T00:00:00Z] AND platformname:Sentinel-5 AND producttype:L2__NO2___ AND processinglevel:L2 AND processingmode:Reprocessing AND (footprint:"Intersects(POLYGON((-30.000000 30.000000, 45.000000 30.000000, 45.000000 76.000000, -30.000000 76.000000, -30.000000 30.000000)))")</title>
                #      ...
                #      <entry>
                #        <title>S5P_RPRO_L2__NO2____20180701T024001_20180701T042329_03699_01_010202_20190211T185158</title>
                #        <link href="https://s5phub.copernicus.eu/dhus/odata/v1/Products('a66d4240-4999-4724-9903-aa2db410bbad')/$value"/>
                #        <str name="filename">S5P_RPRO_L2__NO2____20180701T024001_20180701T042329_03699_01_010202_20190211T185158.nc</str>
                #        <str name="uuid"    >a66d4240-4999-4724-9903-aa2db410bbad</str>
                #        ...
                #      </entry>
                #      ...
                #   </feed>
                #

                # open query result:
                tree = xml.etree.ElementTree.parse( qfile )
                # get root element ("feed"):
                root = tree.getroot()

                # the tag contains an url prefix:
                #   root.tag = "{http://www.w3.org/2005/Atom}feed"
                # prefixed?
                if root.tag.startswith('{') :
                    # extract prefix:
                    prefix,tag = root.tag[1:].split('}')
                else :
                    # no prefix:
                    prefix = ''
                #endif
                # create namespace to translate the "atom:" to the prefix:
                ns = { 'atom' : prefix }

                # find "entry" elements:
                entries = root.findall( 'atom:entry', namespaces=ns )

                # count:
                nrec = len(entries)
                # info ..
                logging.info( indent+'    number of records found: %i' % nrec )
                # check ..
                if (ipage == 1) and (nrec == 0) :
                    logging.warning( '      no records found for this day; something wrong in query? continue with next day ...' )
                    break
                #endif

                # loop over entries:
                for entry in entries :

                    # get target filename, this is the content of the "str" element
                    # with attribute "name" equal to "filename":
                    #        <str name="filename">S5P_RPRO_L2__NO2____20180701T024001_20180701T042329_03699_01_010202_20190211T185158.nc</str>
                    filename = entry.find( 'atom:str[@name="filename"]', ns ).text

                    # full path:
                    output_file = os.path.join( output_dir, filename )

                    # info ...
                    logging.info( indent+'    file : %s' % filename )
                    # (re)new?
                    if (not os.path.isfile(output_file)) or renew :

                        # find first "link"  element:
                        link = entry.find( 'atom:link', ns )
                        # extract download link:
                        href = link.attrib['href']
                        # info ..
                        logging.info( indent+'      download: %s' % href )
                        
                        # sinlge file:
                        CSO_SciHub_DownloadFile( href, output_file, maxtry=maxtry, timeout=timeout, indent=indent )

                    else :
                        # info ..
                        logging.info( indent+'      keep in: %s ...' % output_dir )
                    #endif

                    ## testing ...
                    #break

                #endfor # entries

                # cleanup:
                os.remove( qfile )

                # leave loop over pages?
                if nrec < nrow : break
                # increse offset:
                row0 += nrow

            #endwhile # pages

            # start of next day:
            t = te
        #endwhile # day loop
        
        # info ...
        logging.info( indent+'' )
        logging.info( indent+'** end SciHub Download' )
        logging.info( indent+'' )
        
    #enddef __init__
        
#endclass CSO_SciHub_Download


########################################################################
###
### create listing file
###
########################################################################

class CSO_SciHub_Listing( utopya.UtopyaRc ) :

    """
    Create *listing* file for files download from SciHub or other portals that use equivalent filenames.
    
    A *listing* file contains the names of the converted orbit files,
    the time range of pixels in the file, and other information extracted from the filenames:

        filename                                                                                               ;mission;processing;product_id;start_time         ;end_time           ;orbit;collection;processor_version;processing_time
        RPRO/CH4/2018/04/S5P_RPRO_L2__CH4____20180430T001851_20180430T020219_02818_01_010301_20190513T141133.nc;S5P    ;RPRO      ;L2__CH4___;2018-04-30T00:18:51;2018-04-30T02:02:19;02818;01        ;010301           ;2019-05-13T14:11:33
        RPRO/CH4/2018/04/S5P_RPRO_L2__CH4____20180430T020021_20180430T034349_02819_01_010301_20190513T135953.nc;S5P    ;RPRO      ;L2__CH4___;2018-04-30T02:00:21;2018-04-30T03:43:49;02819;01        ;010301           ;2019-05-13T13:59:53
        :

    This file could be used to scan for available versions and how they were produced.
    
    In the settings, define the name of the file to be created::

        ! csv file that will hold records per file with:
        ! - timerange of pixels in file
        ! - orbit number
        ! time templates are replaced with todays date
        <rcbase>.file        :  /Scratch/Copernicus/S5p/listing-CH4__%Y-%m-%d.csv
        
    An existing listing file is not replaced,
    unless the following flag is set::

        ! renew table?
        <rcbase>.renew           :  True
        
    Orbit files are searched within a timerange::
    
        <rcbase>.timerange.start        :  2018-06-01 00:00
        <rcbase>.timerange.end          :  2018-06-03 23:59

    Specify filename filters to search for orbit files;
    the patterns are relative to the basedir of the listing file,
    and might contain templates for the time values.
    Multiple patterns could be defined; if for a certain orbit number more than one
    file is found, the first match is used.
    This could be explored to create a listing that combines reprocessed data
    with near-real-time data::

        <rcbase>.patterns            :  RPRO/CH4/%Y/%m/S5p_*.nc \
                                        OFFL/CH4/%Y/%m/S5p_*.nc

    """
    
    def __init__( self, rcfile, rcbase='', env={}, indent='' ) :
    
        """
        Convert data.
        """
                        
        # modules:
        import os
        import datetime
        import glob
        import collections
        
        # tools:
        import cso_file

        # info ...
        logging.info( indent+'' )
        logging.info( indent+'** create listing file' )
        logging.info( indent+'' )

        # init base object:
        utopya.UtopyaRc.__init__( self, rcfile=rcfile, rcbase=rcbase, env=env )
        
        # renew output?
        renew = self.GetSetting( 'renew', totype='bool' )

        # table file to be written:
        lst_file = self.GetSetting( 'file' )
        # evaluate current time:
        lst_file = datetime.datetime.now().strftime( lst_file )

        # create?
        if (not os.path.isfile(lst_file)) or renew :        
            # info ..
            logging.info( indent+'create %s ...' % lst_file )
        
            # time range:
            t1 = self.GetSetting( 'timerange.start', totype='datetime' )
            t2 = self.GetSetting( 'timerange.end'  , totype='datetime' )        
            # info ...
            tfmt = '%Y-%m-%d %H:%M'
            logging.info( indent+'  timerange: [%s,%s]' % (t1.strftime(tfmt),t2.strftime(tfmt)) )
            
            # base directory:
            bdir = os.path.dirname( lst_file )
            # create?
            if len(bdir) > 0 :
                if not os.path.isdir(bdir) : os.makedirs( bdir )
            #endif
            # current directory?
            if len(bdir) == 0 : bdir = '.'
            # info ...
            logging.info( indent+'  base directory: %s ...' % bdir )
        
            # initiallize for (re)creation:
            listing = cso_file.CSO_Listing( lst_file, indent=indent+'    ' )

            # info ...
            logging.info( indent+'  cleanup records if necessary ...' )
            # remove entries that do not exist anymore:
            listing.Cleanup( indent=indent+'    ' )

            # filename pattern templates:
            pattern_templates = self.GetSetting( 'patterns' ).split()
            # collection of scanned patterns:
            patterns = []
            
            # loop over days:
            t = t1
            while t <= t2 :

                # loop over patterns:
                for pattern_template in pattern_templates :
                    # expand time values:
                    pattern = t.strftime(pattern_template)
                    # skip if already scanned ...
                    if pattern in patterns : continue
                    # store:
                    patterns.append( pattern )
                    # info ...
                    logging.info( indent+'scan %s ...' % pattern )
                    
                    # list relative to basedir:
                    cwd = os.getcwd()
                    os.chdir( bdir )
                    fnames = glob.glob( pattern )
                    os.chdir( cwd )
                    # empty ?
                    if len(fnames) == 0 :
                        logging.info( indent+'  empty ..' )
                        continue
                    #endif
                    # sort in place:
                    fnames.sort()
                    # loop over files:
                    for fname in fnames :

                        # absolute path:
                        filename = os.path.join(bdir,fname)
                        # already in table?
                        if fname in listing :
                            # info ...
                            logging.info( indent+'  keep entry %s ...' % fname )
                        else :
                            # info ...
                            logging.info( indent+'  add entry %s ...' % fname )
                            # Example filename:
                            #   S5P_RPRO_L2__CH4____20180430T001851_20180430T020219_02818_01_010301_20190513T141133.nc
                            #
                            # Some products have incorrect product id (should be 10 characters):
                            #   S5P_OFFL_L2__CHOCHO___20200101T005246_20200101T023416_11487_01_010000_20210128.nc
                            # The extracted product id is then truncated to 10 characters.
                            #
                            # basename:
                            bname,ext = os.path.splitext( os.path.basename( filename ) )
                            # extract:
                            try :
                                mission,processing,rest = bname.split('_',2)
                                if rest.startswith('L2__CHOCHO__') :
                                    product_id = rest[0:10]
                                    start_time,end_time,orbit,collection,processor_version,prod_time = rest[13:].split('_')
                                else :
                                    product_id = rest[0:10]
                                    start_time,end_time,orbit,collection,processor_version,prod_time = rest[11:].split('_')
                                #endif
                            except :
                                logging.error( 'could not extract filename parts; expected format:' )
                                logging.error( '  S5P_RPRO_L2__CH4____20180430T001851_20180430T020219_02818_01_010301_20190513T141133' )
                                logging.error( 'found:' )
                                logging.error( '  %s' % bname )
                                raise
                            #endif
                            # fill data record:
                            data = collections.OrderedDict()
                            tfmt = '%Y%m%dT%H%M%S'
                            data['start_time'       ] = datetime.datetime.strptime( start_time, tfmt )
                            data['end_time'         ] = datetime.datetime.strptime( end_time  , tfmt )
                            data['mission'          ] = mission
                            data['processing'       ] = processing
                            data['product_id'       ] = product_id
                            data['orbit'            ] = orbit
                            data['collection'       ] = collection
                            data['processor_version'] = processor_version
                            if len(prod_time) == 8 :
                                data['processing_time'  ] = datetime.datetime.strptime( prod_time , '%Y%m%d' )
                            else :
                                data['processing_time'  ] = datetime.datetime.strptime( prod_time , tfmt )
                            #endif
                            # update record:
                            listing.UpdateRecord( fname, data, indent=indent+'    ' )
                        #endif  # new record?

                    #endfor # filenames

                #endfor # patterns
                
                ## testing ...
                #break
                        
                # next
                t = t + datetime.timedelta(0,3600)
            #endwhile
        
            # save:
            listing.Close( indent=indent+'  ' )

        else :
            # info ..
            logging.info( indent+'keep %s ...' % lst_file )
        #endif

        # info ...
        logging.info( indent+'' )
        logging.info( indent+'** end listing' )
        logging.info( indent+'' )
        
    #enddef __init__
    
#endclass CSO_SciHub_Listing


#
# * plot listing
#

class CSO_SciHub_ListingPlot( utopya.UtopyaRc ) :

    """
    Create timeseries plot of number of orbits per processor version.
    Information taken from *listing* file created by :py:class:`CSO_SciHub_Listing` class.
    
    """
    
    def __init__( self, rcfile, rcbase='', env={}, indent='' ) :
    
        """
        Convert data.
        """
                        
        # modules:
        import os
        import pandas
        import matplotlib.pyplot as plt
        import matplotlib.dates  as mdates
        
        # info ...
        logging.info( indent+'' )
        logging.info( indent+'** create listing plot' )
        logging.info( indent+'' )

        # init base object:
        utopya.UtopyaRc.__init__( self, rcfile=rcfile, rcbase=rcbase, env=env )
        
        # renew output?
        renew = self.GetSetting( 'renew', totype='bool' )

        # table file to be written:
        lst_file = self.GetSetting( 'file' )
        # info ..
        logging.info( indent+'listing file: %s' % lst_file )
        
        # target file:
        fig_file = lst_file.replace('.csv','.png')
        # create?
        if (not os.path.isfile(fig_file)) or renew :
        
            # info ..
            logging.info( indent+'create %s ...' % fig_file )

            # frequency:
            #freq = 'MS'  ; freqlabel = 'month'
            freq = 'W'   ; freqlabel = 'week'
            #freq = 'DS'  ; freqlabel = 'day'

            # color list:
            colors = ['purple','blue','cyan','gold','green','red','magenta','brown']

            # read:
            df = pandas.read_csv( lst_file, sep=';', index_col='filename', dtype='str',
                                    parse_dates=['start_time','end_time','processing_time'] )

            # time range:
            t1 = df['start_time'].min()
            t2 = df['start_time'].max()
            # full years, extra space for legends:
            t1 = pandas.Timestamp( year=t1.year  , month=1, day=1 )
            t2 = max( t2+pandas.Timedelta(180,'days'), pandas.Timestamp( year=t2.year+1, month=1, day=1 ) )

            # streams: 'OFFL', 'RPRO', ..
            streams = df['stream'].unique()
            streams.sort()

            # procesors: '010101', ...
            procs = df['processor'].unique()
            procs.sort()
            # count:
            nproc = len(procs)

            # convert processor labels '010203' to version 'v1.2.3':
            proclabs = {}
            for proc in procs :
                proclabs[proc] = 'v%i.%i.%i' % (int(proc[0:2]),int(proc[2:4]),int(proc[4:6]))
            #endfor

            # list of labeled processors:
            labeled = []
            # storage for handles used for legends:
            proch   = {}
            streamh = {}

            # new:
            fig = plt.figure( figsize=(12,4) )
            ax = fig.add_axes( [0.05,0.07,0.92,0.90] )
            # loop:
            for iproc in range(nproc) :
                # current:
                proc  = procs[iproc]
                color = colors[iproc]
                # loop:
                for stream in streams :
                    # select:
                    df2 = df[ (df['stream'] == stream) & (df['processor'] == proc) ]
                    # any?
                    if len(df2) > 0 :
                        # group by month, count number of orbits:
                        nn = df2.set_index('start_time').groupby(pandas.Grouper(freq=freq))['orbit'].count()
                        # annote:
                        proclab = proclabs[proc]
                        if proc not in labeled :
                            label = proclab
                            labeled.append(proc)
                        else :
                            label = None
                        #endif
                        # style:
                        if freq in ['MS'] :
                            style = dict( color=color, linestyle='-', marker='o' )
                            if stream == 'OFFL' : style['linestyle'] = '--'
                        else :
                            style = dict( color=color, linestyle='None', marker='o', markerfacecolor='None' )
                            if stream == 'RPRO' : style['markerfacecolor'] = color
                        #endif
                        # plot non-zero values:
                        p = ax.plot( nn[nn>0], label=label, **style )
                        # store handle for legends:
                        if proclab not in   proch.keys() : proch  [proclab] = p[0]
                        if stream  not in streamh.keys() : streamh[stream ] = p[0]
                    #endif
                #endfor # streams
            #endfor  # processors
            # time axis:
            ax.set_xlim((t1,t2))
            ax.xaxis.set_minor_locator( mdates.MonthLocator() )
            ax.xaxis.set_major_locator( mdates.YearLocator() )
            ax.xaxis.set_major_formatter( mdates.DateFormatter('%Y'))
            ax.grid(axis='x')
            # y-axis:
            ax.set_ylabel( '# orbits / %s' % freqlabel )
            ax.set_ylim((0,None))
            # annote:
            #plt.legend()
            # legend for streams:
            ax.add_artist( plt.legend( list(streamh.values()), list(streamh.keys()), loc='upper right'  ) )
            ax.add_artist( plt.legend( list(  proch.values()), list(  proch.keys()), loc='center right' ) )

            # save:
            fig.savefig( fig_file )
            
        else :
        
            # info ..
            logging.info( indent+'keep  %s ...' % fig_file )
            
        #endif # renew

        # info ...
        logging.info( indent+'' )
        logging.info( indent+'** end listing plot' )
        logging.info( indent+'' )
        
    #enddef __init__
    
#endclass CSO_SciHub_ListingPlot


########################################################################
###
### end
###
########################################################################


