# `cso_dataspace` module

The `cso_dataspace` module provides classes for accessing data from the Copernicus DataSpace.

To browse through the data, use the Browser.

## Account setup

To be able to download data from the DataSpace, first Register and create an account.

On a Linux system, login/passwords for websites can be stored in the users `.netrc` file in the home directory. Create this file if it does not exist yet, and add the following line with the login name of the account (your email) and the chosen password:

    machine zipper.dataspace.copernicus.eu      login Your.Name@institute.org   password ***********

The file should be readible and writable for you only:

    chmod 400 ~/.netrc

## DataSpace API’s

The DataSpace could be access with a number of different APIs.

Currently the OpenSearch API is used as that was the first that worked as needed.

In future the STAC API might be used, as this is becoming more and more the standard in the Earth Observation community. Within CSO it was already used by for example PAL API, but could not get working yet for the DataSpace. See the STAC product catalog for more information.

## Class hierchy

The classes and are defined according to the following hierchy:

- `UtopyaRc`

  - `CSO_DataSpace_Inquire`

- `CSO_DataSpace_Downloader`

- `NullAuth`

## Classes

class cso_dataspace.CSO_DataSpace_Inquire(rcfile, rcbase='', env={}, indent='')

  Bases: `UtopyaRc`

  Inquire available Sentinel data from the Copernicus DataSpace.

  Before data could be downloaded from the DataSpace, setup your Account setup.

  Currently the OpenSearch API is used as that was the first that worked as needed; in future, the STAC product catalog might be used. See also the OpenSearch Description for the various parameters that the API accepts.

  A query is sent to search for products that are available for a certain time and overlap with a specified region. The result is a list with orbit files and instructions on how to download them.

  In the settings, specify the time range over which files should be searched:

      <rcbase>.timerange.start  :  2018-07-01 00:00
      <rcbase>.timerange.end    :  2018-07-01 23:59

  Specify the base url of the API:

      <rcbase>.url              :  https://catalogue.dataspace.copernicus.eu/resto/api

  Define the collection name with:

      <rcbase>.collection       :  Sentinel5P

  Provide a product type:

      ! product type (always 10 characters!):
      <rcbase>.producttype            :  L2__NO2___

  Eventually specify a target area, only orbits with some pixels within the defined box will be downloaded:

      ! target area, leave empty for globe; format:  west,south,east,north
      <rcbase>.area             :
      !<rcbase>.area             :  -30,30,35,76

  The table will also create the url’s to download a file; specifity the template that should be used:

  > ! template for download url given “{product_id}”: <rcbase>.download_url : https://zipper.dataspace.copernicus.eu/odata/v1/Products({product_id})/$value

  Name of output csv file:

      ! output table, here including date of today:
      <rcbase>.output.file            :  ${my.work}/PAL_S5P_NO2_%Y-%m-%d.csv

  Example records (with extra whitespace to show the columns):

      orbit;start_time         ;end_time           ;processing;collection;processor_version;filename                                                                              ;href
      11488;2020-01-01 02:34:16;2020-01-01 04:15:46;RPRO      ;03        ;020400           ;S5P_RPRO_L2__CH4____20200101T023416_20200101T041546_11488_03_020400_20221120T003820.nc;https://zipper.dataspace.copernicus.eu/odata/v1/Products(b3f240e6-505d-4cae-97ea-43a8778a318d)/$value
      11487;2020-01-01 00:52:46;2020-01-01 02:34:16;RPRO      ;03        ;020400           ;S5P_RPRO_L2__CH4____20200101T005246_20200101T023416_11487_03_020400_20221120T003818.nc;https://zipper.dataspace.copernicus.eu/odata/v1/Products(a3d40f81-6c86-44bc-bc4b-457ff069b121)/$value
      :

class cso_dataspace.NullAuth

  Bases: `AuthBase`

  Force requests to ignore the `~/.netrc` file.

  Some sites do not support regular authentication, but we still want to store credentials in the `~/.netrc` file and submit them as form elements. Without this, requests would otherwise use the `~/.netrc` which leads, on some sites, to a 401 error.

  Use with:

      requests.get( url, auth=NullAuth() )

  Source:

  > https://github.com/psf/requests/issues/2773#issuecomment-174312831

class cso_dataspace.CSO_DataSpace_Downloader

  Bases: `object`

  Class to download single file from Copernicus DataSpace. The object defined by the class is used to store an access token that is re-used until it is expired.

  The `DownloadFile` method should be used to actually download a file.

  Usage:

      # initialize downloader:
      downloader = CSO_DataSpace_Downloader()
      # download file:
      downloader.DownloadFile( "https://zipper.dataspace.copernicus.eu/odata/v1/Products('d483baa0-3a61-4985-aa0c-5642a83c9214')/$value", "orbit.nc" )

  CreateToken(href, indent='')

    Create access token.

  DownloadFile(href, output_file, maxtry=10, nsec_wait=5, nsec_wait_max=600, dmode=None, indent='')

    Download file from DataSpace.

    If a request fails it is tried again up to a maximum of `maxtry` times, with a delay of initially `nsec_wait` between requsts.

    Arguments:

    - `href` : download url, for example:

          https://zipper.dataspace.copernicus.eu/odata/v1/Products('d483baa0-3a61-4985-aa0c-5642a83c9214')/$value

    - `output_file` : target file

    Optional arguments:

    - `maxtry` : number of times to try again if download fails

    - `nsec_wait` : initial delay in seconds between requests; with every new attempt, this will be increased up to a maximum of `nsec_wait_max`

    - `dmode`: directory creation mode, e.g. `777`