cso_dataspace module

The cso_dataspace module provides classes for accessing data from the Copernicus DataSpace.

To browse through the data, use the Browser.

Account setup

To be able to download data from the DataSpace, first Register and create an account.

On a Linux system, login/passwords for websites can be stored in the users .netrc file in the home directory. Create this file if it does not exist yet, and add the following line with the login name of the account (your email) and the chosen password:

machine zipper.dataspace.copernicus.eu      login Your.Name@institute.org   password ***********

The file should be readible and writable for you only:

chmod 400 ~/.netrc

DataSpace API’s

The DataSpace could be access with a number of different APIs.

Currently the OpenSearch API is used as that was the first that worked as needed.

In future the STAC API might be used, as this is becoming more and more the standard in the Earth Observation community. Within CSO it was already used by for example PAL API, but could not get working yet for the DataSpace. See the STAC product catalog for more information.

Class hierchy

The classes and are defined according to the following hierchy:

Classes

class cso_dataspace.CSO_DataSpace_Inquire(rcfile, rcbase='', env={}, indent='')

Bases: UtopyaRc

Inquire available Sentinel data from the Copernicus DataSpace.

Before data could be downloaded from the DataSpace, setup your Account setup.

Currently the OpenSearch API is used as that was the first that worked as needed; in future, the STAC product catalog might be used. See also the OpenSearch Description for the various parameters that the API accepts.

A query is sent to search for products that are available for a certain time and overlap with a specified region. The result is a list with orbit files and instructions on how to download them.

In the settings, specify the time range over which files should be searched:

<rcbase>.timerange.start  :  2018-07-01 00:00
<rcbase>.timerange.end    :  2018-07-01 23:59

Specify the base url of the API:

<rcbase>.url              :  https://catalogue.dataspace.copernicus.eu/resto/api

Define the collection name with:

<rcbase>.collection       :  Sentinel5P

Provide a product type:

! product type (always 10 characters!):
<rcbase>.producttype            :  L2__NO2___

Eventually specify a target area, only orbits with some pixels within the defined box will be downloaded:

! target area, leave empty for globe; format:  west,south,east,north
<rcbase>.area             :
!<rcbase>.area             :  -30,30,35,76

The table will also create the url’s to download a file; specifity the template that should be used:

! template for download url given “{product_id}”: <rcbase>.download_url : https://zipper.dataspace.copernicus.eu/odata/v1/Products({product_id})/$value

Name of output csv file:

! output table, here including date of today:
<rcbase>.output.file            :  ${my.work}/PAL_S5P_NO2_%Y-%m-%d.csv

Example records (with extra whitespace to show the columns):

orbit;start_time         ;end_time           ;processing;collection;processor_version;filename                                                                              ;href
11488;2020-01-01 02:34:16;2020-01-01 04:15:46;RPRO      ;03        ;020400           ;S5P_RPRO_L2__CH4____20200101T023416_20200101T041546_11488_03_020400_20221120T003820.nc;https://zipper.dataspace.copernicus.eu/odata/v1/Products(b3f240e6-505d-4cae-97ea-43a8778a318d)/$value
11487;2020-01-01 00:52:46;2020-01-01 02:34:16;RPRO      ;03        ;020400           ;S5P_RPRO_L2__CH4____20200101T005246_20200101T023416_11487_03_020400_20221120T003818.nc;https://zipper.dataspace.copernicus.eu/odata/v1/Products(a3d40f81-6c86-44bc-bc4b-457ff069b121)/$value
:
class cso_dataspace.NullAuth

Bases: AuthBase

Force requests to ignore the ~/.netrc file.

Some sites do not support regular authentication, but we still want to store credentials in the ~/.netrc file and submit them as form elements. Without this, requests would otherwise use the ~/.netrc which leads, on some sites, to a 401 error.

Use with:

requests.get( url, auth=NullAuth() )

Source:

class cso_dataspace.CSO_DataSpace_Downloader

Bases: object

Class to download single file from Copernicus DataSpace. The object defined by the class is used to store an access token that is re-used until it is expired.

The DownloadFile method should be used to actually download a file.

Usage:

# initialize downloader:
downloader = CSO_DataSpace_Downloader()
# download file:
downloader.DownloadFile( "https://zipper.dataspace.copernicus.eu/odata/v1/Products('d483baa0-3a61-4985-aa0c-5642a83c9214')/$value", "orbit.nc" )
CreateToken(href, indent='')

Create access token.

DownloadFile(href, output_file, maxtry=10, nsec_wait=5, nsec_wait_max=600, dmode=None, indent='')

Download file from DataSpace.

If a request fails it is tried again up to a maximum of maxtry times, with a delay of initially nsec_wait between requsts.

Arguments:

  • href : download url, for example:

    https://zipper.dataspace.copernicus.eu/odata/v1/Products('d483baa0-3a61-4985-aa0c-5642a83c9214')/$value
    
  • output_file : target file

Optional arguments:

  • maxtry : number of times to try again if download fails

  • nsec_wait : initial delay in seconds between requests; with every new attempt, this will be increased up to a maximum of nsec_wait_max

  • dmode: directory creation mode, e.g. 777