cso_earthaccess module

The cso_earthaccess module uses the EarthAccess package to inquire and download NAS Earth science data.

Class hierchy

The classes and are defined according to the following hierchy:

Classes

class cso_earthaccess.CSO_EarthAccess_Inquire(rcfile, rcbase='', env={}, indent='')

Bases: UtopyaRc

Create listing table (csv file) with on each line the location and information on a data file available via the EarthAccess package.

As example, a file with VIIRS AOD could be available as:

AERDB_L2_VIIRS_SNPP.A2020001.0000.002.2022244160133.nc

Following the product-information the generic form of a filename is:

ESDT.AYYYYDDD.HHMM.CCC.YYYYDDDHHMMSS.Format

where:

  • ESDT : Earth Science Data Type or Shortname

  • A : Stands for Acquisition

  • YYYYDDD : Data acquisition year and Day-of-year per the Julian Calendar

  • HHMM : Acquisition Hour and Minute

  • CCC : Version ID of the data collection; note that Version ID 002 is in file attributes translated to product version 2.0.0

  • YYYYDDDHHMMSS : Processing year, Day-of-year, UTC time (hour, minutes, seconds)

  • Format : File format suffix, which in the above case represents netCDF4

This inquire class will search for available files, and based on the returned information create a summary table:

filename                                              ;start_time     ;end_time       ;product;platform;processor_version;href
AERDB_L2_VIIRS_SNPP.A2018001.0012.011.2020151235312.nc;20180101T001200;20180101T001800;AERDB  ;SNPP    ;010100           ;https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/5111/AERDB_L2_VIIRS_SNPP/2018/001/AERDB_L2_VIIRS_SNPP.A2018001.0012.011.2020151235312.nc
AERDB_L2_VIIRS_SNPP.A2018001.0012.002.2023076192155.nc;20180101T001200;20180101T001800;AERDB  ;SNPP    ;020000           ;https://data.laadsdaac.earthdatacloud.nasa.gov/prod-lads/AERDB_L2_VIIRS_SNPP/AERDB_L2_VIIRS_SNPP.A2018001.0012.002.2023076192155.nc
:

In here:

  • the product and platform are extracted from the ESDT (first part of the filename);

  • the processor_version is expressed as a 6-digit number derived from the Collection Version in the meta-data; this form is used to have similar numbers as used for Sentinel-5P data;

  • the href is an url that is used for downloading the actual data.

In the settings, specify the time range over which files should be searched:

<rcbase>.timerange.start  :  2018-07-01 00:00
<rcbase>.timerange.end    :  2018-07-01 23:59

Specify the base url of the API:

<rcbase>.url              :  https://catalogue.dataspace.copernicus.eu/resto/api

Define the dataset name (ESDT value); see for example the table on the VIIRS Aerosol page:

<rcbase>.dataset             :  AERDB_L2_VIIRS_SNPP
<rcbase>.dataset             :  AERDB_L2_VIIRS_NOAA20
<rcbase>.dataset             :  AERDT_L2_VIIRS_SNPP
<rcbase>.dataset             :  AERDT_L2_VIIRS_NOAA20

Eventually specify a target area, only orbits with some pixels within the defined box will be downloaded:

! target area, leave empty for globe; format:  west,south,east,north
<rcbase>.area             :
!<rcbase>.area             :  -30,30,35,76

Name of output csv file:

! output table, here including date of today:
<rcbase>.output.file            :  ${my.work}/AERDB_inquiry_%Y-%m-%d.csv

Optionally define a creation mode for the (parent) directories:

! directory creation mode:
<rcbase>.dmode                         :  0o775

An existing listing file is not replaced, unless the following flag is set:

! renew table?
<rcbase>.renew           :  True
class cso_earthaccess.CSO_EarthAccess_Download(rcfile, rcbase='', env={}, indent='')

Bases: UtopyaRc

Download data from NASA EarthData portal using EarthAccess package.

To be able to download data, an account has to be created on EarthData Login. Store the login name and password in the ~/.netrc file in your home directory (or in a file defined by the NETRC environment variable) as:

machine  urs.earthdata.nasa.gov   login your.name@inst.nx  password ********

The download class is initialized by arguments:

  • rcfile, rcbase, env : settings file, prefix for keys, and environment dictionairy

The input files are searched in a table created by the CSO_EarthAccess_Inquire class, which has scanned the EarthData portal to examine which files are available, and stored the result in a csv file. Specify the name of the csv file; this might contain templates for a date that is taken from another key:

! listing of available source files,
! created by for example 'inquire' job:
<rcbase>.inquire.file                   :  /data/EarthData/AERDB_L2_VIIRS_SNPP/inquire_%Y-%m-%d.csv
!! date used in filename, leave empty for today:
!<rcbase>.inquire.filedate               :  2022-01-28

A time range is read from the settings to select data:

<rcbase>.timerange.start        :  2018-06-01 00:00
<rcbase>.timerange.end          :  2018-06-03 23:59

Provide a ‘;’-seperated list of expressions to decide if a particular file should be processed. If more than one file is available for a particular orbit, for example from different processing versions, then the file with the first match will be used. The expressions should include templates ‘%{header}’ for the column values:

<rcbase>.selection           :  %{processor_version} == '020000'

Specify the directory where the input files are to be searched, or where to download them to if not present yet:

! target dir for downloads, including templates for year and julian day:
<rcbase>.dir                             :  /data/EarthData/AERDB_L2_VIIRS_SNPP/%Y/%j

Optionally define a creation mode for the (parent) directories:

! directory creation mode:
<rcbase>.dmode                         :  0o775

Define if existing files need to be replaced:

! download existing files again?
<rcbase>.renew                           :  False