# `cso_earthaccess` module

The `cso_earthaccess` module uses the EarthAccess package to inquire and download NAS Earth science data.

## Class hierchy

The classes and are defined according to the following hierchy:

- `UtopyaRc`

  - `CSO_EarthAccess_Inquire`

  - `CSO_EarthAccess_Download`

## Classes

class cso_earthaccess.CSO_EarthAccess_Inquire(rcfile, rcbase='', env={}, indent='')

  Bases: `UtopyaRc`

  Create listing table (csv file) with on each line the location and information on a data file available via the EarthAccess package.

  As example, a file with VIIRS AOD could be available as:

      AERDB_L2_VIIRS_SNPP.A2020001.0000.002.2022244160133.nc

  Following the product-information the generic form of a filename is:

      ESDT.AYYYYDDD.HHMM.CCC.YYYYDDDHHMMSS.Format

  where:

  - `ESDT` : Earth Science Data Type or Shortname

  - `A` : Stands for Acquisition

  - `YYYYDDD` : Data acquisition year and Day-of-year per the Julian Calendar

  - `HHMM` : Acquisition Hour and Minute

  - `CCC` : Version ID of the data collection; note that Version ID `002` is in file attributes translated to product version `2.0.0`

  - `YYYYDDDHHMMSS` : Processing year, Day-of-year, UTC time (hour, minutes, seconds)

  - `Format` : File format suffix, which in the above case represents netCDF4

  This inquire class will search for available files, and based on the returned information create a summary table:

      filename                                              ;start_time     ;end_time       ;product;platform;processor_version;href
      AERDB_L2_VIIRS_SNPP.A2018001.0012.011.2020151235312.nc;20180101T001200;20180101T001800;AERDB  ;SNPP    ;010100           ;https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/5111/AERDB_L2_VIIRS_SNPP/2018/001/AERDB_L2_VIIRS_SNPP.A2018001.0012.011.2020151235312.nc
      AERDB_L2_VIIRS_SNPP.A2018001.0012.002.2023076192155.nc;20180101T001200;20180101T001800;AERDB  ;SNPP    ;020000           ;https://data.laadsdaac.earthdatacloud.nasa.gov/prod-lads/AERDB_L2_VIIRS_SNPP/AERDB_L2_VIIRS_SNPP.A2018001.0012.002.2023076192155.nc
      :

  In here:

  - the `product` and `platform` are extracted from the `ESDT` (first part of the filename);

  - the `processor_version` is expressed as a 6-digit number derived from the Collection Version in the meta-data; this form is used to have similar numbers as used for Sentinel-5P data;

  - the `href` is an url that is used for downloading the actual data.

  In the settings, specify the time range over which files should be searched:

      <rcbase>.timerange.start  :  2018-07-01 00:00
      <rcbase>.timerange.end    :  2018-07-01 23:59

  Specify the base url of the API:

      <rcbase>.url              :  https://catalogue.dataspace.copernicus.eu/resto/api

  Define the dataset name (ESDT value); see for example the table on the VIIRS Aerosol page:

      <rcbase>.dataset             :  AERDB_L2_VIIRS_SNPP
      <rcbase>.dataset             :  AERDB_L2_VIIRS_NOAA20
      <rcbase>.dataset             :  AERDT_L2_VIIRS_SNPP
      <rcbase>.dataset             :  AERDT_L2_VIIRS_NOAA20

  Eventually specify a target area, only orbits with some pixels within the defined box will be downloaded:

      ! target area, leave empty for globe; format:  west,south,east,north
      <rcbase>.area             :
      !<rcbase>.area             :  -30,30,35,76

  Name of output csv file:

      ! output table, here including date of today:
      <rcbase>.output.file            :  ${my.work}/AERDB_inquiry_%Y-%m-%d.csv

  Optionally define a creation mode for the (parent) directories:

      ! directory creation mode:
      <rcbase>.dmode                         :  0o775

  An existing listing file is not replaced, unless the following flag is set:

      ! renew table?
      <rcbase>.renew           :  True

class cso_earthaccess.CSO_EarthAccess_Download(rcfile, rcbase='', env={}, indent='')

  Bases: `UtopyaRc`

  Download data from NASA EarthData portal using EarthAccess package.

  To be able to download data, an account has to be created on EarthData Login. Store the login name and password in the `~/.netrc` file in your home directory (or in a file defined by the `NETRC` environment variable) as:

      machine  urs.earthdata.nasa.gov   login your.name@inst.nx  password ********

  The download class is initialized by arguments:

  - `rcfile`, `rcbase`, `env` : settings file, prefix for keys, and environment dictionairy

  The input files are searched in a table created by the `CSO_EarthAccess_Inquire` class, which has scanned the EarthData portal to examine which files are available, and stored the result in a csv file. Specify the name of the csv file; this might contain templates for a date that is taken from another key:

      ! listing of available source files,
      ! created by for example 'inquire' job:
      <rcbase>.inquire.file                   :  /data/EarthData/AERDB_L2_VIIRS_SNPP/inquire_%Y-%m-%d.csv
      !! date used in filename, leave empty for today:
      !<rcbase>.inquire.filedate               :  2022-01-28

  A time range is read from the settings to select data:

      <rcbase>.timerange.start        :  2018-06-01 00:00
      <rcbase>.timerange.end          :  2018-06-03 23:59

  Provide a ‘;’-seperated list of expressions to decide if a particular file should be processed. If more than one file is available for a particular orbit, for example from different processing versions, then the file with the first match will be used. The expressions should include templates ‘%{header}’ for the column values:

      <rcbase>.selection           :  %{processor_version} == '020000'

  Specify the directory where the input files are to be searched, or where to download them to if not present yet:

      ! target dir for downloads, including templates for year and julian day:
      <rcbase>.dir                             :  /data/EarthData/AERDB_L2_VIIRS_SNPP/%Y/%j

  Optionally define a creation mode for the (parent) directories:

      ! directory creation mode:
      <rcbase>.dmode                         :  0o775

  Define if existing files need to be replaced:

      ! download existing files again?
      <rcbase>.renew                           :  False