cso_earthaccess
module¶
The cso_earthaccess
module uses the EarthAccess package
to inquire and download NAS Earth science data.
Class hierchy¶
The classes and are defined according to the following hierchy:
Classes¶
- class cso_earthaccess.CSO_EarthAccess_Inquire(rcfile, rcbase='', env={}, indent='')¶
Bases:
UtopyaRc
Create listing table (csv file) with on each line the location and information on a data file available via the EarthAccess package.
As example, a file with VIIRS AOD could be available as:
AERDB_L2_VIIRS_SNPP.A2020001.0000.002.2022244160133.nc
Following the product-information the generic form of a filename is:
ESDT.AYYYYDDD.HHMM.CCC.YYYYDDDHHMMSS.Format
where:
ESDT
: Earth Science Data Type or ShortnameA
: Stands for AcquisitionYYYYDDD
: Data acquisition year and Day-of-year per the Julian CalendarHHMM
: Acquisition Hour and MinuteCCC
: Version ID of the data collection; note that Version ID002
is in file attributes translated to product version2.0.0
YYYYDDDHHMMSS
: Processing year, Day-of-year, UTC time (hour, minutes, seconds)Format
: File format suffix, which in the above case represents netCDF4
This inquire class will search for available files, and based on the returned information create a summary table:
filename ;start_time ;end_time ;product;platform;processor_version;href AERDB_L2_VIIRS_SNPP.A2018001.0012.011.2020151235312.nc;20180101T001200;20180101T001800;AERDB ;SNPP ;010100 ;https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/5111/AERDB_L2_VIIRS_SNPP/2018/001/AERDB_L2_VIIRS_SNPP.A2018001.0012.011.2020151235312.nc AERDB_L2_VIIRS_SNPP.A2018001.0012.002.2023076192155.nc;20180101T001200;20180101T001800;AERDB ;SNPP ;020000 ;https://data.laadsdaac.earthdatacloud.nasa.gov/prod-lads/AERDB_L2_VIIRS_SNPP/AERDB_L2_VIIRS_SNPP.A2018001.0012.002.2023076192155.nc :
In here:
the
product
andplatform
are extracted from theESDT
(first part of the filename);the
processor_version
is expressed as a 6-digit number derived from the Collection Version in the meta-data; this form is used to have similar numbers as used for Sentinel-5P data;the
href
is an url that is used for downloading the actual data.
In the settings, specify the time range over which files should be searched:
<rcbase>.timerange.start : 2018-07-01 00:00 <rcbase>.timerange.end : 2018-07-01 23:59
Specify the base url of the API:
<rcbase>.url : https://catalogue.dataspace.copernicus.eu/resto/api
Define the dataset name (ESDT value); see for example the table on the VIIRS Aerosol page:
<rcbase>.dataset : AERDB_L2_VIIRS_SNPP <rcbase>.dataset : AERDB_L2_VIIRS_NOAA20 <rcbase>.dataset : AERDT_L2_VIIRS_SNPP <rcbase>.dataset : AERDT_L2_VIIRS_NOAA20
Eventually specify a target area, only orbits with some pixels within the defined box will be downloaded:
! target area, leave empty for globe; format: west,south,east,north <rcbase>.area : !<rcbase>.area : -30,30,35,76
Name of output csv file:
! output table, here including date of today: <rcbase>.output.file : ${my.work}/AERDB_inquiry_%Y-%m-%d.csv
Optionally define a creation mode for the (parent) directories:
! directory creation mode: <rcbase>.dmode : 0o775
An existing listing file is not replaced, unless the following flag is set:
! renew table? <rcbase>.renew : True
- class cso_earthaccess.CSO_EarthAccess_Download(rcfile, rcbase='', env={}, indent='')¶
Bases:
UtopyaRc
Download data from NASA EarthData portal using EarthAccess package.
To be able to download data, an account has to be created on EarthData Login. Store the login name and password in the
~/.netrc
file in your home directory (or in a file defined by theNETRC
environment variable) as:machine urs.earthdata.nasa.gov login your.name@inst.nx password ********
The download class is initialized by arguments:
rcfile
,rcbase
,env
: settings file, prefix for keys, and environment dictionairy
The input files are searched in a table created by the
CSO_EarthAccess_Inquire
class, which has scanned the EarthData portal to examine which files are available, and stored the result in a csv file. Specify the name of the csv file; this might contain templates for a date that is taken from another key:! listing of available source files, ! created by for example 'inquire' job: <rcbase>.inquire.file : /data/EarthData/AERDB_L2_VIIRS_SNPP/inquire_%Y-%m-%d.csv !! date used in filename, leave empty for today: !<rcbase>.inquire.filedate : 2022-01-28
A time range is read from the settings to select data:
<rcbase>.timerange.start : 2018-06-01 00:00 <rcbase>.timerange.end : 2018-06-03 23:59
Provide a ‘;’-seperated list of expressions to decide if a particular file should be processed. If more than one file is available for a particular orbit, for example from different processing versions, then the file with the first match will be used. The expressions should include templates ‘%{header}’ for the column values:
<rcbase>.selection : %{processor_version} == '020000'
Specify the directory where the input files are to be searched, or where to download them to if not present yet:
! target dir for downloads, including templates for year and julian day: <rcbase>.dir : /data/EarthData/AERDB_L2_VIIRS_SNPP/%Y/%j
Optionally define a creation mode for the (parent) directories:
! directory creation mode: <rcbase>.dmode : 0o775
Define if existing files need to be replaced:
! download existing files again? <rcbase>.renew : False