cso_file module¶
Create and access file with satellite data extract.
Methods¶
The following methods are defined:
Class hierchy¶
The classes are defined according to the following hierchy:
Methods and classes¶
- cso_file.CheckDir(filename, dmode=None)¶
Check if
filenamehas a directory path; if so, create that directory if it does not exist yet. Optionaldmode(default0o777) is used for creation of directories.
- cso_file.Pack_DataArray(da, dtype='i2')¶
Set encoding parameters of
xarray.DataArrayobjectda; on return, the argument is changed in-place.Argument
dtypedefines the data type in which the array will be written; by default this is a short integer (i2, 2-bytes) instead of a float. Attributes are added that define how to decode values:short pressure(z) ; pressure:add_offset = 53257.7493494103 ; pressure:scale_factor = 1.50569706659902 ; pressure:_FillValue = 32767s ;
The actual values are recalculated using the formula:
values = add_offset + scale_factor * values__short
The packing parameters are computed using:
scale_factor = (vmax - vmin)/( fmax - fmin) add_offset = vmin - scale_factor * fmin
where
[vmin,vmax]is the range of input values, and[fmin,fmax]the range of possible values of the stored data type.The
scale_factorshould not become 0.0 since it is used for division during encoding. This will be the case when the input has the same value in all elements; in that case, the following parameters are used that will lead to zero packed values:scale_factor = 1.0 add_offset = vmin
- class cso_file.CSO_File(filename=None)¶
Bases:
objectStorage for satelite data extract.
If optional
filenameis provided, theRead()method is called to read data from a file. If the optionalvarnamelist is specified too, only selected variables will be read.To create a new track, initialize without arguments and add new variables using the
AddVariable()method. Only certain dimension names and combinations are supported.- Close()¶
Close dataset.
- AddDim(dname, length)¶
Add dimension.
Arguments:
dname: (str) dimension name to be createdlen: (int) dimension length
- GetDimLength(dname)¶
Return dimension length.
- GetShape(dnames)¶
Return shape following specified dimension names.
- AddCoord(cname, da)¶
Add coordinate variable.
Arguments:
cname: coordinate name to be createdda:xarray.DataArrayobject
- GetCoords(cnames)¶
Return dict with coordinate variables as specified in
cnames.
- AddTimeCoord(times, bounds=None)¶
Add time coordinate variable.
Arguments:
times: time stamp series created with for examplepandas.date_range()method
Optional arguments:
* ``bounds=(times1,times2)`` : tupple or list with two timeseries that define the time bounds
- AddVariable(vname, da)¶
Add variable field.
Arguments:
vname: variable name to be createdda:xarray.DataArrayobject
- AddDataVariable(vname, dnames, dtype='f4', attrs={}, values=None)¶
Add variable with specified name and dimensions.
Arguments:
vname: variable name to be createddnames: dimension names, e.g.('time','station')
Optional arguments:
dtype: datatype keywordattrs: dict with variable attributesvalues: array with data values
- Write(filename, unlimited_dims=None, attrs=None, history=[], packed=True, complevel=1)¶
Write data to provided
filename.Optional arguments:
attrs: dictionairy with global attributeshistory: list of str values that describe how the content was created, this will be added to the globalhistoryattributeunlimited_dims: names of unlimited dimesions
Optional arguments to disable file size reduction measures:
packed: by default float arrays are written as short-integers accompanied withadd_offsetandscale_factorattributes; set toFalseto keep original data types.complevel: zlib compression level, default 1 (of 9) to have decent compression without too much overhead; set to 0 to have no compression at all
The following example shows the impact of packing and compression on file sizes; in this exmple, the combination of packing and deflation reduces file sizes with 83%%:
301.009.424 S5p_RPRO_GLYOX_01130.nc 111.648.738 S5p_RPRO_GLYOX_01130__complevel1.nc 104.134.232 S5p_RPRO_GLYOX_01130__complevel9.nc 153.334.891 S5p_RPRO_GLYOX_01130__packed.nc 41.020.139 S5p_RPRO_GLYOX_01130__packed_complevel1.nc
- GetDims(varname)¶
Return variable dimension names.
- GetData(varname)¶
Return data values and units for named variable.
- GetTrack(da)¶
Extract variable on track grid.
Arguments:
da:xarray.DataArrayor variable name
Return values:
xx: corner longitudes (nimage+1,npixel+1)yy: corner latitudes (nimage+1,npixel+1)values: masked array with values at pixel locations (nimage,npixel)units: str units
The corner locations are created using the following variables that are expected to be present:
track_longitude_bounds: 3D array of longitudes with dimensions('scan','pixel','corner')track_latitude_bounds: idem for latitudespixel: compress index that defines the('scan','pixel')location
Example usage to obtain retrieval from data file that includes the track information:
xx,yy,vcd_retr,vcd_units = dfile.GetTrack( 'vcd' )
Example usage for simulated retrieval from state file:
xx,yy,vcd_retr,vcd_units = dfile.GetTrack( sfile.ds['vcd'] )
- GetTimeRange()¶
Return time range of pixels in file. A variable
timeshould be present for this.
- GetTimeAverage(freq=None)¶
Return average of first and last time value.
Optional arguments:
freq: round time to given fequency, for example ‘60min’.
- GetAttr(name, quiet=False)¶
Return global attribute with provided name. If the attribtue is not present an error is raised, unless
quiet=True.
- SelectPixels(rcf, rckey, indent='')¶
Apply filters specified in rcfile.
Arguments:
rcf:RcFileobject with settingsrckey: basename for rcfile keys, e.g. “s5p” for the example below
Return values:
selected: boolean array with shape of track(pixel)which is True if a pixel passed all checks;history: list of character string describing the selelections.
Example configuration:
! Specifiy a list of filter names. ! For each name, specify the variable which values are used for testing, ! the type test, and eventually some thresholds or other settings for this type. ! The units of the thresholds should match with the units in the variables, ! the expected units have to be defined too. ! The examples below show possible types and their settings. ! filters: s5p.filters : lons lats albedo quality detected valid ! select range of values: s5p.filter.lons.var : longitude s5p.filter.lons.type : minmax s5p.filter.lons.minmax : -15.0 35.0 s5p.filter.lons.units : degrees_east ! select above a minimum: s5p.filter.lats.var : latitude s5p.filter.lats.type : minmax s5p.filter.lats.minmax : 35.0 70.0 s5p.filter.lats.units : degrees_north ! select up to a maximum: s5p.filter.albedo.var : albedo s5p.filter.albedo.type : max s5p.filter.albedo.max : 0.3 s5p.filter.albedo.units : 1 ! select from a minimum: s5p.filter.quality.var : qa_value s5p.filter.quality.type : min s5p.filter.quality.max : 0.8 s5p.filter.quality.units : 1 ! select on flag: s5p.filter.detected.var : detection_flag s5p.filter.detected.type : equal s5p.filter.detected.value : 1 ! select only values with data (no "_FillValue"): s5p.filter.valid.var : vmr s5p.filter.valid.type : valid
- class cso_file.CSO_Listing(filename=None, indent='')¶
Bases:
objectStorage for table with file properties. The records describe properties of
CSO_Filefiles, for example the time range of the pixels included. This could be used to quickly scan an archive for filenames with pixels within a requested interval.The content of the table file looks like:
filename;start_time;end_time 2007/06/RAL-IASI-CH4_20070601T003855.nc;2007-06-01 00:45:36.827000000;2007-06-01 02:23:57.567000000 2007/06/RAL-IASI-CH4_20070601T022359.nc;2007-06-01 02:23:59.512000000;2007-06-01 04:05:57.328000000 ...
Optional arguments:
filename: listing file read into table
- Save(filename, dmode=None, indent='')¶
Write table to file.
- GetRecord(irec)¶
Return record for corresponding 0-based row index.
- GetValue(filename, key)¶
Return
keyvalue for record offilename.
- GetValues(name=None)¶
Return
pandas.Seriesobject with all values for provided columnname, or the filenames ifnameis not provided.
- keys()¶
Returns list of column names.
- Cleanup(indent='')¶
Remove records from table if filename is not present anymore.
- Check(filename, indent='')¶
Return exception if no record for
filenameis present.
- Update(filename, csf, tr=None, xcolumns=[], indent='')¶
Add record (or replace) for
filenamewith information from thecsfCSO_Fileobject.The time range of pixels is read from the
csfobject, unless thedatetime.datetimetuppletr=(t1,t2)is supplied.The optional
xcolumnsspecify a list of global attributes to be added as extra columns.
- UpdateRecord(filename, data, indent='')¶
Add record (or replace) for
filenamewith columns and values defined in thedatadict.
- Select(tr=None, method='overlap', expr=None, blacklist=[], indent='', **kwargs)¶
Return
CSO_Listingobjects with selection of records.Optional arguments:
tr=(t1,t2): select records that could be assigned to this timerange; requires themethodkeyword:method='overlap'(default) : select records that have pixels overlapping with intervalmethod='middle': select pixels with the middle ofstart_timeandend_timewithin the inerval(t1,t2]
name=valuearguments select records based on column valuesThe exprssion
exprprovides a list of ‘;’-seperated selection expressions with templates%{..}for the column names:(%{processor_version} == '020400') & (%{processing} == 'RPRO') ; ...
This is evaluted after the previously described selections. The result should be None or exactly one record. Eventually skip files in
blacklists.
- Append(lst, path='')¶
Append content of other listing.
- Sort(by='filename')¶
Sort listing table by filename or other key.