# `cso_colhub` module

The `cso_colhub` module provides classes for accessing data from the Norwegian ColHub archive. This is a (partial) mirror of the Copernicus Open Access Hub with all orbits covering Norway and surrounding areas.

The data can be accessed using a web-interface, but for Met Norway users also directly from the storage. This module provides the `CSO_ColHubMirror_Inquire` to inquire the storage to see what is already available. In addition, the `CSO_ColHubMirror_Missing` is available to create a table that lists files that are in the DataSpace but not yet in the mirror; these could be used to make additional downloads.

## Class hierchy

The classes and are defined according to the following hierchy:

- `UtopyaRc`

  - `CSO_ColHubMirror_Inquire`

  - `CSO_ColHubMirror_Missing`

## Classes

class cso_colhub.CSO_ColHubMirror_Inquire(rcfile, rcbase='', env={}, indent='')

  Bases: `UtopyaRc`

  Create listing file for files available in file archive.

  The format is similar to the output of inquiry classes, with per line a filename, the time range of pixels in the file, and other information extracted from the filenames:

      filename                                                                              ;processing;start_time         ;end_time           ;orbit;collection;processor_version;href
      S5P_RPRO_L2__CH4____20180430T001851_20180430T020219_02818_01_010301_20190513T141133.nc;RPRO      ;2018-04-30T00:18:51;2018-04-30T02:02:19;02818;01        ;010301           ;/archive/mirror/S5P_RPRO_L2__CH4____20180430T001851_20180430T020219_02818_01_010301_20190513T141133.nc
      S5P_RPRO_L2__CH4____20180430T020021_20180430T034349_02819_01_010301_20190513T135953.nc;RPRO      ;2018-04-30T02:00:21;2018-04-30T03:43:49;02819;01        ;010301           ;/archive/mirror/S5P_RPRO_L2__CH4____20180430T020021_20180430T034349_02819_01_010301_20190513T135953.nc
      :

  This file could be used to scan for available versions and how they were produced.

  In the settings, define one or more base directories of the archive:

      <rcbase>.dir           :  /archive/mirror  /scratch/mirror2

  The directories are recursively scanned using the `os.walk` class on files that match a filename pattern::

      ! search S5P CH4 files:
      <rcbase>.pattern       :  S5P_*_L2_CH4___*.nc

  Specifiy the output file:

      ! csv file that will hold records per file with:
      ! - timerange of pixels in file
      ! - orbit number
      ! time templates are replaced with todays date
      <rcbase>.file        :  /Scratch/Copernicus/S5p/listing-CH4__%Y-%m-%d.csv

  Optionally define a creation mode for the (parent) directories:

      ! directory creation mode:
      <rcbase>.dmode                         :  0o775

  An existing listing file is not replaced, unless the following flag is set:

      ! renew table?
      <rcbase>.renew           :  True

class cso_colhub.CSO_ColHubMirror_Missing(rcfile, rcbase='', env={}, indent='')

  Bases: `UtopyaRc`

  Create listing file for files that are in one inquiry table but not in another one. This could be used to complete a mirror archive.

  The format is similar to the output of inquiry classes, with per line a filename, the time range of pixels in the file, and other information extracted from the filenames:

      filename                                                                              ;start_time         ;end_time           ;orbit;processing;collection;processor_version;href
      S5P_RPRO_L2__NO2____20180504T073130_20180504T091300_02879_03_020400_20221208T160012.nc;2018-05-04 07:31:30;2018-05-04 09:13:00;02879 ;RPRO     ;03        ;020400           ;https://zipper.dataspace.copernicus.eu/odata/v1/Products(ae43b35c-0569-4e1f-b8cb-0afc49790716)/$value
      :

  In the settings, define the listing file with all available data, for example the result of an inquiry step; eventually add a timestamp to replace templates in the filename:

      <rcbase>.all.file           :  /work/inquire/Copernicus_S5p_NO2_dataspace__%Y-%m-%d.csv
      !<rcbase>.all.filedate       :  2025-01-24

  Similar specify the name (or “;” seperated list of names of the file that is listing the current mirror(s), probably the output of the `CSO_ColHubMirror_Inquire` class:

      <rcbase>.curr.file           :  /work/inquire/Copernicus_S5p_NO2_colhub-mirror__%Y-%m-%d.csv ; \
                                      /work/inquire/Copernicus_S5p_NO2_colhub-mirror2__%Y-%m-%d.csv
      !<rcbase>.curr.filedate       :  2025-01-24

  Specify a selection filter; this defines which of the orbit files are actually needed:

      ! Provide ';' seperated list of to decide if a particular orbit file should be processed.
      ! If more than one file is available for a particular orbit (from "OFFL" and "RPRO" processing),
      ! the file with the first match will be used.
      ! The expressions should include templates '%{header}' for the column values.
      !
      <rcbase>.selection                     :  (%{collection} == '03') and (%{processing} == 'RPRO') ; \
                                                (%{collection} == '03') and (%{processing} == 'OFFL')

  Specifiy the output file:

      ! csv file that will hold records per file with:
      ! - timerange of pixels in file
      ! - orbit number
      ! time templates are replaced with todays date
      <rcbase>.file        :  /work/inquire/Copernicus_S5p_NO2_colhub-mirror-missing__%Y-%m-%d.csv

  Optionally define a creation mode for the (parent) directories:

      ! directory creation mode:
      <rcbase>.dmode                         :  0o775

  An existing listing file is not replaced, unless the following flag is set:

      ! renew table?
      <rcbase>.renew           :  True