# `utopya_index` module

The `utopya_indexer` module provides tools to create browsable html pages. These are typically used to take a first look at generated images.

## Accessing the index

The index pages are accessed by top level index page that is probably named:

    index.html

Load this page into a web browser to view the result. The top level page could consists of a combination of images and links to lower-level index pages.

Example of top level index with links to sub pages.

Example of index page with figure table.

## Producing index pages

The index pages are defined by rcfile settings. The name of the rcfile, and the base of the keywords should be passed to the `Indexer` class that will create the pages:

    utopya.Indexer( 'postproc.rc', rcbase='index', env={} )

This defines that the settings are read from `postproc.rc`, and that onlys settings with the prefix `index` are used.

## Configuration

The following is an example of a top level definition:

    ! (optional) target location:
    index.outdir                    :  /scratch/flexvar/postproc

    ! (optional) show info on created page?
    index.info                      :  True

    ! (optional) title:
    index.header                    :  WP1

    ! content type:
    index.type                      :  sections

The `outdir` setting defines the (optional) work directory where to create the index pages, default is current directory.

The `info` flag enables that a message is displayed with the url of the index page that should be loaded into a browser. For the top-level page this flag is probably enabled; the displayed message looks like:

    Browse to:
        file:///scratch/flexvar/postproc/index.html

The `header` settings defines an explicit header to be displayed.

The `type` setting defines the content type of an index part as defined below.

## Index parts

The `type` keyword is interpreted by the `GetIndexPart()` method, and refers to a class derived from the `IndexPart` class. The following table shows the `type` keywords and the corresponding class:

type | class | produces …
--- | --- | ---
| `IndexPart` | (basic settings for all classes)
`sections` | `IndexSections` | sections within page
`list` | `IndexList` | bullet list
`table-row` | `IndexTableRow` | table with rows
`table-col` | `IndexTableCol` | columns within table row
`column` | `IndexColumn` | table with only one column
`img` | `IndexImg` | image
`text` | `IndexText` | text
`txtfile` | `IndexTxtFile` | content of file

See the description of the classes for the settings that define an index part.

## Example

As example, the following settings will create index pages for images produced for different experiments. The experiments are named `A`, `B`, and `C`, and for each a number of image files is availble in different directories:

    WP1_A/NL1234_location.png
          NL1234_timeseries_2017-01.png
                                  :
          NL1234_timeseries_2017-12.png
          NL1235_location.png
          NL1235_timeseries_2017-01.png
                                  :
          stations.csv

    WP2_B/NL1234_location.png
          :

    WP2_C/NL1234_location.png
          :

The `stations.csv` files contain information on the station for which images are produced, the first column contains the station codes:

    code;long_name
    NL1234;Oeverloos
    NL1235;Laagwater
    :

A top-level index page is created that has a bullet list with links to sub-pages for different experiments:

    ! ~~ top level index

    ! (optional) target location:
    index.outdir                    :  /scratch/flexvar/postproc

    ! (optional) show info on created page?
    index.info                      :  True

    ! title:
    index.header                    :  WP1

    ! content type:
    index.type                       :  list
    ! generic name, used to define the content:
    index.name                       :  expid
    ! list elements, here experiment names:
    index.values                     :  A B C

For each of the experiments a sub-page should be created:

    ! create new page for each value:
    index.expid.newpage              :  True

The sub-page for an experiment should contain table with images. Each row should contain the images for a different station, the station codes are read from the csv file:

    ! content type:
    index.expid.type                 :  table-row
    ! define row values:
    index.expid.name                 :  scode
    index.expid.values               :  CsvFile( '../WP1_%{expid}/stations.csv' )

The first column should show an image that illustrates the station’s location, the other columns contain time series plots per month:

    ! content type:
    index.expid.scode.type           :  table-col
    ! define column values:
    index.expid.scode.name           :  plot
    index.expid.scode.values         :  location ; TimeSeries( '2017-01', '2017-12', '1 month', 'timeseries_%Y-%m' )

The content is finaly defined by image file names (in experiment specific directories) and optional arguments to limit the heigh of the images in the table:

    ! content type:
    index.expid.scode.plot.type      :  img
    ! define image:
    index.expid.scode.plot.img       :  ../WP1_%{expid}/%{scode}_%{plot}.png
    index.expid.scode.plot.kwargs    :  height=200

## Class hierarchy

- `UtopyaRc`

  - `Indexer`

  - `IndexPart`

    - `IndexSections`

    - `IndexList`

    - `IndexTableRow`

    - `IndexTableCol`

    - `IndexColumn`

    - `IndexImg`

    - `IndexText`

    - `IndexTextFile`

## Methods

- `GetIndexPart()`

## Classes and methods

class utopya_index.Indexer(rcfile, rcbase='', env={}, _indent='')

  Bases: `UtopyaRc`

  Builder for index pages using rcfile settings. See module documentation for description.

  Arguments:

  - `rcfile`, `rcbase`, `env` : settings etc, passed to base class

utopya_index.GetIndexPart(urc, current, _indent='')

  Return object derived from `IndexPart` class given type description.

  Arguments:

  - `urc` : `utopya.UtopyaRc` object for settings

  - `current` : `collections.OrderedDict` object with elements for index levels, ordered from top to current

  The keys of the `current` dict define the full prefix for keywords in the settings. For example, if this method is called for the third index level using a dict:

      current = { 'main' : { .. },
                  'row'  : { .. },
                  'col'  : { .. } }

  then the prefix is:

      main.row.col

  The following keys are read from the settings that have been imported into the `urc` object:

      ! type of index part:
      <prefix>.type       :  list

  See the table Index parts for the classes corresponding to type keyword.

class utopya_index.IndexPart(urc, current, _indent='')

  Bases: `UtopyaRc`

  Base class for index parts of html lines.

  Arguments:

  - `urc` : `utopya.UtopyaRc` object for settings

  - `current` : `collections.OrderedDict` object with elements per level

  An element of the `current` argument is a dict with info for each level:

      current = { 'main' : ...,
                  'row'  : ...,
                  'col'  : { 'value'    : 'timeseries',
                            'filename' : 'index.S1.expA.html',
                            'header'   : None,
                            'alink'    : 'index.S1.expA.html#plot' }
                }

  For the top level index (main page), the `current` is probably empty.

  The values are used to pass information between the levels of the index. For example, a lower level could use headers from higher levels to create upward links.

  The keys of the `current` dict define the full name of the prefix for the settings that define the content. For the above example the prefix is:

      main.row.col

  The keys that are read from the `urc` object by this base class are optional, and are mainly used for fine-tuning the index pages.

  The first (optional) key that is read defines the work directory where the index pages are created; the default is the current directory:

      <prefix>.outdir     :  /scratch/flevar/postproc

  The next flag defines whether this index part should be written to a new file:

      <prefix>.newpage    :  True

  The default is `False`, which means that the index part will be included in a higher level page. An exception is the top level index (empty `current` dict), this will always write a new page. In case `newpage` is enabled, the index part will be written to a filename which has by default the level elements in the name:

      index__main__row__col.html

  Alternatively, the target file could be defined in the settings:

      <prefix>.filename   :  whatever.html

  Each index part could started with a header. For some index parts (derived from this base class) a header is defined automatically; alternatively a header could be define explicitly:

      <prefix>.header      :  Overview of results

  If a header is defined (automatically or explicitly) it will be inserted as the first of the html lines. If the index part is written to a new page, the heading is composed as a list of headers from higher levels, with links to their locations:

      <main-label> / <row-label> / <col-label>

  Optionally a descriptive text could be provided. This is inserted as plain html in the index file, so html-markup might be used. A special formatting is the “n” characters that represent a new-line; these are replace by html line breaks:

      <prefix>.description    :  The purpose of this line is pure informative.\n\
                                 Some users might even read it ...

  For index parts included in the html page of a higher level, the heading will consists of the own header only, with the appropriate heading level within the page:

      <h1>Main label</h1>
      :
        <h2>Row label</h2>
        :
          <h3>Col label</h3>

  GetCurrentLine(key=None, sep='.', alinks=False)

    Return line with concatenation of elements in the `current` dict.

    Arguments:

    - `key` : str value; if defined, it should be one of the possible keys in the `current` elements, for example `value`; if not defined, the level name (`current` keys) is used;

    - `sep` : optional str value inserted between the concatenated elements;

    - `alinks` : if `True`, the elements are surround with links to anchors defined for the elements.

  ReplaceCurrents(line)

    Return line with templates replaced by level values. If the `currrent` dict consists of:

        current = { .., 'row' : { 'value' : 'NL1234', ..}, ..}

    then the input line:

        station_%{row}_location.png

    is returned as:

        station_NL1234_location.png

    Alternative only some selected characters can be inserted:

        station_%{row[0:2]}_location.png
        station_NL_location.png

        station_%{row[:2]}_location.png
        station_NL_location.png

        station_%{row[2:]}_location.png
        station_1234_location.png

  WriteFile()

    If a target file is defined for this object, write the html lines to this file.

    An optional flag is read from the settings, if enabled a message will be displayed to show the url of the index page:

        Browse to:
          file:///scratch/flexvar/postproc/index.html

  GetFilename()

    If this index part is written to a seperate file, return the targe file, or None otherwise.

  AddNavigation(up, prev, next)

    Add navigation buttons for up/previous/next to top and bottom of html lines. The arguments (if not `None`) define the links around the buttons. If an argument is `None`,

  GetValues(key)

    Return value list from settings defined by `key`.

    Supported formats:

    - Space seperated list:

          <key>    :  aa b 1 2 3 4 5

    - Semi-column seperated list; if an element looks like a function (including parantheses) then it is passed to the `utopya_tools.GetValuesFunction()` method:

          <key>    :  aa ; b ; Range(1,5)

class utopya_index.IndexSections(urc, current, _indent='')

  Bases: `IndexPart`

  Define index page that consists of one or more sections.

  Arguments:

  - `urc` : `utopya.UtopyaRc` object for settings

  - `current` : `collections.OrderedDict` object with elements per level

  In the settings, the sectioning is defined by list of values interpreted by the `IndexPart.GetValues()` method:

      <prefix>.values   :  Sec1 Sec2

  For each of these sections, a set of keywords should be present that define their content. The content types are interpreted by the `GetIndexPart()` method, and the entire content definition might look like:

      <prefix>.Sec1.type     :  list
      <prefix>.Sec1.values   :  A B C
      ..

      <prefix>.Sec2.type     :  list
      <prefix>.Sec2.values   :  Range(1,5)
      ..

class utopya_index.IndexList(urc, current, _indent='')

  Bases: `IndexPart`

  Define index part with bullet list.

  Arguments:

  - `urc` : `utopya.UtopyaRc` object for settings

  - `current` : `collections.OrderedDict` object with elements per level

  In the settings, first define a generic name for the list:

      <prefix>.name     :  expid

  The list elements are interpreted by the `IndexPart.GetValues()` method:

      <prefix>.values   :  A B C

  The generic name should be used to define the content under the list item. The content type is interpreted by the `GetIndexPart()` method, and the entire content definition might look like:

      <prefix>.expid.type     :  img
      <prefix>.expid.img      :  figs/results-%{expid}.png
      ..

class utopya_index.IndexTableRow(urc, current, _indent='')

  Bases: `IndexPart`

  Define index part as table with rows.

  Arguments:

  - `urc` : `utopya.UtopyaRc` object for settings

  - `current` : `collections.OrderedDict` object with elements per level

  By default the table has no borders, as it should usually just show images. Optional, define html table properties with:

      <prefix>.kwargs    :   border=1

  In the settings, first define an (optional) generic name for the row elements; the default is:

      <prefix>.name     :  row

  The row elements are interpreted by the `IndexPart.GetValues()` method:

      <prefix>.values   :  A B C

  The generic name should be used to define the content of the rows, which are probably the columns of the table. The content type is interpreted by the `GetIndexPart()` method, and the entire content definition might look like:

      <prefix>.row.type     :  table-col
      <prefix>.row.name     :  col
      <prefix>.row.values   :  1 2 3 4
      ..

class utopya_index.IndexTableCol(urc, current, _indent='')

  Bases: `IndexPart`

  Define index part as column of table.

  Arguments:

  - `urc` : `utopya.UtopyaRc` object for settings

  - `current` : `collections.OrderedDict` object with elements per level

  In the settings, first define an (optional) generic name for the column elements; the default is:

      <prefix>.name     :  col

  The column elements are interpreted by the `IndexPart.GetValues()` method:

      <prefix>.values   :  1 2 3 4

  The generic name should be used to define the content of the columns, which are probably the figures to be displayed. The content type is interpreted by the `GetIndexPart()` method, and the entire content definition might look like:

      <prefix>.col.type     :  img
      <prefix>.col.img      :  figs/timeseries_%{col}_%{row}.png
      <prefix>.col.kwargs   :  height=200
      ..

class utopya_index.IndexColumn(urc, current, _indent='')

  Bases: `IndexPart`

  Define index part as table with single column.

  Arguments:

  - `urc` : `utopya.UtopyaRc` object for settings

  - `current` : `collections.OrderedDict` object with elements per level

  In the settings, first define an (optional) generic name for the row elements; the default is:

      <prefix>.name     :  row

  The row elements are interpreted by the `IndexPart.GetValues()` method:

      <prefix>.values   :  A B C

  The generic name should be used to define the content of the rows, which are probably the columns of the table. The content type is interpreted by the `GetIndexPart()` method, and the entire content definition might look like:

      <prefix>.row.type     :  img
      <prefix>.row.img      :  /data/figure/example-%{row}.png
      ..

class utopya_index.IndexText(urc, current, _indent='')

  Bases: `IndexPart`

  Define index part with text only.

  Arguments:

  - `urc` : `utopya.UtopyaRc` object for settings

  - `current` : `collections.OrderedDict` object with elements per level

  In the settings, the text to be displayed should be defined by:

      <prefix>.value     :  Very important message.

  Eventually also define a link for the text:

      <prefix>.url       :  http://show.it/message

class utopya_index.IndexImg(urc, current, _indent='')

  Bases: `IndexPart`

  Fill html content for image.

  Arguments:

  - `urc` : `utopya.UtopyaRc` object for settings

  - `current` : `collections.OrderedDict` object with elements per level

  The settings should define the image filename, and optional keyword arguments passed to the html `img` tag. The filename probably includes templates to insert index level values:

      <prefix>.col.img      :  figs/timeseries_%{col}_%{row}.png
      <prefix>.col.kwargs   :  height=200

  Eventually define multiple files as semi-colon seperated list, the first existing file is used:

      <prefix>.col.img      :  figs/timeseries_%{col}_%{row}.png  ;  figs/location_%{row}.png

class utopya_index.IndexTxtFile(urc, current, _indent='')

  Bases: `IndexPart`

  Fill html content for showing txt file.

  Arguments:

  - `urc` : `utopya.UtopyaRc` object for settings

  - `current` : `collections.OrderedDict` object with elements per level

  The settings should define the filename to be shown:

      <prefix>.file     :  output/summary.txt
      ..