utopya_index module

The utopya_indexer module provides tools to create browsable html pages. These are typically used to take a first look at generated images.

Accessing the index

The index pages are accessed by top level index page that is probably named:

index.html

Load this page into a web browser to view the result. The top level page could consists of a combination of images and links to lower-level index pages.

Example of index page.

Example of top level index with links to sub pages.

Example of index page.

Example of index page with figure table.

Producing index pages

The index pages are defined by rcfile settings. The name of the rcfile, and the base of the keywords should be passed to the Indexer class that will create the pages:

utopya.Indexer( 'postproc.rc', rcbase='index', env={} )

This defines that the settings are read from postproc.rc, and that onlys settings with the prefix index are used.

Configuration

The following is an example of a top level definition:

! target location:
index.outdir                    :  /scratch/flexvar/postproc

! show info on created page?
index.info                      :  True

! title:
index.header                    :  WP1

! content type:
index.type                      :  sections

The outdir setting defines the (optinal) work directory where to create the index pages, default is current directory.

The info flag enables that a message is displayed with the url of the index page that should be loaded into a browser. For the top-level page this flag is probably enabled; the displayed message looks like:

Browse to:
    file:///scratch/flexvar/postproc/index.html

The header settings defines an explicit header to be displayed.

The type setting defines the content type of an index part as defined below.

Index parts

The type keyword is interpreted by the GetIndexPart() method, and refers to a class derived from the IndexPart class. The following table shows the type keywords and the corresponding class:

type

class

produces …

IndexPart

(basic settings for all classes)

sections

IndexSections

sections within page

list

IndexList

bullet list

table-row

IndexTableRow

table with rows

table-col

IndexTableCol

columns within table row

text

IndexText

text

img

IndexImg

image

See the description of the classes for the settings that define an index part.

Example

As example, the following settings will create index pages for images produced for different experiments. The experiments are named A, B, and C, and for each a number of image files is availble in different directories:

WP1_A/NL1234_location.png
      NL1234_timeseries_2017-01.png
                              :
      NL1234_timeseries_2017-12.png
      NL1235_location.png
      NL1235_timeseries_2017-01.png
                              :
      stations.csv

WP2_B/NL1234_location.png
      :

WP2_C/NL1234_location.png
      :

The stations.csv files contain information on the station for which images are produced, the first column contains the station codes:

code;long_name
NL1234;Oeverloos
NL1235;Laagwater
:

A top-level index page is created that has a bullet list with links to sub-pages for different experiments:

! ~~ top level index

! target location:
index.outdir                    :  /scratch/flexvar/postproc

! show info on created page?
index.info                      :  True

! title:
index.header                    :  WP1

! content type:
index.type                       :  list
! generic name, used to define the content:
index.name                       :  expid
! list elements, here experiment names:
index.values                     :  A B C

For each of the experiments a sub-page should be created:

! create new page for each value:
index.expid.newpage              :  True

The sub-page for an experiment should contain table with images. Each row should contain the images for a different station, the station codes are read from the csv file:

! content type:
index.expid.type                 :  table-row
! define row values:
index.expid.name                 :  scode
index.expid.values               :  CsvFile( '../WP1_%{expid}/stations.csv' )

The first column should show an image that illustrates the station’s location, the other columns contain time series plots per month:

! content type:
index.expid.scode.type           :  table-col
! define column values:
index.expid.scode.name           :  plot
index.expid.scode.values         :  location ; TimeSeries( '2017-01', '2017-12', '1 month', 'timeseries_%Y-%m' )

The content is finaly defined by image file names (in experiment specific directories) and optional arguments to limit the heigh of the images in the table:

! content type:
index.expid.scode.plot.type      :  img
! define image:
index.expid.scode.plot.img       :  ../WP1_%{expid}/%{scode}_%{plot}.png
index.expid.scode.plot.kwargs    :  height=200

Class hierarchy

Methods

Classes and methods

class utopya_index.Indexer(rcfile, rcbase='', env={}, _indent='')

Bases: utopya_rc.UtopyaRc

Builder for index pages using rcfile settings. See module documentation for description.

Arguments:

  • rcfile, rcbase, env : settings etc, passed to base class

utopya_index.GetIndexPart(urc, current, _indent='')

Return object derived from IndexPart class given type description.

Arguments:

  • urc : utopya.UtopyaRc object for settings

  • current : collections.OrderedDict object with elements for index levels, ordered from top to current

The keys of the current dict define the full prefix for keywords in the settings. For example, if this method is called for the third index level using a dict:

current = { 'main' : { .. },
            'row'  : { .. },
            'col'  : { .. } }

then the prefix is:

main.row.col

The following keys are read from the settings that have been imported into the urc object:

! type of index part:
<prefix>.type       :  list

See the table Index parts for the classes corresponding to type keyword.

class utopya_index.IndexPart(urc, current, _indent='')

Bases: utopya_rc.UtopyaRc

Base class for index parts of html lines.

Arguments:

An element of the current argument is a dict with info for each level:

current = { 'main' : ...,
            'row'  : ...,
            'col'  : { 'value'    : 'timeseries', 
                      'filename' : 'index.S1.expA.html', 
                      'header'   : None, 
                      'alink'    : 'index.S1.expA.html#plot' }
          }

For the top level index (main page), the current is probably empty.

The values are used to pass information between the levels of the index. For example, a lower level could use headers from higher levels to create upward links.

The keys of the current dict define the full name of the prefix for the settings that define the content. For the above example the prefix is:

main.row.col

The keys that are read from the urc object by this base class are optional, and are mainly used for fine-tuning the index pages.

The first (optional) key that is read defines the work directory where the index pages are created; the default is the current directory:

<prefix>.outdir     :  /scratch/flevar/postproc

The next flag defines whether this index part should be written to a new file:

<prefix>.newpage    :  True

The default is False, which means that the index part will be included in a higher level page. An exception is the top level index (empty current dict), this will always write a new page. In case newpage is enabled, the index part will be written to a filename which has by default the level elements in the name:

index__main__row__col.html

Alternatively, the target file could be defined in the settings:

<prefix>.filename   :  whatever.html

Each index part could started with a header. For some index parts (derived from this base class) a header is defined automatically; alternatively a header could be define explicitly:

<prefix>.header      :  Overview of results

If a header is defined (automatically or explicitly) it will be inserted as the first of the html lines. If the index part is written to a new page, the heading is composed as a list of headers from higher levels, with links to their locations:

<main-label> / <row-label> / <col-label>

For index parts included in the html page of a higher level, the heading will consists of the own header only, with the appropriate heading level within the page:

<h1>Main label</h1>
:
  <h2>Row label</h2>
  :
    <h3>Col label</h3>
GetCurrentLine(key=None, sep='.', alinks=False)

Return line with concatenation of elements in the current dict.

Arguments:

  • key : str value; if defined, it should be one of the possible keys in the current elements, for example value; if not defined, the level name (current keys) is used;

  • sep : optional str value inserted between the concatenated elements;

  • alinks : if True, the elements are surround with links to anchors defined for the elements.

ReplaceCurrents(line)

Return line with templates replaced by level values. If the currrent dict consists of:

current = { .., 'row' : { 'value' : 'NL1234', ..}, ..}

then the input line:

station_%{row}_location.png

is returned as:

station_NL1234_location.png

Alternative only some selected characters can be inserted:

station_%{row[0:2]}_location.png
station_NL_location.png
WriteFile()

If a target file is defined for this object, write the html lines to this file.

An optional flag is read from the settings, if enabled a message will be displayed to show the url of the index page:

Browse to:
  file:///scratch/flexvar/postproc/index.html
GetFilename()

If this index part is written to a seperate file, return the targe file, or None otherwise.

AddNavigation(up, prev, next)

Add navigation buttons for up/previous/next to top and bottom of html lines. The arguments (if not None) define the links around the buttons. If an argument is None,

GetValues(key)

Return value list from settings defined by key.

Supported formats:

  • Space seperated list:

    <key>    :  aa b 1 2 3 4 5

  • Semi-column seperated list; if an element looks like a function (including parantheses) then it is passed to the GetValuesFunction() method:

    <key>    :  aa ; b ; Range(1,5)
GetValuesFunction(part)

Evaluate function description, return list of values. Supported functions:

  • CsvFile( 'filename.csv' )

    The csv file is supposed to have a header line and semi-colon seperated values; the first column is returned as list of values:

    code;long_name
NL1234;Oeverloos
NL1235;Laagwater
:

  • Range( 0, 23, 1, '%2.2i00' )

    Return list of formatted integer values. The arguments included in the function description are evaluated to a start, end, step increment, and a formatting template.

  • TimeSeries( '2017-01', '2017-12', '1 month', 'conc_timeseries_%Y-%m' )

    Return list of formatted time values. The arguments included in the function description are evaluated to a start time, end time, time step increment, and a formatting template that can be handled by the datetime.datetime.strftime() method.

class utopya_index.IndexSections(urc, current, _indent='')

Bases: utopya_index.IndexPart

Define index page that consists of one or more sections.

Arguments:

In the settings, the sectioning is defined by list of values interpreted by the IndexPart.GetValues() method:

<prefix>.values   :  Sec1 Sec2

For each of these sections, a set of keywords should be present that define their content. The content types are interpreted by the GetIndexPart() method, and the entire content definition might look like:

<prefix>.Sec1.type     :  list
<prefix>.Sec1.values   :  A B C
..

<prefix>.Sec2.type     :  list
<prefix>.Sec2.values   :  Range(1,5)
..
class utopya_index.IndexList(urc, current, _indent='')

Bases: utopya_index.IndexPart

Define index part with bullet list.

Arguments:

In the settings, first define a generic name for the list:

<prefix>.name     :  expid

The list elements are interpreted by the IndexPart.GetValues() method:

<prefix>.values   :  A B C

The generic name should be used to define the content under the list item. The content type is interpreted by the GetIndexPart() method, and the entire content definition might look like:

<prefix>.expid.type     :  img
<prefix>.expid.img      :  figs/results-%{expid}.png
..
class utopya_index.IndexTableRow(urc, current, _indent='')

Bases: utopya_index.IndexPart

Define index part as table with rows.

Arguments:

In the settings, first define an (optional) generic name for the row elements; the default is:

<prefix>.name     :  row

The row elements are interpreted by the IndexPart.GetValues() method:

<prefix>.values   :  A B C

The generic name should be used to define the content of the rows, which are probably the columns of the table. The content type is interpreted by the GetIndexPart() method, and the entire content definition might look like:

<prefix>.row.type     :  table-col
<prefix>.row.name     :  col
<prefix>.row.values   :  1 2 3 4
..
class utopya_index.IndexTableCol(urc, current, _indent='')

Bases: utopya_index.IndexPart

Define index part as column of table.

Arguments:

In the settings, first define an (optional) generic name for the column elements; the default is:

<prefix>.name     :  col

The column elements are interpreted by the IndexPart.GetValues() method:

<prefix>.values   :  1 2 3 4

The generic name should be used to define the content of the columns, which are probably the figures to be displayed. The content type is interpreted by the GetIndexPart() method, and the entire content definition might look like:

<prefix>.col.type     :  img
<prefix>.col.img      :  figs/timeseries_%{col}_%{row}.png
<prefix>.col.kwargs   :  height=200
..
class utopya_index.IndexText(urc, current, _indent='')

Bases: utopya_index.IndexPart

Define index part with text only.

Arguments:

In the settings, the text to be displayed should be defined by:

<prefix>.value     :  Very important message.
class utopya_index.IndexImg(urc, current, _indent='')

Bases: utopya_index.IndexPart

Fill html content for image.

Arguments:

The settings should define the image filename, and optional keyword arguments passed to the html img tag. The filename probably includes templates to insert index level values:

<prefix>.col.img      :  figs/timeseries_%{col}_%{row}.png
<prefix>.col.kwargs   :  height=200
..