# `rc` module

Python module to read settings from a specially formatted text file. This text file will be denoted as a ‘resource’ file, or simpy ‘rcfile’.

## Format of rc-files

An rc-file is a text file with key/value pairs seperated by a colon ‘`:`’. This is the format also used for X-Resources, from which the name resource-file or simply rc-file has been adapted. An example of the format is:

    my.flag    :  T
    my.answer  :  42

The following functionality is supported.

### Line continuation

Long values could be continued at the next line after a ‘`\`’ as last character:

    my.longlist : value1 value2 \
                  value3 value4

### Annotation

The following formatting rules are useful to make the settings readible and understandable.

- Empty lines are ignored.

- Comment lines starting with a ‘`!`’ as first character are ignored.

- Comment starting with ‘`!`’ is stripped from the values. To have a value including exclamation marks, use an escaped version ‘`\!`’:

      my.value      :   -999    ! just an integer value
      my.message    :   This value has 64 characters \! Count if you don't believe it ...

  Note that currently the remainder of the value is not scanned for comment.

### Variable substitution

Both the keys and the values might contain variable expressions, that are substituted by actual values on initialization. In case a variable substitution could not be resolved an error message will be raised.

Variables are enclosed by ‘`${`’ and ‘`}`’, e.g. ‘`${HOME}`’.

The following variable subsitutions are supported, in the order listed here (the first possible substitution is performed):

- User defined variables might be passed on initialization using the optional ‘`env`’ dictionairy; see the initializaton of the `RcFile` class.

- Substitution of operating system environment variables:

      input.dir :   ${HOME}/Documents/data

- Substitution of values assigned to other keys:

      grid                 :  glb300x200
      input.${grid}.path   :  /data/input/${grid}

  The substitutions are performed in a loop until nothing has to be substituted anymore, or some substitutions could not be applied at all; for the later an error is raised. Values to be substituted could therefore be set before and after they are used.

- Special variables:

      ${__filename__}   # absolute path to settings file

      ${__cwd__}        # current work directory

      ${__pid__}        # evaluates to the current process id;
                        # useful for names of log files etc

      ${__script__}     # evaluates to the base name of the calling script,
                        # thus without .py etc

      ${__hostname__}   # evaluates to the hostname

Note that it also possible to define other enclosing marks than ‘`${`’ and ‘`}`’ using the optional ‘`marks`’ argument on initialization.

### Expressions

Evaluation of expressions is applied to all values enclosed by ‘`$((..))`’ . The enclosed value should be a valid Python expression after all variable subsitutions have been applied:

    ntask       :  4
    nthread     :  2
    ncore       :  $(( ${ntask} * ${nthread} ))

### Including other settings

Include the key/value pairs from another file:

    #include path/to/some-other.rc

### Import settings

To import specfic settings from another file, use:

    #from path/to/some-other.rc import key1 key2 ...
    #from path/to/some-other.rc import key as newkey

### Conditional expressions

Case specific settings could be defined using a conditional expression:

    #if ${my.number} == 1
    message    : Welcome
    #elif ${my.number} == 2
    message    : Welcome back
    #else
    message    : Whatever ...
    #endif

The condition should be a valid python expressions that evaluates to a boolean; variable substitutions are performed before evaluation. Examples:

    ${my.number} == 4
    "${my.name}" == "UTOPyA"

Keep it simple! Very complicated and nested if-statements might not be resolved correctly, and are in any case not easy to understand for other users.

### Error messages

In case an undesired condition is found, it is possible to raise have an error message raised using the special ‘`#error`’ mark. Everything behind the ‘`#error`’ mark is displayed as an error message, eventually use ‘`\n`’ for newline and ‘`\t`’ for tab (4 spaces):

    #if ${my.number} < 0
    #error No settings provided for number : ${my.number}
    #endif

### Loop evaluation

A for-loop could be used to quickly set a number of similar settings:

    #for XX in AA BB CC :
    setting.XX   :  This is the value for XX.
    #endfor

This will expand to:

    setting.AA   :  This is the value for AA.
    setting.BB   :  This is the value for BB.
    setting.CC   :  This is the value for CC.

## Usage as Python module

### Initialization of rcfile object

The `RcFile` class is used to read, process, and store all key/value pairs from a rcfile. Initialize an object of this class by passing the name of the rcfile:

    rcf = rc.RcFile( 'settings.rc' )

All substitutions described under formatting are applied on reading, unless the optional ‘`raw`’ flag is enabled:

    rcf = rc.RcFile( 'settings.rc', raw=True )

See the initializaton of the `RcFile` class for details on this and other optional arguments.

### Rcfile keys

The `RcFile.has_key()` method is provided to test if a key is defined:

    if rcf.has_key('my.flag') :
      print 'value of my flag is : ', rcf['my.flag']

Extract a list with all keys using the `RcFile.keys()` method:

    rcf.keys()

### Get values

The `RcFile.get()` method is provided to extract values.

By default, the ‘`get`’ function returns the value as a str type:

    s = rcf.get('my.value')

A second argument is the name of the python type to which the value is converted to:

    i = rcf.get('my.flag','int')

If the return type should be a ‘`bool`’, the result is:

- True for values : ‘True’ , ‘T’, ‘yes’, or ‘1’ ,

- False for values : ‘False’, ‘F’, ‘no’ , or ‘0’ .

For other values that should be converted to a ‘`bool`’ an error is raised.

A default value could be returned if the key is not found, without raising an exception:

    rcf.get( 'version', default='v1.2' )

Print a debug message to the logging system for each extracted key:

    rcf.get( 'my.flag', verbose=True )

### Add new key/value pairs

Use the `RcFile.add()` method to add a new value:

    rcf.add( 'my.iter', 2 )

This function is useful to pass changed settings to other methods, or to write modified rcfiles using the `RcFile.WriteFile()` method. Eventually specify a comment line; if the content of the object is written to a file, this comment will be written before the key/value pair:

    rcf.add( 'my.iter', 2, comment='iteration number for restart' )

### Replace values

Assign a new value to an existing key with the `RcFile.replace()` method:

    rcf.replace( 'my.flag', True )

### Substitute values

The `RcFile.substitute()` method could be used to replace keys by rcfile values in a character line. For example, suppose the rcfile stored had content:

    name       : Model
    version    : v1.2

A character line with the right templates could now be evaluated into a version with values inserted at the right place:

    line = rcf.substitute( 'This is version %{version} of %{name}.' )
    print line
    This is version v1.2 of Model.

### Write content

The `RcFile.WriteFile()` method could be used to write the content of the object to a new file, with all variables expanded and included files included:

    rcf.WriteFile( 'newfile.rc' )

## Usage via script

Use the ‘`rcget`’ script to extract values from a rcfile in non-python application.

Example of usage:

    rcget 'mytest.rc' 'version'

To see all options:

    rcget --help

## History

- 1998? Arjo Segers, TU Delft
  Initial implementation of configuration options in ‘rc’ file style.
  Read routines in Fortran.
- 2001? Arjo Segers, KNMI
  Implementation of shell script to read settings from within a shell script.
- 2008? Andy Jacobson, NOAA
  Translation to python of original shell script ‘go_readrc’ .
- 2009-06 Wouter Peters, WUR
  Support substitution of previously defined variables.
- 2009-06 Arjo Segers, TNO
  Support include files.
- 2009-09 Arjo Segers, TNO
  Re-coded into class.
  Implemented substitution loop.
- 2009-11 Arjo Segers, JRC
  Added main program to run this file as a shell script.
  Added replace and substitute routines.
- 2010-03 Arjo Segers, JRC
  Support simple if-statements.
  Support comment in values.
- 2010-07 Wouter Peters, WUR
  Downgraded to work for python 2.4.3 too.
  Added read/write routines for backwards compatibility.
- 2010-07-27 Arjo Segers, JRC
  Maintain list with rcfile names and line numbers to be displayed with error messages to identify where problematic lines are found.
- 2010-07-28 Andy Jacobson, NOAA
  Add second dictionary of key,linetrace values to help track the provenance of #included keys (to debug multiple key instances).
  Identify duplicate keys by checking on different source lines instead of checking if the values are different.
- 2010-10 Arjo Segers, JRC
  Restructured processing using classes for source lines and rcfile values, and resolve using recursive calls.
  Added evaluation of expression enclosed by $((.)) .
  Added for-loop.
  Removed main program and stored this in the auxilary script ‘rcget’ .
- 2015-04 Arjo Segers, TNO
  Formatted helptext for Sphinx.
- 2015-05 Arjo Segers, TNO
  Load included files relative to including file if necessary.

## Classes

class rc.RcFile(filename, raw=False, marks=('${', '}'), env={}, debug=False)

  Bases: `object`

  Class to store settings read from a rcfile. The filename of the rcfile to be read should be passed as first argument.

  Variable substitutions are applied and special lines are evaluated, unless ‘raw’ is set to True.

  The 2-item tupple (mark1,mark2) could be used to re-define the default substitution pattern ‘${..}’ into something else:

      <mark1>...<mark2>

  An extra environment dictionairy ‘env’ could be passed to substitute variables. For example, if the name of an output directory should depend on an iteration step number that is only available at run time, one could use a setting:

      ! step depended output:
      output.dir     :  /scratch/me/step-${__STEP__}/output

  and the rcfile should then be initialized using argument:

      env = { '__STEP__' : 12 }

  Enable the ‘debug’ flag to have messages printed about the key/value pairs found.

  has_key(key)

    Return bool to test if specified key is defined in rcfile.

  keys()

    Return list of keys defined in rcfile.

  get(key, totype='', default=None, verbose=False)

    Return element `key` from the dictionary.

    If the element is not present but a default is specified, than return this value.

    If `verbose` is set to True, a debug message is send to the logging system on which value is returned for the given key.

    The optional argument `totype` defines the conversion to a Python type.

    - If `totype` is set to `'bool'`, the return value is:

      - True for values `'T'`, `'True'`, `'yes'`, and `'1'`;

      - False for `'F'`, `'False'`, `'no'`, or `'0'`.

    - If `totype` is set to `'datetime'`, the content is read into a `datetime.datetime` object.

    - For `timestamp`, the result is a `pandas.Timestamp` object.

  replace(key, val)

    Replace the value of a key by a new value.

  add(key, val, comment='')

    Add a new key/value pair.

  replace_add(key, val)

    Replace the value of a key by a new value, or add the key/value pair if not present yet.

  substitute(line, marks=('${', '}'))

    Return a line with all ‘${..}’ parts replaced by the corresponding rcfile values. The 2-item tupple (mark1,mark2) could be used to re-define the default key pattern ‘${..}’ into something else:

        <mark1>...<mark2>

  getlines()

    Return list with processed rcfile lines.

  WriteFile(filename)

    Write the dictionary to specified filename.