auto_process_ngs.settings
Classes and functions for handling the collection of configuration settings for automated processing.
The settings are stored in a ‘.ini’-formatted file (by default called ‘auto_process.ini’; this file can be created by making a copy of the ‘auto_process.ini.sample’ file).
The simplest usage example is:
>>> from settings import Settings
>>> s = Settings()
The values of the configuration parameters can then be accessed using e.g.
>>> s.general.max_concurrent_jobs
4
To print the values of all parameters use
>>> s.report_settings()
To import values from a non-standard named file use e.g.
>>> s = Settings('my_auto_process.ini')
The ‘locate_settings_file’ function is used implicitly to locate the settings file if none is given; it can also automatically create a settings file if none is found but there is a sample version on the search path.
To update values once the settings have been read in do e.g.
>>> s.set('general.max_concurrent_jobs',4)
To update the configuration file use the save method e.g.
>>> s.save()
- class auto_process_ngs.settings.Settings(settings_file=None, resolve_undefined=True)
Load parameter values from an external config file
The input file should be in ‘.ini’ format and contain sections and values consistent with the sample ‘auto_process.ini’ file.
- add_section(section)
Add a new section
- Parameters:
section (str) – an identifier of the form SECTION[:SUBSECTION] which specifies the section to add
- fetch_value(param)
Return the value stored against a parameter
NB parameters are referenced by the names that appear in the config file (rather than the internal representation within the Settings instance).
- Parameters:
param (str) – parameter name of the form SECTION[:SUBSECTION][.ATTR]
- get_bcl_converter_config(section, config, attr_dict=None)
Retrieve BCL conversion configuration options from .ini file
Given the name of a section (e.g. ‘bcl_conversion’, ‘platform:miseq’), fetch the BCL converter settings and return in an AttributeDictionary object.
The options that can be extracted are:
bcl_converter
nprocessors
no_lane_splitting
create_empty_fastqs
There are also some legacy options:
default_version
bcl2fastq
- Parameters:
section (str) – name of the section to retrieve the settings from
config (Config) – Config object with settings loaded
attr_dict (AttributeDictionary) – optional, existing AttributeDictionary which will be added to
- Returns:
dictionary of option:value pairs.
- Return type:
AttributeDictionary
- get_destination_config(section, config)
Retrieve ‘destination’ configuration options from .ini file
Given the name of a section (e.g. ‘destination:webserver’), fetch the associated data transfer settings and return in an AttributeDictionary object.
The options that can be extracted are:
directory (compulsory, str)
subdir (optional, str, default ‘None’)
zip_fastqs (optional, boolean, default ‘False’)
max_zip_size (option, str, default ‘None’)
readme_template (optional, str, default ‘None’)
url (optional, str, default ‘None’)
include_downloader (optional, boolean, default ‘False’)
include_qc_report (optional, boolean, default ‘False’)
hard_links (optional, boolean, default ‘False’)
- Parameters:
section (str) – name of the section to retrieve the settings from
config (Config) – Config object with settings loaded
- Returns:
dictionary of option:value pairs.
- Return type:
AttributeDictionary
- get_organism_config(section=None, config=None)
Retrieve ‘organism’ configuration options from .ini file
Given the name of a section (e.g. ‘organism:Human’), fetch the data association with the organism and return in an AttributeDictionary object.
The items that can be extracted are:
star_index (str, path to STAR index)
bowtie_index (str, path to Bowtie index)
annotation_bed (str, path to BED file with annotation)
annotation_gtf (str, path to GTF file with annotation)
cellranger_reference (str)
cellranger_premrna_reference (str)
cellranger_atac_reference (str)
cellranger_arc_reference (str)
cellranger_probe_set (str)
- Parameters:
section (str) – name of the section to retrieve the settings from
config (Config) – Config object with settings loaded
- Returns:
dictionary of option:value pairs.
- Return type:
AttributeDictionary
- get_sequencer_config(section, config)
Retrieve ‘sequencer’ configuration options from .ini file
Given the name of a section (e.g. ‘sequencer:SN7001250’), fetch the data associated with the sequencer instrument and return in an AttributeDictionary object.
The items that can be extracted are:
platform (compulsory, str)
model (str, default ‘None’)
- Parameters:
section (str) – name of the section to retrieve the settings from
config (Config) – Config object with settings loaded
- Returns:
dictionary of option:value pairs.
- Return type:
AttributeDictionary
- has_subsections(section)
Check if section contains subsections
- Parameters:
section (str) – name of the section to check
- list_params(pattern=None, exclude_undefined=False)
Return (yield) all the stored parameters
NB parameters are referenced by the names that appear in the config file (rather than the internal representation within the Settings instance).
- Parameters:
pattern (str) – optional glob-style pattern; if supplied then only parameters matching the pattern will be returned
exclude_undefined (bool) – if True then only undefined parameters (i.e. those with null values) will be returned
- Yields:
String –
- parameter names of the form
SECTION[:SUBSECTION][.ATTR]
- report_settings(exclude_undefined=False)
Report the settings read from the config file
- Parameters:
exclude_undefined (bool) – if True then parameters with null values will not be shown (default: show all parameters)
- Returns:
report of the settings.
- Return type:
String
- resolve_undefined_params()
Set non-null values for all parameters which are null
Resolution has three stages:
Fallback parameters: if a parameter is undefined and has a defined fallback that value is assigned
Default paramaters: if a parameter is undefined after fallbacks are exhausted and has a defined default then that value is assigned
Default runner: any undefined runner parameters are assigned the value of the default runner, if set
Any parameters that are still undefined at this point are then assigned the value of ‘None’.
- save(out_file=None, exclude_undefined=True)
Save the current configuration to the config file
If no config file was specified on initialisation then this method doesn’t do anything.
- Parameters:
out_file (str) – specify output file (default: overwrite initial config file)
exclude_undefined (bool) – if True then parameters with null values will not be written to the output config file (default)
- set(param, value)
Update a configuration parameter value
NB parameters are referenced by the names that appear in the config file (rather than the internal representation within the Settings instance).
- Parameters:
param (str) – an identifier of the form SECTION[:SUBSECTION].ATTR which specifies the parameter to update
value (str) – the new value of the parameter
- auto_process_ngs.settings.fetch_reference_data(s, name)
Fetch specific reference data for all organisms
Given the name of a reference data item (e.g. ‘star_index’), extracts the relevant data for all organisms from the supplied settings object and returns it as a dictionary.
- Parameters:
s (Settings) – populated Settings instance
name (str) – name of the reference data items required (e.g. ‘star_index’)
- Returns:
- keys are organism names and
values are the corresponding reference data.
- Return type:
Dictionary
- auto_process_ngs.settings.get_config_dir()
Return location of config directory
Returns the path to the ‘config’ directory, or None if it doesn’t exist.
- auto_process_ngs.settings.get_install_dir()
Return location of top-level directory of installation
This is a directory one or more level above the location of this module which contains a ‘config’ subdir with an ‘auto_process.ini.sample’ file, for example: if this file is located in
/opt/auto_process/lib/python3.6/site-packages/auto_process_ngs
then each level will be searched until a matching ‘config’ dir is located.
If it can’t be located then the directory of this module is returned.
- auto_process_ngs.settings.locate_settings_file(name='auto_process.ini', create_from_sample=False)
Locate configuration settings file
Look for a configuration settings file (default name ‘auto_process.ini’). The search path is:
file specified by the AUTO_PROCESS_CONF environment variable (if it exists)
current directory
‘config’ subdir of installation location
top-level installation location
The first file with a matching name is returned.
If no matching file is located but one of the locations contains a file with the correct name ending in ‘.sample’, and if the ‘create_from_sample’ argument is set, then use this to make a settings file in the same location.
Returns the path to a settings file, or None if one isn’t found.
- auto_process_ngs.settings.show_dictionary(d, indent=' ', exclude_value=None)
Print the contents of a dictionary
- Parameters:
d (str) – dictionary instance to show
exclude_value (object) – optional, if not ‘None’ then don’t include entries which match this value