splitgraph.config package

Submodules

splitgraph.config.argument_config module

splitgraph.config.argument_config.get_arg_tuples() → List[Tuple[str, str]]

Get the raw (argN, argN+1) tuples from sys.argv

We could also use click to parse the flags ahead of time, and then pass the parsed flag object into SystemConfigGetters. But this way, we avoid having to pass down variables from click just to get config values.

splitgraph.config.argument_config.get_argument_config_value(key: str, default_return: Optional[str] = None) → Optional[str]

Get get the value of an argument, where value is the argument immediately following the argument matching a key in ARG_KEYS, e.g.:

SYS_ARGS = [”–namespace”, “foo”] –> return “foo”

Otherwise, return default_return

splitgraph.config.config module

splitgraph.config.config.chain_getters(getters: Sequence[Callable[[str], Optional[str]]], key: str, default_return: Optional[str] = None) → Optional[str]

splitgraph.config.config.create_config_dict() → Dict[str, Union[str, Dict[str, str], Dict[str, Dict[str, str]]]]

Create and return a dict of all known config values

splitgraph.config.config.get_all_in_section(config: Dict[str, Union[str, Dict[str, str], Dict[str, Dict[str, str]]]], section: str) → Dict[str, Union[Dict[str, str], str]]

Get all subsections from a config (e.g. config[“data_sources”])

splitgraph.config.config.get_all_in_subsection(config: Dict[str, Union[str, Dict[str, str], Dict[str, Dict[str, str]]]], section: str, subsection: str) → Dict[str, str]

splitgraph.config.config.get_from_section(config: Dict[str, Union[str, Dict[str, str], Dict[str, Dict[str, str]]]], section: str, item: str) → str

splitgraph.config.config.get_from_subsection(config: Dict[str, Union[str, Dict[str, str], Dict[str, Dict[str, str]]]], section: str, subsection: str, item: str) → str

Return a singleton variable from a subsection of the config, e.g. config[“remotes”][“data.splitgraph.com”][“SG_ENGINE_HOST”]

splitgraph.config.config.get_singleton(config: Dict[str, Union[str, Dict[str, str], Dict[str, Dict[str, str]]]], item: str) → str

Return a singleton (not a section) variable from the config.

splitgraph.config.config.lazy_get_config_value(key: str, default_return: Optional[str] = None) → Optional[Union[str, Dict[str, Dict[str, str]]]]

Get the config value for a key in the following precedence Otherwise return default_return

splitgraph.config.config.patch_config(config: Dict[str, Union[str, Dict[str, str], Dict[str, Dict[str, str]]]], patch: Dict[str, Union[str, Dict[str, str], Dict[str, Dict[str, str]]]]) → Dict[str, Union[str, Dict[str, str], Dict[str, Dict[str, str]]]]

Recursively updates a nested configuration dictionary:

patch_config(

{“key_1”: “value_1”,

“dict_1”: {“key_1”: “value_1”}},

{“key_1”: “value_2”,

“dict_1”: {“key_2”: “value_2”}}) == {“key_1”: “value_2”, “dict_1”: {“key_1”: “value_1”, “key_2”: “value_2”}}

Parameters

• config – Config dictionary

• patch – Dictionary with the path

Returns

New patched dictionary

splitgraph.config.config.set_in_subsection(config: Dict[str, Union[str, Dict[str, str], Dict[str, Dict[str, str]]]], section: str, subsection: str, item: str, value: str) → None

Set a singleton variable in a subsection of the config, e.g. config[“remotes”][“data.splitgraph.com”][“SG_ENGINE_HOST”]

splitgraph.config.config.update_config_dict_from_arguments(config_dict: Dict[str, Union[str, Dict[str, str], Dict[str, Dict[str, str]]]]) → Dict[str, Union[str, Dict[str, str], Dict[str, Dict[str, str]]]]

Given an existing config_dict, update after reading sys.argv and overwriting any keys.

Return updated copy of config_dict.

splitgraph.config.config.update_config_dict_from_env_vars(config_dict: Dict[str, Union[str, Dict[str, str], Dict[str, Dict[str, str]]]]) → Dict[str, Union[str, Dict[str, str], Dict[str, Dict[str, str]]]]

Given an existing config_dict, update after reading os.environ and overwriting any keys.

Return updated copy of config_dict.

splitgraph.config.config.update_config_dict_from_file(config_dict: Dict[str, Union[str, Dict[str, str], Dict[str, Dict[str, str]]]], sg_config_file: str) → Dict[str, Union[str, Dict[str, str], Dict[str, Dict[str, str]]]]

Given an existing config_dict, update after reading sg_config_file and overwriting any keys according to the rules in config_file_config

Return updated copy of config_dict.

splitgraph.config.config_file_config module

splitgraph.config.config_file_config.accumulate_lists(config_dict: Dict[str, Union[Dict[str, str], str]]) → Dict[str, Union[str, Dict[str, str], Dict[str, Dict[str, str]]]]

Transform a config_dict to “accumulate” objects “nested” via key name

Because ConfigParser does not support nesting, we implement our own syntax via the key names of sections. The ‘:’ character can be used in section names to specify the left and right key. Example:

 

.ini config             new config_dict
 
              <span className="o">-----></span>
                <span className="o">|</span>
[remote: remote1]   |   &lbrace;SG_ENV_VAR=foo      |       **rest_of_config_dict,|       "remotes": &lbrace;[origin: origin1]   |           "remote1": &lbrace;SG_ENV_VAR=bar      |               "SG_ENV_VAR": "foo"|           &rbrace;[origin: origin2]   |       &rbrace;,SG_ENV_VAR=bax      |       "origins": &lbrace;|           "origin1": &lbrace;|               "SG_ENV_VAR": "bar"|           &rbrace;,|           "origin2": &lbrace;|               "SG_ENV_VAR": "bax"|           &rbrace;|       &rbrace;|   &rbrace;|----->

:return a new, updated copy of config_dict

splitgraph.config.config_file_config.get_config_dict_from_config_file(sg_file: str, **kwargs) → Dict[str, Union[str, Dict[str, str], Dict[str, Dict[str, str]]]]

Create a dict from ConfigParser, apply transformations to it.

Return parsed and transformed config_dict.

splitgraph.config.config_file_config.get_config_dict_from_file(sg_file: str, **kwargs) → Dict[str, Dict[str, str]]

splitgraph.config.config_file_config.hoist_section(config_dict: Dict[str, Dict[str, str]], section: str = 'defaults') → Dict[str, Union[Dict[str, str], str]]

If a section exists called <hoist_section>, hoist it to the top level This is useful for overriding default configs from within a config file

Transform config_dict to “hoist” any config values from a section into the top level (thus, overriding environment variables), when the name of the section matches hoist_section.

Return a new, updated copy of config_dict.

splitgraph.config.config_file_config.transform_config_dict(config_dict: Dict[str, Dict[str, str]], **kwargs) → Dict[str, Union[str, Dict[str, str], Dict[str, Dict[str, str]]]]

Apply transformations to the raw ConfigParser.config object

1. hoist_section

2. accumulate_lists

Return the a new, updated copy of config_dict.

splitgraph.config.default_config module

splitgraph.config.default_config.get_default_config_value(key: str, default_return: Optional[str] = None) → Any

Get the hard-coded default value of a config key. Otherwise return default_return.

splitgraph.config.environment_config module

splitgraph.config.environment_config.get_environment_config_value(key: str, default_return: Optional[str] = None) → Optional[str]

Get the environment variable value of the environment variable matching key. Otherwise return default_return.

splitgraph.config.export module

Routines for exporting the config back into text.

splitgraph.config.export.overwrite_config(new_config: Dict[str, Union[str, Dict[str, str], Dict[str, Dict[str, str]]]], config_path: str, include_defaults: bool = False) → None

Serialize the new config dictionary and overwrite the current config file. Note: this will delete all comments in the config!

Parameters

• new_config – Config dictionary.

• config_path – Path to the config file.

• include_defaults – Whether to include values that are the same as their defaults.

splitgraph.config.export.serialize_config(config: Dict[str, Union[str, Dict[str, str], Dict[str, Dict[str, str]]]], config_format: bool, no_shielding: bool, include_defaults: bool = True) → str

Pretty-print the configuration or print it in the Splitgraph config file format.

Parameters

• config – Configuration dictionary.

• config_format – Output configuration in the Splitgraph config file format.

• no_shielding – Don’t replace sensitive values (like passwords) with asterisks

• include_defaults – Emit the config variable even if it’s the same as the default.

Returns

Textual representation of the config.

splitgraph.config.export.serialize_engine_config(engine_name: str, conn_params: Dict[str, str], no_shielding: bool) → str

Output the config section with connection parameters for a single engine.

Parameters

• engine_name – Name of the engine

• conn_params – Dictionary of connection parameters

• no_shielding – Don’t replace passwords with asterisks

splitgraph.config.keys module

splitgraph.config.keys.SENSITIVE_KEY_SUFFIXES = ['_PWD', '_TOKEN']

Warning: Every key in DEFAULTS must have a key in ARGUMENT_KEY_MAP If you add/remove keys from DEFAULTS, make sure to do so here too.

splitgraph.config.management module

splitgraph.config.management.patch_and_save_config(config, patch)

splitgraph.config.system_config module

splitgraph.config.system_config.file_exists(_dir: str, filename: str) → bool

splitgraph.config.system_config.get_config_file(default_return: None = None) → Optional[str]

Get the location of an existing SG_CONFIG_FILE on the system with a valid name. Do not attempt to parse the config file, just return its location.

Otherwise, return default_return

splitgraph.config.system_config.get_explicit_config_file_dirs() → List[str]

Get any explicitly defined config file directories, which are directories where we should search for files from VALID_CONFIG_FILE_NAMES.

This list is defined similar to $PATH, as a colon (:) delimited string, either in:

• argument flag –config-dirs

• or environment key SG_CONFIG_DIRS

Or a single directory defined in either

• argument flag –config-dir

• environment key SG_CONFIG_DIR

If both plural and single are defined, join them together.

Return a list of valid paths that are set, or an empty list.

Print a warning if any paths to not exist.

splitgraph.config.system_config.get_explicit_config_file_location() → Optional[str]

Get the explicitly defined location of config file, if defined.

The location will either be defined in:

• argument flag –config-file

• or environment key SG_CONFIG_FILE

In keeping with assumptions about priority, argument flag has higher priority than environment value.

If the location is set, and points to an existing file, return location.

Otherwise return None

Print a warning if location is set but points to non-existing file.

splitgraph.config.system_config.get_system_config_value(key: str, default_return: None = None) → Optional[str]

splitgraph.config.system_config.is_file(filename: str) → bool

Module contents

The CONFIG object is created and exported once __at import time__Calling CONFIG[“KEY”] directly should be sufficient in most cases, except when a config value has changed since importing CONFIG. In that case, create_config_dict() can provide an updated config dict

Priority (highest to lowest):

1. Command line argument values

2. Environment variable values

3. Config file values in [defaults] section

4. DEFAULTS (see keys.py)