How Kedro Config Loader works

Kedro Config Loader, resolution order
python
kedro
Author

noklam

Published

November 16, 2023

Kedro offers a configuration system, it involves concepts such as - environment (it can be multiple files) - variable interpolation (advance templating) - globals - custom resolvers - runtime parameters (override via CLI or programatically)

The official documentation has plenty of examples, so I am going to focus on explaining the internals.

Config Resolution (Without Globals)

To enable all of the features, the resolution order is actually quite complicated. It involes multiple stages of merging in different scope. This diagram belows describe what happen under the hood. The key is identify two things 1. Some feature apply in local scope (it is not cross environment) 2. There are different merging strategy, some are using OmegaConf, some are pure dictionary merge 3. Configuration are resolved and then merge across environments. i.e. you cannot interpolate a value from another environment.

image.png

Config Resolution (With Globals)