github.com/lyraproj/hiera@v1.0.0-rc4/README.md (about) 1 # Hiera lookup framework 2 3 [](https://goreportcard.com/report/github.com/lyraproj/hiera) 4 [](https://godoc.org/github.com/lyraproj/hiera) 5 [](https://github.com/lyraproj/hiera/actions) 6 7 ## Introduction 8 9 Hiera is a flexible, powerful tool for resolving values for variable lookups, which was first popularised by its use in [Puppet](https://puppet.com/docs/puppet/5.5/hiera.html). 10 11 This module is a "clean-room" Go implementation of the Hiera framework, suitable for use as a library from other tools. 12 13 ## Details 14 15 Hiera uses the concept of "managing by exception": you design a *hierarchy* of data sources, with the most specific source at the top and least-specific defaults at the bottom. Hiera searches for keys starting at the top, allowing more-specific sources to override defaults. Sources are usually YAML files stored on the filesystem, and layers usually use variable interpolation to find the right file, allowing the context of the lookup to pick the right file. 16 17 ## How to run it 18 19 ### Standalone execution 20 21 #### Install the module 22 23 To install the module under $GOPATH/src: 24 25 go get github.com/lyraproj/hiera 26 27 #### Install the lookup binary 28 29 Install the lookup binary under $GOPATH/bin: 30 31 go install github.com/lyraproj/hiera/lookup 32 33 #### Run the binary 34 35 lookup --help 36 37 ### Containerized execution 38 39 #### Download the container 40 41 You can pull the latest containerized version from docker hub: 42 43 docker pull lyraproj/hiera:latest 44 45 The docker repository with previous tags is viewable at https://hub.docker.com/r/lyraproj/hiera . 46 47 #### Run the container 48 49 The docker image accepts environment variables to override default behaviour: 50 51 * *port* - which port to listen on inside the container (default: 8080) 52 * *loglevel* - how much logging to do (default: error, possible values: error, warn, info, debug) 53 * *config* - path to a hiera.yaml configuration (default: /hiera/hiera.yaml) 54 55 Make sure to pass the port on your host through to the container. A directory with a hiera configuration and data files (see below) should be mounted under `/hiera` in the image using a bind mount: 56 57 docker run -p 8080:8080 --mount type=bind,src=$HOME/hiera,dst=/hiera lyraproj/hiera:latest 58 59 #### Query the container 60 61 The web service in the container responds to the `/lookup` endpoint with an additional path element of which key to look up. Nested keys can be looked up using dot-separation notation. Given a yaml map without any overrides like: 62 63 aws: 64 tags: 65 Name: lyra-sample 66 created_by: lyra 67 department: engineering 68 project: incubator 69 lifetime: 1h 70 71 You can get back the entire map or specific parts of it: 72 73 $ curl http://localhost:8080/lookup/aws 74 {"tags":{"Name":"lyra-sample","created_by":"lyra","department":"engineering","lifetime":"1h","project":"incubator"}} 75 $ curl http://localhost:8080/lookup/aws.tags 76 {"Name":"lyra-sample","created_by":"lyra","department":"engineering","lifetime":"1h","project":"incubator"} 77 $ curl http://localhost:8080/lookup/aws.tags.department 78 "engineering" 79 80 ## Pass values for interpolation 81 82 If your hierarchy config contains variable interpolation, you can provide context for the lookup using the `var` query parameter. Repeated `var` parameters will create an array of available parameters. The values should be colon-separated variable-value pairs: 83 84 curl 'http://localhost:8080/lookup/aws.tags?var=environment:production&var=hostname:specialhost' 85 86 TODO: Nested variable lookups such like `os.family` are not yet working. 87 88 ## Hiera configuration and directory structure 89 90 Much of hiera's power lies in its ability to interpolate variables in the hierarchy's configuration. A lookup provides values, and hiera maps the interpolated values onto the filesystem (or other back-end data structure). A common example uses two levels of override: one for specific hosts, a higher layer for environment-wide settings, and finally a fall-through default. A functional `hiera.yaml` which implements this policy looks like: 91 92 --- 93 version: 5 94 defaults: 95 datadir: hiera 96 data_hash: yaml_data 97 98 hierarchy: 99 - name: "Host-specific overrides" 100 path: "hosts/%{hostname}.yaml" 101 - name: "Environmental overrides" 102 path: "environments/%{environment}.yaml" 103 - name: "Fall through defaults" 104 path: "defaults.yaml" 105 106 This maps to a directory structure based in the `hiera` subdirectory (due to the `datadir` top level key) containing yaml files like: 107 108 hiera 109 ├── defaults.yaml 110 ├── environments 111 │ └── production.yaml 112 └── hosts 113 └── specialhost.yaml 114 115 ## Extending Hiera 116 117 When Hiera performs a lookup it uses a lookup function. Unless the function embedded in the hiera binary, it will 118 make an attempt to load a RESTful Plugin that provides the needed function. Such plugins are enabled by using the API 119 provided by the [hierasdk library](https://github.com/lyraproj/hierasdk). 120 121 #### How Hiera finds its plugins 122 The resolution of a plugin can be controlled using a "plugindir" key in the Hiera configuration file. As with "datadir", 123 the "plugindir" can be specified both in the defaults hierarchy or explicitly in a specific hierarchy. Unless specified, 124 the "plugindir" is assumed to be the directory "plugin" present in the same directory as the configuration file. 125 126 In addition to "plugindir", a hierarchy may also specify a "pluginfile". Unless specified, the "pluginfile" is assumed 127 to be equal to the name of the lookup function (with the extension ".exe" in case of Windows). 128 129 ## Environment Variables 130 131 The following environment variables can be set as an alternative to CLI options. 132 133 * `HIERA_CONFIGFILE` - `--config` 134 135 Values passed as CLI options will take precendence over the environment variables. 136 137 The following environment variables can be set as an alternative to setting values in the `defaults` hash. 138 139 * `HIERA_DATADIR` - `datadir` 140 * `HIERA_PLUGINDIR` - `plugindir` 141 142 Values set in `hiera.yaml` will take precedence over the environment variables. 143 144 ### Containerized extension 145 146 In order to include an extension in a Hiera Docker image you need to: 147 148 1. Copy the source (or clone the git repository) of the desired extensions into the hiera plugin directory (don't worry, 149 this directory will be ignored by git when doing commits to hiera). 150 2. For each extension, add a line like the following line to the Hiera Dockerfile below the comment 151 `# Add plugin builds here`: 152 ``` 153 RUN (cd plugin/hiera_terraform && go build -o ../terraform_backend) 154 ``` 155 3. Run the docker build. 156 157 ### Useful extensions 158 159 * [Azure Key Vault lookup_key](https://github.com/lyraproj/hiera_azure). Allows you to lookup single values stored as 160 secrets from the Azure Key Vault. 161 * [Terraform Backend data_hash](https://github.com/lyraproj/hiera_terrform). Allows hiera to query data from a Terraform 162 backend. 163 164 ## Implementation status 165 166 * [x] lookup CLI 167 * [x] lookup function 168 * [x] lookup context 169 * [x] dotted keys (dig functionality) 170 * [x] interpolation using scope, lookup/hiera, alias, or literal function 171 * [x] Hiera version 5 configuration in hiera.yaml 172 * [x] merge strategies (first, unique, hash, deep) 173 * [x] YAML data 174 * [x] JSON data 175 * [x] lookup options stored adjacent to data 176 * [x] convert_to type coercions 177 * [x] Sensitive data 178 * [ ] configurable deep merge 179 * [x] pluggable back ends 180 * [x] `explain` functionality to show traversal 181 * [x] containerized REST-based microservice 182 * [x] JSON and YAML schema for the hiera.yaml config file (see schema/hiera_v5.yaml)