github.com/zntrio/harp/v2@v2.0.9/README.md (about) 1 [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) 2 [![Go Report Card](https://goreportcard.com/badge/github.com/zntrio/harp)](https://goreportcard.com/report/github.com/zntrio/harp) 3 [![made-with-Go](https://img.shields.io/badge/Made%20with-Go-1f425f.svg)](http://golang.org) 4 [![GitHub release](https://img.shields.io/github/release/zntrio/harp.svg)](https://github.com/zntrio/harp/releases/) 5 [![Maintenance](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://github.com/zntrio/harp/graphs/commit-activity) 6 7 - [Harp](#harp) 8 - [TL;DR.](#tldr) 9 - [Visual overview](#visual-overview) 10 - [Why harp?](#why-harp) 11 - [Use cases](#use-cases) 12 - [How does it work?](#how-does-it-work) 13 - [Like a Data pipeline but for secret](#like-a-data-pipeline-but-for-secret) 14 - [Immutable transformation](#immutable-transformation) 15 - [What can I do?](#what-can-i-do) 16 - [FAQ](#faq) 17 - [License](#license) 18 - [Build instructions](#build-instructions) 19 - [Clone repository](#clone-repository) 20 - [Setup dev environment](#setup-dev-environment) 21 - [With nix flake](#with-nix-flake) 22 - [Non-nix managed environment](#non-nix-managed-environment) 23 - [Check your go version](#check-your-go-version) 24 - [Install mage](#install-mage) 25 - [From source](#from-source) 26 - [Bootstrap tools](#bootstrap-tools) 27 - [Mage targets](#mage-targets) 28 - [Plugins](#plugins) 29 - [Community](#community) 30 31 # Harp 32 33 Harp is for Harpocrates (Ancient Greek: Ἁρποκράτης) the god of silence, secrets 34 and confidentiality in the Hellenistic religion. - [Wikipedia](https://en.wikipedia.org/wiki/Harpocrates) 35 36 > This tool was initially developed while I was at Elastic, to be able to continue 37 > to maintain Harp without the upstream dependency, I decided to do a hard-fork 38 > of the Elastic repository. 39 > 40 > I'm going to introduce breaking changes from the Elastic original version. 41 42 ## TL;DR. 43 44 Harp is a tool set to operate secret management by contract. The objective is 45 to reduce the value centric management by handling secret data in a 46 `reproducible` way by providing a technical stack to describe how your value is 47 managed by contracts and pipelines. Furthermore, we know that `naming thing is hard`, 48 as a consequence a secret could be `consistently` associated to a `predictable` 49 secret identifier used as a key to refer to the secret value. Finally, the secret 50 can hold additional metadata (ownership, rotation period, leak severity, etc.) 51 which can be consumed during the pipeline executions. 52 53 These key/value associations (path ⇒ value) form a `Bundle` stored in an 54 immutable file named a `Container`. This `Container` acts as a pivot format to 55 allow Harp commands to communicate and create data management pipelines. 56 57 In addition to that, it provides a `template engine` used to generate various 58 confidence values (password, passphrase, crypto keys, etc.) and allow more 59 sophisticated rendering operations (configuration files, etc.). 60 61 Finally, it provides a `SDK` to allow developers to integrate `Harp` features 62 in their products, and/or extend the Harp pipeline features by creating new 63 [plugins](#plugins). 64 65 ## Visual overview 66 67 ![Visual overview](docs/harp/img/HARP_FLOW.png) 68 69 ## Why harp? 70 71 * Secret management is in essence a collection of processes that must be 72 auditable, executable and reproducible for infosec and operation requirements; 73 * Secret provisioning must be designed with secret rotation as a day one task, 74 due to the fact that secret data must be rotated periodically to keep its 75 secret property; 76 * `Developers` should negotiate secret value for the secret consumer they are 77 currently developing, by the contract based on a path (reference to the secret) 78 and a value specification (for code contract) without the knowledge of the 79 final deployed value; 80 * `Secret Operators` use different set of tools to achieve secret 81 management operation which increases the error/secret exposure probability due to 82 tool count involved in the process (incompatibility, changes, etc.); 83 * Without a defined secret naming convention, the secret storage becomes difficult to 84 handle in time (naming is hard) and secret naming could not be helped to 85 get a consistent, reliable and flexible secret tree; 86 * Secret storage backend can use various implementations in different environments 87 and should be provisioned consistently; 88 * When you use `Terraform` for secret management, you have the cleartext value 89 stored in the state. To protect the state you have to deploy a complex infrastructure. 90 To simplify this we use harp for secret provisioning and use the secret reference 91 in the Terraform topology. 92 93 ## Use cases 94 95 * You want to have a `single secret value` and you are asking yourself 96 `how to generate a strong password` - Harp has a template engine with secret 97 value generation functions to allow you to generate such values. 98 * You have `thousands secrets` to handle to deploy your platform/customers 99 `on multiple cloud providers` with `different secret storages` - Harp will help you 100 to define consistent secret provisioning bundles and pipelines. 101 * You need a `ephemeral secret storage` to `bootstrap` your long term cloud 102 secret storage - Harp will help you to create 103 secret containers that can be consumed on deployment. 104 * You want to `migrate massively` your secrets from one secret storage to 105 another - Harp provides you a secret container to store these secrets while 106 they are going to be distributed in other secret storage implementations. 107 * You have to `alter/modifiy` a secret (rotation/deprecation/renewal) - Harp 108 provides you a `GitOps-able` secret `storage agnostic operation set`, so that you 109 can define a specification to describe how your secret operation is going to 110 be applied offline on the secret container. 111 112 ## How does it work? 113 114 ![Secret management Pipeline](docs/harp/img/SM-HARP-PIPELINE.png) 115 116 ### Like a Data pipeline but for secret 117 118 `harp` allows you to handle secrets using deterministic pipelines expressed 119 using an atomic series of CLI operations applied to a commonly shared container 120 immutable and standalone file system used to store secret collection (Bundle) 121 generated from a template engine via user specification, or external secret 122 value coming from files or external secret storage. 123 124 ![Pipelines](docs/harp/img/SM-HARP.png) 125 126 These pipelines use the immutable container file system as a data exchange 127 protocol and could be extended for new input, intermediary operation or output 128 via plugins created with the `harp` SDK. 129 130 ### Immutable transformation 131 132 Each applied transformation creates a container with transformed data inside. 133 This will enforce container reproducibility by eliminating cumulative 134 side effects applied to the same container. 135 136 The container handles for you the confidentiality and integrity protection applied 137 to the secret collection stored inside and manipulated by copy during the 138 pipeline execution. 139 140 ## What can I do? 141 142 > New to harp, let's start with [onboarding tutorial](docs/onboarding/README.md) ! 143 > TL;DR - [Features overview](FEATURES.md) 144 145 Harp provides : 146 147 * A methodology to design your secret management; 148 * Secret naming convention (CSO); 149 * A defined common language and complete processes to achieve secret management 150 operations; 151 * A SDK to create your own tools to orchestrate your secret management pipelines; 152 * A container manipulation library exposed as `github.com/zntrio/harp/v2/pkg/container`; 153 * A secret bundle specification to store and manipulate secrets exposed as `github.com/zntrio/harp/v2/pkg/bundle`; 154 * An `on-steroid` template engine exposed as `github.com/zntrio/harp/v2/pkg/template` 155 * A path name validation library exposed as `github.com/zntrio/harp/v2/pkg/cso` 156 * A CLI for secret management implementation 157 * CI/CD integration; 158 * Based on human-readable definitions (YAML); 159 * In order to create auditable and reproducible pipelines. 160 * An extensible tool which can be enhanced via [plugins](https://github.com/zntrio/harp-plugins). 161 162 And allows : 163 164 * Bundle level operations 165 * Create a bundle from scratch / template / JSON (more via plugins); 166 * Generate a complete bundle using a YAML Descriptor (`BundleTemplate`) to describe secret and their usages; 167 * Read value stored in the K/V virtual file system; 168 * Update the K/V virtual file system; 169 * Reproducible patch applied on immutable container (copy-on-write); 170 * Import / Export to Vault. 171 * Immutable container level operations 172 * Seal / Unseal a container for integrity and confidentiality property conservation 173 to enforce at-rest encryption (aes256-gcm96 or chacha20-poly1305); 174 * Multiple identities sealing algorithm; 175 176 ## FAQ 177 178 * Is it used internally at zntrio? - Yes. It is used to generate bootstrap 179 secrets used to bootstrap the new region infrastructure components. 180 #ChickenEggProblem 181 182 * Harp is only supporting `Vault`? - No, it has been published with only vault 183 support built-in, but it supports many other secret storage implementations via 184 plugins. 185 186 * What's the difference with `Vault`? - HashiCorp Vault is an encrypted highly 187 available K/V store with advanced authorization engine, it doesn't handle 188 secret provisioning for you. You can't ask Vault to generate secrets for your 189 application and store them using a defined logic. Harp is filling this 190 requirement. 191 192 ## License 193 194 `harp` artifacts and source code is released under [Apache 2.0 Software License](LICENSE). 195 196 # Build instructions 197 198 Download a [release](https://github.com/zntrio/harp/releases) or build from source. 199 200 ## Clone repository 201 202 ```sh 203 $ git clone git@github.com:zntrio/harp.git 204 $ export HARP_REPOSITORY=$(pwd)/harp 205 ``` 206 207 ## Setup dev environment 208 209 ### With nix flake 210 211 Install `nix` on your system, if not already installed. 212 213 ```sh 214 $ sudo install -d -m755 -o $(id -u) -g $(id -g) /nix 215 $ curl -L https://nixos.org/nix/install | sh 216 ``` 217 218 > More information? - <https://nixos.wiki/wiki/Nix_Installation_Guide> 219 220 ```sh 221 $ cd $HARP_REPOSITORY 222 $ nix develop 223 ``` 224 225 ### Non-nix managed environment 226 227 #### Check your go version 228 229 > Only last 2 minor versions of a major are supported. 230 231 `Harp` is compiled with : 232 233 ```sh 234 $ go version 235 go version go1.19.4 linux/amd64 236 ``` 237 238 > Simple go version manager - <https://github.com/stefanmaric/g> 239 240 #### Install mage 241 242 [Mage](https://magefile.org/) is an alternative to Make where language used is Go. 243 You can install it using 2 different methods. 244 245 ##### From source 246 247 ```sh 248 # Install mage 249 git clone https://github.com/magefile/mage 250 cd mage 251 go run bootstrap.go 252 ``` 253 254 #### Bootstrap tools 255 256 ```sh 257 # Go to tools submodule 258 cd $HARP_REPOSITORY/tools 259 # Resolve dependencies 260 go mod tidy 261 go mod vendor 262 # Pull tools sources, compile them and install executable in tools/bin 263 mage 264 ``` 265 266 ## Mage targets 267 268 ```sh 269 ❯ mage -l 270 Targets: 271 api:generate protobuf objects from proto definitions. 272 build* harp executable. 273 code:format source code and process imports. 274 code:generate SDK code (mocks, tests, etc.) 275 code:licenser apply copyright banner to source code. 276 code:lint code using golangci-lint. 277 compile harp code to create an executable. 278 docker:harp build harp docker image 279 docker:tools prepares docker images with go toolchain and project tools. 280 homebrew generates homebrew formula from compiled artifacts. 281 release harp version and cross-compile code to produce all artifacts. 282 releaser:harp releases harp artifacts using docker pipeline. 283 test:cli Test harp application. 284 test:unit Test harp application. 285 286 * default target 287 ``` 288 289 # Plugins 290 291 You can find more Harp feature extensions - <https://github.com/zntrio/harp-plugins> 292 293 # Community 294 295 Here is the list of external projects used as inspiration : 296 297 * [Kubernetes](https://github.com/kubernetes/) 298 * [Helm](https://github.com/helm/) 299 * [Open Policy Agent ConfTest](https://github.com/open-policy-agent/conftest) 300 * [SaltPack](https://github.com/keybase/saltpack) 301 * [Hashicorp Vault](https://github.com/hashicorp/vault) 302 * [AWS SDK Go](https://github.com/aws/aws-sdk-go) 303