github.com/StackExchange/blackbox/v2@v2.0.1-0.20220331193400-d84e904973ab/README.md (about) 1 BlackBox [![CircleCI](https://circleci.com/gh/StackExchange/blackbox.svg?style=shield)](https://circleci.com/gh/StackExchange/workflows/blackbox) [![Build Status](https://github.com/StackExchange/blackbox/workflows/build/badge.svg)](https://github.com/StackExchange/blackbox/actions?query=workflow%3Abuild+branch%3Amaster) 2 ======== 3 4 Safely store secrets in a VCS repo (i.e. Git, Mercurial, Subversion or Perforce). These commands make it easy for you to Gnu Privacy Guard (GPG) encrypt specific files in a repo so they are "encrypted at rest" in your repository. However, the scripts make it easy to decrypt them when you need to view or edit them, and decrypt them for use in production. Originally written for Puppet, BlackBox now works with any Git or Mercurial repository. 5 6 A slide presentation about an older release [is on SlideShare](http://www.slideshare.net/TomLimoncelli/the-blackbox-project-sfae). 7 8 Join our mailing list: [https://groups.google.com/d/forum/blackbox-project](https://groups.google.com/d/forum/blackbox-project) 9 10 Table of Contents 11 ================= 12 13 - [BlackBox](#blackbox) 14 - [Table of Contents](#table-of-contents) 15 - [Overview](#overview) 16 - [Why is this important?](#why-is-this-important) 17 - [Installation Instructions](#installation-instructions) 18 - [Commands](#commands) 19 - [Compatibility](#compatibility) 20 - [How is the encryption done?](#how-is-the-encryption-done) 21 - [What does this look like to the typical user?](#what-does-this-look-like-to-the-typical-user) 22 - Configuration Management 23 - [How to use the secrets with Ansible?](#how-to-use-the-secrets-with-ansible) 24 - [How to use the secrets with Puppet?](#how-to-use-the-secrets-with-puppet) 25 - [Entire files](#entire-files) 26 - [Small strings](#small-strings) 27 - File Management 28 - [How to enroll a new file into the system?](#how-to-enroll-a-new-file-into-the-system) 29 - [How to remove a file from the system?](#how-to-remove-a-file-from-the-system) 30 - User Management 31 - [How to indoctrinate a new user into the system?](#how-to-indoctrinate-a-new-user-into-the-system) 32 - [How to remove a user from the system?](#how-to-remove-a-user-from-the-system) 33 - Repo Management 34 - [Enabling BlackBox For a Repo](#enabling-blackbox-for-a-repo) 35 - [Set up automated users or “role accounts”](#set-up-automated-users-or-role-accounts) 36 - [Replacing expired keys](#replacing-expired-keys) 37 - [Some common errors](#some-common-errors) 38 - [Using BlackBox on Windows](#using-blackbox-on-windows) 39 - [Using BlackBox without a repo](#using-blackbox-without-a-repo) 40 - [Some Subversion gotchas](#some-subversion-gotchas) 41 - [Using Blackbox when gpg2 is installed next to gpg](#using-blackbox-when-gpg2-is-installed-next-to-gpg) 42 - [How to submit bugs or ask questions?](#how-to-submit-bugs-or-ask-questions) 43 - [Developer Info](#developer-info) 44 - [Alternatives](#alternatives) 45 - [License](#license) 46 47 Overview 48 ======== 49 50 Suppose you have a VCS repository (i.e. a Git or Mercurial repo) and certain files contain secrets such as passwords or SSL private keys. Often people just store such files "and hope that nobody finds them in the repo". That's not safe. 51 52 With BlackBox, those files are stored encrypted using GPG. Access to the VCS repo without also having the right GPG keys makes it worthless to have the files. As long as you keep your GPG keys safe, you don't have to worry about storing your VCS repo on an untrusted server. Heck, even if you trust your server, now you don't have to trust the people that do backups of that server, or the people that handle the backup tapes! 53 54 Rather than one GPG passphrase for all the files, each person with access has their own GPG keys in the system. Any file can be decrypted by anyone with their GPG key. This way, if one person leaves the company, you don't have to communicate a new password to everyone with access. Simply disable the one key that should no longer have access. The process for doing this is as easy as running 2 commands (1 to disable their key, 1 to re-encrypt all files.) 55 56 Automated processes often need access to all the decrypted files. This is easy too. For example, suppose Git is being used for Puppet files. The master needs access to the decrypted version of all the files. Simply set up a GPG key for the Puppet master (or the role account that pushes new files to the Puppet master) and have that user run `blackbox_postdeploy` after any files are updated. 57 58 Getting started 59 --------------- 60 61 1. If you don't have a GPG key, set it up using instructions such as: 62 [Set up GPG key](https://help.github.com/articles/generating-a-new-gpg-key/). \ 63 Now you are ready to go. 64 65 1. `cd` into a Git, Mercurial, Subversion or Perforce repository and run `blackbox_initialize`. 66 67 1. If a file is to be encrypted, run `blackbox_register_new_file` and you are done. 68 69 1. Add and remove keys with `blackbox_addadmin` and `blackbox_removeadmin`. 70 71 1. To view and/or edit a file, run `blackbox_edit`; 72 this will decrypt the file and open with whatever is specified by 73 your $EDITOR environment variable. \ 74 When you close the editor the 75 file will automatically be encrypted again and the temporary plaintext 76 file will be shredded. \ 77 If you need to leave the file decrypted while 78 you update you can use the`blackbox_edit_start` to decrypt the file 79 and `blackbox_edit_end` when you want to "put it back in the box." 80 81 Why is this important? 82 ====================== 83 84 OBVIOUSLY we don't want secret things like SSL private keys and passwords to be leaked. 85 86 NOT SO OBVIOUSLY when we store "secrets" in a VCS repo like Git or Mercurial, suddenly we are less able to share our code with other people. Communication between subteams of an organization is hurt. You can't collaborate as well. Either you find yourself emailing individual files around (yuck!), making a special repo with just the files needed by your collaborators (yuck!!), or just deciding that collaboration isn't worth all that effort (yuck!!!). 87 88 The ability to be open and transparent about our code, with the exception of a few specific files, is key to the kind of collaboration that DevOps and modern IT practitioners need to do. 89 90 Installation Instructions 91 ========================= 92 93 - *The hard way (manual)*: Copy all the files in "bin" to your "bin". 94 - *The hard way (automatic)*: `make copy-install` will copy the bin files into $PREFIX/bin, default is /usr/local (uninstall with `make copy-uninstall`). 95 - *The symlinks way*: `make symlinks-install` will make symlinks of the bin files into $PREFIX/bin, default is /usr/local (uninstall with `make copy-uninstall`) (useful when doing development) 96 - *The MacPorts Way*: `sudo port install vcs_blackbox` 97 - *The Homebrew Way*: `brew install blackbox` 98 - *The RPM way*: Check out the repo and make an RPM via `make packages-rpm`; now you can distribute the RPM via local methods. (Requires [fpm](https://github.com/jordansissel/fpm).) 99 - *The Debian/Ubuntu way*: Check out the repo and make a DEB via `make packages-deb`; now you can distribute the DEB via local methods. (Requires [fpm](https://github.com/jordansissel/fpm).) 100 - *The Antigen Way*: Add `antigen bundle StackExchange/blackbox` to your .zshrc 101 - *The Zgenom Way*: Add `zgenom load StackExchange/blackbox` to your .zshrc where you're loading your other plugins. 102 - *The Nix Way*: `nix-shell -p blackbox` 103 - *The Pkgsrc Way*: `pkgin in scm-blackbox` 104 105 Commands 106 ======== 107 108 | Name: | Description: | 109 |-------------------------------------|-------------------------------------------------------------------------| 110 | `blackbox_edit <file>` | Decrypt, run $EDITOR, re-encrypt a file | 111 | `blackbox_edit_start <file>` | Decrypt a file so it can be updated | 112 | `blackbox_edit_end <file>` | Encrypt a file after blackbox_edit_start was used | 113 | `blackbox_cat <file>` | Decrypt and view the contents of a file | 114 | `blackbox_view <file>` | Like blackbox_cat but pipes to `less` or $PAGER | 115 | `blackbox_diff` | Diff decrypted files against their original crypted version | 116 | `blackbox_initialize` | Enable blackbox for a GIT or HG repo | 117 | `blackbox_register_new_file <file>` | Encrypt a file for the first time | 118 | `blackbox_deregister_file <file>` | Remove a file from blackbox | 119 | `blackbox_list_files` | List the files maintained by blackbox | 120 | `blackbox_list_admins` | List admins currently authorized for blackbox | 121 | `blackbox_decrypt_file <file>` | Decrypt a file | 122 | `blackbox_decrypt_all_files` | Decrypt all managed files (INTERACTIVE) | 123 | `blackbox_postdeploy` | Decrypt all managed files (batch) | 124 | `blackbox_addadmin <gpg-key>` | Add someone to the list of people that can encrypt/decrypt secrets | 125 | `blackbox_removeadmin <gpg-key>` | Remove someone from the list of people that can encrypt/decrypt secrets | 126 | `blackbox_shred_all_files` | Safely delete any decrypted files | 127 | `blackbox_update_all_files` | Decrypt then re-encrypt all files. Useful after keys are changed | 128 | `blackbox_whatsnew <file>` | show what has changed in the last commit for a given file | 129 130 Compatibility 131 ============= 132 133 BlackBox automatically determines which VCS you are using and does the right thing. It has a plug-in architecture to make it easy to extend to work with other systems. It has been tested to work with many operating systems. 134 135 - Version Control systems 136 - `git` -- The Git 137 - `hg` -- Mercurial 138 - `svn` -- SubVersion (Thanks, Ben Drasin!) 139 - `p4` -- Perforce 140 - none -- The files can be decrypted outside of a repo if the `.blackbox` directory is intact 141 - Operating system 142 - CentOS / RedHat 143 - MacOS X 144 - Cygwin (Thanks, Ben Drasin!) **See Note Below** 145 - MinGW (git bash on windows) **See Note Below** 146 - NetBSD 147 - SmartOS 148 149 To add or fix support for a VCS system, look for code at the end of `bin/_blackbox_common.sh` 150 151 To add or fix support for a new operating system, look for the case statements in `bin/_blackbox_common.sh` and `bin/_stack_lib.sh` and maybe `tools/confidence_test.sh` 152 153 Using BlackBox on Windows 154 ========================= 155 156 BlackBox can be used with Cygwin or MinGW. 157 158 ### Protect the line endings 159 160 BlackBox assumes that `blackbox-admins.txt` and `blackbox-files.txt` will have 161 LF line endings. Windows users should be careful to configure Git or other systems 162 to not convert or "fix" those files. 163 164 If you use Git, add the following lines to your `.gitattributes` file: 165 166 **/blackbox-admins.txt text eol=lf 167 **/blackbox-files.txt text eol=lf 168 169 The latest version of `blackbox_initialize` will create a `.gitattributes` file in the `$BLACKBOXDATA` 170 directory (usually `.blackbox`) for you. 171 172 ### Cygwin 173 174 Cygwin support requires the following packages: 175 176 Normal operation: 177 178 - gnupg 179 - git or mercurial or subversion or perforce (as appropriate) 180 181 Development (if you will be adding code and want to run the confidence test) 182 183 - procps 184 - make 185 - git (the confidence test currently only tests git) 186 187 ### MinGW 188 189 MinGW (comes with Git for Windows) support requires the following: 190 191 Normal operation: 192 193 - [Git for Windows](https://git-scm.com/) (not tested with Mercurial) 194 - Git Bash MINTTY returns a MinGW console. So when you install make sure you pick `MINTTY` instead of windows console. You'll be executing blackbox from the Git Bash prompt. 195 - You need at least version 2.8.1 of Git for Windows. 196 - [GnuWin32](https://sourceforge.net/projects/getgnuwin32/files/) - needed for various tools not least of which is mktemp which is used by blackbox 197 - after downloading the install just provides you with some batch files. Because of prior issues at sourceforge and to make sure you get the latest version of each package the batch files handle the brunt of the work of getting the correct packages and installing them for you. 198 - from a **windows command prompt** run `download.bat` once it has completed run `install.bat` then add the path for those tools to your PATH (ex: `PATH=%PATH%;c:\GnuWin32\bin`) 199 200 Development: 201 202 - unknown (if you develop Blackbox under MinGW, please let us know if any additional packages are required to run `make test`) 203 204 How is the encryption done? 205 =========================== 206 207 GPG has many different ways to encrypt a file. BlackBox uses the mode that lets you specify a list of keys that can decrypt the message. 208 209 If you have 5 people ("admins") that should be able to access the secrets, each creates a GPG key and adds their public key to the keychain. The GPG command used to encrypt the file lists all 5 key names, and therefore any 1 key can decrypt the file. 210 211 To remove someone's access, remove that admin's key name (i.e. email address) from the list of admins and re-encrypt all the files. They can still read the .gpg file (assuming they have access to the repository) but they can't decrypt it any more. 212 213 *What if they kept a copy of the old repo before you removed access?* Yes, they can decrypt old versions of the file. This is why when an admin leaves the team, you should change all your passwords, SSL certs, and so on. You should have been doing that before BlackBox, right? 214 215 *Why don't you use symmetric keys?* In other words, why mess with all this GPG key stuff and instead why don't we just encrypt all the files with a single passphrase. Yes, GPG supports that, but then we are managing a shared password, which is fraught with problems. If someone "leaves the team" we would have to communicate to everyone a new password. Now we just have to remove their key. This scales better. 216 217 *How do automated processes decrypt without asking for a password?* GPG requires a passphrase on a private key. However, it permits the creation of subkeys that have no passphrase. For automated processes, create a subkey that is only stored on the machine that needs to decrypt the files. For example, at Stack Exchange, when our Continuous Integration (CI) system pushes a code change to our Puppet masters, they run `blackbox_postdeploy` to decrypt all the files. The user that runs this code has a subkey that doesn't require a passphrase. Since we have many masters, each has its own key. And, yes, this means our Puppet Masters have to be very secure. However, they were already secure because, like, dude... if you can break into someone's puppet master you own their network. 218 219 *If you use Puppet, why didn't you just use hiera-eyaml?* There are 4 reasons: 220 221 1. This works with any Git or Mercurial repo, even if you aren't using Puppet. 222 2. hiera-eyaml decrypts "on demand" which means your Puppet Master now uses a lot of CPU to decrypt keys every time it is contacted. It slows down your master, which, in my case, is already slow enough. 223 3. This works with binary files, without having to ASCIIify them and paste them into a YAML file. Have you tried to do this with a cert that is 10K long and changes every few weeks? Ick. 224 4. hiera-eyaml didn't exist when I wrote this. 225 226 What does this look like to the typical user? 227 ============================================= 228 229 - If you need to, start the GPG Agent: `eval $(gpg-agent --daemon)` 230 - Decrypt the file so it is editable: `blackbox_edit_start FILENAME` 231 - (You will need to enter your GPG passphrase.) 232 - Edit FILENAME as you desire: `vim FILENAME` 233 - Re-encrypt the file: `blackbox_edit_end FILENAME` 234 - Commit the changes. `git commit -a` or `hg commit` 235 236 Wait... it can be even easier than that! Run `blackbox_edit FILENAME`, and it'll decrypt the file in a temp file and call `$EDITOR` on it, re-encrypting again after the editor is closed. 237 238 How to use the secrets with Ansible? 239 =================================== 240 241 Ansible Vault provides functionality for encrypting both entire files and strings stored within files; however, 242 keeping track of the password(s) required for decryption is not handled by this module. 243 244 Instead one must specify a password file when running the playbook. 245 246 Ansible example for password file: `my_secret_password.txt.gpg` 247 248 ``` 249 ansible-playbook --vault-password-file my_secret_password.txt site.yml 250 ``` 251 252 Alternatively, one can specify this in the `ANSIBLE_VAULT_PASSWORD_FILE` environment variable. 253 254 How to use the secrets with Puppet? 255 =================================== 256 257 ### Entire files: 258 259 Entire files, such as SSL certs and private keys, are treated just like regular files. You decrypt them any time you push a new release to the puppet master. 260 261 Puppet example for an encrypted file: `secret_file.key.gpg` 262 263 ``` 264 file { '/etc/my_little_secret.key': 265 ensure => 'file', 266 owner => 'root', 267 group => 'puppet', 268 mode => '0760', 269 source => "puppet:///modules/${module_name}/secret_file.key", 270 } 271 ``` 272 273 ### Small strings: 274 275 Small strings, such as passwords and API keys, are stored in a hiera yaml file, which you encrypt with `blackbox_register_new_file`. For example, we use a file called `blackbox.yaml`. You can access them using the hiera() function. 276 277 *Setup:* Configure `hiera.yaml` by adding "blackbox" to the search hierarchy: 278 279 ``` 280 :hierarchy: 281 - ... 282 - blackbox 283 - ... 284 ``` 285 286 In blackbox.yaml specify: 287 288 ``` 289 --- 290 module::test_password: "my secret password" 291 ``` 292 293 In your Puppet Code, access the password as you would any hiera data: 294 295 ``` 296 $the_password = hiera('module::test_password', 'fail') 297 298 file {'/tmp/debug-blackbox.txt': 299 content => $the_password, 300 owner => 'root', 301 group => 'root', 302 mode => '0600', 303 } 304 ``` 305 306 The variable `$the_password` will contain "my secret password" and can be used anywhere strings are used. 307 308 How to enroll a new file into the system? 309 ========================================= 310 311 - If you need to, start the GPG Agent: `eval $(gpg-agent --daemon)` 312 - Add the file to the system: 313 314 ``` 315 blackbox_register_new_file path/to/file.name.key 316 ``` 317 318 Multiple file names can be specified on the command line: 319 320 Example 1: Register 2 files: 321 322 ``` 323 blackbox_register_new_file file1.txt file2.txt 324 ``` 325 326 Example 2: Register all the files in `$DIR`: 327 328 ``` 329 find $DIR -type f -not -name '*.gpg' -print0 | xargs -0 blackbox_register_new_file 330 ``` 331 332 How to remove a file from the system? 333 ===================================== 334 335 This happens quite rarely, but we've got it covered: 336 337 ``` 338 blackbox_deregister_file path/to/file.name.key 339 ``` 340 341 How to indoctrinate a new user into the system? 342 =============================================== 343 344 FYI: Your repo may use `keyrings/live` instead of `.blackbox`. See "Where is the configuration stored?" 345 346 `.blackbox/blackbox-admins.txt` is a file that lists which users are able to decrypt files. (More pedantically, it is a list of the GnuPG key names that the file is encrypted for.) 347 348 To join the list of people that can edit the file requires three steps; You create a GPG key and add it to the key ring. Then, someone that already has access adds you to the system. Lastly, you should test your access. 349 350 ### Step 1: NEW USER creates a GPG key pair on a secure machine and adds to public keychain. 351 352 If you don't already have a GPG key, here's how to generate one: 353 354 ``` 355 gpg --gen-key 356 ``` 357 358 WARNING: New versions of GPG generate keys which are not understood by 359 old versions of GPG. If you generate a key with a new version of GPG, 360 this will cause problems for users of older versions of GPG. 361 Therefore it is recommended that you either assure that everyone using 362 Blackbox have the exact same version of GPG, or generate GPG keys 363 using a version of GPG as old as the oldest version of GPG used by 364 everyone using Blackbox. 365 366 Pick defaults for encryption settings, 0 expiration. Pick a VERY GOOD passphrase. Store a backup of the private key someplace secure. For example, keep the backup copy on a USB drive that is locked in safe. Or, at least put it on a secure machine with little or no internet access, full-disk-encryption, etc. Your employer probably has rules about how to store such things. 367 368 FYI: If generating the key is slow, this is usually because the system 369 isn't generating enough entropy. Tip: Open another window on that 370 machine and run this command: `ls -R /` 371 372 Now that you have a GPG key, add yourself as an admin: 373 374 ``` 375 blackbox_addadmin KEYNAME 376 ``` 377 378 ...where "KEYNAME" is the email address listed in the gpg key you created previously. For example: 379 380 ``` 381 blackbox_addadmin tal@example.com 382 ``` 383 384 When the command completes successfully, instructions on how to commit these changes will be output. Run the command as given to commit the changes. It will look like this: 385 386 ``` 387 git commit -m'NEW ADMIN: tal@example.com' .blackbox/pubring.gpg .blackbox/trustdb.gpg .blackbox/blackbox-admins.txt 388 ``` 389 390 Then push it to the repo: 391 392 ``` 393 git push 394 395 or 396 397 ht push 398 399 (or whatever is appropriate) 400 ``` 401 402 NOTE: Creating a Role Account? If you are adding the pubring.gpg of a role account, you can specify the directory where the pubring.gpg file can be found as a 2nd parameter: `blackbox_addadmin puppetmaster@puppet-master-1.example.com /path/to/the/dir` 403 404 ### Step 2: EXISTING ADMIN adds new user to the system. 405 406 Ask someone that already has access to re-encrypt the data files. This gives you access. They simply decrypt and re-encrypt the data without making any changes. 407 408 Pre-check: Verify the new keys look good. 409 410 ``` 411 git pull # Or whatever is required for your system 412 gpg --homedir=.blackbox --list-keys 413 ``` 414 415 For example, examine the key name (email address) to make sure it conforms to corporate standards. 416 417 Import the keychain into your personal keychain and reencrypt: 418 419 ``` 420 gpg --import .blackbox/pubring.gpg 421 blackbox_update_all_files 422 ``` 423 424 Push the re-encrypted files: 425 426 ``` 427 git commit -a 428 git push 429 430 or 431 432 hg commit 433 hg push 434 ``` 435 436 ### Step 3: NEW USER tests. 437 438 Make sure you can decrypt a file. (Suggestion: Keep a dummy file in VCS just for new people to practice on.) 439 440 How to remove a user from the system? 441 ===================================== 442 443 Simply run `blackbox_removeadmin` with their keyname then re-encrypt: 444 445 Example: 446 447 ``` 448 blackbox_removeadmin olduser@example.com 449 blackbox_update_all_files 450 ``` 451 452 When the command completes, you will be given a reminder to check in the change and push it. 453 454 Note that their keys will still be in the key ring, but they will go unused. If you'd like to clean up the keyring, use the normal GPG commands and check in the file. 455 456 FYI: Your repo may use `keyrings/live` instead of `.blackbox`. See "Where is the configuration stored?" 457 458 ``` 459 gpg --homedir=.blackbox --list-keys 460 gpg --homedir=.blackbox --delete-key olduser@example.com 461 git commit -m'Cleaned olduser@example.com from keyring' .blackbox/* 462 ``` 463 464 FYI: Your repo may use `keyrings/live` instead of `.blackbox`. See "Where is the configuration stored?" 465 466 The key ring only has public keys. There are no secret keys to delete. 467 468 Remember that this person did have access to all the secrets at one time. They could have made a copy. Therefore, to be completely secure, you should change all passwords, generate new SSL keys, and so on just like when anyone that had privileged access leaves an organization. 469 470 Where is the configuration stored? .blackbox vs. keyrings/live 471 ============================================================== 472 473 Blackbox stores its configuration data in the `.blackbox` subdirectory. Older 474 repos use `keyrings/live`. For backwards compatibility either will work. 475 476 All documentation refers to `.blackbox`. 477 478 You can convert an old repo by simply renaming the directory: 479 480 ``` 481 mv keyrings/live .blackbox 482 rmdir keyrings 483 ``` 484 485 There is no technical reason to convert old repos except that it is less 486 confusing to users. 487 488 This change was made in commit 60e782a0, release v1.20180615. 489 490 The details: 491 492 - First Blackbox checks `$BLACKBOXDATA`. If this environment variable is set, this is the directory that will be used. If it lists a directory that does not exist, Blackbox will print an error and exit. 493 - If `$BLACKBOXDATA` is not set: (which is the typical use case) 494 - Blackbox will first try `keyrings/live` and use it if it exists. 495 - Otherwise the default `.blackbox` will be used. If `.blackbox` does not exist, Blackbox will print an error and exit. 496 497 498 Enabling BlackBox For a Repo 499 ============================ 500 501 Overview: 502 503 To add "blackbox" to a git or mercurial repo, you'll need to do the following: 504 505 1. Run the initialize script. This adds a few files to your repo in a directory called ".blackbox". 506 2. For the first user, create a GPG key and add it to the key ring. 507 3. Encrypt the files you want to be "secret". 508 4. For any automated user (one that must be able to decrypt without a passphrase), create a GPG key and create a subkey with an empty passphrase. 509 510 FYI: Your repo may use `keyrings/live` instead of `.blackbox`. See "Where is the configuration stored?" 511 512 ### Run the initialize script. 513 514 You'll want to include blackbox's "bin" directory in your PATH: 515 516 ``` 517 export PATH=$PATH:/the/path/to/blackbox/bin 518 blackbox_initialize 519 ``` 520 521 If you're using antigen, adding `antigen bundle StackExchange/blackbox` to your .zshrc will download this repository and add it to your $PATH. 522 523 ### For the first user, create a GPG key and add it to the key ring. 524 525 Follow the instructions for "[How to indoctrinate a new user into the system?](#how-to-indoctrinate-a-new-user-into-the-system)". Only do Step 1. 526 527 Once that is done, is a good idea to test the system by making sure a file can be added to the system (see "How to enroll a new file into the system?"), and a different user can decrypt the file. 528 529 Make a new file and register it: 530 531 ``` 532 rm -f foo.txt.gpg foo.txt 533 echo This is a test. >foo.txt 534 blackbox_register_new_file foo.txt 535 ``` 536 537 Decrypt it: 538 539 ``` 540 blackbox_edit_start foo.txt.gpg 541 cat foo.txt 542 echo This is the new file contents. >foo.txt 543 ``` 544 545 Re-encrypt it: 546 547 ``` 548 blackbox_edit_end foo.txt.gpg 549 ls -l foo.txt* 550 ``` 551 552 You should only see `foo.txt.gpg` as `foo.txt` should be gone. 553 554 The next step is to commit `foo.txt.gpg` and make sure another user can check out, view, and change the contents of the file. That is left as an exercise for the reader. If you are feel like taking a risk, don't commit `foo.txt.gpg` and delete it instead. 555 556 Set up automated users or "role accounts" 557 ========================================= 558 559 i.e. This is how a Puppet Master can have access to the unencrypted data. 560 561 FYI: Your repo may use `keyrings/live` instead of `.blackbox`. See "Where is the configuration stored?" 562 563 An automated user (a "role account") is one that that must be able to decrypt without a passphrase. In general you'll want to do this for the user that pulls the files from the repo to the master. This may be automated with Jenkins CI or other CI system. 564 565 GPG keys have to have a passphrase. However, passphrases are optional on subkeys. Therefore, we will create a key with a passphrase then create a subkey without a passphrase. Since the subkey is very powerful, it should be created on a very secure machine. 566 567 There's another catch. The role account probably can't check files into Git/Mercurial. It probably only has read-only access to the repo. That's a good security policy. This means that the role account can't be used to upload the subkey public bits into the repo. 568 569 Therefore, we will create the key/subkey on a secure machine as yourself. From there we can commit the public portions into the repo. Also from this account we will export the parts that the role account needs, copy them to where the role account can access them, and import them as the role account. 570 571 ProTip: If asked to generate entropy, consider running this on the same machine in another window: `sudo dd if=/dev/sda of=/dev/null` 572 573 For the rest of this doc, you'll need to make the following substitutions: 574 575 - ROLEUSER: svc_deployacct or whatever your role account's name is. 576 - NEWMASTER: the machine this role account exists on. 577 - SECUREHOST: The machine you use to create the keys. 578 579 NOTE: This should be more automated/scripted. Patches welcome. 580 581 On SECUREHOST, create the puppet master's keys: 582 583 ``` 584 $ mkdir /tmp/NEWMASTER 585 $ cd /tmp/NEWMASTER 586 $ gpg --homedir . --gen-key 587 Your selection? 588 (1) RSA and RSA (default) 589 What keysize do you want? (2048) DEFAULT 590 Key is valid for? (0) DEFAULT 591 592 # Real name: Puppet CI Deploy Account 593 # Email address: svc_deployacct@hostname.domain.name 594 ``` 595 596 NOTE: Rather than a real email address, use the username@FQDN of the host the key will be used on. If you use this role account on many machines, each should have its own key. By using the FQDN of the host, you will be able to know which key is which. In this doc, we'll refer to username@FQDN as $KEYNAME 597 598 Save the passphrase somewhere safe! 599 600 Create a sub-key that has no password: 601 602 ``` 603 $ gpg --homedir . --edit-key svc_deployacct 604 gpg> addkey 605 (enter passphrase) 606 Please select what kind of key you want: 607 (3) DSA (sign only) 608 (4) RSA (sign only) 609 (5) Elgamal (encrypt only) 610 (6) RSA (encrypt only) 611 Your selection? 6 612 What keysize do you want? (2048) 613 Key is valid for? (0) 614 Command> key 2 615 (the new subkey has a "*" next to it) 616 Command> passwd 617 (enter the main key's passphrase) 618 (enter an empty passphrase for the subkey... confirm you want to do this) 619 Command> save 620 ``` 621 622 Now securely export this directory to NEWMASTER: 623 624 ``` 625 gpg --homedir . --export -a svc_sadeploy >/tmp/NEWMASTER/pubkey.txt 626 tar cvf /tmp/keys.tar . 627 rsync -avP /tmp/keys.tar NEWMASTER:/tmp/. 628 ``` 629 630 On NEWMASTER, receive the new GnuPG config: 631 632 ``` 633 sudo -u svc_deployacct bash 634 mkdir -m 0700 -p ~/.gnupg 635 cd ~/.gnupg && tar xpvf /tmp/keys.tar 636 ``` 637 638 <!--- 639 Back on SECUREHOST, import the pubkey into the repository. 640 641 ``` 642 $ cd .blackbox 643 $ gpg --homedir . --import /tmp/NEWMASTER/pubkey.txt 644 ``` 645 --> 646 647 Back on SECUREHOST, add the new email address to .blackbox/blackbox-admins.txt: 648 649 ``` 650 cd /path/to/the/repo 651 blackbox_addadmin $KEYNAME /tmp/NEWMASTER 652 ``` 653 654 Verify that secring.gpg is a zero-length file. If it isn't, you have somehow added a private key to the keyring. Start over. 655 656 ``` 657 cd .blackbox 658 ls -l secring.gpg 659 ``` 660 661 Commit the recent changes: 662 663 ``` 664 cd .blackbox 665 git commit -m"Adding key for KEYNAME" pubring.gpg trustdb.gpg blackbox-admins.txt 666 ``` 667 668 Regenerate all encrypted files with the new key: 669 670 ``` 671 blackbox_update_all_files 672 git status 673 git commit -m"updated encryption" -a 674 git push 675 ``` 676 677 On NEWMASTER, import the keys and decrypt the files: 678 679 ``` 680 sudo -u svc_sadeploy bash # Become the role account. 681 gpg --import /etc/puppet/.blackbox/pubring.gpg 682 export PATH=$PATH:/path/to/blackbox/bin 683 blackbox_postdeploy 684 sudo -u puppet cat /etc/puppet/hieradata/blackbox.yaml # or any encrypted file. 685 ``` 686 687 ProTip: If you get "gpg: decryption failed: No secret key" then you forgot to re-encrypt blackbox.yaml with the new key. 688 689 On SECUREHOST, securely delete your files: 690 691 ``` 692 cd /tmp/NEWMASTER 693 # On machines with the "shred" command: 694 shred -u /tmp/keys.tar 695 find . -type f -print0 | xargs -0 shred -u 696 # All else: 697 rm -rf /tmp/NEWMASTER 698 ``` 699 700 Also shred any other temporary files you may have made. 701 702 Replacing expired keys 703 ====================== 704 705 If someone's key has already expired, blackbox will stop 706 encrypting. You see this error: 707 708 ``` 709 $ blackbox_edit_end modified_file.txt 710 --> Error: can't re-encrypt because a key has expired. 711 ``` 712 713 FYI: Your repo may use `keyrings/live` instead of `.blackbox`. See "Where is the configuration stored?" 714 715 You can also detect keys that are about to expire by issuing this command and manually reviewing the "expired:" dates: 716 717 gpg --homedir=.blackbox --list-keys 718 719 or... list UIDs that will expire within 1 month from today: (Warning: this also lists keys without an expiration date) 720 721 gpg --homedir=.blackbox --list-keys --with-colons --fixed-list-mode | grep ^uid | awk -F: '$6 < '$(( $(date +%s) + 2592000)) 722 723 Here's how to replace the key: 724 725 - Step 1. Administrator removes expired user: 726 727 Warning: This process will erase any unencrypted files that you were in the process of editing. Copy them elsewhere and restore the changes when done. 728 729 ``` 730 blackbox_removeadmin expired_user@example.com 731 # This next command overwrites any changed unencrypted files. See warning above. 732 blackbox_update_all_files 733 git commit -m "Re-encrypt all files" 734 gpg --homedir=.blackbox --delete-key expired_user@example.com 735 git commit -m 'Cleaned expired_user@example.com from keyring' .blackbox/* 736 git push 737 ``` 738 739 - Step 2. Expired user adds an updated key: 740 741 ``` 742 git pull 743 blackbox_addadmin updated_user@example.com 744 git commit -m'NEW ADMIN: updated_user@example.com .blackbox/pubring.gpg .blackbox/trustdb.gpg .blackbox/blackbox-admins.txt 745 git push 746 ``` 747 748 - Step 3. Administrator re-encrypts all files with the updated key of the expired user: 749 750 ``` 751 git pull 752 gpg --import .blackbox/pubring.gpg 753 blackbox_update_all_files 754 git commit -m "Re-encrypt all files" 755 git push 756 ``` 757 758 - Step 4: Clean up: 759 760 Any files that were temporarily copied in the first step so as to not be overwritten can now be copied back and re-encrypted with the `blackbox_edit_end` command. 761 762 (Thanks to @chishaku for finding a solution to this problem!) 763 764 ### Configure git to show diffs in encrypted files 765 766 It's possible to tell Git to decrypt versions of the file before running them through `git diff` or `git log`. To achieve this do: 767 768 - Add the following to `.gitattributes` at the top of the git repository: 769 770 ``` 771 *.gpg diff=blackbox 772 ``` 773 774 - Add the following to `.git/config`: 775 776 ``` 777 [diff "blackbox"] 778 textconv = gpg --use-agent -q --batch --decrypt 779 ```` 780 781 And now commands like `git log -p file.gpg` will show a nice log of the changes in the encrypted file. 782 783 Some common errors 784 ================== 785 786 `gpg: filename: skipped: No public key` -- Usually this means there is an item in `.blackbox/blackbox-admins.txt` that is not the name of the key. Either something invalid was inserted (like a filename instead of a username) or a user has left the organization and their key was removed from the keychain, but their name wasn't removed from the blackbox-admins.txt file. 787 788 `gpg: decryption failed: No secret key` -- Usually means you forgot to re-encrypt the file with the new key. 789 790 `Error: can't re-encrypt because a key has expired.` -- A user's key has expired and can't be used to encrypt any more. Follow the [Replace expired keys](#replace-expired-keys) tip. 791 792 FYI: Your repo may use `keyrings/live` instead of `.blackbox`. See "Where is the configuration stored?" 793 794 Using Blackbox without a repo 795 ============================= 796 797 If the files are copied out of a repo they can still be decrypted and edited. Obviously edits, changes to keys, and such will be lost if they are made outside the repo. Also note that commands are most likely to only work if run from the base directory (i.e. the parent to the .blackbox directory). 798 799 The following commands have been tested outside a repo: 800 801 - `blackbox_postdeploy` 802 - `blackbox_edit_start` 803 - `blackbox_edit_end` 804 805 Some Subversion gotchas 806 ======================= 807 808 The current implementation will store the blackbox in `/keyrings` at the root of the entire repo. This will create an issue between environments that have different roots (i.e. checking out `/` on development vs `/releases/foo` in production). To get around this, you can `export BLACKBOX_REPOBASE=/path/to/repo` and set a specific base for your repo. 809 810 This was originally written for git and supports a two-phase commit, in which `commit` is a local commit and "push" sends the change upstream to the version control server when something is registered or deregistered with the system. The current implementation will immediately `commit` a file (to the upstream subversion server) when you execute a `blackbox_*` command. 811 812 Using Blackbox when gpg2 is installed next to gpg 813 ================================================= 814 815 In some situations, team members or automated roles need to install gpg 816 2.x alongside the system gpg version 1.x to catch up with the team's gpg 817 version. On Ubuntu 16, you can ```apt-get install gnupg2``` which 818 installs the binary gpg2. If you want to use this gpg2 binary, run every 819 blackbox command with GPG=gpg2. 820 821 For example: 822 823 ``` 824 GPG=gpg2 blackbox_postdeploy 825 ``` 826 827 How to submit bugs or ask questions? 828 ==================================== 829 830 We welcome questions, bug reports and feedback! 831 832 The best place to start is to join the [blackbox-project mailing list](https://groups.google.com/d/forum/blackbox-project) and ask there. 833 834 Bugs are tracked here in Github. Please feel free to [report bugs](https://github.com/StackExchange/blackbox/issues) yourself. 835 836 Developer Info 837 ============== 838 839 Code submissions are gladly welcomed! The code is fairly easy to read. 840 841 Get the code: 842 843 ``` 844 git clone git@github.com:StackExchange/blackbox.git 845 ``` 846 847 Test your changes: 848 849 ``` 850 make confidence 851 ``` 852 853 This runs through a number of system tests. It creates a repo, encrypts files, decrypts files, and so on. You can run these tests to verify that the changes you made didn't break anything. You can also use these tests to verify that the system works with a new operating system. 854 855 Please submit tests with code changes: 856 857 The best way to change BlackBox is via Test Driven Development. First add a test to `tools/confidence.sh`. This test should fail, and demonstrate the need for the change you are about to make. Then fix the bug or add the feature you want. When you are done, `make confidence` should pass all tests. The PR you submit should include your code as well as the new test. This way the confidence tests accumulate as the system grows as we know future changes don't break old features. 858 859 Note: The tests currently assume "git" and have been tested only on CentOS, Mac OS X, and Cygwin. Patches welcome! 860 861 Alternatives 862 ============ 863 864 Here are other open source packages that do something similar to BlackBox. If you like them better than BlackBox, please use them. 865 866 - [git-crypt](https://www.agwa.name/projects/git-crypt/) 867 - [Pass](http://www.zx2c4.com/projects/password-store/) 868 - [Transcrypt](https://github.com/elasticdog/transcrypt) 869 - [Keyringer](https://keyringer.pw/) 870 - [git-secret](https://github.com/sobolevn/git-secret) 871 872 git-crypt has the best git integration. Once set up it is nearly transparent to the users. However it only works with git. 873 874 875 License 876 ======= 877 878 This content is released under the MIT License. 879 See the [LICENSE.txt](LICENSE.txt) file.