github.com/git-lfs/git-lfs@v2.5.2+incompatible/docker/README.md (about) 1 # README # 2 3 ## TL;DR version ## 4 1. Run the dockers 5 6 ./docker/run_dockers.bsh 7 8 2. **Enjoy** all your new package files in 9 10 ./repos/ 11 12 ### Slightly longer version ### 13 14 1. Generate GPG keys for everything (See GPG Signing) 15 2. `export REPO_HOSTNAME=repo.something.com` 16 4. Generate git-lfs/repo packages and sign all packages 17 18 ./docker/run_dockers.bsh 19 20 5. Host the `/repo` on the `REPO_HOSTNAME` server 21 6. Test the repos and git-lfs in a client environment 22 23 ./docker/test_dockers.bsh 24 25 ## Using the Dockers ## 26 27 All docker commands need to either be run as root **or** as a user with docker 28 permissions. Adding your user name to the docker group (or setting up boot2docker 29 environment) is probably the easiest. 30 31 For Mac and Windows users, the git-lfs repo needs to be in your Users directory 32 or else boot2docker magic won't work. Alternatively, you could add addition 33 mount points like 34 [this](http://stackoverflow.com/questions/26639968/boot2docker-startup-script-to-mount-local-shared-folder-with-host) 35 36 ### Running Dockers ### 37 38 In order to run the dockers, the docker has to be run with a 39 lot of arguments to get the mount points right, etc... A convenient script is 40 supplied to make this all easy. Simply run 41 42 ./docker/run_docker.bsh 43 44 All the images are pulled automatically, and then run. 45 46 To only run certain docker images, supply them as arguments, e.g. 47 48 ./docker/run_docker.bsh debian_7 49 ./docker/run_docker.bsh centos_7 debian_8 50 ./docker/run_docker.bsh centos_{6,7} 51 52 And only those images will be run. 53 54 ### Development in Dockers ### 55 56 Sometimes you don't want to just build git-lfs and destroy the container, you 57 want to get in there, run a lot of command, debug, develop, etc... To do this, 58 the best command to run is bash, and then you have an interactive shell to use 59 60 ./docker/run_docker.bsh {image name(s)} -- bash 61 62 After listing the image(s) you want to run, add a double dash (--) and then any 63 command (and arguments) you want executed in the docker. Remember, the command 64 you are executing has to be in the docker image. 65 66 ## Docker images ## 67 68 There are currently three type of docker images: 69 70 1. Building images: `{OS NAME}_{OS VERSION}` - These build 71 git-lfs and save the package/repository in the `/repo` direrctory. This image 72 also signs all rpms/debs if gpg signing is setup 73 2. Environment building images: `{OS_NAME}_{OS_VERSION}_env` - 74 These build or install the environment (dependencies) for building git-lfs. These 75 are mostly important for CentOS because without these, many dependencies have 76 to be built by a developer. These containers should create packages for these 77 dependencies and place them in `/repo` 78 3. Testing images: `{OS_NAME}_{OS_VERSION}_test` - These images 79 should install the repo and download the git-lfs packages and dependencies to test 80 that everything is working, including the GPG signatures. Unlike the first two types, 81 testing images are not guaranteed to work without GPG signatures. They should 82 also run the test and integration scripts after installing git-lfs to verify 83 everything is working in a **non-developer** setup. (With the exception that go 84 is needed to build the tests...) 85 86 This default behavior for `./docker/run_dockers.bsh` 87 is to run all of the _building images_. These 88 containers will use the currently checked-out version of git-lfs and copy it 89 into the docker, and run `git clean -xdf` to remove any non-tracked files, 90 (but non-committed changes are kept). git-lfs is built, and a packages/repo is 91 created for each container. 92 93 These are all a developer would need to test the different OSes. And create the 94 git-lfs rpm or deb packages in the `/repo` directory. 95 96 In order to distribute git-lfs **and** build dependencies, the dependencies that 97 that were built to create the docker images need to be saved too. Most of these 98 are downloaded by yum/apt-get and do not need to be saved, but a few are not. 99 In order to save the necessary dependencies, call `./docker/run_dockers.bsh` on 100 `{OS_NAME}_{OS_VERSION}_env` and the rpms 101 will be extracted from the images and saved in the `./repo` directory. 102 (This _can_ be done in one command) 103 104 ./docker/run_dockers.bsh centos_6_env centos_6 105 106 This isn't all that important anymore, unless you want ruby2 and the gems used to 107 make the man pages for CentOS 6 where ruby2 is not natively available. Calling 108 the environment building images only needs to be done once, they should remain in 109 the `./repo` directory afterwards. 110 111 ### Run Docker Environment Variables ### 112 113 There are a few environment variables you can set to easily adjust the behavior 114 of the `run_docker.bsh` script. 115 116 `export` before calling `run_docker.bsh` 117 118 `REPO_HOSTNAME` - Override the hostname for all the repos generated/tested (see below) 119 120 `DOCKER_AUTOPULL` - Default 1. `run_docker.bsh` always pulls the latest version of 121 the lfs dockers. If set to 0, it will not check to see if a new pull is needed, 122 and you will always run off of your currently cached images docker images. 123 124 `AUTO_REMOVE` - Default 1. Docker containers are automatically deleted on 125 exit. If set to 0, the docker containers will not be automatically deleted upon 126 exit. This can be useful for a post mortem analysis (using other docker commands 127 not covered here). Just make sure you clean up the docker containers manually. 128 129 `DOCKER_OTHER_OPTIONS` - Any additional arguments you may want to pass to the 130 docker run command. This can be particularly useful when having to help docker 131 with dns, etc... For example `DOCKER_OTHER_OPTIONS="--dns 8.8.8.8"` 132 133 If for some reason on Windows, you need to add a -v mount, folder names need to 134 start with `//driveleter/dir...` instead of `/driveleter/dir...` to fool MINGW32 135 136 ## Deploying/Building Repositories ## 137 138 When `./docker/run_dockers.bsh` is done building git-lfs and generating packages, 139 it automatically creates a repository for distribution too. Each distro gets a 140 repo generated in `./repos/{DISTRO_NAME}/{VERSION #}`. Just drop the repo 141 directory onto a webserver and you have a fully functioning Linux repo. (See 142 Testing the Repositories below for more detail) 143 144 The two major packages included are: 145 `git-lfs-....*` - the git-lfs package 146 `git-lfs-repo-release....*` - A package to install the repo. 147 148 When building, all **untracked** files are removed during RPM generation (except 149 any stray directories containing a .git folder will not be cleared. This 150 shouldn't be the case, unless you are temporarily storing another git repo in 151 the git repo. This is a safety mechanism in git, so just keep in mind if you 152 are producing packages.) 153 154 ### Setting the website URL ### 155 156 The git-lfs-repo-release must contain the URL where the repo is to be hosted. 157 The current default value is `git-lfs.github.com` but this can be overridden 158 using the `REPO_HOSTNAME` env var, e.g. 159 160 export REPO_HOSTNAME=www.notgithub.uk.co 161 ./docker/run_dockers.bsh 162 163 Now all the `git-lfs-repo-release....*` files will point to that URL instead 164 165 _Hint_: `REPO_HOSTNAME` can also be `www.notgithub.uk.co:2213/not_root_dir` 166 167 ### Testing the Repositories ### 168 169 To test that all the OSes can download the packages, install, and run the tests 170 again, run 171 172 ./test_dockers.bsh 173 174 (which is basically just `./docker/run_dockers.bsh ./docker/git-lfs-test_*`) 175 176 Remember to set `REPO_HOSTNAME` if you changed it for `./docker/build_docker.bsh` 177 This can also be used to run a local test (on `localhost:{Port Number}`, for 178 example) 179 180 An easy way to test the repositories locally, is to run them on a simple webserver such as 181 182 cd ./repos 183 python -m SimpleHTTPServer {Port number} 184 185 or 186 187 cd ./repos 188 ruby -run -ehttpd . -p{Port Number} 189 190 ## GPG signing ### 191 192 For private repo testing, GPG signing can be skipped. apt-get and yum can 193 install .deb/.rpm directly without gpg keys and everything will work (with 194 certain flags). This section is for distribution in a repo. Most if not all 195 this functionality is automatically disabled when there is no signing key 196 (`./docker/git-lfs_*.key`). 197 198 In order to sign packages, you need to generate and place GPG keys in the right 199 place. The general procedure for this is 200 201 gpg --gen-key 202 203 1. 4 - RSA 204 2. 4096 bits 205 3. Some length of time or 0 for infinite 206 4. y for yes 207 5. Signer name (Will become part of the key and uid) 208 6. Email address (Will become part of the key and uid) 209 7. Comment (Will become part of the key) 210 8. O for Okay 211 9. Enter a secure password, make sure you will not forget it 212 10. Generate Entropy! 213 214 gpg --export-secret-key '<key ID>!' > filename.key 215 216 e.g. `gpg --export-secret-key '547CF247!' > ./docker/git-lfs_centos_7.key` 217 218 *NOTE*: the **!** is important in this command 219 220 Keep in mind, .key files must NEVER be accidentally committed to the repo. 221 222 _What if you don't have gpg handy?_ Just enter one of the dockers (-- bash) and 223 generate them in there, and save them in the /src dir to get them out of the docker. 224 Or `docker run -it --rm -v $(pwd):/key OS_NAME:OS_VERSION bash`, and generate in 225 that docker and save to the `/key` directory 226 227 ### GPG Agent ### 228 229 To prevent MANY passphrase entries at random times, a gpg-agent docker is used to 230 cache your signing key. This is done automatically for you, whenever you call 231 `./docker/run_dockers.bsh` on a building image (`git-lfs_*.dockerfile`). It can 232 be manually preloaded by calling `./docker/gpg-agent_preload.bsh`. It will ask 233 you for your passphrase, once for each unique key out of all the dockers. So if 234 you use the same key for every docker, it will only prompt once. If you have 5 235 different keys, you'll have prompts, with only the key ID to tell you which 236 is which. 237 238 The gpg agent TTL is set to 1 year. If this is not acceptable for you, set the 239 `GPG_MAX_CACHE` and `GPG_DEFAULT_CACHE` environment variables (in seconds) before 240 starting the gpg-agent daemon. 241 242 `./docker/gpg-agent_start.bsh` starts the gpg-agent daemon. It is called 243 automatically by `./docker/gpg-agent_preload.bsh` 244 245 `./docker/gpg-agent_stop.bsh` stops the gpg-agent daemon. It is called 246 automatically by `./docker/gpg-agent_preload.bsh` 247 248 `./docker/gpg-agent_preload.bsh` is called automatically by 249 `./docker/run_dockers.bsh` when running any of the signing dockers. 250 251 `./docker/gpg-agent_preload.bsh -r` - Stops and restarts the gpg agent daemon. 252 This is useful for reloading keys when you update them in your host. 253 254 ### GPG capabilities by Distro ### 255 256 Every distro has its own GPG signing capability. This is why every signing 257 docker (`git-lfs_*.dockerfile`) can have an associated key (`git-lfs_*.key`) 258 259 Debian **will** work with 4096 bit RSA signing subkeys like [1] suggests, but will 260 also work with 4096 bit RSA signing keys. 261 262 CentOS will **not** work with subkeys[3]. CentOS 6 and 7 will work with 4096 bit 263 RSA signing keys 264 265 You can make a 4096 RSA key for Debian and CentOS 6/7 (4 for step 1 above, and 266 4096 for step 2). And only have two keys... Or optionally a 4096 RSA subkey for Debain 267 [1]. Or a key for each distro. Dealers choice. 268 269 [1] https://www.digitalocean.com/community/tutorials/how-to-use-reprepro-for-a-secure-package-repository-on-ubuntu-14-04 270 271 [2] https://iuscommunity.org/pages/CreatingAGPGKeyandSigningRPMs.html#exporting-the-public-gpg-key 272 273 [3] http://www.redhat.com/archives/rpm-list/2006-November/msg00105.html 274 275 276 ## Adding additional OSes ## 277 278 To add another operating system, it needs to be added to the lfs_dockers 279 repo and uploaded to docker hub. Then all that is left is to add it to the 280 IMAGES list in `run_dockers.bsh` and `test_dockers.bsh` 281 282 Follow the already existing pattern `{OS NAME}_{OS VERSION #}` where 283 **{OS NAME}** and **{OS VERSION #}** should not contain underscores (\_). 284 285 ## Docker Cheat sheet ## 286 287 Install https://docs.docker.com/installation/ 288 289 * list running dockers 290 291 docker ps 292 293 * list stopped dockers too 294 295 docker ps -a 296 297 * Remove all stopped dockers 298 299 docker rm $(docker ps --filter=status=exited -q) 300 301 * List docker images 302 303 docker images 304 305 * Remove unused docker images 306 307 docker rmi $(docker images -a --filter=dangling=true -q) 308 309 * Run another command (like bash) in a running docker 310 311 docker exec -i {docker name} {command} 312 313 * Stopping a docker (signal 15 to the main pid) 314 315 docker stop {docker name} 316 317 * Killing a docker (signal 9 to the main pid) 318 319 docker kill {docker name} 320 321 # Troubleshooting # 322 323 1. I started one of the script, and am trying to stop it with Ctrl+C. It is 324 ignoring many Ctrl+C's 325 326 This happens a lot when calling programs like apt-get, yum, etc... From the 327 host, you can still use ps, pgrep, kill, pkill, etc... commands to kill the 328 PIDs in a docker. You can also use `docker ps` to find the container 329 name/id and then used `docker stop` (signal 15) or `docker kill` 330 (signal 9) to stop the docker. You can also use 'docker exec' to start another 331 bash or kill command inside that container 332 333 2. How do I re-enter a docker after it failed/succeeded? 334 335 Dockers are immediately deleted upon exit. The best way to work in a docker 336 is to run bash (See Development in Dockers). This will let you to run the 337 main build command and then continue. 338 339 3. That answer's not good enough. How do I resume a docker? 340 341 Well, first you have to set the environment variable `AUTO_REMOVE=0` 342 before running the image you want to resume. This will keep the docker 343 around after stopping. (Be careful! They multiply like rabbits.) Then 344 345 docker commit {container name/id} {new_name} 346 347 Then you can `docker run` that new image.