github.com/charypar/monobuild@v0.0.0-20211122220434-fd884ed50212/README.md (about) 1 # Monobuild 2 3 A build orchestration tool for Continuous Integration in a monorepo. 4 5 NOTE: this is Readme driven development. Not everything described in this readme 6 is fully implemented. 7 8 ## About 9 10 Monobuild is a simple tool that understands a graph of dependencies in 11 a monorepo codebase (where separate components live side by side in folders) 12 and based on it, it can decide what should be built, given a set of changes. 13 14 For help, run 15 16 ```sh 17 $ monobuild help 18 ``` 19 20 ### Versioning 21 22 Monobuild uses [Immutable Versioning](https://imver.github.io) as its versioning 23 scheme. 24 25 ## Usage 26 27 Monobuild constructs the dependency graph from dependency manifests. By 28 default, manifests are files named `Dependencies`, which contain a simple 29 line-by-line list of dependencies for the component in the directory of the 30 file. 31 32 ### Declare dependencies 33 34 An example manifest in `app1/Dependencies` might look like this 35 36 ``` 37 # Content 38 !data/content 39 !shared/images 40 41 # Libs 42 common-lib 43 libs/date-time 44 ``` 45 46 Monobuild will ignore any empty lines and lines starting with `#`. Every other 47 line is considered a dependency and is a path relative to current working 48 directory (typically repository root). Monobuild will expect a dependency 49 manifest (possibly empty) to be present at that path. 50 51 Lines starting with `!` are strong dependencies all other dependencies are 52 considered weak. The difference is in the way the dependency graph is translated 53 to a build schedule. 54 55 One of the benefits of a monorepo, is components and services can be built from 56 code, including their dependencies. Changing a weak dependency of a component 57 means a change to the component, which therefore needs to be rebuilt, but the 58 builds can be run in parallel. Output or result of the dependency does not 59 affect the build of this component. 60 61 A strong dependency has to successfully build first, in order for the build of 62 the component to be possible. If the dependency build fails, the component 63 build does not even start. 64 65 Typically, services are built from source, including their libraries, so the 66 dependencies on libraries are weak (we still want to run the library build to 67 run tests and get a result though). Deploying orchestrations of services 68 typically has a strong dependency on the service builds (as they produce 69 artifacts, e.g. docker images, needed by the deployment). 70 71 ### Visualise dependency graph and build schedule 72 73 To better understand the dependency graphs and build schedules, Monobuild can 74 print them. 75 76 ```sh 77 $ monobuild print 78 ``` 79 80 will print the build schedule, which will ignore weak dependencies 81 82 ```sh 83 $ cd test/fixtures/manifests-test 84 $ monobuild print 85 app4: 86 libs/lib1: 87 libs/lib2: 88 libs/lib3: 89 stack1: app1, app2, app3 90 app1: 91 app2: 92 app3: 93 ``` 94 95 You can also print the dependency structure with one component per line. For example 96 97 ```sh 98 $ cd test/fixtures/manifests-test 99 $ monobuild print --dependencies 100 app4: 101 libs/lib1: libs/lib3 102 libs/lib2: libs/lib3 103 libs/lib3: 104 stack1: app1, app2, app3 105 app1: libs/lib1, libs/lib2 106 app2: libs/lib2, libs/lib3 107 app3: libs/lib3 108 ``` 109 110 Monobuild can also print the entire dependency structure including whether the 111 dependencies are weak or strong. This is useful when 112 [working without a local repo](#working-without-a-local-repository) 113 114 ``` 115 $ monobuild print --full 116 app4: 117 libs/lib1: libs/lib3 118 libs/lib2: libs/lib3 119 libs/lib3: 120 stack1: !app1, !app2, !app3 121 app1: libs/lib1, libs/lib2 122 app2: libs/lib2, libs/lib3 123 app3: libs/lib3 124 ``` 125 126 #### Graphical output 127 128 Print also supports graphical output using GraphViz 129 130 ``` 131 $ cd test/fixtures/manifests-test 132 $ monobuild print --dot 133 ``` 134 135 to produce a PDF, you can pipe the output into the `dot` tool: 136 137 ``` 138 $ cd test/fixtures/manifests-test 139 $ monobuild print --dependencies --dot | dot -Tpdf -o dependencies.pdf 140 digraph dependencies { 141 "app1" -> "libs/lib1" 142 "app1" -> "libs/lib2" 143 "app2" -> "libs/lib2" 144 "app2" -> "libs/lib3" 145 "app3" -> "libs/lib3" 146 "libs/lib1" -> "libs/lib3" 147 "libs/lib2" -> "libs/lib3" 148 "stack1" -> "app1" 149 "stack1" -> "app2" 150 "stack1" -> "app3" 151 } 152 ``` 153 154 ### Change detection 155 156 If the current directory is a git repository, monobuild can decide which 157 components changed (using git). 158 159 ```sh 160 $ monobuild diff 161 app2 162 app2 163 lib3 164 ``` 165 166 Monobuild assumes use of [Mainline Development](https://gitversion.readthedocs.io/en/latest/reference/mainline-development/) 167 and changes are detected in two modes: 168 169 1. for a feature branch, the change detection is equivalent to 170 171 ```sh 172 $ git diff --no-commit-id --name-only -r $(git merge-base master HEAD) 173 ``` 174 175 in other words, list all the changes that happened since the current branch 176 was cut from `master`. 177 178 This is the default mode and the base branch is `master` by default. 179 You can override this with 180 181 ```sh 182 $ monobuild diff --base-branch develop 183 ``` 184 185 2. for a `master` branch (or other main branch) the change detection is equivalent 186 to 187 188 ```sh 189 $ git diff --no-commit-id --name-only -r HEAD^1 190 ``` 191 192 To work in the main-branch mode, use the `--main-branch` flag 193 194 ```sh 195 $ monobuild diff --main-branch 196 ``` 197 198 The main difference between the above `git diff`s and `monobuild diff` is the 199 dependency graph awareness. 200 201 Monobuild will start with the list from `git diff`, filter it down to known 202 components, and then extend it with all components that depend on any of the 203 components in the initial list, including transitive dependencies. 204 205 For the resulting "to do" list, `diff` will then build a build schedule using the 206 strong dependencies. 207 208 You can print the relevant part of the dependency graph (rather than 209 the build schedule) with `--dependencies` 210 211 ``` 212 $ monobuild diff --dependencies 213 ``` 214 215 Both modes also support DOT output with `--dot`. You can also print 216 the entire graph with the affected components with `--dot-highlight`. 217 218 #### Rebuilding strong dependencies 219 220 The assumption behind strong dependencies is that their outcome is required 221 for the dependent builds to proceed. In most cases, this means that if no 222 changes affected a component, the build does not need to run, because the outcome 223 (e.g. a build artifact) already exists from a previous run of the build (when 224 that component was affected). 225 226 In certain situations, it could be useful to run the build again, to ensure its 227 output is present. This will result in wasted work, but ensures builds won't 228 fail because, for example, an artifact cache was lost. The wasted work can 229 also largely be prevented by making builds idempotent. 230 231 Monobuild supports this with an `--rebuild-strong` option on `diff`, which will 232 include strong dependencies of all components affected by the change. 233 234 ### Override the manifest matching 235 236 If you want to use a different filename for the manifest files, you can do so 237 using the global `--manifests` flag. 238 239 ### Filters 240 241 #### Scope 242 243 You can scope the results of both `diff` and `print` to a given component 244 and its dependencies using the `--scope` flag 245 246 #### Top-level components 247 248 Sometimes it's useful to know the "entrypoints" into your dependency graph - 249 the components that nothing depends on (typically services or applications). 250 You can list only those with a `--top-level` flag on both `diff` and `print`. 251 252 ### Creating a Makefile 253 254 **not implemented** 255 256 Monobuild can also generate a `Makefile`, that can be used by individual 257 component builds to build their dependencies. 258 259 You can generate the makefile with 260 261 ```sh 262 $ monobuild makefile 263 ``` 264 265 The resulting Makefile consists of targets like this: 266 267 ```make 268 directory/component1: [dependency1] [dependency2] [dependency3] 269 @cd directory/component1 && make build 270 ``` 271 272 This assumes each component has a minimal `Makefile` which looks like this: 273 274 ```make 275 # directory/component1/Makefile 276 277 default: 278 @cd ../.. && make directory/component1 279 280 build: 281 # steps to make the component available as a dependency of others 282 # this could be empty 283 ``` 284 285 You can also override the build command (`make build` by default) with the 286 `--build-command` flag. 287 288 ## Working without a local repository 289 290 Monobuild can work without a local clone and checkout of your repository. All of 291 the use can be supported with GitHub API calls. 292 293 This is an **optimisation** and may not be necessary on relatively small monorepos or 294 if your CI caches the repo and only pulls changes since last build. 295 296 ### Load dependency graph from a file 297 298 It is possible to keep the dependencies in a central dependency graph file, which 299 is useful for big repos, when you want to work without a full local checkout (e.g. on CI). 300 301 Monobuild supports this by allowing a map file to be specified with the `-f` flag. 302 For example: 303 304 ```sh 305 $ curl -O https://raw.githubusercontent.com/myorg/myrepo/blob/master/dependencies.monobuild 306 $ mononobuild print -f ./dependencies.monobuild --dependencies 307 ``` 308 309 The file can be created with the `print` command itself, which will collect the 310 manifests and produce the full map. 311 312 ```sh 313 $ monobuild print --full 314 app4: 315 libs/lib1: libs/lib3 316 libs/lib2: libs/lib3 317 libs/lib3: 318 stack1: !app1, !app2, !app3 319 app1: libs/lib1, libs/lib2 320 app2: libs/lib2, libs/lib3 321 app3: libs/lib3 322 ``` 323 324 You can update the dependency map as follows (with a local checkout): 325 326 ```sh 327 $ monobuild print --full > dependencies.monobuild 328 ``` 329 330 ### Read changed files from standard input 331 332 Similarly, the changed files for `diff` can be supplied externally, from 333 standard input, e.g. 334 335 ``` 336 $ git diff --no-commit-id --name-only -r $(git merge-base master HEAD) | monobuild diff - 337 ``` 338 339 **note the hyphen at the end**, indicating the diff should be taken from stdin, 340 instead of running the git command. 341 342 ### A full CI example 343 344 A full example may look as follows: 345 346 ``` 347 # Fetch the dependency map 348 $ curl -O https://raw.githubusercontent.com/myorg/myrepo/blob/master/dependencies.monobuild 349 350 # Get a list of changed files remotely from Github API 351 # (note the followig only works up to 300 files) 352 $ curl -s -o changed-files https://api.github.com/repos/charypar/monobuild/pulls/21/files | jq -r .[].filename 353 354 # Get a build schedule 355 $ cat changed-files | monobuild diff -f dependencies.monobuild - 356 ``` 357 358 This is complicated enough that I might wrap it in a small service which can 359 respond to a GitHub webhook, get the right information off of GitHub and then 360 trigger the right builds from a set of webhooks.