github.com/zyedidia/knit@v1.1.2-0.20230901152954-f7d4e39a0e24/README.md (about) 1 # Knit 🧶 2 3  4 [](https://pkg.go.dev/github.com/zyedidia/knit) 5 [](https://goreportcard.com/report/github.com/zyedidia/knit) 6 [](https://github.com/zyedidia/knit/blob/master/LICENSE) 7 8 Knit is a build tool inspired by Make and Plan9 Mk. You define rules with a 9 Make-like embedded syntax within a Lua program. Rules can be passed around as 10 Lua objects, and generated by Lua code. You can use the Lua module system to 11 make reusable modules for building any kind of source code. Knit combines the 12 readability of a Make-style rules language will the power and expressiveness of 13 Lua. If you are familiar with Make, you can learn Knit very quickly. 14 15 Knit tracks more of your build to give you better incremental builds. For 16 example, Knit automatically adds an implicit dependency on a rule's recipe, so 17 if you change a recipe (either directly or through a variable change), Knit 18 will automatically re-run all rules that were affected. Knit will also skip 19 build steps dynamically if it can determine that they will be unchanged from 20 the previous run (even if earlier dependencies were changed). 21 22 Knit has support for namespaced sub-builds that execute relative to their 23 directory, but Knit avoids build fragmentation because sub-builds don't rely on 24 spawning build sub-processes. No more `make -C` to do sub-builds! Everything is 25 tracked by the root Knitfile, but you can still make directory-specific rules. 26 27 Knit's rules language is heavily inspired by [Plan9 28 Mk](https://9p.io/sys/doc/mk.html). In some ways, Knit can be considered a 29 modern version of Mk with a Lua meta-programming system built on top of it 30 (there are some differences compared to Mk). 31 32 Why make yet another build system? Because it's fun and useful to me! Maybe it 33 will be useful to you too. Everyone hates something about their build system so 34 if you have feedback or a request, let me know! The project is new enough that 35 your feedback may be seriously taken into account. 36 37 **Articles**: I have written two articles with more details about Knit 38 [here](https://zyedidia.github.io/blog/posts/3-knit-better-make/) and 39 [here](https://zyedidia.github.io/blog/posts/4-incremental-d-knit/) (this 40 article is specifically about D, but the optimization Knit applies is general 41 and not tied to D specifically). 42 43 # Features 44 45 * Knit uses Lua for customization. This makes it possible to write reusable 46 build libraries, and in general makes it easier to write powerful and 47 expressive builds. 48 * Knit has built-in syntax for a rules language inspired by Make and Plan9 Mk. 49 This makes it very familiar to anyone who has used Make/Mk. 50 * Knit has direct support for sub-builds (compared to Make, which usually 51 involves spawning a separate make sub-process to perform a sub-build). 52 * Knit can hash files to determine if they are out-of-date, rather than just 53 relying on file modification times. 54 * Knit additionally uses hashes for "dynamic task elision": if Knit can 55 dynamically determine that a prerequisite that was rebuilt actually 56 changed nothing, it won't re-run the dependent build step, allowing for 57 even better incremental builds compared to timestamp-based approaches 58 (Make, Ninja, etc.). 59 * Knit tracks recipe changes, so if you update a variable (in the Knitfile or 60 at the command-line), any dependent rules will be automatically rebuilt. 61 * Knit supports `%` meta-rules and regular expression meta-rules. 62 * Knit uses rule attributes instead of using targets such as `.SECONDARY` to 63 indicate special processing. 64 * Knit supports virtual rules that are independent of the file system. 65 * Knit uses sane variable names like `$input`, `$output`, and `$match` instead 66 of Make's `$^`, `$@`, and `$*`. 67 * Knit supports rules with multiple outputs, and treats them like Make's group 68 targets by default. 69 * Knit supports sub-tools that implement various build utilities including: 70 * Generating a graph visualization using graphviz (dot). 71 * Showing build status information (whether targets are out-of-date and 72 why). 73 * Exporting a compile commands database for use with a language server. 74 * Automatically cleaning all build outputs. 75 * Converting your build into a shell script, Makefile, or Ninja file. 76 * Knit will search up the directory hierarchy for a Knitfile, allowing you 77 to run your build from anywhere in your project. 78 * Knit supports parallel builds and uses all cores by default. 79 * Cross-platform support (Windows support is still experimental). 80 * Knit uses a shell to execute commands. By default, Knit searches for `sh` 81 on your system and uses that. If it cannot find `sh`, it uses an internal 82 (cross-platform) shell. 83 84 # Example Knitfile 85 86 Here is a very basic Knitfile for building a simple C project. 87 88 ```lua 89 return b{ 90 $ hello: hello.o 91 cc -O2 $input -o $output 92 $ %.o: %.c 93 cc -O2 -c $input -o $output 94 } 95 ``` 96 97 The syntax for rules is nearly the same as Make, and Knit supports `%` 98 meta-rules just like Make. However, rather than using a custom language 99 to configure the build, Knit uses Lua. 100 101 Here is a more complex example Knitfile used for building a simple C project. 102 This time the Knitfile supports various configurations (changing `cc` and 103 enabling debug flags), and automatically detects the source files. 104 105 ```lua 106 local knit = require("knit") 107 108 local conf = { 109 cc = cli.cc or "gcc", 110 debug = tobool(cli.debug) or false, 111 } 112 113 local cflags := -Wall 114 115 if conf.debug then 116 cflags := $cflags -Og -g 117 else 118 cflags := $cflags -O2 119 end 120 121 local src = knit.glob("*.c") 122 local obj = knit.extrepl(src, ".c", ".o") 123 local prog := hello 124 125 return b{ 126 $ build:VB: $prog 127 128 $ $prog: $obj 129 $(conf.cc) $cflags $input -o $output 130 $ %.o:D[%.d]: %.c 131 $(conf.cc) $cflags -MMD -c $input -o $output 132 } 133 ``` 134 135 Running `knit hello` would build all the necessary `.o` files and then link 136 them together. Running `knit hello debug=1` would change the flags and re-run 137 the affected rules. Running `knit build` will build `hello` (effectively an 138 alias for `knit hello`). The `VB` attributes on the build rule means that it 139 is virtual (not referring to a file on the system), and should always be built 140 (out-of-date). 141 142 Running `knit -t clean` will run a sub-tool that automatically removes all 143 generated files. 144 145 Header dependencies are automatically handled by using the `-MMD` compiler flag 146 with the `D[%.d]` attribute. To explicitly name the dependency file (e.g., to 147 put it in a `.dep` folder), you could instead use: 148 149 ``` 150 $ %.o:D[.dep/%.dep]: %.c 151 $(conf.cc) $cflags -MMD -MF $dep -c $input -o $output 152 ``` 153 154 Note that Knitfiles are Lua programs with some modified syntax: special syntax 155 using `$` for defining rules, and special syntax using `:=` for defining raw 156 strings (no quotes) with interpolation. 157 158 See the [docs](./docs/knit.md) for more information. 159 160 See [examples](./examples) for a few examples, and see this repository's 161 Knitfile and the tests for even more examples. 162 163 # Installation 164 165 Prebuilt binaries are available from the [release page](https://github.com/zyedidia/knit/releases). 166 167 You can install one automatically using [eget](https://github.com/zyedidia/eget). 168 169 ``` 170 eget zyedidia/knit 171 ``` 172 173 Or you can build from source (requires Go 1.19): 174 175 ``` 176 go install github.com/zyedidia/knit/cmd/knit@latest 177 ``` 178 179 # Experimental or future possible features 180 181 * Ninja to Knit converter (for compatibility with cmake, and for benchmarking). 182 See [knitja](https://github.com/zyedidia/knitja) for the converter tool. 183 * Performance optimizations. 184 * Knit can already be used to build large projects, such as CVC5 (using the 185 knitja converter). For massive builds though, like LLVM, Knit suffers 186 from some performance problems that could be improved. 187 * Better support for dynamic dependencies. Currently it is possible to handle 188 dynamic dependencies by generating rules, but I would like to explore the 189 possibility of a more clean and cohesive solution. 190 * Ptrace enabled automatic dependency discovery (Linux-only feature). 191 See the [xkvt](https://github.com/zyedidia/xkvt) project for some 192 experiments on this front. 193 * Global build file cache (similar to `ccache`, but for every command that is 194 executed). 195 * A restrictive mode for build sandboxing. 196 197 # Feedback 198 199 It is always useful useful to get feedback from others to improve Knit. If you 200 have feedback, or questions about how to use it, please open a discussion. It 201 would be great to discuss the good and bad parts of the current design, and how 202 it can be improved. 203 204 # Usage 205 206 ``` 207 Usage of knit: 208 knit [TARGETS] [ARGS] 209 210 Options: 211 -B, --always-build unconditionally build all targets 212 --cache string directory for caching internal build information (default ".") 213 --cpuprofile string write cpu profile to 'file' 214 -D, --debug print debug information 215 -C, --directory string run command from directory 216 -n, --dry-run print commands without actually executing 217 -f, --file string knitfile to use (default "knitfile") 218 --hash hash files to determine if they are out-of-date (default true) 219 -h, --help show this help message 220 --keep-going keep going even if recipes fail 221 -q, --quiet don't print commands 222 --shell string shell to use when executing commands (default "sh") 223 -s, --style string printer style to use (basic, steps, progress) (default "basic") 224 -j, --threads int number of cores to use (default 8) 225 -t, --tool string subtool to invoke (use '-t list' to list subtools); further flags are passed to the subtool 226 -u, --updated strings treat files as updated 227 -v, --version show version information 228 ``` 229 230 Available sub-tools (`knit -t list`): 231 232 ``` 233 list - list all available tools 234 graph - print build graph in specified format: text, tree, dot, pdf 235 clean - remove all files produced by the build 236 targets - list all targets (pass 'virtual' for just virtual targets) 237 compdb - output a compile commands database 238 commands - output the build commands (formats: knit, json, make, ninja, shell) 239 status - output dependency status information 240 ``` 241 242 # Contributing 243 244 If you find a bug or have a feature request please open an issue for 245 discussion. I am sometimes prone to being unresponsive to pull requests, so I 246 apologize in advance. Please ping me if I forget to respond. If you have a 247 feature you would like to implement, please double check with me about the 248 feature before investing lots of time into implementing it. 249 250 If you have a question or feedback about the current design, please open a 251 discussion.