github.com/zyedidia/knit@v1.1.2-0.20230901152954-f7d4e39a0e24/docs/knit.md

## Rules

A rule consists of four parts:

* Targets: a list of values that are generated by this rule.
* Prereqs: a list of values that must be generated before this rule can be
  executed.
* Attributes: a list of flags that customize how this rule is interpreted.
* Recipe: a list of commands that are run to execute the rule.

The syntax for a rule is

```
targets:attributes: prereqs
    recipe
```

If the rule has no attributes, it may be omitted, leaving just `targets:
prereqs` on the first line.

When a rule is embedded in Lua, it is prefixed with a `$`:

```
$ targets:attributes: prereqs
    recipe
```

The rule continues until it is de-indented to the original indentation of the
`$`.

Direct rules are rules where the targets and prereqs are explicit names of
values. For example,

```
foo.o: foo.c
    gcc -c foo.c -o foo.o
```

Specifies that the file `foo.o` is built from `foo.c`, using the command `gcc
-c foo.c -o foo.o`.

Within a rule, `#` denotes the start of a comment. Outside of rules, `--` is
the start of a comment (the standard Lua syntax).

### Meta rules

A meta rule is a special rule that describes a generic way to create direct
rules.  The meta rule describes how to match targets and prereqs into a direct
rule.  For example, a `%` meta rule uses `%` to match any sequence of
characters.

Thus the meta rule

```
%.o: %.c
    ...
```

describes that a direct rule

```
foo.o: foo.c
    ...
```

could be created. This direct rule would be created if another rule needed
`foo.o` to be generated.

Meta rules may also specify the match by using regular expressions, if the `R`
attribute is provided. For example:

```
foo-(.*)-(.*).tar.gz:R: $1/$2/foo
    ...
```

could create a direct rule

```
foo-linux-amd64.tar.gz: linux/amd64/foo
    ...
```

### Attributes

Knit supports the following attributes:

* `Q` (quiet): do not print this rule's recipe when executing.
* `R` (regex): this meta rule uses regular expression matching.
* `V` (virtual): the value this rule generates is not a file, just a name.
* `M` (no meta): this rule's targets cannot be matched by meta rules.
* `E` (non-stop): this rule does not stop if one of its commands fails.
* `B` (build): this rule must always be built (it is always out-of-date).
* `L` (linked): this rule always runs if an out-of-date sub-rule requires it as
  a prereq.
* `O` (order-only): this rule's prereqs are not considered automatically
  up-to-date even if this rule is up-to-date.
* `D[depfile]` (dependency): include `depfile` as an additional list of
  dependencies for this rule.

The `D` attribute takes an argument. It is used for including `.d` files for
C headers. For example, this rule

```
$ %.o:D[%.d]: %.c
    gcc -MMD -c $input -o $output
```

specifies that it should read the file `%.d` as a rule file and include any
additional rules (without recipes) as dependencies for the current rule. If the
file does not exist it is ignored, and any rules from the file that can't be
satisfied are ignored instead of returned as errors.

Attributes can also be applied to particular prerequisites rather than to an
entire rule, using the syntax `prereq[attributes]`. For example:

```
foo:V:
    echo foo
bar:V: foo[Q]
    echo bar
```

The `foo` rule will be quiet only when used as a prerequisite to the `bar`
rule.

Some attributes can only be applied in this way:

* `I` (implicit): this prereq does not appear in `$input`.

The `[...][attributes]` syntax can be used to apply attributes to groups of
prerequisites. For example, in the following rule all three prerequisites are
implicit.

```
foo: [a b c][I]
    ...
```

### Recipes

A recipe is a list of commands to execute, each separated by a newline. They
are executed within the `sh` shell. Each command is executed in a separate
process, spawned with `sh -c <cmd>`. One implication of this is that a `cd`
command will not persist to future commands. If you wish to run multiple
commands in the same shell, you should use the shell's features (`&&`, `||`, or
`;`) for this. For example, `cd foo; cat bar.txt`. Note that you can use `\` to
escape newlines, so that one command can span multiple lines in the recipe.

Recipes may use variables that will be expanded before the recipe executes.
Variables are written with `$var`, or full Lua expressions can be written with
`$(expr)`. Variables/expressions are expanded eagerly when the rule is created.
If expansion causes an error, the expansion is delayed until rule evaluation
(when special build variables are available). You can use `$$` to escape a
dollar sign. For example, the recipe `echo $$PWD` will echo the environment
variable `PWD`, while `echo $PWD` would attempt to replace `$PWD` with the Lua
variable `PWD` when the rule is elaborated.

Some special variables are available during recipe expansion:

* `input`: a string of rule's prereqs.
* `inputs`: an array of this rule's prereqs.
* `output`: a string of this rule's targets.
* `outputs`: an array of this rule's targets.
* `match`: the value captured with `%` in a meta rule.
* `matches`: a list of matches captured by a regular expression meta rule.
* `dep`: the name of the dependency file (if it exists) for the rule, defined
  by the `D[...]` attribute.

When a rule is encountered, `$` expressions are immediately expanded. If
expansion fails, expansion is re-tried when the rule is evaluated (once the
inputs/outputs are known).

Lua expressions should not mix uses of special build variables and Lua local
variables. Local variables are only available during immediate expansion, and
special build variables are only available during lazy expansion. This
constraint may be relaxed in the future if it turns out to be a useful feature.

### Out-of-date calculation

To determine if a rule must be re-run, Knit computes whether its output is
up-to-date or not. There are two mechanisms for this: a hash-based one and a
timestamp-based one. By default, Knit uses the hash-based mechanism: if a
file's hash has changed since the previous build, it is considered out-of-date,
and all rules that depend on it must be re-run. When depending on a directory,
its hash is the recursive hash of all files within it. The hash-based method is
good for small builds, but can become too slow for large builds. In those cases
you can disable hashing and use the timestamp-based calculation.

When hashing is disabled, Knit uses a similar computation to Make that is based
on file modification times. If a rule's prerequisites refer to files that have
been modified more recently (according to the system's file modification
timestamp) than the output file, then the rule is determined to be out-of-date
and must be re-run. Other factors may also cause a rule to be re-run, such as
if its recipe has changed, or if it has a rebuild attribute. If the file refers
to a directory, the system's timestamp is not used. Instead, the modification
time of a directory is the timestamp of the most recently modified file in it
(or in any sub-directory). Depending on a very large directory may hinder
performance.

Hashing can be disabled on a per-project basis or globally by using the
`.knit.toml` configuration file, described the "Configuration" section of this
documentation.

## Knitfiles

A Knitfile is a Lua 5.1 program with additional support for rule expressions.
The Knitfile ultimately must return a "buildset" -- a list of build rules that
are used to construct the build graph.

A rule is defined in Lua with the `$` syntax:

```
$ targets:attributes: prereqs
    ...
```

A rule expression may be assigned to a variable

```
local rule = $ foo.o: foo.c
    gcc -c $input -o $output
```

More often, rule expressions are gathered together in a Lua table:

```
local rules = {
    $ foo.o: foo.c
        gcc -c $input -o $output
    $ foo: foo.o
        gcc $input -o $output
}
```

When you run `knit`, Knit will automatically look for a file called `Knitfile`
or `knitfile` (or you can specify a custom name with `-f`). Knit will look in
the current directory, and up to ancestor directories if one does not exist in
the current directory. This allows you to execute the build from anywhere in
your project without fragmenting the build system with multiple Knitfiles. You
can still make directory-specific targets that are namespaced relative to that
directory with sub-builds.

If you run `knit foo.o` from the `foo` directory, and there is a file
`../Knitfile` that defines a rule for building `foo/foo.o`, Knit will
automatically figure out that you mean to build `foo/foo.o` (relative to `..`),
since you specified `foo.o` (relative to `foo`). In other words, building
sub-files just works.

Likewise, if you run `knit all` from the `foo` directory, and the `all` rule
is only defined for the root directory, Knit will automatically use that rule
instead of trying to use `foo/all`.

### Deviations from Lua 5.1

There are some differences between Knit Lua and Lua 5.1:

* Knit supports `$` for creating rules.
* Knit supports `:=` for creating raw strings.
* Knit will give an error message when attempting to access an undeclared
  variable, whereas Lua 5.1 will just return nil for the value.

## Rulesets

A table of rules can be converted into a "ruleset" by using the special `r`
function, which converts the table into a Lua "userdata" object representing a
list of build rules.


```
local ruleset = r{
    $ foo.o: foo.c
        gcc -c $input -o $output
    $ foo: foo.o
        gcc $input -o $output
}
```

Note that two rulesets may be combined with the `+` operator.

```
ruleset = ruleset + r{
    $ build:V: foo
}
```

## Buildsets

A buildset is a set of rules associated with a particular directory. A buildset
may also contain other buildsets (rules from other directories). All rules in
the buildset are executed relative to its directory. A buildset can be
constructed by using the special `b` function, which constructs a buildset from
a table of rules, rulesets, or other buildsets.

```
local buildset = b{
    $ foo.o: foo.c
        gcc -c $input -o $output
    $ foo: foo.o
        gcc $input -o $output
}
```

By default the buildset's directory is the current working directory when it is
constructed. A second argument may also be passed to `b` to directly specify the
build directory.

```
local buildset = b({
    $ foo.o: foo.c
        gcc -c $input -o $output
    $ foo: foo.o
        gcc $input -o $output
}, "directory")
```

A buildset must be returned by the Knitfile for a build to take place. When a
buildset is returned, knit expands it and all the buildsets that it returns
into the full set of rules, where each rule is relative to the buildset
directory that it came from. Rules may have cross-buildset dependencies.

These facilities for making rules relative to directories are for enabling
sub-builds, discussed in the next section.

### Sub-builds

A build may use several buildsets.

For example:

```lua
-- this buildset is relative to the "libfoo" directory
local foorules = b({
    $ foo.o: foo.c
        gcc -c $input -o $output
}, "libfoo")

return b{
    $ prog.o: prog.c
        gcc -c $input -o $output
    -- libfoo/foo.o is automatically resolved to correspond to the rule in foorules
    $ prog: prog.o libfoo/foo.o
        gcc $input -o $output

    -- include the foorules buildset
    foorules
}
```

This Knitfile assumes the build consists of `prog.c` and `libfoo/foo.c`. It
builds `libfoo/foo.o` using a sub-build and automatically determines that the
`foorules` buildset contains the rule for building `libfoo/foo.o`. Note that
the recipe for `foo.o` is run in the `libfoo` directory. Including a buildset
inside another will automatically including all of its rules namespaced into
the directory that the buildset came from.

It is also useful to combine sub-builds with the `include(x)` function, which
runs the knit program `x` from the directory where it exists, and returns the
value that `x` produces. This means you can easily use a sub-directory's
Knitfile to create a buildset for use in a sub-build.

For example, for the previous build we could use the following file system
structure:

`libfoo/build.knit` contains:

```lua
-- this buildset's directory will be the current working directory
return b{
    $ foo.o: foo.c
        gcc -c $input -o $output
}
```

`Knitfile` contains:

```lua
return b{
    $ prog.o: prog.c
        gcc -c $input -o $output
    -- libfoo/foo.o is automatically resolved to correspond to the rule in foorules
    $ prog: prog.o libfoo/foo.o
        gcc $input -o $output

    -- include the libfoo rules: this will change directory into libfoo, execute
    -- build.knit, and change back to the current directory, thus giving us a buildset
    -- for the libfoo directory automatically
    include("libfoo/build.knit")
}
```

Note that since knit looks upwards for the nearest Knitfile, you can run `knit
foo.o` from inside `libfoo`, and knit will correctly build `libfoo/foo.o`.

Since managing the current working directory is important for easily creating
buildsets that automatically reference the correct directory, there are several
functions for this:

* `include(x)`: runs a Lua file from the directory where it exists.
* `dcall(fn, args)`: calls a Lua function from the directory where it is defined.
* `dcallfrom(dir, fn, args)`: calls a Lua function from a specified directory.
* `rel(files)`: makes all input files relative to the build's root directory.

## Configuration

Knit will search the current directory for a Knitfile called `knitfile` or
`Knitfile`. If one is not found, it will use the Knitfile in
`~/.config/knit/Knitfile.def`, or if that does not exist it will throw an
error.

Several options are available as command-line flags. They may also be specified
in a `.knit.toml` file. Knit will search upwards from the current directory
for `.knit.toml` files, and use the options set in those files. It will also
search `~/.config/knit/.knit.toml`.

The default set of flags is:

```toml
knitfile = "knitfile"
ncpu = 8 # depends on the number of logical cores on your machine
dryrun = false
directory = ""
always = false
quiet = false
style = "basic"
cache = ""
hash = true
updated = []
root = false
keepgoing = false
shell = "sh"
```

## Sub-tools

Running `knit [TARGET]` will create a build graph for the target. By default,
knit will then execute that build graph. Using the `-t TOOL` option, you may
specify a sub-tool to run instead of building:

* `list` - list all available tools
* `graph` - print build graph in specified format: text, tree, dot, pdf
* `clean` - remove all files produced by the build
* `targets` - list all targets (pass 'virtual' for just virtual targets)
* `compdb` - output a compile commands database
* `commands` - output the build commands (formats: knit, json, make, ninja, shell)
* `status` - lists dependencies and whether they are up-to-date
* `path` - shows the path of the current knitfile

The special target `:all` depends on every target in the build. Thus `knit :all
-t targets` will list all targets.

Some examples are shown below.

### Automatic cleaning

```
knit target -t clean
```

### Output a shell script for the build

```
knit target -t commands shell
```

### Output a Ninja build file

```
knit target -t commands ninja
```

### Output a compile commands database

```
knit target -t compdb
```

### Output a PDF build graph

```
knit target -t graph pdf > graph.pdf
```

## Special rules

Knit automatically defines two special rules: `:all` and `:build`.

The `:all` rule depends on all possible targets in the build (except those
attainable only from meta-rules). For example `knit :all -t targets` will list
all possible targets, and `knit :all -t clean` will clean all possible outputs.

The `:build` rule is the root rule of the build and depends on all requested
targets. For example `knit a b c` will generate a `:build` rule that depends on
`a`, `b`, and `c`. In general, you should never refer to the `:build` rule
since doing so will usually create a build cycle.

## Default rule

If you run `knit` without a target, Knit will build the first non-meta rule.

## Rule priority

If several rules with recipes that could be used to build a file, Knit uses the
last one. In other words, defining a rule later will override previous
definitions of a rule. However, if a later rule does not have a recipe, it will
not override the rule for the target, but instead just add the new
prerequisites for that target to the previous rule.

If several rules from different buildsets could be used to build a target, the
rule from the buildset for the target's directory is attempted first. If it
does not exist, then rules are attempted from all other buildsets and the first
buildset to have a matching rule is used. If a meta-rule is used, it is
attmpted in the current buildset before looking in other buildsets.

## Built-in Lua syntax

* `$ ...`: creates a rule. The rule is formatted using string interpolation.
  The rule continues until indentation returns to the level of the `$`.

* `x := ...`: creates a string without quotes. The string value continues until
  the end of the line, and is automatically formatted using string
  interpolation.

Both of these expressions are implicitly terminated with a `;`, allowing them
to be used in tables.

## Built-in Lua functions

Note: several functions throw errors. Use the built-in Lua `pcall` function to
perform error handling. `local ok, result = pcall(fn, args)` returns whether
there was an error during the execution of `fn(args)`. The `result` variable
will contain the result, or the error value.

* `rule(rule)`: define a rule. The `$` syntax is shorthand for this function.

* `rulefile(file)`: define a rule by reading it from `file`. Throws an error if
  the file does not exist.

* `include(file)`: run a Knitfile from its directory (changes the
  current working directory while the file is being executed) and return the
  generated ruleset. Throws an error if `file` does not exist.

* `dcall(fn, args)`: call `fn(args)` from the directory where `fn` is defined.

* `dcallfrom(dir, fn, args)`: call `fn(args)` from `dir`.

* `rel(files)`: make all paths in the table `files` relative to the build root
  (the location of the Knitfile).

* `r{$ ...}`, `r({...})`: turn a table of rules into a ruleset.

* `b{...}`, b({...}, dir): turn a table of rules, rulesets, or buildsets
  into a buildset associated with directory `dir`. `dir` is optional, and if
  not specified will be the current working directory.

* `tobool(value)` bool: convert an arbitrary value to a boolean. A nil value
  will return nil, the strings `false`, `off`, or `0` will become false. A
  boolean will not be converted. Anything else will be true.

* `eval(code)`: evaluates a Lua expression in the global scope
  and returns the result. Throws an error if the code has an error.

* `f"..."`, `f(s)`: formats a string using `$var` or `$(expr)` to
  expand variables/expressions. Throws an error if the variable does not exist,
  or the expression has an error.

* `expand(s)`: formats a string in the same way as `f`, but if there is an
  error, it does not expand that particular `$...` expression.

* `use(pkg)`: imports all fields of `pkg` into the global namespace. Meant to
  be used with `require`: `use(require("knit"))`.

* `sel(cond, a, b)`: if `cond` is true return `a`, otherwise return `b`.

* `choose(a, b, c...)`: return the first value in the list of arguments that is
  not nil.

* `r{} + r{}`: you may use the `+` operator to combine rulesets together.

* `b{} + val`: you may use the `+` operator to combine buildsets with
  rules/rulesets/buildsets.

* `{s} + {s}`: string tables returned by knit functions can be added together.

## The `knit` Lua package

The `knit` package can be imported with `require("knit")`, and provides the following functions:

* `repl(in, patstr, repl)`: replace all occurrences of the Go regular
  expression `patstr` with `repl` within the array `in`. Throws an error if
  there is an error with `patstr`.

* `extrepl(in, ext, repl)`: replace all occurrences of the literal string `ext`
  as a suffix with `repl` within the array `in`.

* `glob(pat)`: return all files in the current working directory that match the
  glob `pat`.

* `suffix(in, suffix)`: add the string `suffix` to the end of every string in
  the array `in`, and return the new array.

* `prefix(in, prefix)`: add the string `prefix` to the beginning of every
  string in the array `in`, and return the new array.

* `filterout(in, exclude)`: returns a new table containing all the elements of
  `in`, except those in `exclude`.

* `shell(cmd) string`: execute a command with the shell and return its
  output. Throws an error if the command exits with an error.

* `trim(s)`: trim leading and trailing whitespace from a string.

* `abs(path)`: return the absolute path of a path.

* `dir(path)`: return the directory part of a path.

* `base(path)`: return the basename of a path.

* `os`: a string containing the operating system name.

* `arch`: a string containing the machine architecture name.

* `flags`: a struct containing the values of the flags when Knit was invoked.
  See https://pkg.go.dev/github.com/zyedidia/knit#Flags.

* `addpath(p)`: adds the path `p` to the global require path. Files with ending
  with `.lua` or `.knit` are added.

* `knit(flags)`: executes the shell command `knit flags` (where `flags` is a
  string of CLI arguments) using the current instance of Knit.

## CLI and environment variables

Variables may be set at the command-line when invoking Knit with the syntax
`var=value`. These variables will be available in the Knitfile in the `cli`
table. Environment variables are similarly available in the `env` table.