github.com/hugorut/terraform@v1.1.3/docs/architecture.md (about) 1 # Terraform Core Architecture Summary 2 3 This document is a summary of the main components of Terraform Core and how 4 data and requests flow between these components. It's intended as a primer 5 to help navigate the codebase to dig into more details. 6 7 We assume some familiarity with user-facing Terraform concepts like 8 configuration, state, CLI workflow, etc. The Terraform website has 9 documentation on these ideas. 10 11 ## Terraform Request Flow 12 13 The following diagram shows an approximation of how a user command is 14 executed in Terraform: 15 16 ![Terraform Architecture Diagram, described in text below](./images/architecture-overview.png) 17 18 Each of the different subsystems (solid boxes) in this diagram is described 19 in more detail in a corresponding section below. 20 21 ## CLI (`command` package) 22 23 Each time a user runs the `terraform` program, aside from some initial 24 bootstrapping in the root package (not shown in the diagram) execution 25 transfers immediately into one of the "command" implementations in 26 [the `command` package](https://pkg.go.dev/github.com/hugorut/terraform/internal/command). 27 The mapping between the user-facing command names and 28 their corresponding `command` package types can be found in the `commands.go` 29 file in the root of the repository. 30 31 The full flow illustrated above does not actually apply to _all_ commands, 32 but it applies to the main Terraform workflow commands `terraform plan` and 33 `terraform apply`, along with a few others. 34 35 For these commands, the role of the command implementation is to read and parse 36 any command line arguments, command line options, and environment variables 37 that are needed for the given command and use them to produce a 38 [`backend.Operation`](https://pkg.go.dev/github.com/hugorut/terraform/internal/backend#Operation) 39 object that describes an action to be taken. 40 41 An _operation_ consists of: 42 43 * The action to be taken (e.g. "plan", "apply"). 44 * The name of the [workspace](https://www.terraform.io/docs/state/workspaces.html) 45 where the action will be taken. 46 * Root module input variables to use for the action. 47 * For the "plan" operation, a path to the directory containing the configuration's root module. 48 * For the "apply" operation, the plan to apply. 49 * Various other less-common options/settings such as `-target` addresses, the 50 "force" flag, etc. 51 52 The operation is then passed to the currently-selected 53 [backend](https://www.terraform.io/docs/backends/index.html). Each backend name 54 corresponds to an implementation of 55 [`backend.Backend`](https://pkg.go.dev/github.com/hugorut/terraform/internal/backend#Backend), using a 56 mapping table in 57 [the `backend/init` package](https://pkg.go.dev/github.com/hugorut/terraform/internal/backend/init). 58 59 Backends that are able to execute operations additionally implement 60 [`backend.Enhanced`](https://pkg.go.dev/github.com/hugorut/terraform/internal/backend#Enhanced); 61 the command-handling code calls `Operation` with the operation it has 62 constructed, and then the backend is responsible for executing that action. 63 64 Backends that execute operations, however, do so as an architectural implementation detail and not a 65 general feature of backends. That is, the term 'backend' as a Terraform feature is used to refer to 66 a plugin that determines where Terraform stores its state snapshots - only the default `local` 67 backend and Terraform Cloud's backends (`remote`, `cloud`) perform operations. 68 69 Thus, most backends do _not_ implement this interface, and so the `command` package wraps these 70 backends in an instance of 71 [`local.Local`](https://pkg.go.dev/github.com/hugorut/terraform/internal/backend/local#Local), 72 causing the operation to be executed locally within the `terraform` process itself. 73 74 ## Backends 75 76 A _backend_ determines where Terraform should store its state snapshots. 77 78 As described above, the `local` backend also executes operations on behalf of most other 79 backends. It uses a _state manager_ 80 (either 81 [`statemgr.Filesystem`](https://pkg.go.dev/github.com/hugorut/terraform/internal/states/statemgr#Filesystem) if the 82 local backend is being used directly, or an implementation provided by whatever 83 backend is being wrapped) to retrieve the current state for the workspace 84 specified in the operation, then uses the _config loader_ to load and do 85 initial processing/validation of the configuration specified in the 86 operation. It then uses these, along with the other settings given in the 87 operation, to construct a 88 [`terraform.Context`](https://pkg.go.dev/github.com/hugorut/terraform/internal/terraform#Context), 89 which is the main object that actually performs Terraform operations. 90 91 The `local` backend finally calls an appropriate method on that context to 92 begin execution of the relevant command, such as 93 [`Plan`](https://pkg.go.dev/github.com/hugorut/terraform/internal/terraform#Context.Plan) 94 or 95 [`Apply`](), which in turn constructs a graph using a _graph builder_, 96 described in a later section. 97 98 ## Configuration Loader 99 100 The top-level configuration structure is represented by model types in 101 [package `configs`](https://pkg.go.dev/github.com/hugorut/terraform/internal/configs). 102 A whole configuration (the root module plus all of its descendent modules) 103 is represented by 104 [`configs.Config`](https://pkg.go.dev/github.com/hugorut/terraform/internal/configs#Config). 105 106 The `configs` package contains some low-level functionality for constructing 107 configuration objects, but the main entry point is in the sub-package 108 [`configload`](https://pkg.go.dev/github.com/hugorut/terraform/internal/configs/configload]), 109 via 110 [`configload.Loader`](https://pkg.go.dev/github.com/hugorut/terraform/internal/configs/configload#Loader). 111 A loader deals with all of the details of installing child modules 112 (during `terraform init`) and then locating those modules again when a 113 configuration is loaded by a backend. It takes the path to a root module 114 and recursively loads all of the child modules to produce a single 115 [`configs.Config`](https://pkg.go.dev/github.com/hugorut/terraform/internal/configs#Config) 116 representing the entire configuration. 117 118 Terraform expects configuration files written in the Terraform language, which 119 is a DSL built on top of 120 [HCL](https://github.com/hashicorp/hcl). Some parts of the configuration 121 cannot be interpreted until we build and walk the graph, since they depend 122 on the outcome of other parts of the configuration, and so these parts of 123 the configuration remain represented as the low-level HCL types 124 [`hcl.Body`](https://pkg.go.dev/github.com/hashicorp/hcl/v2/#Body) 125 and 126 [`hcl.Expression`](https://pkg.go.dev/github.com/hashicorp/hcl/v2/#Expression), 127 allowing Terraform to interpret them at a more appropriate time. 128 129 ## State Manager 130 131 A _state manager_ is responsible for storing and retrieving snapshots of the 132 [Terraform state](https://www.terraform.io/docs/language/state/index.html) 133 for a particular workspace. Each manager is an implementation of 134 some combination of interfaces in 135 [the `statemgr` package](https://pkg.go.dev/github.com/hugorut/terraform/internal/states/statemgr), 136 with most practical managers implementing the full set of operations 137 described by 138 [`statemgr.Full`](https://pkg.go.dev/github.com/hugorut/terraform/internal/states/statemgr#Full) 139 provided by a _backend_. The smaller interfaces exist primarily for use in 140 other function signatures to be explicit about what actions the function might 141 take on the state manager; there is little reason to write a state manager 142 that does not implement all of `statemgr.Full`. 143 144 The implementation 145 [`statemgr.Filesystem`](https://pkg.go.dev/github.com/hugorut/terraform/internal/states/statemgr#Filesystem) is used 146 by default (by the `local` backend) and is responsible for the familiar 147 `terraform.tfstate` local file that most Terraform users start with, before 148 they switch to [remote state](https://www.terraform.io/docs/language/state/remote.html). 149 Other implementations of `statemgr.Full` are used to implement remote state. 150 Each of these saves and retrieves state via a remote network service 151 appropriate to the backend that creates it. 152 153 A state manager accepts and returns a state snapshot as a 154 [`states.State`](https://pkg.go.dev/github.com/hugorut/terraform/internal/states#State) 155 object. The state manager is responsible for exactly how that object is 156 serialized and stored, but all state managers at the time of writing use 157 the same JSON serialization format, storing the resulting JSON bytes in some 158 kind of arbitrary blob store. 159 160 ## Graph Builder 161 162 A _graph builder_ is called by a 163 [`terraform.Context`](https://pkg.go.dev/github.com/hugorut/terraform/internal/terraform#Context) 164 method (e.g. `Plan` or `Apply`) to produce the graph that will be used 165 to represent the necessary steps for that operation and the dependency 166 relationships between them. 167 168 In most cases, the 169 [vertices](https://en.wikipedia.org/wiki/Vertex_(graph_theory)) of Terraform's 170 graphs each represent a specific object in the configuration, or something 171 derived from those configuration objects. For example, each `resource` block 172 in the configuration has one corresponding 173 [`GraphNodeConfigResource`](https://pkg.go.dev/github.com/hugorut/terraform/internal/terraform#GraphNodeConfigResource) 174 vertex representing it in the "plan" graph. (Terraform Core uses terminology 175 inconsistently, describing graph _vertices_ also as graph _nodes_ in various 176 places. These both describe the same concept.) 177 178 The [edges](https://en.wikipedia.org/wiki/Glossary_of_graph_theory_terms#edge) 179 in the graph represent "must happen after" relationships. These define the 180 order in which the vertices are evaluated, ensuring that e.g. one resource is 181 created before another resource that depends on it. 182 183 Each operation has its own graph builder, because the graph building process 184 is different for each. For example, a "plan" operation needs a graph built 185 directly from the configuration, but an "apply" operation instead builds its 186 graph from the set of changes described in the plan that is being applied. 187 188 The graph builders all work in terms of a sequence of _transforms_, which 189 are implementations of 190 [`terraform.GraphTransformer`](https://pkg.go.dev/github.com/hugorut/terraform/internal/terraform#GraphTransformer). 191 Implementations of this interface just take a graph and mutate it in any 192 way needed, and so the set of available transforms is quite varied. Some 193 important examples include: 194 195 * [`ConfigTransformer`](https://pkg.go.dev/github.com/hugorut/terraform/internal/terraform#ConfigTransformer), 196 which creates a graph vertex for each `resource` block in the configuration. 197 198 * [`StateTransformer`](https://pkg.go.dev/github.com/hugorut/terraform/internal/terraform#StateTransformer), 199 which creates a graph vertex for each resource instance currently tracked 200 in the state. 201 202 * [`ReferenceTransformer`](https://pkg.go.dev/github.com/hugorut/terraform/internal/terraform#ReferenceTransformer), 203 which analyses the configuration to find dependencies between resources and 204 other objects and creates any necessary "happens after" edges for these. 205 206 * [`ProviderTransformer`](https://pkg.go.dev/github.com/hugorut/terraform/internal/terraform#ProviderTransformer), 207 which associates each resource or resource instance with exactly one 208 provider configuration (implementing 209 [the inheritance rules](https://www.terraform.io/docs/language/modules/develop/providers.html)) 210 and then creates "happens after" edges to ensure that the providers are 211 initialized before taking any actions with the resources that belong to 212 them. 213 214 There are many more different graph transforms, which can be discovered 215 by reading the source code for the different graph builders. Each graph 216 builder uses a different subset of these depending on the needs of the 217 operation that is being performed. 218 219 The result of graph building is a 220 [`terraform.Graph`](https://pkg.go.dev/github.com/hugorut/terraform/internal/terraform#Graph), which 221 can then be processed using a _graph walker_. 222 223 ## Graph Walk 224 225 The process of walking the graph visits each vertex of that graph in a way 226 which respects the "happens after" edges in the graph. The walk algorithm 227 itself is implemented in 228 [the low-level `dag` package](https://pkg.go.dev/github.com/hugorut/terraform/internal/dag#AcyclicGraph.Walk) 229 (where "DAG" is short for [_Directed Acyclic Graph_](https://en.wikipedia.org/wiki/Directed_acyclic_graph)), in 230 [`AcyclicGraph.Walk`](https://pkg.go.dev/github.com/hugorut/terraform/internal/dag#AcyclicGraph.Walk). 231 However, the "interesting" Terraform walk functionality is implemented in 232 [`terraform.ContextGraphWalker`](https://pkg.go.dev/github.com/hugorut/terraform/internal/terraform#ContextGraphWalker), 233 which implements a small set of higher-level operations that are performed 234 during the graph walk: 235 236 * `EnterPath` is called once for each module in the configuration, taking a 237 module address and returning a 238 [`terraform.EvalContext`](https://pkg.go.dev/github.com/hugorut/terraform/internal/terraform#EvalContext) 239 that tracks objects within that module. `terraform.Context` is the _global_ 240 context for the entire operation, while `terraform.EvalContext` is a 241 context for processing within a single module, and is the primary means 242 by which the namespaces in each module are kept separate. 243 244 Each vertex in the graph is evaluated, in an order that guarantees that the 245 "happens after" edges will be respected. If possible, the graph walk algorithm 246 will evaluate multiple vertices concurrently. Vertex evaluation code must 247 therefore make careful use of concurrency primitives such as mutexes in order 248 to coordinate access to shared objects such as the `states.State` object. 249 In most cases, we use the helper wrapper 250 [`states.SyncState`](https://pkg.go.dev/github.com/hugorut/terraform/internal/states#SyncState) 251 to safely implement concurrent reads and writes from the shared state. 252 253 ## Vertex Evaluation 254 255 The action taken for each vertex during the graph walk is called 256 _execution_. Execution runs a sequence of arbitrary actions that make sense 257 for a particular vertex type. 258 259 For example, evaluation of a vertex representing a resource instance during 260 a plan operation would include the following high-level steps: 261 262 * Retrieve the resource's associated provider from the `EvalContext`. This 263 should already be initialized earlier by the provider's own graph vertex, 264 due to the "happens after" edge between the resource node and the provider 265 node. 266 267 * Retrieve from the state the portion relevant to the specific resource 268 instance being evaluated. 269 270 * Evaluate the attribute expressions given for the resource in configuration. 271 This often involves retrieving the state of _other_ resource instances so 272 that their values can be copied or transformed into the current instance's 273 attributes, which is coordinated by the `EvalContext`. 274 275 * Pass the current instance state and the resource configuration to the 276 provider, asking the provider to produce an _instance diff_ representing the 277 differences between the state and the configuration. 278 279 * Save the instance diff as part of the plan that is being constructed by 280 this operation. 281 282 Each execution step for a vertex is an implementation of 283 [`terraform.Execute`](https://pkg.go.dev/github.com/hugorut/terraform/internal/erraform#Execute). 284 As with graph transforms, the behavior of these implementations varies widely: 285 whereas graph transforms can take any action against the graph, an `Execute` 286 implementation can take any action against the `EvalContext`. 287 288 The implementation of `terraform.EvalContext` used in real processing 289 (as opposed to testing) is 290 [`terraform.BuiltinEvalContext`](https://pkg.go.dev/github.com/hugorut/terraform/internal/terraform#BuiltinEvalContext). 291 It provides coordinated access to plugins, the current state, and the current 292 plan via the `EvalContext` interface methods. 293 294 In order to be executed, a vertex must implement 295 [`terraform.GraphNodeExecutable`](https://pkg.go.dev/github.com/hugorut/terraform/internal/terraform#GraphNodeExecutable), 296 which has a single `Execute` method that handles. There are numerous `Execute` 297 implementations with different behaviors, but some prominent examples are: 298 299 * [NodePlannableResource.Execute](https://pkg.go.dev/github.com/hugorut/terraform/internal/terraform#NodePlannableResourceInstance.Execute), which handles the `plan` operation. 300 301 * [`NodeApplyableResourceInstance.Execute`](https://pkg.go.dev/github.com/hugorut/terraform/internal/terraform#NodeApplyableResourceInstance.Execute), which handles the main `apply` operation. 302 303 * [`NodeDestroyResourceInstance.Execute`](https://pkg.go.dev/github.com/hugorut/terraform/internal/terraform#EvalWriteState), which handles the main `destroy` operation. 304 305 A vertex must complete successfully before the graph walk will begin evaluation 306 for other vertices that have "happens after" edges. Evaluation can fail with one 307 or more errors, in which case the graph walk is halted and the errors are 308 returned to the user. 309 310 ### Expression Evaluation 311 312 An important part of vertex evaluation for most vertex types is evaluating 313 any expressions in the configuration block associated with the vertex. This 314 completes the processing of the portions of the configuration that were not 315 processed by the configuration loader. 316 317 The high-level process for expression evaluation is: 318 319 1. Analyze the configuration expressions to see which other objects they refer 320 to. For example, the expression `aws_instance.example[1]` refers to one of 321 the instances created by a `resource "aws_instance" "example"` block in 322 configuration. This analysis is performed by 323 [`lang.References`](https://pkg.go.dev/github.com/hugorut/terraform/internal/lang#References), 324 or more often one of the helper wrappers around it: 325 [`lang.ReferencesInBlock`](https://pkg.go.dev/github.com/hugorut/terraform/internal/lang#ReferencesInBlock) 326 or 327 [`lang.ReferencesInExpr`](https://pkg.go.dev/github.com/hugorut/terraform/internal/lang#ReferencesInExpr) 328 329 1. Retrieve from the state the data for the objects that are referred to and 330 create a lookup table of the values from these objects that the 331 HCL evaluation code can refer to. 332 333 1. Prepare the table of built-in functions so that HCL evaluation can refer to 334 them. 335 336 1. Ask HCL to evaluate each attribute's expression (a 337 [`hcl.Expression`](https://pkg.go.dev/github.com/hashicorp/hcl/v2/#Expression) 338 object) against the data and function lookup tables. 339 340 In practice, steps 2 through 4 are usually run all together using one 341 of the methods on [`lang.Scope`](https://pkg.go.dev/github.com/hugorut/terraform/internal/lang#Scope); 342 most commonly, 343 [`lang.EvalBlock`](https://pkg.go.dev/github.com/hugorut/terraform/internal/lang#Scope.EvalBlock) 344 or 345 [`lang.EvalExpr`](https://pkg.go.dev/github.com/hugorut/terraform/internal/lang#Scope.EvalExpr). 346 347 Expression evaluation produces a dynamic value represented as a 348 [`cty.Value`](https://pkg.go.dev/github.com/zclconf/go-cty/cty#Value). 349 This Go type represents values from the Terraform language and such values 350 are eventually passed to provider plugins. 351 352 ### Sub-graphs 353 354 Some vertices have a special additional behavior that happens after their 355 evaluation steps are complete, where the vertex implementation is given 356 the opportunity to build another separate graph which will be walked as part 357 of the evaluation of the vertex. 358 359 The main example of this is when a `resource` block has the `count` argument 360 set. In that case, the plan graph initially contains one vertex for each 361 `resource` block, but that graph then _dynamically expands_ to have a sub-graph 362 containing one vertex for each instance requested by the count. That is, the 363 sub-graph of `aws_instance.example` might contain vertices for 364 `aws_instance.example[0]`, `aws_instance.example[1]`, etc. This is necessary 365 because the `count` argument may refer to other objects whose values are not 366 known when the main graph is constructed, but become known while evaluating 367 other vertices in the main graph. 368 369 This special behavior applies to vertex objects that implement 370 [`terraform.GraphNodeDynamicExpandable`](https://pkg.go.dev/github.com/hugorut/terraform/internal/terraform#GraphNodeDynamicExpandable). 371 Such vertices have their own nested _graph builder_, _graph walk_, 372 and _vertex evaluation_ steps, with the same behaviors as described in these 373 sections for the main graph. The difference is in which graph transforms 374 are used to construct the graph and in which evaluation steps apply to the 375 nodes in that sub-graph.