github.com/crossplane/upjet@v1.3.0/docs/configuring-a-resource.md (about)

     1  <!--
     2  SPDX-FileCopyrightText: 2023 The Crossplane Authors <https://crossplane.io>
     3  
     4  SPDX-License-Identifier: CC-BY-4.0
     5  -->
     6  # Configuring a resource
     7  
     8  [Upjet] generates as much as it could using the available information in the
     9  Terraform resource schema. This includes an XRM-conformant schema of the
    10  resource, controller logic, late initialization, sensitive data handling, etc.
    11  However, there are still information that requires some input configuration
    12  which can be found by checking the Terraform documentation of the resource:
    13  
    14  - [External name]
    15  - [Cross Resource Referencing]
    16  - [Additional Sensitive Fields and Custom Connection Details]
    17  - [Late Initialization Behavior]
    18  - [Overriding Terraform Resource Schema]
    19  - [Initializers]
    20  
    21  ## External Name
    22  
    23  Crossplane uses an annotation in managed resource CR to identify the external
    24  resource which is managed by Crossplane. See [the external name documentation]
    25  for more details. The format and source of the external name depends on the
    26  cloud provider; sometimes it could simply be the name of resource (e.g. S3
    27  Bucket), and sometimes it is an auto-generated id by cloud API (e.g. VPC id ).
    28  This is something specific to resource, and we need some input configuration for
    29  upjet to appropriately generate a resource.
    30  
    31  Since Terraform already needs [a similar identifier] to import a resource, most
    32  helpful part of resource documentation is the [import section].
    33  
    34  Upjet performs some back and forth conversions between Crossplane resource model
    35  and Terraform configuration. We need a custom, per resource configuration to
    36  adapt Crossplane `external name` from Terraform `id`.
    37  
    38  Here are [the types for the External Name configuration]:
    39  
    40  ```go
    41  // SetIdentifierArgumentsFn sets the name of the resource in Terraform attributes map,
    42  // i.e. Main HCL file.
    43  type SetIdentifierArgumentsFn func(base map[string]any, externalName string)
    44  // GetExternalNameFn returns the external name extracted from the TF State.
    45  type GetExternalNameFn func(tfstate map[string]any) (string, error)
    46  // GetIDFn returns the ID to be used in TF State file, i.e. "id" field in
    47  // terraform.tfstate.
    48  type GetIDFn func(ctx context.Context, externalName string, parameters map[string]any, providerConfig map[string]any) (string, error)
    49  
    50  // ExternalName contains all information that is necessary for naming operations,
    51  // such as removal of those fields from spec schema and calling Configure function
    52  // to fill attributes with information given in external name.
    53  type ExternalName struct {
    54    // SetIdentifierArgumentFn sets the name of the resource in Terraform argument
    55    // map. In many cases, there is a field called "name" in the HCL schema, however,
    56    // there are cases like RDS DB Cluster where the name field in HCL is called
    57    // "cluster_identifier". This function is the place that you can take external
    58    // name and assign it to that specific key for that resource type.
    59    SetIdentifierArgumentFn SetIdentifierArgumentsFn
    60  
    61    // GetExternalNameFn returns the external name extracted from TF State. In most cases,
    62    // "id" field contains all the information you need. You'll need to extract
    63    // the format that is decided for external name annotation to use.
    64    // For example the following is an Azure resource ID:
    65    // /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/mygroup1
    66    // The function should return "mygroup1" so that it can be used to set external
    67    // name if it was not set already.
    68    GetExternalNameFn GetExternalNameFn
    69  
    70    // GetIDFn returns the string that will be used as "id" key in TF state. In
    71    // many cases, external name format is the same as "id" but when it is not
    72    // we may need information from other places to construct it. For example,
    73    // the following is an Azure resource ID:
    74    // /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/mygroup1
    75    // The function here should use information from supplied arguments to
    76    // construct this ID, i.e. "mygroup1" from external name, subscription ID
    77    // from providerConfig, and others from parameters map if needed.
    78    GetIDFn GetIDFn
    79  
    80    // OmittedFields are the ones you'd like to be removed from the schema since
    81    // they are specified via external name. For example, if you set
    82    // "cluster_identifier" in SetIdentifierArgumentFn, then you need to omit
    83    // that field.
    84    // You can omit only the top level fields.
    85    // No field is omitted by default.
    86    OmittedFields []string
    87  
    88    // DisableNameInitializer allows you to specify whether the name initializer
    89    // that sets external name to metadata.name if none specified should be disabled.
    90    // It needs to be disabled for resources whose external identifier is randomly
    91    // assigned by the provider, like AWS VPC where it gets vpc-21kn123 identifier
    92    // and not let you name it.
    93    DisableNameInitializer bool
    94  }
    95  ```
    96  
    97  Comments explain the purpose of each field but let's clarify further with some
    98  example cases.
    99  
   100  ### Case 1: Name as External Name and Terraform ID
   101  
   102  This is the simplest and most straightforward case with the following
   103  conditions:
   104  
   105  - Terraform resource uses the `name` argument to identify the resources
   106  - Terraform resource can be imported with `name`, i.e. `id`=`name`
   107  
   108  [aws_iam_user] is a good example here. In this case, we can just use the
   109  [NameAsIdentifier] config of Upjet as follows:
   110  
   111  ```go
   112  import (
   113   "github.com/crossplane/upjet/pkg/config"
   114   ...
   115  )
   116  
   117  ...
   118      p.AddResourceConfigurator("aws_iam_user", func(r *config.Resource) {
   119          r.ExternalName = config.NameAsIdentifier
   120    ...
   121      }
   122  ```
   123  
   124  There are some resources which fits into this case with an exception by
   125  expecting an argument other than `name` to name/identify a resource, for
   126  example, [bucket] for [aws_s3_bucket] and [cluster_identifier] for
   127  [aws_rds_cluster].
   128  
   129  Let's check [aws_s3_bucket] further. Reading the [import section of s3 bucket]
   130  we see that bucket is imported with its **name**, however, checking _arguments_
   131  section we see that this name is provided with the [bucket] argument. We also
   132  notice, there is also another argument as `bucket_prefix` which conflicts with
   133  `bucket` argument. We can just use the [NameAsIdentifier] config, however, we
   134  also need to configure the `bucket` argument with `SetIdentifierArgumentFn` and
   135  also omit `bucket` and `bucket_prefix` arguments from the spec with
   136  `OmittedFields`:
   137  
   138  ```go
   139  import (
   140   "github.com/crossplane/upjet/pkg/config"
   141   ...
   142  )
   143  
   144  ...
   145      p.AddResourceConfigurator("aws_s3_bucket", func(r *config.Resource) {
   146          r.ExternalName = config.NameAsIdentifier
   147          r.ExternalName.SetIdentifierArgumentFn = func(base map[string]any, externalName string) {
   148              base["bucket"] = externalName
   149          }
   150          r.ExternalName.OmittedFields = []string{
   151              "bucket",
   152              "bucket_prefix",
   153          }
   154    ...
   155      }
   156  ```
   157  
   158  ### Case 2: Identifier from Provider
   159  
   160  In this case, the (cloud) provider generates an identifier for the resource
   161  independent of what we provided as arguments.
   162  
   163  Checking the [import section of aws_vpc], we see that this resource is being
   164  imported with `vpc id`. When we check the [arguments list] and provided [example
   165  usages], it is clear that this **id** is **not** something that user provides,
   166  rather generated by AWS API.
   167  
   168  Here, we can just use [IdentifierFromProvider] configuration:
   169  
   170  ```go
   171  import (
   172   "github.com/crossplane/upjet/pkg/config"
   173   ...
   174  )
   175  
   176  ...
   177      p.AddResourceConfigurator("aws_vpc", func(r *config.Resource) {
   178          r.ExternalName = config.IdentifierFromProvider
   179    ...
   180      }
   181  ```
   182  
   183  ### Case 3: Terraform ID as a Formatted String
   184  
   185  For some resources, Terraform uses a formatted string as `id` which include
   186  resource identifier that Crossplane uses as external name but may also contain
   187  some other parameters.
   188  
   189  Most `azurerm` resources fall into this category. Checking the [import section
   190  of azurerm_sql_server], we see that can be imported with an `id` in the
   191  following format:
   192  
   193  ```text
   194  /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myresourcegroup/providers/Microsoft.Sql/servers/myserver
   195  ```
   196  
   197  To properly set external name for such a resource, we need to configure how to
   198  extract external name from this string (`GetExternalNameFn`) and how to build
   199  this id back (`GetIDFn`).
   200  
   201  ```go
   202  import (
   203   "github.com/crossplane/upjet/pkg/config"
   204   ...
   205  )
   206  
   207  func getNameFromFullyQualifiedID(tfstate map[string]any) (string, error) {
   208   id, ok := tfstate["id"]
   209   if !ok {
   210    return "", errors.Errorf(ErrFmtNoAttribute, "id")
   211   }
   212   idStr, ok := id.(string)
   213   if !ok {
   214    return "", errors.Errorf(ErrFmtUnexpectedType, "id")
   215   }
   216   words := strings.Split(idStr, "/")
   217   return words[len(words)-1], nil
   218  }
   219  
   220  func getFullyQualifiedIDfunc(ctx context.Context, externalName string, parameters map[string]any, providerConfig map[string]any) (string, error) {
   221   subID, ok := providerConfig["subscription_id"]
   222      if !ok {
   223          return "", errors.Errorf(ErrFmtNoAttribute, "subscription_id")
   224      }
   225      subIDStr, ok := subID.(string)
   226      if !ok {
   227          return "", errors.Errorf(ErrFmtUnexpectedType, "subscription_id")
   228      }
   229      rg, ok := parameters["resource_group_name"]
   230      if !ok {
   231          return "", errors.Errorf(ErrFmtNoAttribute, "resource_group_name")
   232      }
   233      rgStr, ok := rg.(string)
   234      if !ok {
   235          return "", errors.Errorf(ErrFmtUnexpectedType, "resource_group_name")
   236      }
   237  
   238   name, ok := parameters["name"]
   239      if !ok {
   240          return "", errors.Errorf(ErrFmtNoAttribute, "name")
   241      }
   242      nameStr, ok := rg.(string)
   243      if !ok {
   244          return "", errors.Errorf(ErrFmtUnexpectedType, "name")
   245      }
   246  
   247      return fmt.Sprintf("/subscriptions/%s/resourceGroups/%s/providers/Microsoft.Sql/servers/%s", subIDStr, rgStr, nameStr), nil
   248  }
   249  
   250  ...
   251      p.AddResourceConfigurator("azurerm_sql_server", func(r *config.Resource) {
   252          r.ExternalName = config.NameAsIdentifier
   253          r.ExternalName.GetExternalNameFn = getNameFromFullyQualifiedID
   254          r.ExternalName.GetIDFn = getFullyQualifiedIDfunc
   255    ...
   256      }
   257  ```
   258  
   259  With this, we have covered most common scenarios for configuring external name.
   260  You can always check resource configurations of existing jet Providers as
   261  further examples under `config/<group>/config.go` in their repositories.
   262  
   263  _Please see [this figure] to understand why we really need 3 different functions
   264  to configure external names and, it visualizes which is used how:_
   265  ![Alt text](../docs/images/upjet-externalname.png) _Note that, initially, GetIDFn
   266  will use the external-name annotation to set the terraform.tfstate id and, after
   267  that, it uses the terraform.tfstate id to update the external-name annotation.
   268  For cases where both values are different, both GetIDFn and GetExternalNameFn
   269  must be set in order to have the correct configuration._
   270  
   271  ### Cross Resource Referencing
   272  
   273  Crossplane uses cross resource referencing to [handle dependencies] between
   274  managed resources. For example, if you have an IAM User defined as a Crossplane
   275  managed resource, and you want to create an Access Key for that user, you would
   276  need to refer to the User CR from the Access Key resource. This is handled by
   277  cross resource referencing.
   278  
   279  See how the [user] referenced at `forProvider.userRef.name` field of the Access
   280  Key in the following example:
   281  
   282  ```yaml
   283  apiVersion: iam.aws.tf.crossplane.io/v1alpha1
   284  kind: User
   285  metadata:
   286    name: sample-user
   287  spec:
   288    forProvider: {}
   289  ---
   290  apiVersion: iam.aws.tf.crossplane.io/v1alpha1
   291  kind: AccessKey
   292  metadata:
   293    name: sample-access-key
   294  spec:
   295    forProvider:
   296      userRef:
   297        name: sample-user
   298    writeConnectionSecretToRef:
   299      name: sample-access-key-secret
   300      namespace: crossplane-system
   301  ```
   302  
   303  Historically, reference resolution method were written by hand which requires
   304  some effort, however, with the latest Crossplane code generation tooling, it is
   305  now possible to [generate reference resolution methods] by just adding some
   306  marker on the fields. Now, the only manual step for generating cross resource
   307  references is to provide which field of a resource depends on which information
   308  (e.g. `id`, `name`, `arn` etc.) from the other.
   309  
   310  In Upjet, we have a [configuration] to provide this information for a field:
   311  
   312  ```go
   313  // Reference represents the Crossplane options used to generate
   314  // reference resolvers for fields
   315  type Reference struct {
   316      // Type is the type name of the CRD if it is in the same package or
   317      // <package-path>.<type-name> if it is in a different package.
   318      Type string
   319      // TerraformName is the name of the Terraform resource
   320      // which will be referenced. The supplied resource name is
   321      // converted to a type name of the corresponding CRD using
   322      // the configured TerraformTypeMapper.
   323      TerraformName string
   324      // Extractor is the function to be used to extract value from the
   325      // referenced type. Defaults to getting external name.
   326      // Optional
   327      Extractor string
   328      // RefFieldName is the field name for the Reference field. Defaults to
   329      // <field-name>Ref or <field-name>Refs.
   330      // Optional
   331      RefFieldName string
   332      // SelectorFieldName is the field name for the Selector field. Defaults to
   333      // <field-name>Selector.
   334      // Optional
   335      SelectorFieldName string
   336  }
   337  ```
   338  
   339  For a resource that we want to generate, we need to check its argument list in
   340  Terraform documentation and figure out which field needs reference to which
   341  resource.
   342  
   343  Let's check [iam_access_key] as an example. In the argument list, we see the
   344  [user] field which requires a reference to a IAM user. So, we need to the
   345  following referencing configuration:
   346  
   347  ```go
   348  func Configure(p *config.Provider) {
   349      p.AddResourceConfigurator("aws_iam_access_key", func (r *config.Resource) {
   350          r.References["user"] = config.Reference{
   351              Type: "User",
   352          }
   353      })
   354  }
   355  ```
   356  
   357  Please note the value of `Type` field needs to be a string representing the Go
   358  type of the resource. Since, `AccessKey` and `User` resources are under the same
   359  go package, we don't need to provide the package path. However, this is not
   360  always the case and referenced resources might be in different package. In that
   361  case, we would need to provide the full path. Referencing to a [kms key] from
   362  `aws_ebs_volume` resource is a good example here:
   363  
   364  ```go
   365  func Configure(p *config.Provider) {
   366      p.AddResourceConfigurator("aws_ebs_volume", func(r *config.Resource) {
   367          r.References["kms_key_id"] = config.Reference{
   368             Type: "github.com/crossplane-contrib/provider-tf-aws/apis/kms/v1alpha1.Key",
   369          }
   370      })
   371  }
   372  ```
   373  
   374  ### Auto Cross Resource Reference Generation
   375  
   376  Cross Resource Referencing is one of the key concepts of the resource
   377  configuration. As a very common case, cloud services depend on other cloud
   378  services. For example, AWS Subnet resource needs an AWS VPC for creation. So,
   379  for creating a Subnet successfully, before you have to create a VPC resource.
   380  Please see the [Managed Resources] documentation for more details.
   381  
   382  These documentations focus on the general concepts and manual configurations
   383  of Cross Resource References. However, the main topic of this documentation is
   384  automatic example&reference generation.
   385  
   386  Upjet has a scraper tool for scraping provider metadata from the Terraform
   387  Registry. The scraped metadata are:
   388  
   389  - Resource Descriptions
   390  - Examples of Resources (in HCL format)
   391  - Field Documentations
   392  - Import Statements
   393  
   394  These are very critical information for our automation processes. We use this
   395  scraped metadata in many contexts. For example, field documentation of
   396  resources and descriptions are used as Golang comments for schema fields and
   397  CRDs.
   398  
   399  Another important scraped information is examples of resources. As a part
   400  of testing efforts, finding the correct combination of field values is not easy
   401  for every scenario. So, having a working example (combination) is very important
   402  for easy testing.
   403  
   404  At this point, this example that is in HCL format is converted to a Managed
   405  Resource manifest, and we can use this manifest in our test efforts.
   406  
   407  This is an example from Terraform Registry AWS Ebs Volume resource:
   408  
   409  ```hcl
   410  resource "aws_ebs_volume" "example" {
   411    availability_zone = "us-west-2a"
   412    size              = 40
   413  
   414    tags = {
   415      Name = "HelloWorld"
   416    }
   417  }
   418  
   419  resource "aws_ebs_snapshot" "example_snapshot" {
   420    volume_id = aws_ebs_volume.example.id
   421  
   422    tags = {
   423      Name = "HelloWorld_snap"
   424    }
   425  }
   426  ```
   427  
   428  The generated example:
   429  
   430  ```yaml
   431  apiVersion: ec2.aws.upbound.io/v1beta1
   432  kind: EBSSnapshot
   433  metadata:
   434    annotations:
   435      meta.upbound.io/example-id: ec2/v1beta1/ebssnapshot
   436    labels:
   437      testing.upbound.io/example-name: example_snapshot
   438    name: example-snapshot
   439  spec:
   440    forProvider:
   441      region: us-west-1
   442      tags:
   443        Name: HelloWorld_snap
   444      volumeIdSelector:
   445        matchLabels:
   446          testing.upbound.io/example-name: example
   447  
   448  ---
   449  
   450  apiVersion: ec2.aws.upbound.io/v1beta1
   451  kind: EBSVolume
   452  metadata:
   453    annotations:
   454      meta.upbound.io/example-id: ec2/v1beta1/ebssnapshot
   455    labels:
   456      testing.upbound.io/example-name: example
   457    name: example
   458  spec:
   459    forProvider:
   460      availabilityZone: us-west-2a
   461      region: us-west-1
   462      size: 40
   463      tags:
   464        Name: HelloWorld
   465  ```
   466  
   467  Here, there are three very important points that scraper makes easy our life:
   468  
   469  - We do not have to find the correct value combinations for fields. So, we can
   470    easily use the generated example manifest in our tests.
   471  - The HCL example was scraped from registry documentation of the
   472    `aws_ebs_snapshot` resource. In the example, you also see the `aws_ebs_volume`
   473    resource manifest because, for the creation of an EBS Snapshot, you need an
   474    EBS Volume resource. Thanks to the source Registry, (in many cases, there are
   475    the dependent resources of target resources) we can also scrape the
   476    dependencies of target resources.
   477  - The last item is actually what is intended to be explained in this document.
   478    For using the Cross Resource References, as I mentioned above, you need to add
   479    some references to the resource configuration. But, in many cases, if in the
   480    scraped example, the mentioned dependencies are already described you do not
   481    have to write explicit references to resource configuration. The Cross
   482    Resource Reference generator generates the mentioned references.
   483  
   484  ### Validating the Cross Resource References
   485  
   486  As I mentioned, many references are generated from scraped metadata by an auto
   487  reference generator. However, there are two cases where we miss generating the
   488  references.
   489  
   490  The first one is related to some bugs or improvement points in the generator.
   491  This means that the generator can handle many references in the scraped examples
   492  and correctly generate them. But we cannot say that the ratio is 100%. For some
   493  cases, the generator cannot generate references, even though they are in the
   494  scraped example manifests.
   495  
   496  The second one is related to the scraped example itself. As I mentioned above,
   497  the source of the generator is the scraped example manifest. So, it checks the
   498  manifest and tries to generate the found cross-resource references. In some
   499  cases, although there are other reference fields, these do not exist in the
   500  example manifest. They can only be mentioned in schema/field documentation.
   501  
   502  For these types of situations, you must configure cross-resource references
   503  explicitly.
   504  
   505  ### Removing Auto-Generated Cross Resource References In Some Corner Cases
   506  
   507  In some cases, the generated references can narrow the reference pool covered by
   508  the field. For example, X resource has an A field and Y and Z resources can be
   509  referenced via this field. However, since the reference to Y is mentioned in the
   510  example manifest, this reference field will only be defined over Y. In this
   511  case, since the reference pool of the relevant field will be narrowed, it would
   512  be more appropriate to delete this reference. For example,
   513  
   514  ```hcl
   515  resource "aws_route53_record" "www" {
   516    zone_id = aws_route53_zone.primary.zone_id
   517    name    = "example.com"
   518    type    = "A"
   519  
   520    alias {
   521      name                   = aws_elb.main.dns_name
   522      zone_id                = aws_elb.main.zone_id
   523      evaluate_target_health = true
   524    }
   525  }
   526  ```
   527  
   528  Route53 Record resource’s alias.name field has a reference. In the example, this
   529  reference is shown by using the `aws_elb` resource. However, when we check the
   530  field documentation, we see that this field can also be referenced by other
   531  resources:
   532  
   533  ```text
   534  Alias
   535  Alias records support the following:
   536  
   537  name - (Required) DNS domain name for a CloudFront distribution, S3 bucket, ELB,
   538  or another resource record set in this hosted zone.
   539  ```
   540  
   541  ### Conclusion
   542  
   543  As a result, mentioned scraper and example&reference generators are very useful
   544  for making easy the test efforts. But using these functionalities, we must be
   545  careful to avoid undesired states.
   546  
   547  [Managed Resources]: https://docs.crossplane.io/latest/concepts/managed-resources/#referencing-other-resources
   548  
   549  ## Additional Sensitive Fields and Custom Connection Details
   550  
   551  Crossplane stores sensitive information of a managed resource in a Kubernetes
   552  secret, together with some additional fields that would help consumption of the
   553  resource, a.k.a. [connection details].
   554  
   555  In Upjet, we already handle sensitive fields that are marked as sensitive in
   556  Terraform schema and no further action required for them. Upjet will properly
   557  hide these fields from CRD spec and status by converting to a secret reference
   558  or storing in connection details secret respectively. However, we still have
   559  some custom configuration API that would allow including additional fields into
   560  connection details secret no matter they are sensitive or not.
   561  
   562  As an example, let's use `aws_iam_access_key`. Currently, Upjet stores all
   563  sensitive fields in Terraform schema as prefixed with `attribute.`, so without
   564  any `AdditionalConnectionDetailsFn`, connection resource will have
   565  `attribute.id` and `attribute.secret` corresponding to [id] and [secret] fields
   566  respectively. To see them with more common keys, i.e. `aws_access_key_id` and
   567  `aws_secret_access_key`, we would need to make the following configuration:
   568  
   569  ```go
   570  func Configure(p *config.Provider) {
   571   p.AddResourceConfigurator("aws_iam_access_key", func(r *config.Resource) {
   572    r.Sensitive.AdditionalConnectionDetailsFn = func(attr map[string]any) (map[string][]byte, error) {
   573     conn := map[string][]byte{}
   574     if a, ok := attr["id"].(string); ok {
   575      conn["aws_access_key_id"] = []byte(a)
   576     }
   577     if a, ok := attr["secret"].(string); ok {
   578      conn["aws_secret_access_key"] = []byte(a)
   579     }
   580     return conn, nil
   581    }
   582   })
   583  }
   584  ```
   585  
   586  This will produce a connection details secret as follows:
   587  
   588  ```yaml
   589  apiVersion: v1
   590  data:
   591    attribute.id: QUtJQVk0QUZUVFNFNDI2TlhKS0I=
   592    attribute.secret: ABCxyzRedacted==
   593    attribute.ses_smtp_password_v4: QQ00REDACTED==
   594    aws_access_key_id: QUtJQVk0QUZUVFNFNDI2TlhKS0I=
   595    aws_secret_access_key: ABCxyzRedacted==
   596  kind: Secret
   597  ```
   598  
   599  ### Late Initialization Configuration
   600  
   601  Late initialization configuration is only required if there are conflicting
   602  arguments in terraform resource configuration. Unfortunately, there is _no easy
   603  way_ to figure that out without testing the resource, _so feel free to skip this
   604  configuration_ at the first place and revisit _only if_ you have errors like
   605  below while testing the resource.
   606  
   607  ```text
   608  observe failed: cannot run refresh: refresh failed: Invalid combination of arguments:
   609    "address_prefix": only one of `address_prefix,address_prefixes` can be specified, but `address_prefix,address_prefixes` were specified.: File name: main.tf.json
   610  ```
   611  
   612  If you would like to have the late-initialization library _not_ to process the
   613  [`address_prefix`] parameter field, then the following configuration where we
   614  specify the parameter field path is sufficient:
   615  
   616  ```go
   617  func Configure(p *config.Provider) {
   618   p.AddResourceConfigurator("azurerm_subnet", func(r *config.Resource) {
   619    r.LateInitializer = config.LateInitializer{
   620     IgnoredFields: []string{"address_prefix"},
   621    }
   622   })
   623  }
   624  ```
   625  
   626  _Please note that, there could be errors looking slightly different from above,
   627  so please consider configuring late initialization behaviour whenever you got
   628  some unexpected error starting with `observe failed:`, once you are sure that
   629  you provided all necessary parameters to your resource._
   630  
   631  ### Further details on Late Initialization
   632  
   633  Upjet runtime automatically performs late-initialization during an
   634  [`external.Observe`] call with means of runtime reflection. State of the world
   635  observed by Terraform CLI is used to initialize any `nil`-valued pointer
   636  parameters in the managed resource's `spec`. In most of the cases no custom
   637  configuration should be necessary for late-initialization to work. However,
   638  there are certain cases where you will want/need to customize
   639  late-initialization behaviour. Thus, Upjet provides an extensible
   640  [late-initialization customization API] that controls late-initialization
   641  behaviour.
   642  
   643  The associated resource struct is defined
   644  [here](https://github.com/crossplane/upjet/blob/c9e21387298d8ed59fcd71c7f753ec401a3383a5/pkg/config/resource.go#L91)
   645  as follows:
   646  
   647  ```go
   648  // LateInitializer represents configurations that control
   649  // late-initialization behaviour
   650  type LateInitializer struct {
   651      // IgnoredFields are the canonical field names to be skipped during
   652      // late-initialization
   653      IgnoredFields []string
   654  }
   655  ```
   656  
   657  Currently, it only involves a configuration option to specify certain `spec`
   658  parameters to be ignored during late-initialization. Each element of the
   659  `LateInitializer.IgnoredFields` slice represents the canonical path relative to
   660  the parameters struct for the managed resource's `Spec` using `Go` type names as
   661  path elements. As an example, with the following type definitions:
   662  
   663  ```go
   664  type Subnet struct {
   665      metav1.TypeMeta   `json:",inline"`
   666      metav1.ObjectMeta `json:"metadata,omitempty"`
   667      Spec              SubnetSpec   `json:"spec"`
   668      Status            SubnetStatus `json:"status,omitempty"`
   669  }
   670  
   671  type SubnetSpec struct {
   672      ForProvider     SubnetParameters `json:"forProvider"`
   673      ...
   674  }
   675  
   676  type DelegationParameters struct {
   677      // +kubebuilder:validation:Required
   678      Name *string `json:"name" tf:"name,omitempty"`
   679      ...
   680  }
   681  
   682  type SubnetParameters struct {
   683      // +kubebuilder:validation:Optional
   684      AddressPrefix *string `json:"addressPrefix,omitempty" tf:"address_prefix,omitempty"`
   685      // +kubebuilder:validation:Optional
   686      Delegation []DelegationParameters `json:"delegation,omitempty" tf:"delegation,omitempty"`
   687      ...
   688  }
   689  ```
   690  
   691  In most cases, custom late-initialization configuration will not be necessary.
   692  However, after generating a new managed resource and observing its behaviour (at
   693  runtime), it may turn out that late-initialization behaviour needs
   694  customization. For certain resources like the `provider-tf-azure`'s
   695  `PostgresqlServer` resource, we have observed that Terraform state contains
   696  values for mutually exclusive parameters, e.g., for `PostgresqlServer`, both
   697  `StorageMb` and `StorageProfile[].StorageMb` get late-initialized. Upon next
   698  reconciliation, we generate values for both parameters in the Terraform
   699  configuration, and although they happen to have the same value, Terraform
   700  configuration validation requires them to be mutually exclusive. Currently, we
   701  observe this behaviour at runtime, and upon observing that the resource cannot
   702  transition to the `Ready` state and acquires the Terraform validation error
   703  message in its `status.conditions`, we do the `LateInitializer.IgnoreFields`
   704  custom configuration detailed above to skip one of the mutually exclusive fields
   705  during late-initialization.
   706  
   707  ## Overriding Terraform Resource Schema
   708  
   709  Upjet generates Crossplane resource schemas (CR spec/status) using the
   710  [Terraform schema of the resource]. As of today, Upjet leverages the following
   711  attributes in the schema:
   712  
   713  - [Type] and [Elem] to identify the type of the field.
   714  - [Sensitive] to see if we need to keep it in a Secret instead of CR.
   715  - [Description] to add as a description to the field in CRD.
   716  - [Optional] and [Computed] to identify whether the fields go under spec or
   717    status:
   718    - Not Optional & Not Computed => Spec (required)
   719    - Optional & Not Computed => Spec (optional)
   720    - Optional & Computed => Spec (optional, to be late-initialized)
   721    - Not Optional & Computed => Status
   722  
   723  Usually, we don't need to make any modifications in the resource schema and
   724  resource schema just works as is. However, there could be some rare edge cases
   725  like:
   726  
   727  - Field contains sensitive information but not marked as `Sensitive` or vice
   728    versa.
   729  - An attribute does not make sense to have in CRD schema, like [tags_all for jet
   730    AWS resources].
   731  - Moving parameters from Terraform provider config to resources schema to fit
   732    Crossplane model, e.g. [AWS region] parameter is part of provider config in
   733    Terraform but Crossplane expects it in CR spec.
   734  
   735  Schema of a resource could be overridden as follows:
   736  
   737  ```go
   738  p.AddResourceConfigurator("aws_autoscaling_group", func(r *config.Resource) {
   739      // Managed by Attachment resource.
   740      if s, ok := r.TerraformResource.Schema["load_balancers"]; ok {
   741          s.Optional = false
   742          s.Computed = true
   743      }
   744      if s, ok := r.TerraformResource.Schema["target_group_arns"]; ok {
   745          s.Optional = false
   746          s.Computed = true
   747      }
   748  })
   749  ```
   750  
   751  ## Initializers
   752  
   753  Initializers involve the operations that run before beginning of reconciliation.
   754  This configuration option will provide that setting initializers for per
   755  resource.
   756  
   757  Many resources in aws have `tags` field in their schema. Also, in Crossplane
   758  there is a [tagging convention]. To implement the tagging convention for jet-aws
   759  provider, this initializer configuration support was provided.
   760  
   761  There is a common struct (`Tagger`) in upjet to use the tagging convention:
   762  
   763  ```go
   764  // Tagger implements the Initialize function to set external tags
   765  type Tagger struct {
   766   kube      client.Client
   767   fieldName string
   768  }
   769  
   770  // NewTagger returns a Tagger object.
   771  func NewTagger(kube client.Client, fieldName string) *Tagger {
   772   return &Tagger{kube: kube, fieldName: fieldName}
   773  }
   774  
   775  // Initialize is a custom initializer for setting external tags
   776  func (t *Tagger) Initialize(ctx context.Context, mg xpresource.Managed) error {
   777   paved, err := fieldpath.PaveObject(mg)
   778   if err != nil {
   779    return err
   780   }
   781   pavedByte, err := setExternalTagsWithPaved(xpresource.GetExternalTags(mg), paved, t.fieldName)
   782   if err != nil {
   783    return err
   784   }
   785   if err := json.Unmarshal(pavedByte, mg); err != nil {
   786    return err
   787   }
   788   if err := t.kube.Update(ctx, mg); err != nil {
   789    return err
   790   }
   791   return nil
   792  }
   793  ```
   794  
   795  As seen above, the `Tagger` struct accepts a `fieldName`. This `fieldName`
   796  specifies which value of field to set in the resource's spec. You can use the
   797  common `Initializer` by specifying the field name that points to the external
   798  tags in the configured resource.
   799  
   800  There is also a default initializer for tagging convention, `TagInitializer`. It
   801  sets the value of `fieldName` to `tags` as default:
   802  
   803  ```go
   804  // TagInitializer returns a tagger to use default tag initializer.
   805  var TagInitializer NewInitializerFn = func(client client.Client) managed.Initializer {
   806   return NewTagger(client, "tags")
   807  }
   808  ```
   809  
   810  In jet-aws provider, as a default process, if a resource has `tags` field in its
   811  schema, then the default initializer (`TagInitializer`) is added to Initializer
   812  list of resource:
   813  
   814  ```go
   815  // AddExternalTagsField adds ExternalTagsFieldName configuration for resources that have tags field.
   816  func AddExternalTagsField() tjconfig.ResourceOption {
   817   return func(r *tjconfig.Resource) {
   818    if s, ok := r.TerraformResource.Schema["tags"]; ok && s.Type == schema.TypeMap {
   819     r.InitializerFns = append(r.InitializerFns, tjconfig.TagInitializer)
   820    }
   821   }
   822  }
   823  ```
   824  
   825  However, if the field name that used for the external label is different from
   826  the `tags`, the `NewTagger` function can be called and the specific `fieldName`
   827  can be passed to this:
   828  
   829  ```go
   830  r.InitializerFns = append(r.InitializerFns, func(client client.Client) managed.Initializer {
   831   return tjconfig.NewTagger(client, "example_tags_name")
   832  })
   833  ```
   834  
   835  If the above tagging convention logic does not work for you, and you want to use
   836  this configuration option for a reason other than tagging convention (for
   837  another custom initializer operation), you need to write your own struct in
   838  provider and have this struct implement the `Initializer` function with a custom
   839  logic.
   840  
   841  This configuration option is set by using the [InitializerFns] field that is a
   842  list of [NewInitializerFn]:
   843  
   844  ```go
   845  // NewInitializerFn returns the Initializer with a client.
   846  type NewInitializerFn func(client client.Client) managed.Initializer
   847  ```
   848  
   849  Initializer is an interface in [crossplane-runtime]:
   850  
   851  ```go
   852  type Initializer interface {
   853   Initialize(ctx context.Context, mg resource.Managed) error
   854  }
   855  ```
   856  
   857  So, an interface must be passed to the related configuration field for adding
   858  initializers for a resource.
   859  
   860  [Upjet]: https://github.com/crossplane/upjet
   861  [External name]: #external-name
   862  [Cross Resource Referencing]: #cross-resource-referencing
   863  [Additional Sensitive Fields and Custom Connection Details]: #additional-sensitive-fields-and-custom-connection-details
   864  [Late Initialization Behavior]: #late-initialization-configuration
   865  [Overriding Terraform Resource Schema]: #overriding-terraform-resource-schema
   866  [the external name documentation]: https://docs.crossplane.io/master/concepts/managed-resources/#naming-external-resources
   867  [import section]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_access_key#import
   868  [the types for the External Name configuration]: https://github.com/crossplane/upjet/blob/main/pkg/config/resource.go#L68
   869  [aws_iam_user]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_user
   870  [NameAsIdentifier]: https://github.com/crossplane/upjet/blob/main/pkg/config/externalname.go#L28
   871  [aws_s3_bucket]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket
   872  [import section of s3 bucket]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket#import
   873  [bucket]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket#bucket
   874  [cluster_identifier]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/rds_cluster#cluster_identifier
   875  [aws_rds_cluster]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/rds_cluster.
   876  [import section of aws_vpc]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/vpc#import
   877  [arguments list]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/vpc#argument-reference
   878  [example usages]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/vpc#example-usage
   879  [IdentifierFromProvider]: https://github.com/crossplane/upjet/blob/main/pkg/config/externalname.go#L42
   880  [a similar identifier]: https://www.terraform.io/docs/glossary#id
   881  [import section of azurerm_sql_server]: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/sql_server#import
   882  [handle dependencies]: https://docs.crossplane.io/master/concepts/managed-resources/#referencing-other-resources
   883  [user]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_access_key#user
   884  [generate reference resolution methods]: https://github.com/crossplane/crossplane-tools/pull/35
   885  [configuration]: https://github.com/crossplane/upjet/blob/main/pkg/config/resource.go#L123
   886  [iam_access_key]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_access_key#argument-reference
   887  [kms key]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ebs_volume#kms_key_id
   888  [connection details]: https://docs.crossplane.io/master/concepts/managed-resources/#writeconnectionsecrettoref
   889  [id]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_access_key#id
   890  [secret]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_access_key#secret
   891  [`external.Observe`]: https://github.com/crossplane/upjet/blob/main/pkg/controller/external.go#L175
   892  [late-initialization customization API]: https://github.com/crossplane/upjet/blob/main/pkg/resource/lateinit.go#L45
   893  [`address_prefix`]: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/subnet#address_prefix
   894  [Terraform schema of the resource]: https://github.com/hashicorp/terraform-plugin-sdk/blob/e3325b095ef501cf551f7935254ce942c44c1af0/helper/schema/schema.go#L34
   895  [Type]: https://github.com/hashicorp/terraform-plugin-sdk/blob/e3325b095ef501cf551f7935254ce942c44c1af0/helper/schema/schema.go#L52
   896  [Elem]: https://github.com/hashicorp/terraform-plugin-sdk/blob/e3325b095ef501cf551f7935254ce942c44c1af0/helper/schema/schema.go#L151
   897  [Sensitive]: https://github.com/hashicorp/terraform-plugin-sdk/blob/e3325b095ef501cf551f7935254ce942c44c1af0/helper/schema/schema.go#L244
   898  [Description]: https://github.com/hashicorp/terraform-plugin-sdk/blob/e3325b095ef501cf551f7935254ce942c44c1af0/helper/schema/schema.go#L120
   899  [Optional]: https://github.com/hashicorp/terraform-plugin-sdk/blob/e3325b095ef501cf551f7935254ce942c44c1af0/helper/schema/schema.go#L80
   900  [Computed]: https://github.com/hashicorp/terraform-plugin-sdk/blob/e3325b095ef501cf551f7935254ce942c44c1af0/helper/schema/schema.go#L139
   901  [tags_all for jet AWS resources]: https://github.com/upbound/provider-aws/blob/main/config/overrides.go#L62
   902  [AWS region]: https://github.com/upbound/provider-aws/blob/main/config/overrides.go#L32
   903  [this figure]: ../docs/images/upjet-externalname.png
   904  [Initializers]: #initializers
   905  [InitializerFns]: https://github.com/crossplane/upjet/blob/main/pkg/config/resource.go#L297
   906  [NewInitializerFn]: https://github.com/crossplane/upjet/blob/main/pkg/config/resource.go#L210
   907  [crossplane-runtime]: https://github.com/crossplane/crossplane-runtime/blob/428b7c3903756bb0dcf5330f40298e1fa0c34301/pkg/reconciler/managed/reconciler.go#L138
   908  [tagging convention]: https://github.com/crossplane/crossplane/blob/60c7df9/design/one-pager-managed-resource-api-design.md#external-resource-labeling