github.com/Racer159/jackal@v0.32.7-0.20240401174413-0bd2339e4f2e/adr/0017-jackal-bundle.md (about)

     1  # 17. Bundles
     2  
     3  Date: 2023-06-13
     4  
     5  ## Status
     6  
     7  [Migrated](https://github.com/defenseunicorns/uds-cli)
     8  
     9  ## Context
    10  
    11  Orchestrating capabilities from multiple Jackal packages into meta-packages is a current weak point for Jackal. The core of Jackal was built around components as capabilities, but as Jackal packages have scaled, there has been a need to create a new boundary to manage these capabilities efficiently.
    12  
    13  Currently there is no official way to enable the deployment, publishing, pulling, and creation of multiple Jackal packages together, and due to this some in the community have resorted to patterns such as:
    14  
    15  ```yaml
    16  - name: init
    17    required: true
    18    files:
    19      - source: jackal-init-amd64-v0.27.0.tar.zst
    20        target: jackal-init-amd64-v0.27.0.tar.zst
    21    actions:
    22      onDeploy:
    23        after:
    24          - cmd: jackal package deploy jackal-init-amd64-v0.27.0.tar.zst --components git-server --confirm -l warn
    25  ```
    26  
    27  While this _does_ fulfill the need to deploy two packages in one command, it does so in such a way that is verbose within the `jackal.yaml`, brittle across Jackal versions, inefficient within the package structure (it doesn't share layers), and is difficult to use `variables` with.
    28  
    29  ### Proposed Solutions
    30  
    31  There are currently three proposed solutions to this, each with their own pros and cons:
    32  
    33  #### Jackal Bundle
    34  
    35  Uses OCI and a separate Jackal schema / declarative YAML definition to pull packages into a single artifact and orchestrate them together.
    36  
    37  Pros:
    38  
    39  - maintains efficient OCI layering / deduping of shared package resources
    40  - allows flexibility in defining what `jackal bundle` (or a separate command) would look like as its own command without polluting `jackal package`
    41  
    42  Cons:
    43  
    44  - variables set within packages with `setVariables` may be difficult to share across packages
    45  - package sources in bundles would be best to keep as OCI _only_, without support for local packages. This would help us ensure there are versions for packages and would help with efficiency by taking advantage of things like layer deduping.
    46  
    47  #### Super Jackal Packages
    48  
    49  Adds a packages key or another way to overlay packages into a larger package with the same internal structure as current Jackal packages.
    50  
    51  Pros:
    52  
    53  - packages would maintain the same syntax under `jackal package` between normal / meta packages.
    54  
    55  Cons:
    56  
    57  - it would be difficult to properly scope things like variables and helm chart names properly across packages.
    58  - this continues to add to `jackal package` making it more complex and harder to test
    59  
    60  #### Jackal Package Manager
    61  
    62  Uses a separate binary (not `jackal`) to pull and manage packages together - this would also include dependency declaration and resolution between packages.
    63  
    64  Pros:
    65  
    66  - this is a familiar/expressive way to solve the package problem and would be familiar to developers and system administrators
    67  - allows flexibility in defining what the package manager would look like as its own command without polluting `jackal package`
    68  
    69  Cons:
    70  
    71  - dependencies may be difficult to determine whether they are "installed"/"deployed" - particularly for pre-cluster resources
    72  - variables set within packages with `setVariables` may be difficult to share across packages
    73  - this would necessitate a separate binary with it's own CLI that would need its own release schedule and maintenance
    74  
    75  > :warning: **NOTE**: The package manager could also be made to be OCI-only but would come with the same OCI pros/cons.
    76  
    77  ## Decision
    78  
    79  > :warning: **NOTE**: This functionality was migrated to [uds-cli](https://github.com/defenseunicorns/uds-cli) - this ADR is kept here for historical purposes.
    80  
    81  The current proposition (subject to change before acceptance) is **Jackal Bundles**, which a following PR will focus on and create a POC of.
    82  
    83  In essense the `jackal-bundle.yaml` would look something like so:
    84  
    85  ```yaml
    86  metadata:
    87    name: omnibus
    88    description: an example Jackal bundle
    89    version: 0.0.1
    90    architecture: amd64
    91  
    92  packages:
    93    - repository: localhost:888/init
    94      ref: "###JACKAL_BNDL_TMPL_INIT_VERSION###"
    95      optional-components:
    96        - git-server
    97    - repository: ghcr.io/defenseunicorns/packages/dubbd
    98      ref: 0.0.1 # OCI spec compliant reference
    99      # arch is not needed as it will use w/e arch is set in the bundle's metadata
   100      optional-components: # by default, all required components will be included
   101        - "*" # include all optional components
   102    - repository: docker.io/<namespace>/<name>
   103      ref: 0.0.1
   104      optional-components:
   105        - preflight
   106        - aws-* # include all optional components that start with "aws-"
   107  ```
   108  
   109  Bundle would be a new top-level command in Jackal with a full compliment of sub-commands (mirroring the pattern of `jackal package`):
   110  
   111  - `jackal bundle create <path> -o oci://<reference>|<path>`
   112  - `jackal bundle deploy oci://<reference>|<path>`
   113  - `jackal bundle inspect oci://<reference>|<path>`
   114  - `jackal bundle list` --> will probably just show the same as `jackal package list`
   115  - ~~`jackal bundle publish`~~ --> Bundles will be OCI only, so there is no need for a publish command, `create` will handle that.
   116  - `jackal bundle pull oci://<reference> -o <dir>`
   117  - `jackal bundle remove oci://<reference>|<path>`
   118  
   119  ## Consequences
   120  
   121  This does add complexity to the Jackal codebase, as it is the addition of an entire suite of commands, JSON schema, schema docs, CLI docs, and a chunk of library code + tests.  It is a good litmus test of the current packager and OCI codebase to see how ready it is to be consumed as a library.
   122  
   123  Additionally, this does add a new layer of complexity to the Jackal ecosystem, as meta-package maintainers must now also be aware of this bundling process, syntax and schema.  This is a necessary evil however, as the current pattern of using `jackal package deploy` to deploy multiple packages is not sustainable at the scale we are seeing.
   124  
   125  There is also the hard requirement that packages bundled must be first published to a registry available to the person performing the bundle operation. This removes some ability to develop bundles on an air gapped environment, but the team believes that in such scenarios, the air gapped environment should be _receiving_ a bundle, rather than developing one internally.  If this assumption is incorrect however there are options for us to allow the creation of bundles from OCI directories on local systems if we need to or we could provide more provisions within Jackal to make it easier to connect to air gap registries to mirror bundles.