github.com/quay/claircore@v1.5.28/docs/howto/add_dist.md (about)

     1  # Adding Distribution And Language Support
     2  
     3  *Note: If terms in this document sound foreign check out [Getting Started](../getting_started.md) to acquaint yourself with "indexing", "scanning", and "matching"*
     4  
     5  The claircore team is always open to adding more distributions and languages to the library.
     6  
     7  Generally, distributions or languages must provide a security tracker. 
     8  
     9  All officially supported distributions and languages provide a database of security vulnerabilities. 
    10  
    11  These databases are maintained by the distribution or language developers and reflect up-to-date CVE and advisory data for their packages.
    12  
    13  If your distribution or language does not provide a security tracker or piggy-backs off another distribution's start an issue in our Github issue tracker to discuss further.
    14  
    15  ## Implementing an Updater
    16  
    17  The first step to adding your distribution or language to claircore is getting your security tracker's data ingested by Libvuln. 
    18  
    19  The Updater interfaces are responsible for this task.
    20  
    21  An implementer must consider several design points:
    22  * Does the security database provide enough information to parse each entry into a claircore.Vulnerability?
    23      * Each entry must parse into a claircore.Vulnerability. 
    24      * Each Vulnerability **must** contain a package **and** a repository **or** distribution field. 
    25  * Will the Updater need to be configured at runtime?
    26      * Your updater may implement the Configurable interface. Your matcher will have its "Configuration" method called before use, giving you an opportunity for run time configuration.
    27  * What fields in a parsed claircore.Vulnerability will be present when indexing layer artifacts.
    28      * When implementing an updater you must keep in mind how packages/distributions/repositories will be parsed during index.
    29  	  When indexing a layer a common data model **must** exist between the possible package/distribution/repository and the parsed Vulnerabilitie's package/distribution/repository fields.
    30  
    31  If you are having trouble figuring out these requirements do not hesitate to reach out to us for help. 
    32  
    33  After you have taken the design points into consideration, you are ready to implement your updater.
    34  
    35  Typically you will create a new package named after the source you are adding support for. 
    36  
    37  Inside this package you can begin implementing the [Updater](../reference/updater.md) and [Updater Set Factory](../reference/updatersetfactory.md) interfaces.
    38  
    39  Optionally you may implement the [Configurable](../reference/updater.md) interface if you need runtime configuration.
    40  
    41  It will undoubtly be helpful to look at the examples in the "ubuntu", "rhel", and "debian" packages to get yourself started.
    42  
    43  ## Implementing a Package Scanner
    44  
    45  At this point you hopefully have your Updater working, writing vulnerability data into Libvuln's database. 
    46  
    47  We can now move our attention to package scanning.
    48  
    49  A package scanner is responsible for taking a claircore.Layer and parsing the contents for a particular package database or set of files inside the provided tar archive.
    50  Once the target files are located the package scanner should parse these files into claircore.Packages and return a slice of these data structures. 
    51  
    52  Package scanning is context free, meaning no distribution classification has happened yet.
    53  This is because manifests are made up of layers, and a layer which holds a package database may not hold distribution information such as an os-release file.
    54  A package scanner need only parse a target package database and return claircore.Packages.
    55  
    56  You need to implement the [Package Scanner](../reference/packagescanner.md) interface to achieve this.
    57  
    58  Optionally, you may implement the [Configurable Scanner](../reference/configurable_scanner.md) if the scanner needs to perform runtime configuration before use.
    59  
    60  Keep in mind that its very common for distributions to utilize an existing package manager such as RPM. 
    61  
    62  If this is the case there's a high likelihood that you can utilize the existing "rpm" or "dpkg" package scanner implementations.
    63  
    64  ## Implementing a Distribution Scanner
    65  
    66  Once the package scanner is implemented, tested, and working you can begin implementing a Distribution Scanner.
    67  
    68  Implementing a distribution scanner is a design choice.
    69  Distributions and repositories are the way claircore matches packages to vulnerabilities. 
    70  
    71  If your implemented Updater parses vulnerabilities with distribution information you will likely need to implement a distribution scanner.
    72  Likewise, if your Updater parses vulnerabilities with repository information (typical with language vulnerabilities) you will likely need to implement a repository scanner.
    73  
    74  A distribution scanner, like a package scanner, is provided a claircore.Layer. 
    75  
    76  The distribution scanner will parse the provided tar archive exhaustively searching for any clue that this layer was derived from your distribution.
    77  If you identify that it is, you should return a common distribution used by your Updater implementation.
    78  This ensures that claircore can match the output of your distribution scanner with your parsed vulnerabilities. 
    79  
    80  Optionally, you may implement the [Configurable Scanner](../reference/configurable_scanner.md) if the scanner needs to perform runtime configuration before use.
    81  
    82  ## Implementing a Repository Scanner
    83  
    84  As mentioned above, implementing a repository scanner is a design choice, often times applicable for language package managers.
    85  
    86  If your Updater parses vulnerabilities with a repository field you will likely want to implement a repository scanner.
    87  
    88  A repository scanner is used just like a distribution scanner however you will search for any clues that a layer contains your repository and if so return a common data model identifying the repository.
    89  
    90  Optionally, you may implement the [Configurable Scanner](../reference/configurable_scanner.md) if the scanner needs to perform runtime configuration before use.
    91  
    92  ## Implementing a Coalescer
    93  
    94  As you may have noticed, the process of scanning a layer for packages, distribution, and repository information is distinct and separate.
    95  
    96  At some point, claircore will need to take all the context-free information returned from layer scanners and create a complete view of the manifest.
    97  A coalescer performs this computation. 
    98  
    99  It's unlikely you will need to implement your own coalescer.
   100  Claircore provides a default "linux" coalescer which will work if your package database is rewritten when modified.
   101  For example, if a Dockerfile's `RUN` command causes a change to to dpkg's `/var/lib/dpkg/status` database, the resulting manifest will have a copy placed in the associated layer.
   102  
   103  However, if your package database does not fit into this model, implementing a coalescer may be necessary.
   104  
   105  To implement a coalescer, several details must be understood:
   106  * Each layer only provides a "piece" of the final manifest.
   107      * Because manifests are comprised of multiple copy-on-write layers, some layers may contain package information, distribution information, repository information, any combination of those, or no information at all.
   108  * An OS may have a "dist-upgrade" performed and the implications of this on the package management system is distribution or language dependent.
   109      * The coalescer must deal with distribution upgrades in a sane way.
   110  	  If your distribution or language does a dist-upgrade, are all packages bumped?
   111  	  Are they simply left alone?
   112  	  The coalescer must understand what happens and compute the final manifest's content correctly.
   113  * Packages may be removed and added between layers.
   114      * When the package database is a regular file on disk, this case is simpler: the database file found in the most recent layer holds the ultimate set of packages for all previous layers.
   115  	  However, in the case where the package database is realized by adding and removing files on disk it becomes trickier.
   116  	  Claircore has no special handling of whiteout files, currently.
   117  	  We will address this in upcoming releases.
   118  
   119  If your distribution or language cannot utilize a default coalescer, you will need to implement the [Coalescer interface](../reference/coalescer.md)
   120  
   121  ## Implementing or Adding To An Ecosystem
   122  
   123  An Ecosystem provides a set of coalescers, package scanners, distribution scanners, and repository scanners to Libindex at the time of indexing.
   124  
   125  Libindex will take the [Ecosystem](../reference/ecosystem.md) and scan each layer with all provided scanners.
   126  When Libindex is ready to coalesce the results of each scanner into an [IndexReport](../reference/index_report.md) the provided coalescer is given the output of the configured scanners.
   127  
   128  This allows Libindex to segment the input to the coalescing step to particular scanners that a coalescer understands. 
   129  
   130  For instance, if we only wanted a (fictitious) Haskell coalescer to evaluate artifacts returned from a (fictitious) Haskell package and repository scanner we would create an ecosystem similar to:
   131  
   132  ```go
   133  {{#include ../howto_test.go:example}}
   134  ```
   135  This ensures that Libindex will only provide Haskell artifacts to the Haskell coalescer and avoid calling the coalescer with rpm packages for example.
   136  
   137  If your distribution uses an already implemented package manager such as "rpm" or "dpkg", it's likely you will simply add your scanners to the existing ecosystem in one of those packages.
   138  
   139  ## Alternative Implementations
   140  
   141  This how-to guide is a "perfect world" scenario.
   142  
   143  Working on claircore has made us realize that this domain is a bit messy.
   144  Security trackers are not developed with package managers in mind, security databases do not follow correct specs, distribution maintainers spin their own tools, etc.
   145  
   146  We understand that supporting your distribution or language may take some bending of claircore's architecture and business logic.
   147  If this is the case, start a conversation with us.
   148  We are open to design discussions.
   149  
   150  ## Getting Help
   151  
   152  At this point, you have implemented all the necessary components to integrate your distribution or language with claircore.
   153  
   154  If you struggle with the design phase or are getting stuck at the implementation phases do not hesitate to reach out to us.
   155  Here are some links:
   156  
   157  - [Clair SIG](https://groups.google.com/g/clair-dev?pli=1)
   158  - [Github Issues](https://github.com/quay/claircore)
   159  - [RedHat Issues](https://issues.redhat.com/projects/PROJQUAY)