github.com/crowdsecurity/crowdsec@v1.6.1/pkg/setup/README.md (about)

     1  
     2  > **_NOTE_**: The following document describes an experimental, work-in-progress feature. To enable the `cscli setup` command, set the environment variable `CROWDSEC_FEATURE_CSCLI_SETUP=true` or add the line " - cscli_setup" to `/etc/crowdsec/feature.yaml`. Any feedback is welcome.
     3  
     4  ---
     5  
     6  # cscli setup
     7  
     8  The "cscli setup" command can configure a crowdsec instance based on the services that are installed or running on the server.
     9  
    10  There are three main subcommands:
    11  
    12  - `cscli setup detect`: *detect* the services, the OS family, version or the Linux distribution
    13  - `cscli setup install-hub`: *install* the recommended collections, parsers, etc. based on the detection result
    14  - `cscli setup datasources`: *generate* the appropriate acquisition rules
    15  
    16  The setup command is used in the `wizard.sh` script, but can also be invoked by hand or customized via a configuration file
    17  by adding new services, log locations and detection rules.
    18  
    19  Detection and installation are performed as separate steps, as you can see in the following diagram:
    20  
    21  ```
    22   +-------------+
    23   |             |
    24   | detect.yaml |
    25   |             |
    26   +-------------+
    27          |
    28          v
    29    setup detect
    30          |
    31          v
    32   +--------------+
    33   |              +---> setup install-hub     +-----------------------+
    34   |  setup.yaml  |                           |                       |
    35   |              +---> setup datasources --->| etc/crowdsec/acquis.d |
    36   +--------------+                           |                       |
    37                                              +-----------------------+
    38  ```
    39  
    40  You can inspect and customize the intermediary file (`setup.yaml`), which is useful
    41  in case of many instances, deployment automation or unusual setups.
    42  
    43  A subcommand can be used to check your changes in this case:
    44  
    45  - `cscli setup validate`: *validate* or report errors on a setup file
    46  
    47  ## Basic usage
    48  
    49  Identify the existing services and write out what was detected:
    50  
    51  ```console
    52  # cscli setup detect > setup.yaml
    53  ```
    54  
    55  See what was found.
    56  
    57  ```console
    58  # cscli setup install-hub setup.yaml --dry-run
    59  dry-run: would install collection crowdsecurity/apache2
    60  dry-run: would install collection crowdsecurity/linux
    61  dry-run: would install collection crowdsecurity/pgsql
    62  dry-run: would install parser crowdsecurity/whitelists
    63  ```
    64  
    65  Install the objects (parsers, scenarios...) required to support the detected services:
    66  
    67  ```console
    68  # cscli setup install-hub setup.yaml
    69  INFO[29-06-2022 03:16:14 PM] crowdsecurity/apache2-logs : OK              
    70  INFO[29-06-2022 03:16:14 PM] Enabled parsers : crowdsecurity/apache2-logs 
    71  INFO[29-06-2022 03:16:14 PM] crowdsecurity/http-logs : OK             
    72  [...]
    73  INFO[29-06-2022 03:16:18 PM] Enabled crowdsecurity/linux      
    74  ```
    75  
    76  Generate the datasource configuration:
    77  
    78  ```console
    79  # cscli setup datasources setup.yaml --to-dir /etc/crowdsec/acquis.d
    80  ```
    81  
    82  With the above command, each detected service gets a corresponding file in the
    83  `acquis.d` directory. Running `cscli setup` again may add more services as they
    84  are detected, but datasource files or hub items are never removed
    85  automatically.
    86  
    87  
    88  ## The detect.yaml file
    89  
    90  A detect.yaml file is downloaded when you first install crowdsec, and is updated by the `cscli hub update`
    91  command.
    92  
    93  > **_NOTE_**: XXX XXX - this is currently not the case, the file is distributed in the crowdsec repository, but it should change.
    94  
    95  You can see the default location with `cscli setup detect --help | grep detect-config`
    96  
    97  The YAML file contains a version number (always 1.0) and a list of sections, one per supported service.
    98  
    99  Each service defines its detection rules, the recommended hub items and
   100  recommended datasources. The same software can be defined in multiple service
   101  sections: for example, apache on debian and fedora have different detection
   102  rules and different datasources so it requires two sections to support both platforms.
   103  
   104  The following are minimal `detect.yaml` examples just to show a few concepts.
   105  
   106  ```yaml
   107  version: 1.0
   108  
   109  services:
   110  
   111    apache2:
   112      when:
   113        - ProcessRunning("apache2")
   114      install:
   115        collections:
   116          - crowdsecurity/apache2
   117      datasources:
   118        source: file
   119        labels:
   120          type: apache2
   121        filenames:
   122          - /var/log/apache2/*.log
   123          - /var/log/httpd/*.log
   124  ```
   125  
   126  
   127  - `ProcessRunning()` matches the process name of a running application. The
   128  `when:` clause can contain any number of expressions, they are all evaluated
   129  and must all return true for a service to be detected (implied *and* clause, no
   130  short-circuit). A missing or empty `when:` section is evaluated as true.
   131  The [expression
   132  engine](https://github.com/antonmedv/expr/blob/master/docs/language-definition.md)
   133  is the same one used by CrowdSec parser filters. You can force the detection of
   134  a process by using the `cscli setup detect... --force-process <processname>`
   135  flag. It will always behave as if `<processname>` was running.
   136  
   137  The `install:` section can contain any number of collections, parsers, scenarios
   138  and postoverflows. In practices, it's most often a single collection.
   139  
   140  The `datasource:` section is copied as-is in the acquisition file.
   141  
   142  > **_NOTE_**: XXX TODO - the current version does not validate the `datasource:` mapping. Bad content is written to acquis.d until crowdsec chokes on it.
   143  
   144  Detecting a running process may seem a good idea, but if a process manager like
   145  systemd is available it's better to ask it for the information we want.
   146  
   147  
   148  ```yaml
   149  version: 1.0
   150  
   151  services:
   152  
   153    apache2-systemd:
   154      when:
   155        - UnitFound("apache2.service")
   156        - OS.ID != "centos"
   157      install:
   158        collections:
   159          - crowdsecurity/apache2
   160      datasource:
   161        source: file
   162        labels:
   163          type: syslog
   164        filenames:
   165          - /var/log/apache2/*.log
   166  
   167    apache2-systemd-centos:
   168      when:
   169        - UnitFound("httpd.service")
   170        - OS.ID == "centos"
   171      install:
   172        collections:
   173          - crowdsecurity/apache2
   174      datasource:
   175        source: file
   176        labels:
   177          type: syslog
   178        filenames:
   179          - /var/log/httpd/*.log
   180  ```
   181  
   182  Here we see two more detection methods:
   183  
   184  - `UnitFound()` matches the name of systemd units, if the are in state enabled,
   185    generated or static. You can see here that CentOS is using a different unit
   186    name for Apache so it must have its own service section. You can force the
   187    detection of a unit by using the `cscli setup detect... --force-unit <unitname>` flag.
   188  
   189  - OS.Family, OS.ID and OS.RawVersion are read from /etc/os-release in case of
   190    Linux, and detected by other methods for FreeBSD and Windows. Under FreeBSD
   191    and Windows, the value of OS.ID is the same as OS.Family. If OS detection
   192    fails, it can be overridden with the flags `--force-os-family`, `--force-os-id`
   193    and `--force-os-version`.
   194  
   195  If you want to ignore one or more services (i.e. not install anything and not
   196  generate acquisition rules) you can specify it with `cscli setup detect...
   197  --skip-service <servicename>`. For example, `--skip-service apache2-systemd`.
   198  If you want to disable systemd unit detection, use `cscli setup detect... --snub-systemd`.
   199  
   200  If you used the `--force-process` or `--force-unit` flags, but none of the
   201  defined services is looking for them, you'll have an error like "detecting
   202  services: process(es) forced but not supported".
   203  
   204  > **_NOTE_**: XXX XXX - having an error for this is maybe too much, but can tell that a configuration is outdated. Could this be a warning with optional flag to make it an error?
   205  
   206  We used the `OS.ID` value to check for the linux distribution, but since the same configuration
   207  is required for CentOS and the other RedHat derivatives, it's better to check for the existence
   208  of a file that is known to exist in all of them:
   209  
   210  ```yaml
   211  version: 1.0
   212  
   213  services:
   214  
   215    apache2-systemd-deb:
   216      when:
   217        - UnitFound("apache2.service")
   218        - PathExists("/etc/debian_version")
   219      install:
   220      # [...]
   221  
   222    apache2-systemd-rpm:
   223      when:
   224        - UnitFound("httpd.service")
   225        - PathExists("/etc/redhat-release")
   226      install:
   227      # [...]
   228  ```
   229  
   230  - `PathExists()` evaluates to true if a file, directory or link exists at the
   231    given path. It does not check for broken links.
   232  
   233  
   234  
   235  Rules can be used to detect operating systems and environments:
   236  
   237  ```yaml
   238  version: 1.0
   239  
   240  services:
   241  
   242    linux:
   243      when:
   244        - OS.Family == "linux"
   245      install:
   246        collections:
   247          - crowdsecurity/linux
   248      datasource:
   249        type: file
   250        labels:
   251          type: syslog
   252        log_files:
   253        - /var/log/syslog
   254        - /var/log/kern.log
   255        - /var/log/messages
   256  
   257    freebsd:
   258      when:
   259        - OS.Family == "freebsd"
   260      install:
   261        collections:
   262          - crowdsecurity/freebsd
   263  
   264    windows:
   265      when:
   266        - OS.Family == "windows"
   267      install:
   268        collections:
   269          - crowdsecurity/windows
   270  ```
   271  
   272  The OS object contains a methods to check for version numbers:
   273  `OS.VersionCheck("<constraint>")`. It uses the
   274  [Masterminds/semver](https://github.com/Masterminds/semver) package and accepts
   275  a variety of operators.
   276  
   277  Instead of: OS.RawVersion == "1.2.3" you should use `OS.VersionCheck("~1")`,
   278  `OS.VersionCheck("~1.2")` depending if you want to match the major or the minor
   279  version. It's unlikely that you need to match the exact patch level.
   280  
   281  Leading zeroes are permitted, to allow comparison of Ubuntu versions: strict semver rules would treat "22.04" as invalid.
   282  
   283  
   284  # The `setup.yaml` file
   285  
   286  This file does not actually have a specific name, as it's usually written to standard output.
   287  
   288  For example, on a Debian system running Apache under systemd you can execute:
   289  
   290  ```console
   291  $ cscli setup detect --yaml
   292  setup:
   293    - detected_service: apache2-systemd-deb
   294      install:
   295        collections:
   296          - crowdsecurity/apache2
   297      datasource:
   298        filenames:
   299          - /var/log/apache2/*.log
   300        labels:
   301          type: apache2
   302    - detected_service: linux
   303      install:
   304        collections:
   305          - crowdsecurity/linux
   306      datasource:
   307        filenames:
   308          - /var/log/syslog
   309          - /var/log/kern.log
   310          - /var/log/messages
   311        labels:
   312          type: syslog
   313    - detected_service: whitelists
   314      install:
   315        parsers:
   316          - crowdsecurity/whitelists
   317  ```
   318  
   319  The default output format is JSON, which is compatible with YAML but less readable to humans.
   320  
   321   - `detected_service`: used to generate a name for the files written to `acquis.d`
   322   - `install`: can contain collections, parsers, scenarios, postoverflows
   323   - `datasource`: copied to `acquis.d`
   324  
   325  
   326  ```console
   327  $ cscli setup datasources --help
   328  generate datasource (acquisition) configuration from a setup file
   329  
   330  Usage:
   331    cscli setup datasources [setup_file] [flags]
   332  
   333  Flags:
   334    -h, --help            help for datasources
   335        --to-dir string   write the configuration to a directory, in multiple files
   336  [...]
   337  ```
   338  
   339  If the `--to-dir` option is not specified, a single monolithic `acquis.yaml` is printed to the standard output.
   340