github.com/anth0d/nomad@v0.0.0-20221214183521-ae3a0a2cad06/website/content/tools/autoscaling/concepts/policy-eval/checks.mdx (about)

     1  ---
     2  layout: docs
     3  page_title: Checks
     4  description: Learn about how the Autoscaler deals with policy checks.
     5  ---
     6  
     7  # Scaling Policy Checks
     8  
     9  A scaling policy can include several [checks][policy_check] all of which
    10  produce a scaling suggestion. Each check can specify its own source of metrics
    11  data and apply different strategies based on the desired outcome.
    12  
    13  ```hcl
    14  policy {
    15    # ...
    16    check "cpu_allocated_percentage" {
    17      source = "prometheus"
    18      query  = "..."
    19  
    20      strategy "target-value" {
    21        target = 70
    22      }
    23    }
    24  
    25    check "high-memory-usage" {
    26      source = "prometheus"
    27      query  = "..."
    28      group  = "memory-usage"
    29  
    30      strategy "threshold" {
    31        upper_bound = 100
    32        lower_bound = 70
    33        delta       = 1
    34      }
    35    }
    36  
    37    check "low-memory-usage" {
    38      source = "prometheus"
    39      query  = "..."
    40      group  = "memory-usage"
    41  
    42      strategy "threshold" {
    43        upper_bound = 30
    44        lower_bound = 0
    45        delta       = -1
    46      }
    47    }
    48  }
    49  ```
    50  
    51  ## Resolving Conflicts
    52  
    53  The checks are all executed at the same time during a policy evaluation and
    54  they can generate conflicting scaling actions. In a scenario like this, the
    55  Autoscaler iterates over the results and chooses the safest option, which is
    56  defined as the action that results in retaining the most capacity of the
    57  resource.
    58  
    59  In a scenario where two checks return different desired scaling directions, the
    60  following logic is applied.
    61  
    62  - `ScaleOut and ScaleIn => ScaleOut`
    63  - `ScaleOut and ScaleNone => ScaleOut`
    64  - `ScaleIn and ScaleNone => ScaleNone`
    65  
    66  In situations where the same actions are suggested, but with different counts
    67  the following logic is applied, where the count is the final desired value.
    68  
    69  - `ScaleOut(10) and ScaleOut(9) => ScaleOut(10)`
    70  - `ScaleIn(3) and ScaleIn(4) => ScaleIn(4)`
    71  
    72  ## Check Grouping
    73  
    74  The above logic for resolving conflicts only works when the checks are
    75  independent from each other. If you use the same `query` in multiple `check`
    76  blocks, or if the underlying data being queried is somehow correlated, only
    77  one check will result in a scaling action.
    78  
    79  In the example above, the `high-memory-usage` and `low-memory-usage` checks use
    80  the same query to retrieve memory usage information. We expect that memory
    81  usage is either low or high (or neither), but never both at the same time.
    82  
    83  Without grouping the target is never be able to reduce its count, since the
    84  possible resulting actions and the final scaling outcome can only be one of the
    85  following:
    86  
    87  - `ScaleOut and ScaleNone => ScaleOut`
    88  - `ScaleIn and ScaleNone => ScaleNone`
    89  - `ScaleNone and ScaleNone => ScaleNone`
    90  
    91  To fix this problem, the correlated checks need to be set to the same `group`.
    92  The Nomad Autoscaler then computes a single scaling action for the entire group
    93  by applying a slightly different logic:
    94  
    95  - `ScaleOut and ScaleIn => ScaleOut`
    96  - `ScaleOut and ScaleNone => ScaleOut`
    97  - `ScaleIn and ScaleNone => ScaleIn`
    98  - `ScaleNone and ScaleNone => ScaleNone`
    99  
   100  `ScaleNone` results are ignored unless all checks in the group return it and so
   101  a group is able to `ScaleIn` a target even when all other checks results in no
   102  action.
   103  
   104  [policy_check]: /tools/autoscaling/policy#check-options