bitbucket.org/Aishee/synsec@v0.0.0-20210414005726-236fc01a153d/pkg/leakybucket/README.md (about)

     1  # Leakybuckets
     2  
     3  ## Bucket concepts
     4  
     5  Leakybucket is used for decision making. Under certain conditions
     6  enriched events are poured in these buckets. When these buckets are
     7  full, we raise a new event. After this event is raised the bucket is
     8  destroyed. There are many types of buckets, and we welcome any new
     9  useful design of buckets.
    10  
    11  Usually the bucket configuration generates the creation of many
    12  buckets. They are differenciated by a field called stackkey. When two
    13  events arrives with the same stackkey they go in the same matching
    14  bucket.
    15  
    16  The very purpose of these buckets is to detect clients that exceed a
    17  certain rate of attemps to do something (ssh connection, http
    18  authentication failure, etc...). Thus, the most use stackkey field is
    19  often the source_ip.
    20  
    21  ## Standard leaky buckets
    22  
    23  Default buckets have two main configuration options:
    24   * capacity: number of events the bucket can hold. When the capacity
    25     is reached and a new event is poured, a new event is raised. We
    26     call this type of event overflow. This is an int.
    27   * leakspeed: duration needed for an event to leak. When an event
    28     leaks, it disappear from the bucket.
    29  
    30  ## Trigger
    31  
    32  It's a special type of bucket with a zero capacity. Thus, when an
    33  event is poured in a trigger, it always raises an overflow.
    34  
    35  ## Uniq
    36  
    37  It's a bucket working as the standard leaky bucket except for one
    38  thing: a filter returns a property for each event and only one
    39  occurence of this property is allowed in the bucket, thus the bucket
    40  is called uniq.
    41  
    42  ## Counter
    43  
    44  It's a special type of bucket with an infinite capacity and an
    45  infinite leakspeed (it never overflows, neither leaks). Nevertheless,
    46  the event is raised after a fixed duration. The option is called
    47  duration.
    48  
    49  ## Available configuration options for buckets
    50  
    51  ### Fields for standard buckets
    52  
    53  * type: mandatory field. Must be one of "leaky", "trigger", "uniq" or
    54    "counter"
    55  * name: mandatory field, but the value is totally open. Nevertheless
    56    this value will tag the events raised by the bucket.
    57  * filter: mandatory field. It's a filter that is run when the decision
    58    to make an event match the bucket or not. The filter have to return
    59    a boolean. As a filter implementation we use
    60    https://github.com/antonmedv/expr
    61  * capacity: [mandatory for now, shouldn't be mandatory in the final
    62    version] it's the size of the bucket. When pouring in a bucket
    63    already with size events, it overflows.
    64  * leakspeed: leakspeed is a time duration (has to be parseable by
    65    https://golang.org/pkg/time/#ParseDuration). After each interval an
    66    event is leaked from the bucket.
    67  * stackkey: mandatory field. This field is used to differenciate on
    68    which bucket ongoing events will be poured. When an unknows stackkey
    69    is seen in an event a new bucekt is created.
    70  * on_overflow: optional field, that tells the what to do when the
    71    bucket is returning the overflow event. As of today, the possibility
    72    are these: "ban,1h", "Reprocess", "Delete".
    73    Reprocess is used to send the raised event back in the event pool to
    74    be matched agains buckets
    75  
    76  ### Fields for special buckets
    77  
    78  #### Uniq
    79  
    80  Uniq has an extra field uniq_filter which is too use the filter
    81  implementation from https://github.com/antonmedv/expr. The filter must
    82  return a string. All strins returned by this filter in the same
    83  buckets have to be different. Thus, if a string is seen twice it is
    84  dismissed.
    85  
    86  #### Trigger
    87  
    88  Capacity and leakspeed are not relevant for this kind of bucket.
    89  
    90  #### Counter
    91  
    92  It's a special kind of bucket that raise an event and is destroyed
    93  after a fixed duration. The configuration field used is duration and
    94  must be parseable by https://golang.org/pkg/time/#ParseDuration.
    95  Nevertheless, this kind of bucket is often used with an infinite
    96  leakspeed and an infinite capacity [capacity set to -1 for now].
    97  
    98  
    99  ## Add exemples here
   100  
   101  ```
   102  # ssh bruteforce
   103  - type: leaky
   104    name: ssh_bruteforce
   105    filter: "Meta.log_type == 'ssh_failed-auth'"
   106    leakspeed: "10s"
   107    capacity: 5
   108    stackkey: "source_ip"
   109    on_overflow: ban,1h
   110  
   111  # reporting of src_ip,dest_port seen
   112  - type: counter
   113    name: counter
   114    filter: "Meta.service == 'tcp' && Event.new_connection == 'true'"
   115    distinct: "Meta.source_ip + ':' + Meta.dest_port"
   116    duration: 5m
   117    capacity: -1
   118  
   119  - type: trigger
   120    name: "New connection"
   121    filter: "Meta.service == 'tcp' && Event.new_connection == 'true'"
   122    on_overflow: Reprocess
   123  ```
   124  
   125  # Note on leakybuckets implementation
   126  
   127  [This is not dry enough to have many details here, but:]
   128  
   129  The bucket code is triggered by `InfiniBucketify` in main.go.
   130  There's one struct called buckets which is for now a
   131  `map[string]interface{}` that holds all buckets. The key of this map
   132  is derivated from the filter configured for the bucket and its
   133  stackkey. This looks like complicated, but in fact it allows us to use
   134  only one structs. This is done in buckets.go.
   135  
   136  On top of that the implementation define only the standard leaky
   137  bucket. A goroutine is launched for every buckets (bucket.go). This
   138  goroutine manages the life of the bucket.
   139  
   140  For special buckets, hooks are defined at initialization time in
   141  manager.go. Hooks are called when relevant by the bucket gorourine
   142  when events are poured and/or when bucket overflows.