github.com/outbrain/consul@v1.4.5/website/source/docs/guides/semaphore.html.md (about)

     1  ---
     2  layout: "docs"
     3  page_title: "Semaphore"
     4  sidebar_current: "docs-guides-semaphore"
     5  description: |-
     6    This guide demonstrates how to implement a distributed semaphore using the Consul KV store.
     7  ---
     8  
     9  # Semaphore
    10  
    11  This guide demonstrates how to implement a distributed semaphore using the Consul
    12  KV store. This is useful when you want to coordinate many services while
    13  restricting access to certain resources.
    14  
    15  ~>  If you only need mutual exclusion or leader election,
    16  [this guide](/docs/guides/leader-election.html)
    17  provides a simpler algorithm that can be used instead.
    18  
    19  There are a number of ways that a semaphore can be built, so our goal is not to
    20  cover all the possible methods. Instead, we will focus on using Consul's support for
    21  [sessions](/docs/internals/sessions.html). Sessions allow us to build a system that
    22  can gracefully handle failures.
    23  
    24  -> **Note:** JSON output in this guide has been pretty-printed for easier reading. Actual values returned from the API will not be formatted.
    25  
    26  ## Contending Nodes
    27  
    28  Let's imagine we have a set of nodes who are attempting to acquire a slot in the
    29  semaphore. All nodes that are participating should agree on three decisions: the
    30  prefix in the KV store used to coordinate, a single key to use as a lock,
    31  and a limit on the number of slot holders.
    32  
    33  For the prefix we will be using for coordination, a good pattern is simply:
    34  
    35  ```text
    36  service/<service name>
    37  ```
    38  
    39  We'll abbreviate this pattern as simply `<prefix>` for the rest of this guide.
    40  
    41  The first step is for each contender to create a session. This is done using the
    42  [Session HTTP API](/api/session.html#session_create):
    43  
    44  ```text
    45  curl  -X PUT -d '{"Name": "db-semaphore"}' \
    46    http://localhost:8500/v1/session/create
    47   ```
    48  
    49  This will return a JSON object contain the session ID:
    50  
    51  ```text
    52  {
    53    "ID": "4ca8e74b-6350-7587-addf-a18084928f3c"
    54  }
    55  ```
    56  
    57  ->  **Note:** Sessions by default only make use of the gossip failure detector. That is, the session is considered held by a node as long as the default Serf health check has not declared the node unhealthy. Additional checks can be specified at session creation if desired.
    58  
    59  Next, we create a lock contender entry. Each contender creates a kv entry that is tied
    60  to a session. This is done so that if a contender is holding a slot and fails, its session
    61  is detached from the key, which can then be detected by the other contenders.
    62  
    63  Create the contender key by doing an `acquire` on `<prefix>/<session>` via `PUT`.
    64  This is something like:
    65  
    66  ```text
    67  curl -X PUT -d <body> http://localhost:8500/v1/kv/<prefix>/<session>?acquire=<session>
    68   ```
    69  
    70  `body` can be used to associate a meaningful value with the contender, such as its node’s name. 
    71  This body is opaque to Consul but can be useful for human operators.
    72  
    73  The `<session>` value is the ID returned by the call to
    74  [`/v1/session/create`](/api/session.html#session_create).
    75  
    76  The call will either return `true` or `false`. If `true`, the contender entry has been
    77  created. If `false`, the contender node was not created; it's likely that this indicates
    78  a session invalidation.
    79  
    80  The next step is to create a single key to coordinate which holders are currently
    81  reserving a slot. A good choice for this lock key is simply `<prefix>/.lock`. We will
    82  refer to this special coordinating key as `<lock>`.
    83  
    84  This is done with:
    85  
    86  ```text
    87  curl -X PUT -d <body> http://localhost:8500/v1/kv/<lock>?cas=0
    88   ```
    89  
    90  Since the lock is being created, a `cas` index of 0 is used so that the key is only put if it does not exist.
    91  
    92  `body` should contain both the intended slot limit for the semaphore and the session ids
    93  of the current holders (initially only of the creator). A simple JSON body like the following works:
    94  
    95  ```text
    96  {
    97      "Limit": 2,
    98      "Holders": [
    99        "<session>"
   100      ]
   101  }
   102  ```
   103  
   104  The current state of the semaphore is read by doing a `GET` on the entire `<prefix>`:
   105  
   106  ```text
   107  curl http://localhost:8500/v1/kv/<prefix>?recurse
   108   ```
   109  
   110  Within the list of the entries, we should find two keys: the `<lock>` and the
   111  contender key ‘<prefix>/<session>’. 
   112  
   113  ```text
   114  [
   115    {
   116      "LockIndex": 0,
   117      "Key": "<lock>",
   118      "Flags": 0,
   119      "Value": "eyJMaW1pdCI6IDIsIkhvbGRlcnMiOlsiPHNlc3Npb24+Il19",
   120      "Session": "",
   121      "CreateIndex": 898,
   122      "ModifyIndex": 901
   123    },
   124    {
   125      "LockIndex": 1,
   126      "Key": "<prefix>/<session>",
   127      "Flags": 0,
   128      "Value": null,
   129      "Session": "<session>",
   130      "CreateIndex": 897,
   131      "ModifyIndex": 897
   132    }
   133  ]
   134  ```
   135  Note that the `Value` we embedded into `<lock>` is Base64 encoded when returned by the API.
   136  
   137  When the `<lock>` is read and its `Value` is decoded, we can verify the `Limit` agrees with the `Holders` count. 
   138  This is used to detect a potential conflict. The next step is to determine which of the current
   139  slot holders are still alive. As part of the results of the `GET`, we also have all the contender
   140  entries. By scanning those entries, we create a set of all the `Session` values. Any of the
   141  `Holders` that are not in that set are pruned. In effect, we are creating a set of live contenders
   142  based on the list results and doing a set difference with the `Holders` to detect and prune
   143  any potentially failed holders. In this example `<session>` is present in `Holders` and 
   144  is attached to the key `<prefix>/<session>`, so no pruning is required.
   145  
   146  If the number of holders after pruning is less than the limit, a contender attempts acquisition
   147  by adding its own session to the `Holders` list and doing a Check-And-Set update of the `<lock>`. 
   148  This performs an optimistic update.
   149  
   150  This is done with:
   151  
   152  ```text
   153  curl -X PUT -d <Updated Lock Body> http://localhost:8500/v1/kv/<lock>?cas=<lock-modify-index>
   154   ```
   155  `lock-modify-index` is the latest `ModifyIndex` value known for `<lock>`, 901 in this example.
   156  
   157  If this request succeeds with `true`, the contender now holds a slot in the semaphore. 
   158  If this fails with `false`, then likely there was a race with another contender to acquire the slot.
   159  
   160  To re-attempt the acquisition, we watch for changes on `<prefix>`. This is because a slot
   161  may be released, a node may fail, etc. Watching for changes is done via a blocking query
   162  against `/kv/<prefix>?recurse`. 
   163  
   164  Slot holders **must** continously watch for changes to `<prefix>` since their slot can be 
   165  released by an operator or automatically released due to a false positive in the failure detector. 
   166  On changes to `<prefix>` the lock’s `Holders` list must be re-checked to ensure the slot
   167  is still held. Additionally, if the watch fails to connect the slot should be considered lost. 
   168  
   169  This semaphore system is purely *advisory*. Therefore it is up to the client to verify
   170  that a slot is held before (and during) execution of some critical operation.
   171  
   172  Lastly, if a slot holder ever wishes to release its slot voluntarily, it should be done by doing a
   173  Check-And-Set operation against `<lock>` to remove its session from the `Holders` object.
   174  Once that is done, both its contender key `<prefix>/<session>` and session should be deleted.