github.com/outbrain/consul@v1.4.5/website/source/docs/guides/geo-failover.html.md (about)

     1  ---
     2  layout: "docs"
     3  page_title: "Geo Failover"
     4  sidebar_current: "docs-guides-geo-failover"
     5  description: |-
     6    Consul provides a prepared query capability that makes it easy to implement automatic geo failover policies for services.
     7  ---
     8  
     9  # Geo Failover
    10  
    11  Within a datacenter, Consul provides automatic failover for services by omitting failed service instances from DNS lookups, and by providing service health information in APIs. When there are no more instances of a service available in the local datacenter, it can be challenging to implement failover policies to other datacenters because typically that logic would need to be written into each application.
    12  
    13  Fortunately, Consul has a [prepared query](/api/query.html) capability that lets users define failover policies in a centralized way. It's easy to expose these to applications using Consul's DNS interface and it's also available to applications that consume Consul's APIs. These policies range from fully static lists of alternate datacenters to fully dynamic policies that make use of Consul's [network coordinate](/docs/internals/coordinates.html) subsystem to automatically determine the next best datacenter to fail over to based on network round trip time. Prepared queries can be made with policies specific to certain services and prepared query templates allow one policy to apply to many, or even all services, with just a small number of templates.
    14  
    15  This guide shows how to build geo failover policies using prepared queries through a set of examples.
    16  
    17  ## Prepared Queries
    18  
    19  Prepared queries are objects that are defined at the datacenter level, similar to the values in Consul's KV store. They are created once and then invoked by applications to perform the query and get the latest results.
    20  
    21  Here's an example request to create a prepared query:
    22  
    23  ```
    24  $ curl \
    25      --request POST \
    26      --data \
    27  '{
    28    "Name": "api",
    29    "Service": {
    30      "Service": "api",
    31      "Tags": ["v1.2.3"]
    32    }
    33  }' http://127.0.0.1:8500/v1/query
    34  
    35  {"ID":"fe3b8d40-0ee0-8783-6cc2-ab1aa9bb16c1"}
    36  ```
    37  
    38  This creates a prepared query called "api" that does a lookup for all instances of the "api" service with the tag "v1.2.3". This policy could be used to control which version of a "api" applications should be using in a centralized way. By [updating this prepared query](/api/query.html#update-prepared-query) to look for the tag "v1.2.4" applications could start to find the newer version of the service without having to reconfigure anything.
    39  
    40  Applications can make use of this query in two ways. Since we gave the prepared query a name, they can simply do a DNS lookup for "api.query.consul" instead of "api.service.consul". Now with the prepared query, there's the additional filter policy working behind the scenes that the application doesn't have to know about. Queries can also be executed using the [prepared query execute API](/api/query.html#execute-prepared-query) for applications that integrate with Consul's APIs directly.
    41  
    42  ## Failover Policies
    43  
    44  Using the techniques in this section we will develop prepared queries with failover policies where simply changing application configurations to look up "api.query.consul" instead of "api.service.consul" via DNS will result in automatic geo failover to the next closest federated Consul datacenters, in order of increasing network round trip time.
    45  
    46  Failover is just another policy choice for a prepared query, it works in the same manner as the previous example and is similarly transparent to applications. The failover policy is configured using the `Failover` structure, which contains two fields, both of which are optional, and determine what happens if no healthy nodes are available in the local datacenter when the query is executed.
    47  
    48  - `NearestN` `(int: 0)` - Specifies that the query will be forwarded to up to `NearestN` other datacenters based on their estimated network round trip time using [network coordinates](/docs/internals/coordinates.html).
    49  
    50  - `Datacenters` `(array<string>: nil)` - Specifies a fixed list of remote datacenters to forward the query to if there are no healthy nodes in the local datacenter. Datacenters are queried in the order given in the list.
    51  
    52  The following sections show examples using these fields to implement different geo failover policies.
    53  
    54  ### Static Policy
    55  
    56  A static failover policy includes a fixed list of datacenters to contact once there are no healthy instances in the local datacenter.
    57  
    58  Here's the example from the introduction, expanded with a static failover policy:
    59  
    60  ```
    61  $ curl \
    62      --request POST \
    63      --data \
    64  '{
    65    "Name": "api",
    66    "Service": {
    67      "Service": "api",
    68      "Tags": ["v1.2.3"],
    69      "Failover": {
    70        "Datacenters": ["dc1", "dc2"]
    71      }
    72    }
    73  }' http://127.0.0.1:8500/v1/query
    74  
    75  {"ID":"fe3b8d40-0ee0-8783-6cc2-ab1aa9bb16c1"}
    76  ```
    77  
    78  When this query is executed, such as with a DNS lookup to "api.query.consul", the following actions will occur:
    79  
    80  1. Consul servers in the local datacenter will attempt to find healthy instances of the "api" service with the required tag.
    81  2. If none are available locally, the Consul servers will make an RPC request to the Consul servers in "dc1" to perform the query there.
    82  3. If none are available in "dc1", then an RPC will be made to the Consul servers in "dc2" to perform the query there.
    83  4. Finally an error will be returned if none of these datacenters had any instances available.
    84  
    85  ### Dynamic Policy
    86  
    87  In a complex federated environment with many Consul datacenters, it can be cumbersome to set static failover policies, so Consul offers a dynamic option based on Consul's [network coordinate](/docs/internals/coordinates.html) subsystem. Consul continuously maintains an estimate of the network round trip time from the local datacenter to the servers in other datacenters it is federated with. Each server uses the median round trip time from itself to the servers in the remote datacenter. This means that failover can simply try other remote datacenters in order of increasing network round trip time, and if datacenters come and go, or experience network issues, this order will adjust automatically.
    88  
    89  Here's the example from the introduction, expanded with a dynamic failover policy:
    90  
    91  ```
    92  $ curl \
    93      --request POST \
    94      --data \
    95  '{
    96    "Name": "api",
    97    "Service": {
    98      "Service": "api",
    99      "Tags": ["v1.2.3"],
   100      "Failover": {
   101        "NearestN": 2
   102      }
   103    }
   104  }' http://127.0.0.1:8500/v1/query
   105  
   106  {"ID":"fe3b8d40-0ee0-8783-6cc2-ab1aa9bb16c1"}
   107  ```
   108  
   109  This query is resolved in a similar fashion to the previous example, except the choice of "dc1" or "dc2", or possibly some other datacenter, is made automatically.
   110  
   111  ### Hybrid Policy
   112  
   113  It is possible to combine `Datacenters` and `NearestN` in the same policy. The `NearestN` queries will be performed first, followed by the list given by `Datacenters`. A given datacenter will only be queried one time during a failover, even if it is selected by both `NearestN` and is listed in `Datacenters`. This is useful for allowing a limited number of round trip-based attempts, followed by a static configuration for some known datacenter to failover to.
   114  
   115  ## Templates
   116  
   117  For datacenters with many services, it can be cumbersome to define a prepared query to apply a geo failover policy for each service. Consul provides a [prepared query template](/api/query.html#prepared-query-templates) capability to allow one prepared query to apply to many, and even all, services.
   118  
   119  Here's an example request to create a prepared query template that applies a dynamic geo failover policy to all services:
   120  
   121  ```
   122  $ curl \
   123      --request POST \
   124      --data \
   125  '{
   126    "Name": "",
   127    "Template": {
   128      "Type": "name_prefix_match"
   129    },
   130    "Service": {
   131      "Service": "${name.full}",
   132      "Failover": {
   133        "NearestN": 2
   134      }
   135    }
   136  }' http://127.0.0.1:8500/v1/query
   137  
   138  {"ID":"fe3b8d40-0ee0-8783-6cc2-ab1aa9bb16c1"}
   139  ```
   140  
   141  Templates can match on prefixes or use full regular expressions to determine which services they match. In this case, we've chosen the `name_prefix_match` type and given it an empty name, which means that it will match any service. If multiple queries are registered, the most specific one will be selected, so it's possible to have a template like this as a catch-all, and then apply more specific policies to certain services.
   142  
   143  With this one prepared query template in place, simply changing application configurations to look up "api.query.consul" instead of "api.service.consul" via DNS will result in automatic geo failover to the next closest federated Consul datacenters, in order of increasing network round trip time.