github.com/cilium/cilium@v1.16.2/Documentation/kvstore.rst (about) 1 Key-Value Store 2 ############### 3 4 Cilium uses an external key-value store to exchange information across multiple 5 Cilium instances: 6 7 Layout 8 ====== 9 10 All data is stored under a common key prefix: 11 12 ===================== ==================== 13 Prefix Description 14 ===================== ==================== 15 ``cilium/`` All keys share this common prefix. 16 ``cilium/state/`` State stored by agents, data is automatically recreated on removal or corruption. 17 ===================== ==================== 18 19 20 Cluster Nodes 21 ------------- 22 23 Every agent will register itself as a node in the kvstore and make the 24 following information available to other agents: 25 26 - Name 27 - IP addresses of the node 28 - Health checking IP addresses 29 - Allocation range of endpoints on the node 30 31 ============================================================ ==================== 32 Key Value 33 ============================================================ ==================== 34 ``cilium/state/nodes/v1/<cluster>/<node>`` node.Node_ 35 ============================================================ ==================== 36 37 .. _node.Node: https://pkg.go.dev/github.com/cilium/cilium/pkg/node/types#Node 38 39 All node keys are attached to a lease owned by the agent of the respective 40 node. 41 42 43 Services 44 -------- 45 46 All Kubernetes services are mirrored into the kvstore by the Cilium operator. This is 47 required to implement multi cluster service discovery. 48 49 ============================================================= ==================== 50 Key Value 51 ============================================================= ==================== 52 ``cilium/state/services/v1/<cluster>/<namespace>/<service>`` serviceStore.ClusterService_ 53 ============================================================= ==================== 54 55 .. _serviceStore.ClusterService: https://pkg.go.dev/github.com/cilium/cilium/pkg/service/store#ClusterService 56 57 Identities 58 ---------- 59 60 Any time a new endpoint is started on a Cilium node, it will determine whether 61 the labels for the endpoint are unique and allocate an identity for that set of 62 labels. These identities are only meaningful within the local cluster. 63 64 ============================================================= ==================== 65 Key Value 66 ============================================================= ==================== 67 ``cilium/state/identities/v1/id/<identity>`` labels.LabelArray_ 68 ``cilium/state/identities/v1/value/<labels>/<node>`` identity.NumericIdentity_ 69 ============================================================= ==================== 70 71 .. _identity.NumericIdentity: https://pkg.go.dev/github.com/cilium/cilium/pkg/identity#NumericIdentity 72 .. _labels.LabelArray: https://pkg.go.dev/github.com/cilium/cilium/pkg/labels#LabelArray 73 74 Endpoints 75 --------- 76 77 All endpoint IPs and corresponding identities are mirrored to the kvstore by 78 the agent on the node where the endpoint is launched, to allow peer nodes to 79 configure egress policies to endpoints backed by these IPs. 80 81 ============================================================= ==================== 82 Key Value 83 ============================================================= ==================== 84 ``cilium/state/ip/v1/<cluster>/<ip>`` identity.IPIdentityPair_ 85 ============================================================= ==================== 86 87 .. _identity.IPIdentityPair: https://pkg.go.dev/github.com/cilium/cilium/pkg/identity#IPIdentityPair 88 89 CiliumNetworkPolicyNodeStatus 90 ----------------------------- 91 92 If handover to Kubernetes is enabled, then each ``cilium-agent`` will propagate 93 the state of whether it has realized a given CNP to the key-value store instead 94 of directly writing to ``kube-apiserver``. ``cilium-operator`` will listen for 95 updates to this prefix from the key-value store, and will be the sole updater 96 of statuses for CNPs in the cluster. 97 98 ================================================================ ==================== 99 Key Value 100 ================================================================ ==================== 101 ``cilium/state/cnpstatuses/v2/<UID>/<namespace>/<name>/<node>`` k8s.CNPNSWithMeta_ 102 ================================================================ ==================== 103 104 .. _k8s.CNPNSWithMeta: https://pkg.go.dev/github.com/cilium/cilium/pkg/k8s#CNPNSWithMeta 105 106 Heartbeat 107 --------- 108 109 The heartbeat key is periodically updated by the operator to contain the 110 current time and date. It is used by agents to validate that kvstore updates 111 can be received. 112 113 ====================== ====================== 114 Key Value 115 ====================== ====================== 116 ``cilium/.heartbeat`` Current time and date 117 ====================== ====================== 118 119 120 Leases 121 ====== 122 123 With a few exceptions, all keys in the key-value store are owned by a 124 particular agent running on a node. All such keys have a lease attached. The 125 lease is renewed automatically. When the lease expires, the key is removed from 126 the key-value store. This guarantees that keys are removed from the key-value 127 store in the event that an agent dies on a particular and never reappears. 128 129 The lease lifetime is set to 15 minutes. The exact expiration behavior is 130 dependent on the kvstore implementation but the expiration typically occurs 131 after double the lease lifetime. 132 133 In addition to regular entry leases, all locks in the key-value store are 134 owned by a particular agent running on the node with a separate "lock lease" 135 attached. The lock lease has a default lifetime of 25 seconds. 136 137 =============================================================== ================ ======================================== 138 Key Lease Timeout Default expiry 139 =============================================================== ================ ======================================== 140 ``cilium/.initlock/<random>/<lease-ID>`` LockLeaseTTL_ 25 seconds 141 ``cilium/.heartbeat`` KVstoreLeaseTTL 15 minutes 142 ``cilium/state/cnpstatuses/v2/<UID>/<namespace>/<name>/<node>`` KVstoreLeaseTTL_ 15 minutes 143 ``cilium/state/identities/v1/id/<identity>`` None Garbage collected by ``cilium-operator`` 144 ``cilium/state/identities/v1/value/<labels>/<node>`` KVstoreLeaseTTL_ 15 minutes 145 ``cilium/state/ip/v1/<cluster>/<ip>`` KVstoreLeaseTTL_ 15 minutes 146 ``cilium/state/nodes/v1/<cluster>/<node>`` KVstoreLeaseTTL_ 15 minutes 147 ``cilium/state/services/v1/<cluster>/<namespace>/<service>`` KVstoreLeaseTTL_ 15 minutes 148 =============================================================== ================ ======================================== 149 150 .. _LockLeaseTTL: https://pkg.go.dev/github.com/cilium/cilium/pkg/defaults?tab=doc#LockLeaseTTL 151 .. _KVstoreLeaseTTL: https://pkg.go.dev/github.com/cilium/cilium/pkg/defaults?tab=doc#KVstoreLeaseTTL 152 153 Debugging 154 ========= 155 156 The contents stored in the kvstore can be queued and manipulate using the 157 ``cilium kvstore`` command. For additional details, see the command reference. 158 159 Example: 160 161 .. code-block:: shell-session 162 163 $ cilium kvstore get --recursive cilium/state/nodes/ 164 cilium/state/nodes/v1/default/runtime1 => {"Name":"runtime1","IPAddresses":[{"AddressType":"InternalIP","IP":"10.0.2.15"}],"IPv4AllocCIDR":{"IP":"10.11.0.0","Mask":"//8AAA=="},"IPv6AllocCIDR":{"IP":"f00d::a0f:0:0:0","Mask":"//////////////////8AAA=="},"IPv4HealthIP":"","IPv6HealthIP":""}