github.com/mika/distribution@v2.2.2-0.20160108133430-a75790e3d8e0+incompatible/docs/notifications.md (about) 1 <!--[metadata]> 2 +++ 3 title = "Working with notifications" 4 description = "Explains how to work with registry notifications" 5 keywords = ["registry, on-prem, images, tags, repository, distribution, notifications, advanced"] 6 [menu.main] 7 parent="smn_registry" 8 weight=5 9 +++ 10 <![end-metadata]--> 11 12 # Notifications 13 14 The Registry supports sending webhook notifications in response to events 15 happening within the registry. Notifications are sent in response to manifest 16 pushes and pulls and layer pushes and pulls. These actions are serialized into 17 events. The events are queued into a registry-internal broadcast system which 18 queues and dispatches events to [_Endpoints_](#endpoints). 19 20  21 22 ## Endpoints 23 24 Notifications are sent to _endpoints_ via HTTP requests. Each configured 25 endpoint has isolated queues, retry configuration and http targets within each 26 instance of a registry. When an action happens within the registry, it is 27 converted into an event which is dropped into an inmemory queue. When the 28 event reaches the end of the queue, an http request is made to the endpoint 29 until the request succeeds. The events are sent serially to each endpoint but 30 order is not guaranteed. 31 32 ## Configuration 33 34 To setup a registry instance to send notifications to endpoints, one must add 35 them to the configuration. A simple example follows: 36 37 notifications: 38 endpoints: 39 - name: alistener 40 url: https://mylistener.example.com/event 41 headers: 42 Authorization: [Bearer <your token, if needed>] 43 timeout: 500ms 44 threshold: 5 45 backoff: 1s 46 47 The above would configure the registry with an endpoint to send events to 48 `https://mylistener.example.com/event`, with the header "Authorization: Bearer 49 <your token, if needed>". The request would timeout after 500 milliseconds. If 50 5 failures happen consecutively, the registry will backoff for 1 second before 51 trying again. 52 53 For details on the fields, please see the [configuration documentation](configuration.md#notifications). 54 55 A properly configured endpoint should lead to a log message from the registry 56 upon startup: 57 58 ``` 59 INFO[0000] configuring endpoint alistener (https://mylistener.example.com/event), timeout=500ms, headers=map[Authorization:[Bearer <your token if needed>]] app.id=812bfeb2-62d6-43cf-b0c6-152f541618a3 environment=development service=registry 60 ``` 61 62 ## Events 63 64 Events have a well-defined JSON structure and are sent as the body of 65 notification requests. One or more events are sent in a structure called an 66 envelope. Each event has a unique id that can be used to uniquely identify incoming 67 requests, if required. Along with that, an _action_ is provided with a 68 _target, identifying the object mutated during the event. 69 70 The fields available in an event are described in detail in the 71 [godoc](http://godoc.org/github.com/docker/distribution/notifications#Event). 72 73 **TODO:** Let's break out the fields here rather than rely on the godoc. 74 75 The following is an example of a JSON event, sent in response to the push of a 76 manifest: 77 78 ```json 79 { 80 "id": "asdf-asdf-asdf-asdf-0", 81 "timestamp": "2006-01-02T15:04:05Z", 82 "action": "push", 83 "target": { 84 "mediaType": "application/vnd.docker.distribution.manifest.v1+json", 85 "size": 1, 86 "digest": "sha256:0123456789abcdef0", 87 "length": 1, 88 "repository": "library/test", 89 "url": "http://example.com/v2/library/test/manifests/latest" 90 }, 91 "request": { 92 "id": "asdfasdf", 93 "addr": "client.local", 94 "host": "registrycluster.local", 95 "method": "PUT", 96 "useragent": "test/0.1" 97 }, 98 "actor": { 99 "name": "test-actor" 100 }, 101 "source": { 102 "addr": "hostname.local:port" 103 } 104 } 105 ``` 106 107 > __NOTE:__ As of version 2.1, the `length` field for event targets 108 > is being deprecated for the `size` field, bringing the target in line with 109 > common nomenclature. Both will continue to be set for the foreseeable 110 > future. Newer code should favor `size` but accept either. 111 112 ## Envelope 113 114 The envelope contains one or more events, with the following json structure: 115 116 ```json 117 { 118 "events": [ ... ], 119 } 120 ``` 121 122 While events may be sent in the same envelope, the set of events within that 123 envelope have no implied relationship. For example, the registry may choose to 124 group unrelated events and send them in the same envelope to reduce the total 125 number of requests. 126 127 The full package has the mediatype 128 "application/vnd.docker.distribution.events.v1+json", which will be set on the 129 request coming to an endpoint. 130 131 An example of a full event may look as follows: 132 133 ```json 134 GET /callback 135 Host: application/vnd.docker.distribution.events.v1+json 136 Authorization: Bearer <your token, if needed> 137 Content-Type: application/vnd.docker.distribution.events.v1+json 138 139 { 140 "events": [ 141 { 142 "id": "asdf-asdf-asdf-asdf-0", 143 "timestamp": "2006-01-02T15:04:05Z", 144 "action": "push", 145 "target": { 146 "mediaType": "application/vnd.docker.distribution.manifest.v1+json", 147 "length": 1, 148 "digest": "sha256:0123456789abcdef0", 149 "repository": "library/test", 150 "url": "http://example.com/v2/library/test/manifests/latest" 151 }, 152 "request": { 153 "id": "asdfasdf", 154 "addr": "client.local", 155 "host": "registrycluster.local", 156 "method": "PUT", 157 "useragent": "test/0.1" 158 }, 159 "actor": { 160 "name": "test-actor" 161 }, 162 "source": { 163 "addr": "hostname.local:port" 164 } 165 }, 166 { 167 "id": "asdf-asdf-asdf-asdf-1", 168 "timestamp": "2006-01-02T15:04:05Z", 169 "action": "push", 170 "target": { 171 "mediaType": "application/vnd.docker.container.image.rootfs.diff+x-gtar", 172 "length": 2, 173 "digest": "sha256:3b3692957d439ac1928219a83fac91e7bf96c153725526874673ae1f2023f8d5", 174 "repository": "library/test", 175 "url": "http://example.com/v2/library/test/manifests/latest" 176 }, 177 "request": { 178 "id": "asdfasdf", 179 "addr": "client.local", 180 "host": "registrycluster.local", 181 "method": "PUT", 182 "useragent": "test/0.1" 183 }, 184 "actor": { 185 "name": "test-actor" 186 }, 187 "source": { 188 "addr": "hostname.local:port" 189 } 190 }, 191 { 192 "id": "asdf-asdf-asdf-asdf-2", 193 "timestamp": "2006-01-02T15:04:05Z", 194 "action": "push", 195 "target": { 196 "mediaType": "application/vnd.docker.container.image.rootfs.diff+x-gtar", 197 "length": 3, 198 "digest": "sha256:3b3692957d439ac1928219a83fac91e7bf96c153725526874673ae1f2023f8d6", 199 "repository": "library/test", 200 "url": "http://example.com/v2/library/test/manifests/latest" 201 }, 202 "request": { 203 "id": "asdfasdf", 204 "addr": "client.local", 205 "host": "registrycluster.local", 206 "method": "PUT", 207 "useragent": "test/0.1" 208 }, 209 "actor": { 210 "name": "test-actor" 211 }, 212 "source": { 213 "addr": "hostname.local:port" 214 } 215 } 216 ] 217 } 218 ``` 219 220 ## Responses 221 222 The registry is fairly accepting of the response codes from endpoints. If an 223 endpoint responds with any 2xx or 3xx response code (after following 224 redirects), the message will be considered delivered and discarded. 225 226 In turn, it is recommended that endpoints are accepting of incoming responses, 227 as well. While the format of event envelopes are standardized by media type, 228 any "pickyness" about validation may cause the queue to backup on the 229 registry. 230 231 ## Monitoring 232 233 The state of the endpoints are reported via the debug/vars http interface, 234 usually configured to `http://localhost:5001/debug/vars`. Information such as 235 configuration and metrics are available by endpoint. 236 237 The following provides an example of a few endpoints that have experienced 238 several failures and have since recovered: 239 240 ```json 241 "notifications":{ 242 "endpoints":[ 243 { 244 "name":"local-5003", 245 "url":"http://localhost:5003/callback", 246 "Headers":{ 247 "Authorization":[ 248 "Bearer \u003can example token\u003e" 249 ] 250 }, 251 "Timeout":1000000000, 252 "Threshold":10, 253 "Backoff":1000000000, 254 "Metrics":{ 255 "Pending":76, 256 "Events":76, 257 "Successes":0, 258 "Failures":0, 259 "Errors":46, 260 "Statuses":{ 261 262 } 263 } 264 }, 265 { 266 "name":"local-8083", 267 "url":"http://localhost:8083/callback", 268 "Headers":null, 269 "Timeout":1000000000, 270 "Threshold":10, 271 "Backoff":1000000000, 272 "Metrics":{ 273 "Pending":0, 274 "Events":76, 275 "Successes":76, 276 "Failures":0, 277 "Errors":28, 278 "Statuses":{ 279 "202 Accepted":76 280 } 281 } 282 } 283 ] 284 } 285 ``` 286 287 If using notification as part of a larger application, it is _critical_ to 288 monitor the size ("Pending" above) of the endpoint queues. If failures or 289 queue sizes are increasing, it can indicate a larger problem. 290 291 The logs are also a valuable resource for monitoring problems. A failing 292 endpoint will lead to messages similar to the following: 293 294 ``` 295 ERRO[0340] retryingsink: error writing events: httpSink{http://localhost:5003/callback}: error posting: Post http://localhost:5003/callback: dial tcp 127.0.0.1:5003: connection refused, retrying 296 WARN[0340] httpSink{http://localhost:5003/callback} encountered too many errors, backing off 297 ``` 298 299 The above indicates that several errors have led to a backoff and the registry 300 will wait before retrying. 301 302 ## Considerations 303 304 Currently, the queues are inmemory, so endpoints should be _reasonably 305 reliable_. They are designed to make a best-effort to send the messages but if 306 an instance is lost, messages may be dropped. If an endpoint goes down, care 307 should be taken to ensure that the registry instance is not terminated before 308 the endpoint comes back up or messages will be lost. 309 310 This can be mitigated by running endpoints in close proximity to the registry 311 instances. One could run an endpoint that pages to disk and then forwards a 312 request to provide better durability. 313 314 The notification system is designed around a series of interchangeable _sinks_ 315 which can be wired up to achieve interesting behavior. If this system doesn't 316 provide acceptable guarantees, adding a transactional `Sink` to the registry 317 is a possibility, although it may have an effect on request service time. 318 Please see the 319 [godoc](http://godoc.org/github.com/docker/distribution/notifications#Sink) 320 for more information.