github.com/cozy/cozy-stack@v0.0.0-20240603063001-31110fa4cae1/docs/archives/realtime.md (about) 1 [Table of contents](../README.md#table-of-contents) 2 3 This document was written in january 2017. 4 5 # Realtime design 6 7 ## Context 8 9 ### Definitions 10 11 - **Event:** Something happening in the stack. Most of them will come from 12 couchdb, some jobs and user actions might also trigger them. 13 - **Events feed:** the feed of occurring events. There is two types of feeds: 14 - **continuous** allows to follow events as they occurs 15 - **interval** allow the see the history of the feed from any given time 16 - **Realtime:** user experienced updates of the interface from change 17 happening from another source. _Ie, I have a folder opened in cozy-files on 18 the browser, I take some pictures from my smartphone, the pictures appears 19 in the folder without me needing to refresh the browser tab._ 20 21 ### What couchdb offers 22 23 Couchdb supports with its `_changes` API both events feeds types: 24 25 - using `since=now&continuous=true` we get all events **continuous**ly as they 26 happen (SSE) 27 - using `since=(last known seq_number)` we get all changes in the **interval** 28 between last known `seq_number` and now. 29 30 Couchdb also offers a `_db_updates` route, which give us **continuous** changes 31 at the database level. This routes does not support a since parameter, as there 32 is no global `seq_number`. 33 34 Other events will be generated from the stack itself, such as session close or 35 jobs activity. 36 37 ### Performance limitation 38 39 **We cannot have a continuous `_changes` feed open to every databases** 40 41 ## Use cases for interval events feeds 42 43 ### Replication 44 45 Couchdb replication algorithm can work in one-shot mode, where it replicates 46 changes since last sync up until now, or in continuous mode where it replicates 47 changes as they happens. 48 49 - The stack will not allow continuous mode for replication. 50 - This is already supported with the `_changes` route 51 52 ### Sharing 53 54 Our sharing is based on couchdb replication rules, so also depends on `_changes` 55 feed to ensure all changes have been applied. 56 57 **Considering these use cases, there is no need for non-couchdb event to be part 58 of the interval events feed.** 59 60 ## Use cases for continuous events feeds 61 62 ### Realtime 63 64 Some events should be send to the client to update views as data change. 65 66 ### `@event` jobs trigger 67 68 Some event will trigger the activation of a job (ie. When a photo has been 69 uploaded, generate thumbnails and extract EXIF metadatas). This should be done 70 as soon as possible after the events 71 72 ### Sharing? 73 74 While not absolutely necessary, having cozy A notify cozy B when a shared 75 document is changed allows for both better user experience (faster propagation) 76 and better performance (no need to poll every X minutes, the N cozy we are 77 sharing from). 78 79 ## Client realtime tech choice 80 81 ### Options 82 83 - **Polling:** regularly ask the server what happened since last time. 84 - **COMET:** Leaving a normal HTTP connection open sending data and heartbeets 85 regularly to keep it open, reading xhr.responseText at intervals without 86 waiting for readyState == 4. Restart the connection when it breaks. 87 - **SSE:** Normalized & standardized version of COMET with 88 [half-decent browser support (86% users)](http://caniuse.com/#feat=eventsource) 89 but easily polyfillable (it's just COMET). It is simpler and easier to 90 debug. It has some limitations (no HTTP headers in JS api, counts toward the 91 maximum number of http connection per domain). 92 - **Websocket:** keep a socket open, it allows 2 way data communication which 93 we do not need, has 94 [better server support (92% users)](http://caniuse.com/#feat=websockets) but 95 is impossible to polyfill client side, more popular, there is a better 96 [golang package](https://pkg.go.dev/github.com/gorilla/websocket) 97 - **SockJS & cie** they are **a lot** of packages which imitate Websocket API 98 while using complicated client&server polyfill to allow support of older 99 browser. [SockJS](https://github.com/sockjs/) is a drop-in websocket 100 replacement with a go package and javascript client. 101 102 ### Choice = Websocket 103 104 While SSE appears at first glance like a better fit for our use case, its 105 limitation and lack of browser priority makes us choose websocket. In the event 106 older browser supports becomes necessary we can use SockJS. 107 108 ### optimization paths (future) 109 110 - **bandwidth** Limiting the number of events sent by allowing the client to 111 specified it is only interested in events matching a selector _(files app 112 only care about changes in the files of the current folder view)_ 113 - **number of connections** Instead of 1 socket / tab, we can probably make 1 114 socket / browser using some hackish combination of SharedWorker / 115 iframe.postMessage and a client-side demultiplexer. 116 - **both** No need for realtime if the user is not using the tab (for most 117 usecases), we could cut the realtime feed depending on 118 [Page Visibility API](https://www.w3.org/TR/2011/WD-page-visibility-20110602/) 119 120 ## Go/Stack architecture 121 122 - We assume all couchdb changes will originate from the stack 123 - Events are generated at the stack level 124 - We do **NOT** rely on couchdb `_changes?continuous` nor `_db_udpates` 125 126 We create a realtime.Event interface, which we call in other packages. We accept 127 websocket connection and bind them to a realtime.Dispatcher object. 128 129 ### Small cozy version 130 131 It all happens in RAM, realtime.Event are immediately transmited to the 132 dispatcher. 133 134 ### Big cozy version (ie. multiple stack instance) 135 136 Redis pub/sub