github.com/cozy/cozy-stack@v0.0.0-20240603063001-31110fa4cae1/docs/archives/realtime.md (about)

     1  [Table of contents](../README.md#table-of-contents)
     2  
     3  This document was written in january 2017.
     4  
     5  # Realtime design
     6  
     7  ## Context
     8  
     9  ### Definitions
    10  
    11  -   **Event:** Something happening in the stack. Most of them will come from
    12      couchdb, some jobs and user actions might also trigger them.
    13  -   **Events feed:** the feed of occurring events. There is two types of feeds:
    14      -   **continuous** allows to follow events as they occurs
    15      -   **interval** allow the see the history of the feed from any given time
    16  -   **Realtime:** user experienced updates of the interface from change
    17      happening from another source. _Ie, I have a folder opened in cozy-files on
    18      the browser, I take some pictures from my smartphone, the pictures appears
    19      in the folder without me needing to refresh the browser tab._
    20  
    21  ### What couchdb offers
    22  
    23  Couchdb supports with its `_changes` API both events feeds types:
    24  
    25  -   using `since=now&continuous=true` we get all events **continuous**ly as they
    26      happen (SSE)
    27  -   using `since=(last known seq_number)` we get all changes in the **interval**
    28      between last known `seq_number` and now.
    29  
    30  Couchdb also offers a `_db_updates` route, which give us **continuous** changes
    31  at the database level. This routes does not support a since parameter, as there
    32  is no global `seq_number`.
    33  
    34  Other events will be generated from the stack itself, such as session close or
    35  jobs activity.
    36  
    37  ### Performance limitation
    38  
    39  **We cannot have a continuous `_changes` feed open to every databases**
    40  
    41  ## Use cases for interval events feeds
    42  
    43  ### Replication
    44  
    45  Couchdb replication algorithm can work in one-shot mode, where it replicates
    46  changes since last sync up until now, or in continuous mode where it replicates
    47  changes as they happens.
    48  
    49  -   The stack will not allow continuous mode for replication.
    50  -   This is already supported with the `_changes` route
    51  
    52  ### Sharing
    53  
    54  Our sharing is based on couchdb replication rules, so also depends on `_changes`
    55  feed to ensure all changes have been applied.
    56  
    57  **Considering these use cases, there is no need for non-couchdb event to be part
    58  of the interval events feed.**
    59  
    60  ## Use cases for continuous events feeds
    61  
    62  ### Realtime
    63  
    64  Some events should be send to the client to update views as data change.
    65  
    66  ### `@event` jobs trigger
    67  
    68  Some event will trigger the activation of a job (ie. When a photo has been
    69  uploaded, generate thumbnails and extract EXIF metadatas). This should be done
    70  as soon as possible after the events
    71  
    72  ### Sharing?
    73  
    74  While not absolutely necessary, having cozy A notify cozy B when a shared
    75  document is changed allows for both better user experience (faster propagation)
    76  and better performance (no need to poll every X minutes, the N cozy we are
    77  sharing from).
    78  
    79  ## Client realtime tech choice
    80  
    81  ### Options
    82  
    83  -   **Polling:** regularly ask the server what happened since last time.
    84  -   **COMET:** Leaving a normal HTTP connection open sending data and heartbeets
    85      regularly to keep it open, reading xhr.responseText at intervals without
    86      waiting for readyState == 4. Restart the connection when it breaks.
    87  -   **SSE:** Normalized & standardized version of COMET with
    88      [half-decent browser support (86% users)](http://caniuse.com/#feat=eventsource)
    89      but easily polyfillable (it's just COMET). It is simpler and easier to
    90      debug. It has some limitations (no HTTP headers in JS api, counts toward the
    91      maximum number of http connection per domain).
    92  -   **Websocket:** keep a socket open, it allows 2 way data communication which
    93      we do not need, has
    94      [better server support (92% users)](http://caniuse.com/#feat=websockets) but
    95      is impossible to polyfill client side, more popular, there is a better
    96      [golang package](https://pkg.go.dev/github.com/gorilla/websocket)
    97  -   **SockJS & cie** they are **a lot** of packages which imitate Websocket API
    98      while using complicated client&server polyfill to allow support of older
    99      browser. [SockJS](https://github.com/sockjs/) is a drop-in websocket
   100      replacement with a go package and javascript client.
   101  
   102  ### Choice = Websocket
   103  
   104  While SSE appears at first glance like a better fit for our use case, its
   105  limitation and lack of browser priority makes us choose websocket. In the event
   106  older browser supports becomes necessary we can use SockJS.
   107  
   108  ### optimization paths (future)
   109  
   110  -   **bandwidth** Limiting the number of events sent by allowing the client to
   111      specified it is only interested in events matching a selector _(files app
   112      only care about changes in the files of the current folder view)_
   113  -   **number of connections** Instead of 1 socket / tab, we can probably make 1
   114      socket / browser using some hackish combination of SharedWorker /
   115      iframe.postMessage and a client-side demultiplexer.
   116  -   **both** No need for realtime if the user is not using the tab (for most
   117      usecases), we could cut the realtime feed depending on
   118      [Page Visibility API](https://www.w3.org/TR/2011/WD-page-visibility-20110602/)
   119  
   120  ## Go/Stack architecture
   121  
   122  -   We assume all couchdb changes will originate from the stack
   123  -   Events are generated at the stack level
   124  -   We do **NOT** rely on couchdb `_changes?continuous` nor `_db_udpates`
   125  
   126  We create a realtime.Event interface, which we call in other packages. We accept
   127  websocket connection and bind them to a realtime.Dispatcher object.
   128  
   129  ### Small cozy version
   130  
   131  It all happens in RAM, realtime.Event are immediately transmited to the
   132  dispatcher.
   133  
   134  ### Big cozy version (ie. multiple stack instance)
   135  
   136  Redis pub/sub