github.com/randomizedcoder/goTrackRTP@v0.0.2/README.md (about) 1 # goTrackingRing 2 3 This repo contains a library of code to monitor [RTC 3550 Real Time Protocoo (RTP)](https://www.rfc-editor.org/rfc/rfc3550) sequence numbers. 4 5 ## RTP Sequence Reporting 6 7 It provides the ability to report on: 8 9 - RTP sequence loss over time 10 - Specifically, it monitors loss within a particular "acceptable window" of packets. e.g. How many packets were lost out of the last 100? 11 - How many packets were out of order? 12 - How many packets were late? 13 - How many jumps ( continuous gaps ) in sequence numbers were there? 14 - How many duplicate packets were received? 15 16 This library is intended to used in conjunction with other code that handles packets and decodes the RTP header. Essentially, this library only concerns itself with tracking uint16 sequence numbers. The intended use is to expose Prometheus metrics to allow longer term reporting over time. e.g. Allows a network operator to monitor trends in packet losses. 17 18 There is also a very simple example implmentation. 19 20 ### RTP Sequence numbers 21 22 Reminder on RTP sequence numbers: 23 24 - RTP sequence numbers are 2^16 25 - Encoders should be randomly select the starting sequence number on startup 26 - With no network losses, the RTP sequence numbers should increment by one 27 - In real networks there are delays, reordering, which this code aims to track and report upon 28 29 This library aims to be efficent from a CPU and memory perspective, and so uses a [B-tree](https://en.wikipedia.org/wiki/B-tree) structure. 30 31 ## Overview 32 33 The solution essentially tracks RTP sequence numbers and then categorizes an arriving packet into various categories. 34 35 This diagram provides an overview of the categories defined. 36 37 <img src="./rtp_sequence_numbers.png" alt="RTP Sequence Numbers" width="80%" height="80%"/> 38 39 ### Definitions 40 41 | Definition | Variable | Description | 42 | ----------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | 43 | Max() | Max() | Highest RTP sequence and current reference point. Future packets are all relative to this packet | 44 | Ahead | | Packet with a higher sequnce number than >Max() | 45 | Behind | | Packet with a lower sequnce number than <Max() | 46 | Acceptable Window | <aw,>bw | Packets with sequence number in this range are accaptable, which is to total length bw+aw | 47 | Ahead Window | aw | Sequence number is within +aw packets of <Max() | 48 | Behind Window | bw | Sequence number is within -bw packets of >Max() | 49 | Safety Buffers | | To avoid erronously jumping to a new tracking window, "safety" buffers behind/ahead allow the operator to make sure tracker doesn't reinitilize the window | 50 | Ahead Buffer | ab | Ahead buffer is a logical gap, where if a packet arrives within this range the existing Max() and acceptable window will NOT be regenerated | 51 | Behind Buffer | bb | Behind buffer is a logical gap, where if a packet arrives within this range the existing Max() and acceptable window will NOT be regenerated | 52 | Restart | | If the sequence number jumps ahead/behind by a large amount then the encoder has restarted, so the acceptable window needs to be reinitilized | 53 | Behind Restart | >ab | Sequence number arriving in this range reinitilizes the window | 54 | Ahead Restart | <bb | Sequence number arriving in this range reinitilizes the window | 55 | Sequence Roll | | For now, 2^16 sequence roll will be treated like a restart | 56 57 ### Positions 58 59 | Position | Description | 60 | -------- | ---------------------------------------- | 61 | Unknown | | 62 | Ahead | Within the acceptable | 63 | Behind | Within the safety buffer, and so ignored | 64 65 ### Categories 66 67 | Category | Description | 68 | -------- | ------------------------------------------------------------------------------- | 69 | Unknown | | 70 | Window | Within the acceptable | 71 | Buffer | Within the safety buffer, and so ignored | 72 | Reset | Outside the acceptable window and buffer, causing reinitilization of the window | 73 74 ### SubCategories 75 76 | SubCategory | Description | 77 | ------------- | ------------------------------------------------------------------------ | 78 | None | No additional sub catagorization | 79 | Next Sequence | Next sequence is the packet that will ideally arrive next and is Max()+1 | 80 | Duplicate | Duplicate packets are also identified | 81 82 > **Please note:** 83 > 84 > All the windows and buffers are defined in terms of _packets_ NOT _time_ 85 86 ## Roughly how this code works 87 88 ### Items in the "acceptable window" are added to the B-tree 89 90 Items are added using [.ReplaceOrInsert()](https://pkg.go.dev/github.com/google/btree#BTreeG.ReplaceOrInsert). 91 92 For the happy path, this is the next sequence number, and so .Max() advances by one, so the B-tree will become slightly longer to the left. 93 94 If the packet is not in sequence, it will also be added to the tree, and may adjust the .Max() forward. 95 96 Typically, we expect either complete packet loss, or slight delays in packets, so the expectation is there are gaps in the sequence numbers, or "behind" packets, neither of which will advance the .Max(). 97 98 This means the B-tree always holds: 99 100 - All the sequence numbers seen within the "acceptable window" 101 - Iteration is required to get a list of the missing items, although for small window sizes the iteration is relatively inexpensive, although unless there is a very specific debugging scenario this is probably not required. 102 - Count of the number of packets in the "acceptable window" ( .Len() ). The number of missing packets is merely the "acceptable window" size, minus the number of packets seen. 103 104 ### Automatic rebalancing of the B-tree 105 106 Overtime, the B-tree will become longer on the left ( more items on the left of the tree ), and so the B-tree will rebalanced via rotation+merging. The rebalance is mostly pointer moves, so it's reasonably efficient. 107 108 ### Items fall off the back of the "behind window" 109 110 Of course, as sequence numbers fall off the back of the "behind window", these items need to be removed. This is done by a simple [.Delete()](https://pkg.go.dev/github.com/google/btree#BTreeG.Delete) of a single item, which is just finding the minimum item, so it's efficent. 111 112 ### Batch deletes ( jump ahead ) 113 114 If a new item jumps forward the current position of .Max() by more than +1, then essentially multiple items need to be deleted. To make this operation more efficient the [.DescendLessOrEqual()](https://pkg.go.dev/github.com/google/btree#BTreeG.DescendLessOrEqual) iterator is used, deleting as it visits the node. B-trees are efficent at finding the next lower/higher, so this iteration is reasonably efficent. 115 116 ( An alternative implmentation would be to repeatedly call [.DeleteMin()](https://pkg.go.dev/github.com/google/btree#BTreeG.DeleteMin) until the tail of the "behind window" is reached ( .Max() - bw ), but each delete would traverse the full tree and would not be as efficient as the .DescendLess. ) 117 118 ### Restart of window ( large jump behind/ahead ) 119 120 If items arrive that are beyond the behind or ahead buffer ( wihtin the "Restart" zones in the diagram ), then these are deemed to be a restart of the RTP encoder. The existing tree is cleared via [.Clear()](https://pkg.go.dev/github.com/google/btree#BTreeG.Clear), and items are put back on the Freelist. 121 122 ( Honestly, I haven't looked to closely at how the library manages the memory or garbage collection tuning, but hopefully this library is being used with relatively small windows like <=100, so this should be a pretty small memory footprint. I assume from reading words like "freelist" in the documnetation that the library is holding on to memory, which should keep the garbage collection low. I should probably do some profiling and update the finds here.) 123 124 ### Safety buffers ( large jump behind/ahead ) 125 126 Given that restarting the window/B-tree will wipe all the packet sequence history, there is a risk that if the window configuration is smaller than packets that may actually arrive, the window will be wiped. 127 128 e.g. You could imagine that occationally a packet gets delayed more than expected ( for some unknown reason ), and even if your audio/video decoder may ignore this late packet, the RTP sequence tracker may restart the window and wipe all your useful RTP sequence data. 129 130 You probably don't want this, so to protect against this, the "behind and ahead buffers" exist. Essentially, this allows you to configure a bit more space, most importantly the "behind buffer", so reduce this risk. 131 132 Of course, the downside of this approach is that there is a small risk the RTP encoder could legitimately restart and start with a new random sequence number that's within acceptable window + buffer range (bb+bw+aw+ab), but with 2^16 the chances are pretty slim, assuming you keep pretty small windows+buffers. 133 134 ### Configuration comments 135 136 The intention is to allow an network operator to tune the monitoring windows to suit the particular network and reporting requirements. 137 138 When configuring the various window and buffer seetings, implementors should consider: 139 140 - Objectives of the monitoring, 141 - RTP packet rates, 142 - Network design in terms of redundant paths, and particularly how the network topology may change impacting end to end latency, particularly during re-convergence 143 144 Please keep in mind that the entire "acceptable window" worth of packet sequence numbers is held within the B-tree. 145 146 Please refer to this sheet for some simple Mb/s and packet rate calculations 147 https://docs.google.com/spreadsheets/d/16Wcjm8JVv4121QuZAHokMtMJ6n_b4_QZN73iNydAT5w/edit?usp=sharing 148 149 ### Example configuration 150 151 #### Network with modest variations 152 153 For the following environment: 154 155 - Video rate of ~10 Mb/s ( estimated packet size of 1380 bytes is ~725 packets per second ) 156 - Maximum network delay of up to 100 milliseconds ( ~725 packets ) 157 - Maximum network path length change of 100 milliseconds ( ~725 packets ) 158 159 The following configuration might be a good place to start: 160 161 | Variable | Packets | Comment | 162 | -------- | ------- | -------------- | 163 | aw | 725 | ~100 ms ahead | 164 | bw | 725 | ~100 ms behind | 165 | ab | 3600 | ~500 ms | 166 | bb | 3600 | ~500 ms | 167 168 This would allow for ~0.2 seconds ( 200 ms ) or ~1449 packets of "acceptable window". 169 170 #### Crazy network with pretty large variations 171 172 For the following environment: 173 174 - Video rate of ~1 Mb/s ( estimated packet size of 1380 bytes is ~725 packets per second ) 175 - Maximum network delay of up to 1 second ( 1000 ms is ~725 packets ) 176 - Maximum network path length change of 0.5 seconds ( 500 ms is ~362 packets ) 177 178 The following configuration might be a good place to start: 179 180 | Variable | Packets | Comment | 181 | -------- | ------- | --------------- | 182 | aw | 362 | ~500 ms ahead | 183 | bw | 725 | ~1000 ms behind | 184 | ab | 500 | ~690 ms | 185 | bb | 500 | ~690 ms | 186 187 This would allow for ~1.5 seconds ( ~1500 ms ) or ~1087 packets of "acceptable window". 188 189 Diagram Google Slides link: https://docs.google.com/presentation/d/1gkgs0uZ6YDqRBUeYwPjZWI2JgWBN_54CXdNvueBpjXc/edit?usp=sharing 190 191 ## Performance considerations 192 193 This library was originally designed to monitor RTP video at rates <20 Mb/s, and has not been tested for video rates higher than this. e.g. Not tested with SMPTE-2110 video transport. The b-tree operation times should mostly be <200 ns, so there's a chance it will work ok, but it would need to be carefully tested and potentially some tuning could be done. 194 195 Please also note that B-Tree "degree" is currently hard coded to three (3). Tuning this is likely to be required for higher packet rates. 196 197 ## RTP Header 198 199 https://www.rfc-editor.org/rfc/rfc3550#section-5.1 200 201 ```bash 202 5.1 RTP Fixed Header Fields 203 204 The RTP header has the following format: 205 206 0 1 2 3 207 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 208 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 209 |V=2|P|X| CC |M| PT | sequence number | 210 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 211 | timestamp | 212 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 213 | synchronization source (SSRC) identifier | 214 +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ 215 | contributing source (CSRC) identifiers | 216 | .... | 217 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 218 ``` 219 220 ## Btree libraries for golang 221 222 This is a list of some golang btree implmentations. 223 224 For now, I've decided to use [Google's golang btree](https://pkg.go.dev/github.com/google/btree) implmentation, until I know a reason not too. 225 226 Functions: https://pkg.go.dev/github.com/google/btree#pkg-functions 227 228 The scylladb writeup make it look like a reasonable library: 229 230 https://www.scylladb.com/2022/04/27/shaving-40-off-googles-b-tree-implementation-with-go-generics/ 231 232 We are using the "generic" implemention: https://github.com/google/btree/issues/41 233 234 Other AVL tree implementations 235 236 https://github.com/VictorLowther/btree 237 238 https://github.com/tsuzu/go-avl/blob/master/avl_test.go 239 240 https://github.com/ross-oreto/go-tree 241 242 ### Note for myself 243 244 Markdown syntax link: https://www.markdownguide.org/basic-syntax/