github.com/anycable/anycable-go@v1.5.1/etc/k6/results/2023-10-broker.md (about) 1 # Broker benchmarks 2 3 Scenario: broadcast + history retrieval. 4 5 Key metrics: rtt trend, broadcast trend, history trend. 6 7 No RPC server (gobench-cable binary). 8 9 Multi-node setup uses Traefik as a load balancer and 3 AnyCable-Go nodes. 10 11 ## NUM_CHANNELS=5, SENDING_RATE=0.7, SENDERS_RATIO=0.5 12 13 This configuration helps to see the difference between brokers. Having less channels or broadcast per seconds works fine for all brokers. 14 15 Speaking of the number we have: 16 17 - ~125-150 broadcast/s; 18 - ~120-150k historical messages received out of ~1kk in total (70-80k/s). 19 20 ## Multi-node, no broker, Redis pub/sub 21 22 This setup doesn't use any broker; we can refer to it as baseline setup for multi-node tests. 23 24 ```sh 25 ✓ successful connection 26 ✓ successful subscription 27 ✗ history received 28 ↳ 0% — ✓ 0 / ✗ 3267 29 30 acks_rcvd............: 7198 49.775111/s 31 broadcast_duration...: avg=4.13ms min=0s med=3ms max=43ms p(90)=7ms p(95)=10ms 32 broadcasts_rcvd......: 1688013 11672.830561/s 33 broadcasts_sent......: 7198 49.775111/s 34 checks...............: 66.66% ✓ 6534 ✗ 3267 35 data_received........: 271 MB 1.9 MB/s 36 data_sent............: 2.9 MB 20 kB/s 37 history_duration.....: avg=266.91µs min=0s med=0s max=17ms p(90)=0s p(95)=1ms 38 history_rcvd.........: 19 0.131387/s 39 iteration_duration...: avg=42.11s min=39.07s med=42.12s max=45.18s p(90)=43.37s p(95)=43.74s 40 iterations...........: 3267 22.591732/s 41 rtt..................: avg=2.93ms min=0s med=2ms max=33ms p(90)=6ms p(95)=8ms 42 suback...............: avg=335.47µs min=0s med=0s max=19ms p(90)=0s p(95)=2ms 43 vus..................: 6 min=6 max=1500 44 vus_max..............: 1500 min=1500 max=1500 45 ws_connecting........: avg=4.14ms min=594.08µs med=1.78ms max=113.94ms p(90)=8.48ms p(95)=11.67ms 46 ws_msgs_received.....: 1759570 12167.65657/s 47 ws_msgs_sent.........: 13732 94.958575/s 48 ws_sessions..........: 3267 22.591732/s 49 ``` 50 51 ## Multi-node, Redis broker, Redis pub/sub 52 53 ```sh 54 ✓ successful connection 55 ✓ successful subscription 56 ✓ history received 57 58 acks_rcvd............: 7183 49.690276/s 59 broadcast_duration...: avg=4.21ms min=0s med=3ms max=44ms p(90)=8ms p(95)=10ms 60 broadcasts_rcvd......: 1673008 11573.469106/s 61 broadcasts_sent......: 7183 49.690276/s 62 checks...............: 100.00% ✓ 9798 ✗ 0 63 data_received........: 406 MB 2.8 MB/s 64 data_sent............: 2.9 MB 20 kB/s 65 history_duration.....: avg=1.71ms min=0s med=1ms max=26ms p(90)=3ms p(95)=5ms 66 history_rcvd.........: 269789 1866.33576/s 67 iteration_duration...: avg=42.07s min=38.46s med=42.07s max=45.3s p(90)=43.3s p(95)=43.63s 68 iterations...........: 3266 22.593407/s 69 rtt..................: avg=3.15ms min=0s med=3ms max=27ms p(90)=6ms p(95)=8ms 70 suback...............: avg=132.57µs min=0s med=0s max=12ms p(90)=0s p(95)=1ms 71 vus..................: 2 min=2 max=1500 72 vus_max..............: 1500 min=1500 max=1500 73 ws_connecting........: avg=3.01ms min=584.7µs med=1.61ms max=30.99ms p(90)=6.93ms p(95)=9.54ms 74 ws_msgs_received.....: 2014672 13937.018921/s 75 ws_msgs_sent.........: 13715 94.877089/s 76 ws_sessions..........: 3266 22.593407/s 77 ``` 78 79 ## Multi-node, embedded NATS broker and pub/sub 80 81 ```sh 82 ✓ successful connection 83 ✓ successful subscription 84 ✓ history received 85 86 acks_rcvd............: 7206 49.568904/s 87 broadcast_duration...: avg=4.78ms min=0s med=4ms max=221ms p(90)=9ms p(95)=11ms 88 broadcasts_rcvd......: 1692092 11639.626116/s 89 broadcasts_sent......: 7206 49.568904/s 90 checks...............: 100.00% ✓ 9792 ✗ 0 91 data_received........: 407 MB 2.8 MB/s 92 data_sent............: 2.9 MB 20 kB/s 93 history_duration.....: avg=22.44ms min=0s med=3ms max=237ms p(90)=67.7ms p(95)=92ms 94 history_rcvd.........: 266712 1834.668541/s 95 iteration_duration...: avg=42.13s min=38.9s med=42.12s max=45.35s p(90)=43.42s p(95)=43.77s 96 iterations...........: 3264 22.452526/s 97 rtt..................: avg=1.92ms min=0s med=1ms max=63ms p(90)=5ms p(95)=7ms 98 suback...............: avg=181.37µs min=0s med=0s max=16ms p(90)=0s p(95)=1ms 99 vus..................: 2 min=2 max=1500 100 vus_max..............: 1500 min=1500 max=1500 101 ws_connecting........: avg=3.3ms min=473.95µs med=1.67ms max=44.94ms p(90)=7.85ms p(95)=10.64ms 102 ws_msgs_received.....: 2030319 13966.23473/s 103 ws_msgs_sent.........: 13734 94.473956/s 104 ws_sessions..........: 3264 22.452526/s 105 ``` 106 107 ## Single node, memory broker 108 109 ```sh 110 ✓ successful connection 111 ✓ successful subscription 112 ✓ history received 113 114 acks_rcvd............: 7157 49.461266/s 115 broadcast_duration...: avg=1.77ms min=0s med=1ms max=16ms p(90)=3ms p(95)=4ms 116 broadcasts_rcvd......: 1671812 11553.715077/s 117 broadcasts_sent......: 7157 49.461266/s 118 checks...............: 100.00% ✓ 9786 ✗ 0 119 data_received........: 404 MB 2.8 MB/s 120 data_sent............: 2.9 MB 20 kB/s 121 history_duration.....: avg=931.94µs min=0s med=1ms max=21ms p(90)=2ms p(95)=3ms 122 history_rcvd.........: 267642 1849.645421/s 123 iteration_duration...: avg=42.13s min=39.04s med=42.14s max=45.28s p(90)=43.34s p(95)=43.68s 124 iterations...........: 3262 22.543335/s 125 rtt..................: avg=585.72µs min=0s med=0s max=11ms p(90)=2ms p(95)=3ms 126 suback...............: avg=77.25µs min=0s med=0s max=8ms p(90)=0s p(95)=0s 127 vus..................: 6 min=6 max=1500 128 vus_max..............: 1500 min=1500 max=1500 129 ws_connecting........: avg=948.18µs min=167µs med=521.35µs max=11.82ms p(90)=2.06ms p(95)=3.08ms 130 ws_msgs_received.....: 2010091 13891.525299/s 131 ws_msgs_sent.........: 13681 94.547937/s 132 ws_sessions..........: 3262 22.543335/s 133 ``` 134 135 ## Single node, Redis broker 136 137 ```sh 138 ✓ successful connection 139 ✓ successful subscription 140 ✓ history received 141 142 acks_rcvd............: 7181 49.690377/s 143 broadcast_duration...: avg=2.32ms min=0s med=2ms max=32ms p(90)=4ms p(95)=5ms 144 broadcasts_rcvd......: 1671940 11569.325909/s 145 broadcasts_sent......: 7181 49.690377/s 146 checks...............: 100.00% ✓ 9804 ✗ 0 147 data_received........: 404 MB 2.8 MB/s 148 data_sent............: 2.9 MB 20 kB/s 149 history_duration.....: avg=1.55ms min=0s med=1ms max=18ms p(90)=3ms p(95)=4ms 150 history_rcvd.........: 269239 1863.053542/s 151 iteration_duration...: avg=42.11s min=38.69s med=42.11s max=45.73s p(90)=43.33s p(95)=43.69s 152 iterations...........: 3268 22.613585/s 153 rtt..................: avg=863.66µs min=0s med=1ms max=31ms p(90)=2ms p(95)=3ms 154 suback...............: avg=110.77µs min=0s med=0s max=19ms p(90)=0s p(95)=1ms 155 vus..................: 5 min=5 max=1500 156 vus_max..............: 1500 min=1500 max=1500 157 ws_connecting........: avg=876.93µs min=115.54µs med=524.25µs max=12.09ms p(90)=2.02ms p(95)=2.81ms 158 ws_msgs_received.....: 2012477 13925.740336/s 159 ws_msgs_sent.........: 13717 94.917547/s 160 ws_sessions..........: 3268 22.613585/s 161 ``` 162 163 ## Single node, embedded NATS broker 164 165 ```sh 166 ✓ successful connection 167 ✓ successful subscription 168 ✓ history received 169 170 acks_rcvd............: 7205 49.640288/s 171 broadcast_duration...: avg=2.16ms min=0s med=2ms max=195ms p(90)=4ms p(95)=5ms 172 broadcasts_rcvd......: 1676679 11551.815128/s 173 broadcasts_sent......: 7205 49.640288/s 174 checks...............: 100.00% ✓ 9798 ✗ 0 175 data_received........: 404 MB 2.8 MB/s 176 data_sent............: 2.9 MB 20 kB/s 177 history_duration.....: avg=20.16ms min=0s med=2ms max=213ms p(90)=60ms p(95)=86ms 178 history_rcvd.........: 268459 1849.601944/s 179 iteration_duration...: avg=42.1s min=39.17s med=42.09s max=45.55s p(90)=43.41s p(95)=43.78s 180 iterations...........: 3266 22.50176/s 181 rtt..................: avg=934.21µs min=0s med=1ms max=64ms p(90)=2ms p(95)=3ms 182 suback...............: avg=96.44µs min=0s med=0s max=20ms p(90)=0s p(95)=0s 183 vus..................: 1 min=1 max=1500 184 vus_max..............: 1500 min=1500 max=1500 185 ws_connecting........: avg=914.67µs min=177.33µs med=512.52µs max=31.01ms p(90)=2.13ms p(95)=3ms 186 ws_msgs_received.....: 2016894 13895.794378/s 187 ws_msgs_sent.........: 13737 94.643807/s 188 ws_sessions..........: 3266 22.50176/s 189 ``` 190 191 ## Notes 192 193 Commands used to run server with different configurations: 194 195 ```sh 196 # Single node, no broker 197 PORT=8080 ANYCABLE_DEBUG=0 ANYCABLE_LOG_LEVEL=info make run-gobench 198 199 # Single node, memory 200 ANYCABLE_BROKER=memory PORT=8080 ANYCABLE_DEBUG=0 ANYCABLE_LOG_LEVEL=info make run-gobench 201 202 # Single node, Redis 203 ANYCABLE_BROKER=redis PORT=8080 ANYCABLE_DEBUG=0 ANYCABLE_LOG_LEVEL=info make run-gobench 204 205 # Multi-node 206 PORT=8081 ANYCABLE_BROKER=redis ANYCABLE_PUBSUB=redis ANYCABLE_DEBUG=0 ANYCABLE_LOG_LEVEL=info make run-gobench 207 PORT=8082 ANYCABLE_BROKER=redis ANYCABLE_PUBSUB=redis ANYCABLE_DEBUG=0 ANYCABLE_LOG_LEVEL=info make run-gobench 208 PORT=8083 ANYCABLE_BROKER=redis ANYCABLE_PUBSUB=redis ANYCABLE_DEBUG=0 ANYCABLE_LOG_LEVEL=info make run-gobench 209 (cd etc/traefik && traefik --configFile=traefik.yml) 210 211 # Multi-node w/ eNATS 212 ANYCABLE_EMBED_NATS=1 ANYCABLE_ENATS_CLUSTER=nats://localhost:4342 ANYCABLE_ENATS_ADDR=nats://localhost:4242 ANYCABLE_ENATS_CLUSTER_ROUTES=nats://localhost:4342 PORT=8081 ANYCABLE_PRESETS=broker ANYCABLE_BROADCAST_ADAPTER=nats ANYCABLE_DEBUG=0 ANYCABLE_LOG_LEVEL=info ANYCABLE_ENATS_DEBUG=true make run-gobench 213 ANYCABLE_EMBED_NATS=1 ANYCABLE_ENATS_CLUSTER=nats://localhost:4343 ANYCABLE_ENATS_CLUSTER_ROUTES=nats://localhost:4342 ANYCABLE_ENATS_ADDR=nats://localhost:4243 PORT=8082 ANYCABLE_PRESETS=broker ANYCABLE_BROADCAST_ADAPTER=nats ANYCABLE_DEBUG=0 ANYCABLE_LOG_LEVEL=info ANYCABLE_ENATS_DEBUG=true make run-gobench 214 ANYCABLE_EMBED_NATS=1 ANYCABLE_ENATS_CLUSTER=nats://localhost:4344 ANYCABLE_ENATS_CLUSTER_ROUTES=nats://localhost:4342 ANYCABLE_ENATS_ADDR=nats://localhost:4244 PORT=8083 ANYCABLE_PRESETS=broker ANYCABLE_BROADCAST_ADAPTER=nats ANYCABLE_DEBUG=0 ANYCABLE_LOG_LEVEL=info ANYCABLE_ENATS_DEBUG=true make run-gobench 215 ```