vitess.io/vitess@v0.16.2/changelog/11.0/11.0.0/release_notes.md (about) 1 This release complies with VEP-3 which removes the upgrade order requirement. Components can be upgraded in any order. It is recommended that the upgrade order should still be followed if possible, except to canary test the new version of VTGate before upgrading the rest of the components. 2 3 ## Known Issues 4 5 - A critical vulnerability [CVE-2021-44228](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228) in the Apache Log4j logging library was disclosed on Dec 9 2021. 6 The project provided release `2.15.0` with a patch that mitigates the impact of this CVE. It was quickly found that the initial patch was insufficient, and additional CVEs 7 [CVE-2021-45046](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45046) and [CVE-2021-44832](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44832) followed. 8 These have been fixed in release `2.17.1`. This release of Vitess, `v11.0.0`, uses a version of Log4j below `2.17.1`, for this reason, we encourage you to use version `v11.0.4` instead, to benefit from the vulnerability patches. 9 10 - An issue where the value of the `-force` flag is used instead of `-keep_data` flag's value in v2 vreplication workflows (#9174) is known to be present in this release. A workaround is available in the description of issue #9174. 11 12 ## Bug fixes 13 ### Build/CI 14 * update moby/term to fix darwin build issue #7787 15 * Removing Linux/amd64 specific subdependencies #7796 16 * CI: run some tests inside docker to workaround GH Actions issue #7868 17 * Fix flaky race condition in vtexplain #7930 18 * Fixing flaky endtoend/tablegc tests #7947 19 * Add libwww-perl to allow pt-osc wrapper to work in vitess/lite #8141 20 ### Cluster management 21 * Add missing return on a failed `GetShard` call in `FindAllShardsInKeyspace` #7992 22 * Increase the default srv_topo_timeout #8011 23 * parser: support index name in FOREIGN KEY clause; Online DDL to reject FK clauses #8058 24 * Correctly parse cell names with dashes in tablet aliases #8167 25 * /healthz should not report ok when vttablet is not connected to mysql #8238 26 * [topo] Refactor `ExpandCells` to not error on valid aliases #8291 27 * tm state: don't populate metadata in updateLocked #8362 28 * [grpcvtctldserver] Fix backup detail limit math #8402 29 ### Query Serving 30 * fix select star, col1 orderby col1 bug. #7743 31 * VTExplain: Add support for multi table join query #7829 32 * Routing of Information Schema Queries #7841 33 * Fix + integration test for `keyspaces_to_watch` routing regression [fixes #7882] #7873 34 * Revert "[tablet, queryrules] Extend query rules to check comments" #7897 35 * vtctl: return error on invalid ddl_strategy #7923 36 * Panic when EOF after @ symbol #7925 37 * Fixes encoding of sql strings #8033 38 * Make TwoPC engine initialization async. #8048 39 * Fix for issue with information_schema queries with both table name and schema name predicates #8087 40 * Fix for transactions not allowed to finish during PlannedReparentShard #8089 41 * Set fully parsed to false when ignoring errors #8094 42 * Fix for query and sub query with limits #8097 43 * Fix buffering when using reserved connections #8102 44 * Parse generated columns in DDL #8117 45 * More explicit message on online DDL parsing error #8118 46 * healthcheck: attempt to update primary only if the current tablet is serving #8121 47 * PRIMARY in index hint list for master #8160 48 * Signed int parse #8189 49 * Delete table reference alias support #8393 50 * Fix for function calls in DEFAULT value of CREATE TABLE statement in release-11.0 #8476 51 * Backport: Fixing multiple issues related to onlineddl/lifecycle #8517 52 * boolean values should not be parenthesised in default clause - release 11 #8531 53 54 55 ### VReplication 56 * VDiff: Use byte compare if weight_string() returns null for either source or target #7696 57 * Rowlog: Update rowlog for the API change made for the vstream skew alignment feature #7809 58 * Pad binlog values for binary() columns to match the value returned by mysql selects #7969 59 * Change vreplication error metric name to start with a string to pass prometheus validations #7983 60 * Pass the provided keyspace through to `UpdateDisableQueryService` rather than hard-coding the sourceKeyspace #8020 61 * Fix vreplication timing metrics #8024 62 * Switchwrites: error if no tablets available on target for reverse replication #8142 63 * Remove noisy vexec logs #8144 64 * VReplicationExec: don't create new stats objects on a select #8166 65 * Binlog JSON Parser: handle inline types correctly for large documents #8187 66 * Schema Tracking Flaky Test: Ignore unrelated gtid progress events #8283 67 * Adds padding to keyrange comparison #8296 68 * VReplication Reverse Workflows: add keyspace scope to vindex while creating reverse vreplication streams #8385 69 * OnlineDDL/Vreplication stress test: investigating failures #8390 70 * Ignore SBR statements from pt-table-checksum #8396 71 * Return from throttler goroutine if context is cancelled to prevent goroutine leaks #8489 72 * VDiff: Add BIT datatype to list of byte comparable types #8401 73 * Fix excessive VReplication logging to file and db #8521 74 75 ### VTAdmin 76 * Add missing return in `vtctld-*` DSN case, and log any flag that gets ignored #7872 77 * [vtadmin-web] Do not parse numbers/booleans in URL query parameters by default #8100 78 * [vtadmin-web] Small bugfix where stream source displayed instead of target #8311 79 ## CI/Build 80 ### Build/CI 81 * Planbuilder: Fix fuzzer #7952 82 * java: Bump SNAPSHOT version to 11.0.0-SNAPSHOT after Vitess release v10 #7968 83 * Add optional TLS feature to gRPC servers #8049 84 * Add image and base image arguments to build.sh #8064 85 * [fuzzing] Add report #8128 86 * CI: fail PR if it does not have required labels #8147 87 * trigger pr-labels workflow when labels added/removed from PR #8157 88 * Add vmg as a maintainer #8186 89 * Refactor vtorc endtoend tests #8215 90 * Use shared testing helper which guards against races in tmclient construction #8300 91 * moved multiple vtgate and tabletgateway to individual shards #8305 92 * include squashed PRs in release notes #8326 93 * pr-labels workflow: only check actual PRs #8327 94 * Heuristic fix for a flaky test: allow for some noise to pass through #8339 95 * ci: upgrade to Go 1.16 #8274 96 * mod: upgrade etcd to stable version #8357 97 * Resolve go.mod issue with direct dependency on planetscale/tengo #8383 98 * Initial GitHub Docker Build Setup #8399 99 100 ### Cluster management 101 * make: build vitess as static binaries by default #7795 102 * Fix TableGC flaky test by reducing check interval #8270 103 ### Java 104 * Bump commons-io from 2.6 to 2.7 in /java #7953 105 ### Other 106 * Bump MySQL version for docker builds to 8.0.23 #7811 107 ### Query Serving 108 * Online DDL Vreplication test suite: adding tests #8213 109 * Online DDL/VReplication: more passing tests (no UK/PK) #8225 110 * Online DDL: reject ALTER TABLE...RENAME statements #8227 111 * Online DDL/VReplication: fail on existence of FOREIGN KEY #8228 112 * Online DDL/VReplication: test PK column case change #8336 113 * Online DDL/VReplication: add PK DATETIME->TIMESTAMP test #8338 114 * Bugfix: assign cexpr.references for column CONVERTed to utf8mb4 #8355 115 116 117 ### VReplication 118 * Towards a VReplication/OnlineDDL testing suite #8181 119 * Online DDL/Vreplication: column type awareness #8239 120 * remove duplicate ReadMigrations, fixing build #8258 121 * VReplication (and by product, Online DDL): support GENERATED column as part of PRIMARY KEY #8335 122 ### vttestserver 123 * docker/vttestserver: Set max_connections in default-fast.cnf at container build time #7810 124 ## Documentation 125 ### Cluster management 126 * Enhance k8stopo flag documentation #8458 127 ### Build/CI 128 * Update version for latest snapshot #7801 129 * v10 GA Release Notes #7964 130 * Updates and Corrections to v9 Release Notes #8295 131 ### Query Serving 132 * Update link to 'Why FK not supported in Online DDL' blog post #8372 133 ### Other 134 * correct several wrong words #7822 135 * MAINTAINERS.md: Update enisoc's email. #7909 136 * Modification of the Markdown list format in the release notes template #7962 137 * Update vreplicator docs #8014 138 * Release notes: add Known Issues #8027 139 ## Enhancement 140 ### Observability 141 * [trace] Add logging support to opentracing plugins #8289 142 143 144 ### Build/CI 145 * Makefile: add cross-build target for cross-compiling client binaries #7806 146 * git: improve signoff detection #7852 147 * Release notes generation #7932 148 * vitess/lite docker build fails for mysql80 #7943 149 * Fix a gofmt warning #7959 150 * docker/vttestserver/run.sh: Add $CHARSET environment variable #7970 151 * docker/lite/install_dependencies.sh: If the dependency install loop reaches its max number of retries, consider that a failure; exit the script nonzero so the build halts. #7976 152 * Add commit count and authors to the release notes #7982 153 * Automate version naming and release tagging #8034 154 * Make sure to only allow codegen when the rest of the code compiles #8169 155 * Add options to vttestserver to pass -foreign_key_mode, -enable_online_ddl, and -enable_direct_ddl through to vtcombo #8177 156 ### Cluster management 157 * Dynamic throttle metric threshold #7742 158 * Bump Bootstrap version, per CVE-2018-14040 (orchestrator/vtorc) #7824 159 * Check tablet alias before removing after error stream #7915 160 * vttablet/tabletmanager - add additional test for tmstate.Open() #7993 161 * Add `vttablet_restore_done` hook #8007 162 * Added ValidateVSchema #8012 163 * Add RemoteOperationTimeout to both legacy and grpc `ChangeTabletType` implementations. #8052 164 * [vtctldserver] Add guard against self-reparent, plus misc updates #8084 165 * [tm_state] updateLocked should re-populate local metadata tables to reflect promotion rule changes #8107 166 * [vtctld] Migrate `ApplyVSchema` to `VtctldServer` #8113 167 * vtctl.generateShardRanges -> key.GenerateShardRanges #8134 168 * etcd: add grpc.WithBlock to client config #8205 169 * [workflow] Add tracing to `GetWorkflows` endpoint #8266 170 * [vtctldclient] Add legacy shim #8284 171 * [vtctldserver] Add tracing #8285 172 * [vtctldserver] Add additional backup info fields #8321 173 * [workflow] Call `scanWorkflow` concurrently #8272 174 ### Query Serving 175 * [Gen4] Implemented table alias functionality in the semantic analysis #7629 176 * Improved error messages (tabletserver) #7747 177 * Allow modification of tablet unhealthy_threshold via debugEnv #7753 178 * [Gen4] Initial Horizon Planning #7775 179 * [tablet, queryrules] Extend query rules to check comments #7784 180 * Add support for showing global gtid executed per shard #7856 181 * Minor cleanups around errors on the vtgate #7864 182 * log unsupported queries #7865 183 * Show databases like #7912 184 * Add rank as reserved keyword #7944 185 * Online DDL: introducing ddl_strategy `-singleton-context` flag #7946 186 * Ignore the error and log as warn if not able to validate the current system setting for check and ignore case #8004 187 * Detect and signal schema changes on vttablets #8005 188 * DDL bypass plan #8013 189 * Online DDL: progress & ETA for Vreplication migrations #8015 190 * Scatter errors as warning in olap query #8018 191 * livequeryz: livequeryz/terminate link should be relative #8025 192 * Handle online DDL user creation if we do not have SUPER privs (e.g. AWS Aurora) #8038 193 * Update the schema copy with minimal changes #8067 194 * Gen4: Support for order by column not present in projection #8070 195 * Schema tracking in vtgate #8074 196 * protobuf: upgrade #8075 197 * added mysql 8.0 reserved keywords #8086 198 * Fix ghost/pt-osc in the external DB case where MySQL might be reporting #8115 199 * Support for vtgate -enable_direct_ddl flag #8116 200 * add transaction ID to query log for acquisition and analysis #8133 201 * Inline reference #8136 202 * Improve ScatterErrorsAsWarnings functionality #8139 203 * Gen4: planning Select IN #8155 204 * Primary key name #8188 205 * Online DDL: read and publish gh-ost FATAL message where possible #8192 206 * Expose inner net.Conn to be able to write better unit tests #8217 207 * vtgate: validate important flags during startup #8218 208 * Online DDL/VReplication: AUTO_INCREMENT support and tests #8223 209 * [Gen4] some renaming from v4 to Gen4 #8234 210 * Schema Tracking: Optimize and Bug Fix #8243 211 * gen4: minor refactoring to improve readability #8245 212 * gen4: plan more opcodes #8254 213 * Online DDL: report rows_copied for migration (initial support in gh-ost) #8255 214 * Schema tracking: new tables in sharded keyspace #8256 215 * [parser] use table_alias for ENGINE option in CREATE TABLE stmt #8307 216 * Add /debug/env for vtgate #8292 217 * gen4: outer joins #8312 218 * Gen4: expand star in projection list #8325 219 * gen4: Fail all queries not handled well by gen4 #8359 220 * Gen4 fail more2 #8382 221 * SHOW VITESS_MIGRATION '...' LOGS, retain logs for 24 hours #8532 222 * [11.0] query serving to continue when topo server restarts #8533 223 * [11.0] Disable allowing set statements on system settings by default #8540 224 ### VReplication 225 * Change local example to use v2 vreplication flows and make v2 flows as the default #8527 226 * Use Dba user when Vexec is runAsAdmin #7731 227 * Add table for logging stream errors, state changes and key workflow steps #7831 228 * Fix some static check warning #7960 229 * VSchema Validation on ReshardWorkflow Creation #7977 230 * Tracking `rows_copied` #7980 231 * Vdiff formatting improvements #8079 232 * Ignore generated columns in workflows #8129 233 * VReplication Copy Phase: Increase default replica lag tolerance. Also make it and copy timeout modifiable via flags #8130 234 * Materialize: Add additional comparison operators in Materialize and fix bug where they not applied for sharded keyspaces #8247 235 * Copy Phase: turn on OptimizeInserts by default #8248 236 * Tracker/VStreamer: only reload schema for tables in current database and not for internal table artifacts #8257 237 * Online DDL/Vreplication suite: support ENUM->VARCHAR/TEXT type change #8275 238 * Added TableName to DiffReport struct. #8279 239 * New VReplication lag metric #8306 240 * VStream API: add heartbeat for idle streams #8244 241 * Online DDL/VReplication: support non-UTF8 character sets #8322 242 * Online DDL/Vreplication suite: fix test for no shared UK #8334 243 * Online DDL/VReplication: support DROP+ADD column of same name #8337 244 * Online DDL/VReplication test suite: support ENUM as part of PRIMARY KEY #8345 245 * Change local example to use v2 vreplication flows #8527 246 * Tablet Picker: add metric to record lack of available tablets #8403 247 ### VTAdmin 248 * [vtadmin-web] Add useSyncedURLParam hook to persist filter parameter in the URL #7857 249 * [vtadmin-web] Display more data on /gates view and add filtering #7876 250 * [vtadmin] Promote ErrNoSchema to a TypedError which returns http 404 #7885 251 * [vtadmin] gate/tablet/vtctld FQDNs #7886 252 * [vtadmin-web] Display vindex data on Schema view #7917 253 * [vtadmin-web] Add filtering and source/target shards to Workflows view #7948 254 * [vtadmin-web] Add filtering + shard counts/status to Keyspaces view #7991 255 * [vtadmin-web] Display shard state on Tablets view + extract tablet utilities #7999 256 * [vtadmin] experimental tabletdebug #8003 257 * [vtadmin-web] Display timestamps + stream counts on Workflows view #8009 258 * [vtadmin-web] Updates to table styling + other CSS #8072 259 * [vtadmin-web] Add Tab components #8119 260 * [vtadmin-api] Update GetTablet to use alias instead of hostname #8163 261 * [vtadmin-api] Rename flag 'http-tablet-fqdn-tmpl' to 'http-tablet-url-tmpl' + update vtadmin flags for local example #8164 262 * Add `--tracer` flag to vtadmin and actually start tracing #8165 263 * [vtadmin-web] The hastiest Tablet view (+ experimental debug vars) #8170 264 * [vtadmin-web] Add tabs to Workflow view #8203 265 * [vtctld] Add GetSrvVSchemas command #8221 266 * [vtadmin-api] Add HTTP endpoints for /api/srvvschemas and /api/srvvschema/{cluster}/{cell} #8226 267 * [vtadmin-web] Add QPS and VReplicationQPS charts to Tablet view #8263 268 * [vtadmin-web] Add client-side error handling interface + Bugsnag implementation #8287 269 * [vtadmin-web] Add chart for stream vreplication lag across all streams in a workflow #8331 270 * [vtctldproxy] Add more annotations to vtctld Dial calls #8346 271 272 273 ### vttestserver 274 * Vttest create db #7989 275 * Adds an environment variable to set the MySQL max connections limit in vttestserver docker image #8210 276 ## Feature Request 277 ### Build/CI 278 * [trace] Add optional flags to support flexible jaeger configs #8199 279 ### Cluster management 280 * Add VtctldServer to vtcombo #7896 281 * [vtctldserver] Migrate routing rules RPCs, and also `RebuildVSchemaGraph` #8197 282 * [vtctldserver] Migrate `CellInfo`, `CellAlias` rw RPCs #8219 283 * [vtctldserver] Add RefreshState RPCs #8232 284 ### Query Serving 285 * VTGate grpc implementation of Prepare and CloseSession #8211 286 * Schema tracking: One schema load at a time per keyspace #8224 287 ### VTAdmin 288 * [vtadmin-web] Add DataFilter + Workspace layout components #8032 289 * [vtadmin-web] Add Tooltip + HelpTooltip components #8076 290 * [vtadmin-web] Add initial Stream view, render streams on Workflow view #8091 291 * [vtadmin-web] The hastiest-ever VTExplain UI #8092 292 * [vtadmin-web] Add source-map-explorer util #8093 293 * [vtadmin-web] Add Keyspace detail view #8111 294 * [vtadmin-api] Add GetKeyspace endpoint #8125 295 * [workflow] Add vreplication_log data to workflow protos, and `VtctldServer.GetWorkflows` method #8261 296 * [vtadmin] Add debug endpoints #8268 297 ## Internal Cleanup 298 ### Build/CI 299 * Makefile: fix cross-build comments #8246 300 * remove unused hooks with refs to master branch #8250 301 * [vtctldserver] Update tests to prevent races during tmclient init #8320 302 * sks-keyservers.net seems to be finally dead, replace with #8363 303 * endtoend: change log level of etcd start from error to info #8370 304 ### Cluster management 305 * Online DDL: code cleanup #7589 306 * [wrangler|topotools] Migrate `UpdateShardRecords`, `RefreshTabletsByShard`, and `{Get,Save}RoutingRules` #7965 307 * Tear down old stream_migrater shim, now that we're fully in `package workflow` #8073 308 * [mysqlctl] Restructure `MetadataManager` to reduce public API surface area #8152 309 * naming: master to primary #8251 310 * vtorc: code cleanup #8269 311 ### Query Serving 312 * Plan StreamExecute Queries #7941 313 ### VReplication 314 * [workflow] extract migration targets from wrangler #7934 315 * [wrangler|workflow] Extract vrStream type to workflow.VReplicationStream #7966 316 * [wrangler|workflow] Extract `workflowState` and `workflowType` out to `package workflow` #7967 317 * [wrangler|workflow] extract `*wrangler.streamMigrater` to `workflow.StreamMigrator` #8008 318 * [workflow] Migrate `getCellsWith{Shard,Table}ReadsSwitched`, `TrafficSwitchDirection` and `TableRemovalType` to package workflow #8190 319 * [workflow] Cleanup wrangler wrappers, migrate `checkIfJournalExistsOnTablet` to package workflow #8193 320 * Backports of #8403 #8483 #8489 #8401 #8521 #8396 from main into release 11.0 #8536 321 ### VTAdmin 322 * [vtadmin-api] Replace magic numbers with `net/http` constants #8127 323 * [vtadmin-web] Move single-entity view components into subfolders #8202 324 * [vtadmin] Ensure we log any errors when closing the tracer #8262 325 ## Other 326 ### Build/CI 327 * add vtctldclient to binary directory #7889 328 ### Cluster management 329 * vttablet/tabletmanager: add isInSRVKeyspace/isShardServing #7929 330 ### Other 331 * Change VitessInputFormat key type #199 332 * Online DDL plan via Send; "singleton" migrations on tablets #7785 333 * flaky onlineddl tests: reduce -online_ddl_check_interval #7847 334 * Looking into flaky endtoend upgrade test #7900 335 * fixing flaky upgrade test #7901 336 ### Query Serving 337 * Introduce Concatenated Fixed-width Composite or CFC vindex #7537 338 * Add common tags for stats backends that support it #7651 339 * Add support for showing global vgtid executed #7797 340 * perf: vttablet/mysql optimizations #7800 341 * Fix bug with reserved connections to stale tablets #7879 342 * Memory Sort to close the goroutines when callback returns error #7903 343 * OnlineDDL: more migration check ticks upon migration start #7961 344 * Update gh-ost binary to v1.1.3 #8021 345 * VReplication Online DDL: fix classification of virtual columns #8043 346 ### VReplication 347 * VReplicationErrors metric: use . as delimiter instead of _ to behave well with Prometheus #7807 348 ### VTAdmin 349 * Update `GetSchema` filtering to exclude shards where `IsMasterServing` but no `MasterAlias` #7805 350 * [vtadmin-api] Reintroduce include_non_serving_shards opt to GetSchema #7814 351 * [vtadmin-web] Add DataCell component #7817 352 * Rewrite useTableDefinitions hook as getTableDefinitions util #7821 353 * [vtadmin-web] Display (approximate) table sizes + row count on /schemas view #7826 354 * [vtadmin-web] Add Pip + TabletServingPip components #7827 355 ## Performance 356 ### Query Serving 357 * vttablet: stream consolidation #7752 358 * perf: optimize bind var generation #7828 359 ### VReplication 360 * Optimize the catchup phases by filtering out rows which not within range of copied pks during inserts #7708 361 * Ability to compress gtid when stored in _vt.vreplication's pos column #7877 362 * Performance & benchmarks (table copying) #7881 363 * Dynamic packet sizing #7933 364 * perf: vreplication client CPU usage #7951 365 * VDIff: fix performance regression introduced by progress logging #8016 366 * proto: Generate faster code using vtprotobuf #8173 367 * proto: enable pooling for vreplication #8273 368 ## Testing 369 ### Build/CI 370 * Attempt to fix TLS Server Flaky test #7842 371 * TestMakeCommonTags Flaky Test: match elements #7843 372 * upgrade tests: test against v9.0.0 #7848 373 * FOSSA scan added #7862 374 * ci: fix all racy tests #7904 375 * CI: Revert docker change for unit tests from #7868 #7940 376 * Add online ddl on start queries to schema list in vtexplain tablet #8397 377 ### Java 378 * [Java] JDBC mysql driver test #8154 379 ### Other 380 * Fuzzing: Fixup oss-fuzz build script #7782 381 * Wrangler tests: Return a fake tablet in the wrangler test dialer to avoid tablet picker errors spamming the test logs #7863 382 ### Query Serving 383 * clean up test #7816 384 * tabletserver: fix flaky test #7851 385 * Planbuilder: Add fuzzer #7902 386 * mysql: Small adjustments to fuzzer #7907 387 * vtgate/engine: Add fuzzer #7914 388 * Adding Fuzzer Test Cases #8106 389 * Addition of fuzzer issues #8195 390 ### VReplication 391 * vstreamer: Add fuzzer #7918 392 ### vttestserver 393 * Speedup new vttestserver tests #8229 394 * Vttestserver docker test #8253 395 ### Cluster management 396 * Make timestamp authoritative for master information #8381 397 398 399 The release includes 1080 commits (excluding merges) 400 401 Thanks to all our contributors: @AdamKorcz, @GuptaManan100, @Hellcatlk, @Johnny-Three, @acharisshopify, @ajm188, @alexrs, @aquarapid, @askdba, @deepthi, @doeg, @dyv, @enisoc, @frouioui, @gedgar, @guidoiaquinti, @harshit-gangal, @hkdsun, @idvoretskyi, @jmoldow, @kirs, @mcronce, @narcsfz, @noxiouz, @rafael, @rohit-nayak-ps, @setassociative, @shlomi-noach, @systay, @tokikanno, @vmg, @wangmeng99, @yangxuanjia, @zhangshj-inspur