github.com/kaptinlin/jsonschema@v0.4.6/testdata/JSON-Schema-Test-Suite/README.md (about) 1 # JSON Schema Test Suite 2 3 [](https://github.com/json-schema-org/.github/blob/main/CODE_OF_CONDUCT.md) 4 [](https://www.repostatus.org/#active) 5 [](https://opencollective.com/json-schema) 6 7 [](https://zenodo.org/badge/latestdoi/5952934) 8 [](https://github.com/json-schema-org/JSON-Schema-Test-Suite/actions?query=workflow%3A%22Test+Suite+Sanity+Checking%22) 9 10 This repository contains a set of JSON objects that implementers of JSON Schema validation libraries can use to test their evaluators. 11 12 It is meant to be language agnostic and should require only a JSON parser. 13 The conversion of the JSON objects into tests within a specific language and test framework of choice is left to be done by the validator implementer. 14 15 The recommended workflow of this test suite is to clone the `main` branch of this repository as a `git submodule` or `git subtree`. The `main` branch is always stable. 16 17 ## Coverage 18 19 All JSON Schema specification releases should be well covered by this suite, including drafts 2020-12, 2019-09, 07, 06, 04 and 03. 20 Drafts 04 and 03 are considered "frozen" in that less effort is put in to backport new tests to these versions. 21 22 Additional coverage is always welcome, particularly for bugs encountered in real-world implementations. 23 If you see anything missing or incorrect, please feel free to [file an issue](https://github.com/json-schema-org/JSON-Schema-Test-Suite/issues) or [submit a PR](https://github.com/json-schema-org/JSON-Schema-Test-Suite). 24 25 @gregsdennis has also started a separate [test suite](https://github.com/gregsdennis/json-schema-vocab-test-suites) that is modelled after this suite to cover third-party vocabularies. 26 27 ## Introduction to the Test Suite Structure 28 29 The tests in this suite are contained in the `tests` directory at the root of this repository. 30 Inside that directory is a subdirectory for each released version of the specification. 31 32 The structure and contents of each file in these directories is described below. 33 34 In addition to the version-specific subdirectories, two additional directories are present: 35 36 1. `draft-next/`: containing tests for the next version of the specification whilst it is in development 37 2. `latest/`: a symbolic link which points to the directory which is the most recent release (which may be useful for implementations providing specific entry points for validating against the latest version of the specification) 38 39 Inside each version directory there are a number of `.json` files each containing a collection of related tests. 40 Often the grouping is by property under test, but not always. 41 In addition to the `.json` files, each version directory contains one or more special subdirectories whose purpose is [described below](#subdirectories-within-each-draft), and which contain additional `.json` files. 42 43 Each `.json` file consists of a single JSON array of test cases. 44 45 ### Terminology 46 47 For clarity, we first define this document's usage of some testing terminology: 48 49 | term | definition | 50 |-----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------| 51 | **test suite** | the entirety of the contents of this repository, containing tests for multiple different releases of the JSON Schema specification | 52 | **test case** | a single schema, along with a description and an array of *test*s | 53 | **test** | within a *test case*, a single test example, containing a description, instance and a boolean indicating whether the instance is valid under the test case schema | 54 | **test runner** | a program, external to this repository and authored by a user of this suite, which is executing each of the tests in the suite | 55 56 An example illustrating this structure is immediately below, and a JSON Schema containing a formal definition of the contents of test cases can be found [alongside this README](./test-schema.json). 57 58 ### Sample Test Case 59 60 Here is a single *test case*, containing one or more tests: 61 62 ```json 63 { 64 "description": "The test case description", 65 "schema": { "type": "string" }, 66 "tests": [ 67 { 68 "description": "a test with a valid instance", 69 "data": "a string", 70 "valid": true 71 }, 72 { 73 "description": "a test with an invalid instance", 74 "data": 15, 75 "valid": false 76 } 77 ] 78 } 79 ``` 80 81 ### Subdirectories Within Each Draft 82 83 There is currently only one additional subdirectory that may exist within each draft test directory. 84 85 This is: 86 87 1. `optional/`: Contains tests that are considered optional. 88 89 Note, the `optional/` subdirectory today conflates many reasons why a test may be optional -- it may be because tests within a particular file are indeed not required by the specification but still potentially useful to an implementer, or it may be because tests within it only apply to programming languages with particular functionality (in 90 which case they are not truly optional in such a language). 91 In the future this directory structure will be made richer to reflect these differences more clearly. 92 93 ## Using the Suite to Test a Validator Implementation 94 95 The test suite structure was described [above](#introduction-to-the-test-suite-structure). 96 97 If you are authoring a new validator implementation, or adding support for an additional version of the specification, this section describes: 98 99 1. How to implement a test runner which passes tests to your validator 100 2. Assumptions the suite makes about how the test runner will configure your validator 101 3. Invariants the test suite claims to hold for its tests 102 103 ### How to Implement a Test Runner 104 105 Presented here is a possible implementation of a test runner. 106 The precise steps described do not need to be followed exactly, but the results of your own procedure should produce the same effects. 107 108 To test a specific version: 109 110 * For 2019-09 and later published drafts, implementations that are able to detect the draft of each schema via `$schema` SHOULD be configured to do so 111 * For draft-07 and earlier, draft-next, and implementations unable to detect via `$schema`, implementations MUST be configured to expect the draft matching the test directory name 112 * Load any remote references [described below](#additional-assumptions) and configure your implementation to retrieve them via their URIs 113 * Walk the filesystem tree for that version's subdirectory and for each `.json` file found: 114 115 * if the file is located in the root of the version directory: 116 117 * for each test case present in the file: 118 119 * load the schema from the `"schema"` property 120 * load (or log) the test case description from the `"description"` property for debugging or outputting 121 * for each test in the `"tests"` property: 122 123 * load the instance to be tested from the `"data"` property 124 * load (or log) the individual test description from the `"description"` property for debugging or outputting 125 126 * use the schema loaded above to validate whether the instance is considered valid under your implementation 127 128 * if the result from your implementation matches the value found in the `"valid"` property, your implementation correctly implements the specific example 129 * if the result does not match, or your implementation errors or crashes, your implementation does not correctly implement the specific example 130 131 * otherwise it is located in a special subdirectory as described above. 132 Follow the additional assumptions and restrictions for the containing subdirectory, then run the test case as above. 133 134 If your implementation supports multiple versions, run the above procedure for each version supported, configuring your implementation as appropriate to call each version individually. 135 136 ### Additional Assumptions 137 138 1. The suite, notably in its `refRemote.json` file in each draft, expects a number of remote references to be configured. 139 These are JSON documents, identified by URI, which are used by the suite to test the behavior of the `$ref` keyword (and related keywords). 140 Depending on your implementation, you may configure how to "register" these *either*: 141 142 * by directly retrieving them off the filesystem from the `remotes/` directory, in which case you should load each schema with a retrieval URI of `http://localhost:1234` followed by the relative path from the remotes directory -- e.g. a `$ref` to `http://localhost:1234/foo/bar/baz.json` is expected to resolve to the contents of the file at `remotes/foo/bar/baz.json` 143 144 * or alternatively, by executing `bin/jsonschema_suite remotes` using the executable in the `bin/` directory, which will output a JSON object containing all of the remotes combined, e.g.: 145 146 ``` 147 148 $ bin/jsonschema_suite remotes 149 ``` 150 ```json 151 { 152 "http://localhost:1234/baseUriChange/folderInteger.json": { 153 "type": "integer" 154 }, 155 "http://localhost:1234/baseUriChangeFolder/folderInteger.json": { 156 "type": "integer" 157 } 158 } 159 ``` 160 161 2. Test cases found within [special subdirectories](#subdirectories-within-each-draft) may require additional configuration to run. 162 In particular, when running tests within the `optional/format` subdirectory, test runners should configure implementations to enable format validation, where the implementation supports it. 163 164 ### Invariants & Guarantees 165 166 The test suite guarantees a number of things about tests it defines. 167 Any deviation from the below is generally considered a bug. 168 If you suspect one, please [file an issue](https://github.com/json-schema-org/JSON-Schema-Test-Suite/issues/new): 169 170 1. All files containing test cases are valid JSON. 171 2. The contents of the `"schema"` property in a test case are always valid 172 JSON Schemas under the corresponding specification. 173 174 The rationale behind this is that we are testing instances in a test's `"data"` element, and not the schema itself. 175 A number of tests *do* test the validity of a schema itself, but do so by representing the schema as an instance inside a test, with the associated meta-schema in the `"schema"` property (via the `"$ref"` keyword): 176 177 ```json 178 { 179 "description": "Test the \"type\" schema keyword", 180 "schema": { 181 "$ref": "https://json-schema.org/draft/2019-09/schema" 182 }, 183 "tests": [ 184 { 185 "description": "Valid: string", 186 "data": { 187 "type": "string" 188 }, 189 "valid": true 190 }, 191 { 192 "description": "Invalid: null", 193 "data": { 194 "type": null 195 }, 196 "valid": false 197 } 198 ] 199 } 200 ``` 201 See below for some [known limitations](#known-limitations). 202 203 ## Known Limitations 204 205 This suite expresses its assertions about the behavior of an implementation *within* JSON Schema itself. 206 Each test is the application of a schema to a particular instance. 207 This means that the suite of tests can test against any behavior a schema can describe, and conversely cannot test against any behavior which a schema is incapable of representing, even if the behavior is mandated by the specification. 208 209 For example, a schema can require that a string is a _URI-reference_ and even that it matches a certain pattern, but even though the specification contains [recommendations about URIs being normalized](https://json-schema.org/draft/2020-12/json-schema-core.html#name-the-id-keyword), a JSON schema cannot today represent this assertion within the core vocabularies of the specifications, so no test covers this behavior. 210 211 ## Who Uses the Test Suite 212 213 This suite is being used by: 214 215 ### Clojure 216 217 * [jinx](https://github.com/juxt/jinx) 218 * [json-schema](https://github.com/tatut/json-schema) 219 220 ### Coffeescript 221 222 * [jsck](https://github.com/pandastrike/jsck) 223 224 ### Common Lisp 225 226 * [json-schema](https://github.com/fisxoj/json-schema) 227 228 ### C++ 229 230 * [Modern C++ JSON schema validator](https://github.com/pboettch/json-schema-validator) 231 * [Valijson](https://github.com/tristanpenman/valijson) 232 233 ### Dart 234 235 * [json\_schema](https://github.com/patefacio/json_schema) 236 237 ### Elixir 238 239 * [ex\_json\_schema](https://github.com/jonasschmidt/ex_json_schema) 240 241 ### Erlang 242 243 * [jesse](https://github.com/for-GET/jesse) 244 245 ### Go 246 247 * [gojsonschema](https://github.com/sigu-399/gojsonschema) 248 * [validate-json](https://github.com/cesanta/validate-json) 249 250 ### Haskell 251 252 * [aeson-schema](https://github.com/timjb/aeson-schema) 253 * [hjsonschema](https://github.com/seagreen/hjsonschema) 254 255 ### Java 256 257 * [json-schema-validation-comparison](https://www.creekservice.org/json-schema-validation-comparison/functional) (Comparison site for JVM-based validator implementations) 258 * [json-schema-validator](https://github.com/daveclayton/json-schema-validator) 259 * [everit-org/json-schema](https://github.com/everit-org/json-schema) 260 * [networknt/json-schema-validator](https://github.com/networknt/json-schema-validator) 261 * [Justify](https://github.com/leadpony/justify) 262 * [Snow](https://github.com/ssilverman/snowy-json) 263 * [jsonschemafriend](https://github.com/jimblackler/jsonschemafriend) 264 * [OpenAPI JSON Schema Generator](https://github.com/openapi-json-schema-tools/openapi-json-schema-generator) 265 266 ### JavaScript 267 268 * [json-schema-benchmark](https://github.com/Muscula/json-schema-benchmark) 269 * [direct-schema](https://github.com/IreneKnapp/direct-schema) 270 * [is-my-json-valid](https://github.com/mafintosh/is-my-json-valid) 271 * [jassi](https://github.com/iclanzan/jassi) 272 * [JaySchema](https://github.com/natesilva/jayschema) 273 * [json-schema-valid](https://github.com/ericgj/json-schema-valid) 274 * [Jsonary](https://github.com/jsonary-js/jsonary) 275 * [jsonschema](https://github.com/tdegrunt/jsonschema) 276 * [request-validator](https://github.com/bugventure/request-validator) 277 * [skeemas](https://github.com/Prestaul/skeemas) 278 * [tv4](https://github.com/geraintluff/tv4) 279 * [z-schema](https://github.com/zaggino/z-schema) 280 * [jsen](https://github.com/bugventure/jsen) 281 * [ajv](https://github.com/epoberezkin/ajv) 282 * [djv](https://github.com/korzio/djv) 283 284 ### Kotlin 285 286 * [json-schema-validation-comparison](https://www.creekservice.org/json-schema-validation-comparison/functional) (Comparison site for JVM-based validator implementations) 287 288 ### Node.js 289 290 For node.js developers, the suite is also available as an [npm](https://www.npmjs.com/package/@json-schema-org/tests) package. 291 292 Node-specific support is maintained in a [separate repository](https://github.com/json-schema-org/json-schema-test-suite-npm) which also welcomes your contributions! 293 294 ### .NET 295 296 * [JsonSchema.Net](https://github.com/gregsdennis/json-everything) 297 * [Newtonsoft.Json.Schema](https://github.com/JamesNK/Newtonsoft.Json.Schema) 298 299 ### Perl 300 301 * [Test::JSON::Schema::Acceptance](https://github.com/karenetheridge/Test-JSON-Schema-Acceptance) (a wrapper of this test suite) 302 * [JSON::Schema::Modern](https://github.com/karenetheridge/JSON-Schema-Modern) 303 * [JSON::Schema::Tiny](https://github.com/karenetheridge/JSON-Schema-Tiny) 304 305 ### PHP 306 307 * [opis/json-schema](https://github.com/opis/json-schema) 308 * [json-schema](https://github.com/justinrainbow/json-schema) 309 * [json-guard](https://github.com/thephpleague/json-guard) 310 311 ### PostgreSQL 312 313 * [postgres-json-schema](https://github.com/gavinwahl/postgres-json-schema) 314 * [is\_jsonb\_valid](https://github.com/furstenheim/is_jsonb_valid) 315 316 ### Python 317 318 * [jsonschema](https://github.com/Julian/jsonschema) 319 * [fastjsonschema](https://github.com/seznam/python-fastjsonschema) 320 * [hypothesis-jsonschema](https://github.com/Zac-HD/hypothesis-jsonschema) 321 * [jschon](https://github.com/marksparkza/jschon) 322 * [OpenAPI JSON Schema Generator](https://github.com/openapi-json-schema-tools/openapi-json-schema-generator) 323 324 ### Ruby 325 326 * [json-schema](https://github.com/hoxworth/json-schema) 327 * [json\_schemer](https://github.com/davishmcclurg/json_schemer) 328 329 ### Rust 330 331 * [jsonschema](https://github.com/Stranger6667/jsonschema-rs) 332 * [valico](https://github.com/rustless/valico) 333 334 ### Scala 335 336 * [json-schema-validation-comparison](https://www.creekservice.org/json-schema-validation-comparison/functional) (Comparison site for JVM-based validator implementations) 337 * [typed-json](https://github.com/frawa/typed-json) 338 339 ### Swift 340 341 * [JSONSchema](https://github.com/kylef/JSONSchema.swift) 342 343 If you use it as well, please fork and send a pull request adding yourself to 344 the list :). 345 346 ## Contributing 347 348 If you see something missing or incorrect, a pull request is most welcome! 349 350 There are some sanity checks in place for testing the test suite. You can run 351 them with `bin/jsonschema_suite check` or `tox`. They will be run automatically 352 by [GitHub Actions](https://github.com/json-schema-org/JSON-Schema-Test-Suite/actions?query=workflow%3A%22Test+Suite+Sanity+Checking%22) 353 as well. 354 355 This repository is maintained by the JSON Schema organization, and will be governed by the JSON Schema steering committee (once it exists).