github.com/keltia/go-ipfs@v0.3.8-0.20150909044612-210793031c63/test/sharness/t0051-object-data/UTF-8-test.txt (about) 1 UTF-8 decoder capability and stress test 2 ---------------------------------------- 3 4 Markus Kuhn <http://www.cl.cam.ac.uk/~mgk25/> - 2015-08-24 - CC BY 4.0 5 6 This test file can help you examine, how your UTF-8 decoder handles 7 various types of correct, malformed, or otherwise interesting UTF-8 8 sequences. This file is not meant to be a conformance test. It does 9 not prescribes any particular outcome and therefore there is no way to 10 "pass" or "fail" this test file, even though the text suggests a 11 preferable decoder behaviour at some places. The aim is instead to 12 help you think about and test the behaviour of your UTF-8 decoder on a 13 systematic collection of unusual inputs. Experience so far suggests 14 that most first-time authors of UTF-8 decoders find at least one 15 serious problem in their decoder using this file. 16 17 The test lines below cover boundary conditions, malformed UTF-8 18 sequences, as well as correctly encoded UTF-8 sequences of Unicode code 19 points that should never occur in a correct UTF-8 file. 20 21 According to ISO 10646-1:2000, sections D.7 and 2.3c, a device 22 receiving UTF-8 shall interpret a "malformed sequence in the same way 23 that it interprets a character that is outside the adopted subset" and 24 "characters that are not within the adopted subset shall be indicated 25 to the user" by a receiving device. One commonly used approach in 26 UTF-8 decoders is to replace any malformed UTF-8 sequence by a 27 replacement character (U+FFFD), which looks a bit like an inverted 28 question mark, or a similar symbol. It might be a good idea to 29 visually distinguish a malformed UTF-8 sequence from a correctly 30 encoded Unicode character that is just not available in the current 31 font but otherwise fully legal, even though ISO 10646-1 doesn't 32 mandate this. In any case, just ignoring malformed sequences or 33 unavailable characters does not conform to ISO 10646, will make 34 debugging more difficult, and can lead to user confusion. 35 36 Please check, whether a malformed UTF-8 sequence is (1) represented at 37 all, (2) represented by exactly one single replacement character (or 38 equivalent signal), and (3) the following quotation mark after an 39 illegal UTF-8 sequence is correctly displayed, i.e. proper 40 resynchronization takes place immediately after any malformed 41 sequence. This file says "THE END" in the last line, so if you don't 42 see that, your decoder crashed somehow before, which should always be 43 cause for concern. 44 45 All lines in this file are exactly 79 characters long (plus the line 46 feed). In addition, all lines end with "|", except for the two test 47 lines 2.1.1 and 2.2.1, which contain non-printable ASCII controls 48 U+0000 and U+007F. If you display this file with a fixed-width font, 49 these "|" characters should all line up in column 79 (right margin). 50 This allows you to test quickly, whether your UTF-8 decoder finds the 51 correct number of characters in every line, that is whether each 52 malformed sequences is replaced by a single replacement character. 53 54 Note that, as an alternative to the notion of malformed sequence used 55 here, it is also a perfectly acceptable (and in some situations even 56 preferable) solution to represent each individual byte of a malformed 57 sequence with a replacement character. If you follow this strategy in 58 your decoder, then please ignore the "|" column. 59 60 61 Here come the tests: | 62 | 63 1 Some correct UTF-8 text | 64 | 65 You should see the Greek word 'kosme': "κόσμε" | 66 | 67 2 Boundary condition test cases | 68 | 69 2.1 First possible sequence of a certain length | 70 | 71 2.1.1 1 byte (U-00000000): " " 72 2.1.2 2 bytes (U-00000080): "" | 73 2.1.3 3 bytes (U-00000800): "ࠀ" | 74 2.1.4 4 bytes (U-00010000): "𐀀" | 75 2.1.5 5 bytes (U-00200000): "" | 76 2.1.6 6 bytes (U-04000000): "" | 77 | 78 2.2 Last possible sequence of a certain length | 79 | 80 2.2.1 1 byte (U-0000007F): "" 81 2.2.2 2 bytes (U-000007FF): "߿" | 82 2.2.3 3 bytes (U-0000FFFF): "" | 83 2.2.4 4 bytes (U-001FFFFF): "" | 84 2.2.5 5 bytes (U-03FFFFFF): "" | 85 2.2.6 6 bytes (U-7FFFFFFF): "" | 86 | 87 2.3 Other boundary conditions | 88 | 89 2.3.1 U-0000D7FF = ed 9f bf = "" | 90 2.3.2 U-0000E000 = ee 80 80 = "" | 91 2.3.3 U-0000FFFD = ef bf bd = "�" | 92 2.3.4 U-0010FFFF = f4 8f bf bf = "" | 93 2.3.5 U-00110000 = f4 90 80 80 = "" | 94 | 95 3 Malformed sequences | 96 | 97 3.1 Unexpected continuation bytes | 98 | 99 Each unexpected continuation byte should be separately signalled as a | 100 malformed sequence of its own. | 101 | 102 3.1.1 First continuation byte 0x80: "" | 103 3.1.2 Last continuation byte 0xbf: "" | 104 | 105 3.1.3 2 continuation bytes: "" | 106 3.1.4 3 continuation bytes: "" | 107 3.1.5 4 continuation bytes: "" | 108 3.1.6 5 continuation bytes: "" | 109 3.1.7 6 continuation bytes: "" | 110 3.1.8 7 continuation bytes: "" | 111 | 112 3.1.9 Sequence of all 64 possible continuation bytes (0x80-0xbf): | 113 | 114 " | 115 | 116 | 117 " | 118 | 119 3.2 Lonely start characters | 120 | 121 3.2.1 All 32 first bytes of 2-byte sequences (0xc0-0xdf), | 122 each followed by a space character: | 123 | 124 " | 125 " | 126 | 127 3.2.2 All 16 first bytes of 3-byte sequences (0xe0-0xef), | 128 each followed by a space character: | 129 | 130 " " | 131 | 132 3.2.3 All 8 first bytes of 4-byte sequences (0xf0-0xf7), | 133 each followed by a space character: | 134 | 135 " " | 136 | 137 3.2.4 All 4 first bytes of 5-byte sequences (0xf8-0xfb), | 138 each followed by a space character: | 139 | 140 " " | 141 | 142 3.2.5 All 2 first bytes of 6-byte sequences (0xfc-0xfd), | 143 each followed by a space character: | 144 | 145 " " | 146 | 147 3.3 Sequences with last continuation byte missing | 148 | 149 All bytes of an incomplete sequence should be signalled as a single | 150 malformed sequence, i.e., you should see only a single replacement | 151 character in each of the next 10 tests. (Characters as in section 2) | 152 | 153 3.3.1 2-byte sequence with last byte missing (U+0000): "" | 154 3.3.2 3-byte sequence with last byte missing (U+0000): "" | 155 3.3.3 4-byte sequence with last byte missing (U+0000): "" | 156 3.3.4 5-byte sequence with last byte missing (U+0000): "" | 157 3.3.5 6-byte sequence with last byte missing (U+0000): "" | 158 3.3.6 2-byte sequence with last byte missing (U-000007FF): "" | 159 3.3.7 3-byte sequence with last byte missing (U-0000FFFF): "" | 160 3.3.8 4-byte sequence with last byte missing (U-001FFFFF): "" | 161 3.3.9 5-byte sequence with last byte missing (U-03FFFFFF): "" | 162 3.3.10 6-byte sequence with last byte missing (U-7FFFFFFF): "" | 163 | 164 3.4 Concatenation of incomplete sequences | 165 | 166 All the 10 sequences of 3.3 concatenated, you should see 10 malformed | 167 sequences being signalled: | 168 | 169 "" | 170 | 171 3.5 Impossible bytes | 172 | 173 The following two bytes cannot appear in a correct UTF-8 string | 174 | 175 3.5.1 fe = "" | 176 3.5.2 ff = "" | 177 3.5.3 fe fe ff ff = "" | 178 | 179 4 Overlong sequences | 180 | 181 The following sequences are not malformed according to the letter of | 182 the Unicode 2.0 standard. However, they are longer then necessary and | 183 a correct UTF-8 encoder is not allowed to produce them. A "safe UTF-8 | 184 decoder" should reject them just like malformed sequences for two | 185 reasons: (1) It helps to debug applications if overlong sequences are | 186 not treated as valid representations of characters, because this helps | 187 to spot problems more quickly. (2) Overlong sequences provide | 188 alternative representations of characters, that could maliciously be | 189 used to bypass filters that check only for ASCII characters. For | 190 instance, a 2-byte encoded line feed (LF) would not be caught by a | 191 line counter that counts only 0x0a bytes, but it would still be | 192 processed as a line feed by an unsafe UTF-8 decoder later in the | 193 pipeline. From a security point of view, ASCII compatibility of UTF-8 | 194 sequences means also, that ASCII characters are *only* allowed to be | 195 represented by ASCII bytes in the range 0x00-0x7f. To ensure this | 196 aspect of ASCII compatibility, use only "safe UTF-8 decoders" that | 197 reject overlong UTF-8 sequences for which a shorter encoding exists. | 198 | 199 4.1 Examples of an overlong ASCII character | 200 | 201 With a safe UTF-8 decoder, all of the following five overlong | 202 representations of the ASCII character slash ("/") should be rejected | 203 like a malformed UTF-8 sequence, for instance by substituting it with | 204 a replacement character. If you see a slash below, you do not have a | 205 safe UTF-8 decoder! | 206 | 207 4.1.1 U+002F = c0 af = "" | 208 4.1.2 U+002F = e0 80 af = "" | 209 4.1.3 U+002F = f0 80 80 af = "" | 210 4.1.4 U+002F = f8 80 80 80 af = "" | 211 4.1.5 U+002F = fc 80 80 80 80 af = "" | 212 | 213 4.2 Maximum overlong sequences | 214 | 215 Below you see the highest Unicode value that is still resulting in an | 216 overlong sequence if represented with the given number of bytes. This | 217 is a boundary test for safe UTF-8 decoders. All five characters should | 218 be rejected like malformed UTF-8 sequences. | 219 | 220 4.2.1 U-0000007F = c1 bf = "" | 221 4.2.2 U-000007FF = e0 9f bf = "" | 222 4.2.3 U-0000FFFF = f0 8f bf bf = "" | 223 4.2.4 U-001FFFFF = f8 87 bf bf bf = "" | 224 4.2.5 U-03FFFFFF = fc 83 bf bf bf bf = "" | 225 | 226 4.3 Overlong representation of the NUL character | 227 | 228 The following five sequences should also be rejected like malformed | 229 UTF-8 sequences and should not be treated like the ASCII NUL | 230 character. | 231 | 232 4.3.1 U+0000 = c0 80 = "" | 233 4.3.2 U+0000 = e0 80 80 = "" | 234 4.3.3 U+0000 = f0 80 80 80 = "" | 235 4.3.4 U+0000 = f8 80 80 80 80 = "" | 236 4.3.5 U+0000 = fc 80 80 80 80 80 = "" | 237 | 238 5 Illegal code positions | 239 | 240 The following UTF-8 sequences should be rejected like malformed | 241 sequences, because they never represent valid ISO 10646 characters and | 242 a UTF-8 decoder that accepts them might introduce security problems | 243 comparable to overlong UTF-8 sequences. | 244 | 245 5.1 Single UTF-16 surrogates | 246 | 247 5.1.1 U+D800 = ed a0 80 = "" | 248 5.1.2 U+DB7F = ed ad bf = "" | 249 5.1.3 U+DB80 = ed ae 80 = "" | 250 5.1.4 U+DBFF = ed af bf = "" | 251 5.1.5 U+DC00 = ed b0 80 = "" | 252 5.1.6 U+DF80 = ed be 80 = "" | 253 5.1.7 U+DFFF = ed bf bf = "" | 254 | 255 5.2 Paired UTF-16 surrogates | 256 | 257 5.2.1 U+D800 U+DC00 = ed a0 80 ed b0 80 = "" | 258 5.2.2 U+D800 U+DFFF = ed a0 80 ed bf bf = "" | 259 5.2.3 U+DB7F U+DC00 = ed ad bf ed b0 80 = "" | 260 5.2.4 U+DB7F U+DFFF = ed ad bf ed bf bf = "" | 261 5.2.5 U+DB80 U+DC00 = ed ae 80 ed b0 80 = "" | 262 5.2.6 U+DB80 U+DFFF = ed ae 80 ed bf bf = "" | 263 5.2.7 U+DBFF U+DC00 = ed af bf ed b0 80 = "" | 264 5.2.8 U+DBFF U+DFFF = ed af bf ed bf bf = "" | 265 | 266 5.3 Other illegal code positions | 267 | 268 5.3.1 U+FFFE = ef bf be = "" | 269 5.3.2 U+FFFF = ef bf bf = "" | 270 | 271 THE END |