Naughty KeyboardEncoding and mojibake test values
Text that went through the wrong encoding and never came back. 11 values, each with what it breaks and what a correct application does instead.
Type them one at a time with AltShiftN
in the palette, or take them all into a script with nkb emit encoding-mojibake.
Text encoded twice
encoding-mojibake/double-encoded
Żółć
Żółć
- 8 graphemes
- 8 code points
- 18 bytes
- 8 UTF-16 units
- What it breaks
- Text encoded twice: it displays as nonsense and, once saved in this state, the original cannot be recovered.
- What a correct application does
- Never produced - and if received, stored verbatim rather than "fixed" by guessing.
Byte-order mark before a name
encoding-mojibake/bom-leading
Jan
\uFEFFJan
- 1 zero-width character
- 4 graphemes
- 4 code points
- 6 bytes
- 4 UTF-16 units
- What it breaks
- A byte-order mark at the start of a value makes exact comparison fail and sorts the record before everything else.
- What a correct application does
- Stripped on input, consistently in every layer that reads files.
Replacement character in a name
encoding-mojibake/replacement-char
Jan�Kowalski
Jan�Kowalski
- 12 graphemes
- 12 code points
- 14 bytes
- 12 UTF-16 units
- What it breaks
- The character that appears where decoding already failed; its presence means data was lost earlier.
- What a correct application does
- Detected and reported rather than stored as if it were ordinary text.
Question marks for lost letters
encoding-mojibake/question-mark-fallback
Jan????????
Jan ????????
- 12 graphemes
- 12 code points
- 12 bytes
- 12 UTF-16 units
- What it breaks
- What narrow encodings produce instead of characters they cannot represent - indistinguishable from a user typing question marks.
- What a correct application does
- Never produced: the full character set supported end to end.
Accented letter at the end
encoding-mojibake/latin1-tail
café
café
- 4 graphemes
- 4 code points
- 5 bytes
- 4 UTF-16 units
- What it breaks
- A common accented word that survives or breaks depending on the encoding of each layer it passes through.
- What a correct application does
- Identical before and after a save-and-reload round trip.
Letters from regional encodings
encoding-mojibake/cp-specific-letters
źżćńó
źżćńó
- 5 graphemes
- 5 code points
- 10 bytes
- 5 UTF-16 units
- What it breaks
- Letters present in regional encodings and absent from others; the first thing to break in an export.
- What a correct application does
- Preserved in exports, printouts and file names.
One Cyrillic letter in a Latin name
encoding-mojibake/mixed-scripts
JanКowalski
Jan Кowalski
- 12 graphemes
- 12 code points
- 13 bytes
- 12 UTF-16 units
- What it breaks
- One Cyrillic letter inside a Latin word - the value looks correct and never matches a search.
- What a correct application does
- Either normalised or flagged; not silently accepted as if identical.
Emoji, ideograph and Arabic together
encoding-mojibake/high-codepoint
😀中ال
😀中ال
- 4 graphemes
- 4 code points
- 11 bytes
- 5 UTF-16 units
- What it breaks
- Emoji, an ideograph and an Arabic letter in one value: three different storage assumptions at once.
- What a correct application does
- Stored and returned unchanged.
Ideographic space in a name
encoding-mojibake/overlong-space
JanKowalski
Jan\u3000Kowalski
- 1 space that is not an ordinary space
- 12 graphemes
- 12 code points
- 14 bytes
- 12 UTF-16 units
- What it breaks
- An ideographic space: a real space character that most trimming routines do not know about.
- What a correct application does
- Trimmed like any other whitespace, or preserved consistently.
Letter with five combining accents
encoding-mojibake/combining-overflow
á́́́́
á́́́́
- 1 grapheme
- 6 code points
- 11 bytes
- 6 UTF-16 units
- What it breaks
- Many combining marks on one letter: normalisation and length limits disagree about how many characters this is.
- What a correct application does
- Counted consistently, and rendered without covering neighbouring interface.
Every risky class in one value
encoding-mojibake/roundtrip-check
Żółć😀中ą
Żółć 😀 中 ą
- 10 graphemes
- 10 code points
- 20 bytes
- 11 UTF-16 units
- What it breaks
- A single value combining every risky class, for a quick save-and-reload check.
- What a correct application does
- Returned byte-for-byte identical after a full round trip.