UTF-9 and UTF-18

UTF-9 and UTF-18 (9- and 18-bit Unicode Transformation Format, respectively) were two April Fools' Day RFC joke specifications for encoding Unicode on systems where the nonet (nine bit group) is a better fit for the native word size than the octet, such as the 36-bit PDP-10 and the UNIVAC 1100/2200 series. Both encodings were specified in RFC 4042, written by Mark Crispin (inventor of IMAP) and released on April 1, 2005. The encodings suffer from a number of flaws and it is confirmed by their author that they were intended as a joke.[1]

However, unlike some of the "specifications" given in other April 1 RFCs, UTF-9 and UTF-18 are actually technically possible to implement, and have in fact been implemented in PDP-10 assembly language. They are however not endorsed by the Unicode Consortium.

Technical details

Like the 8-bit code unit Unicode Transformation Format – UTF-8 – commonly called variable-length quantity, UTF-9 uses a system of putting an octet in the low 8 bits of each nonet and using the high bit to indicate continuation. This means that ASCII and Latin 1 characters take one nonet each, the rest of the BMP characters take two nonets each and non-BMP code points take three. Code points that require multiple nonets are stored starting with the most significant non-zero octet.

This table shows the UTF-9 encoding scheme (the x characters are replaced by the bits of the code point):

Number
of nonets		Bits for
code point		First
code point		Last
code point		Nonet 1Nonet 2Nonet 3
1 8 U+0000 U+00FF 0xxxxxxxx
2 16 U+0100 U+FFFF 1xxxxxxxx0xxxxxxxx
3 21 U+10000 U+10FFFF 1000xxxxx1xxxxxxxx0xxxxxxxx

The details of the encoding scheme used in UTF-9 differ from UTF-8 in a non-ideal way, as UTF-9 is not self-synchronizing: the end of a longer sequence can be confused with a shorter sequence. For instance, U+0041 is represented in octal as 101 and U+E0041 as 416 400 101. This stems from the lack of distinction between the beginning of a sequence and the subsequent continuation nonets, as both simply have their most significant bit set, and the lack of distinction between a one-nonet sequence and the last nonet of a multi-nonet sequence. In contrast, in UTF-8, the three different kinds of octets are trivially distinguishable from each other, making the scheme self-synchronizing. Searching within a UTF-9 encoded string or splitting one requires special care, as it is always necessary to search backwards to find the beginning of the current sequence.

UTF-18 is a fixed length encoding using an 18 bit integer per code point. This allows representation of four planes, which are mapped to the four planes currently used by Unicode (planes 0–2 and 14). This means that the two private use planes (15 and 16) and the currently unused planes (3–13) are not supported. The UTF-18 specification does not say why they did not allow surrogates to be used for these code points, though when talking about UTF-16 earlier in the RFC, it says "This transformation format requires complex surrogates to represent code points outside the BMP". After complaining about their complexity, it would have looked a bit hypocritical to use surrogates in their new standard. It is unlikely that planes 3–13 will be assigned by Unicode any time in the foreseeable future. Thus, UTF-18, like UCS-2 and UCS-4, guarantees a fixed width for all code points (although not for all glyphs).

See also

Notes

  1. ↑ "Mark Crispin's Web Page". Retrieved 2006-09-17. Points out April Fool's Day for two of his RFCs.
This article is issued from Wikipedia - version of the 10/13/2016. The text is available under the Creative Commons Attribution/Share Alike but additional terms may apply for the media files.