UTF-8

There are several possible representations of Unicode data, including UTF-8 , UTF-16 and UTF-32 . They are all able to represent all of Unicode, but they differ for example in the number of bits for their constituent code units . A Unicode transformation format (UTF ) is an algorithmic mapping from every Unicode code point (except surrogate code points ) to a unique byte sequence. The ISO/IEC 10646 standard uses the term “UCS transformation format ” for UTF; the two terms are merely synonyms for the same concept. Each UTF is reversible, thus every UTF supports lossless round tripping : mapping from any Unicode coded character sequence S to a sequence of bytes and back will produce S again. To ensure round tripping, a UTF mapping must have a mapping for all code points (except surrogate code points). This includes reserved or unassigned code points and the 66 noncharacters (including U+FFFE and U+FFFF). In addition to being lossless, UTFs are unique: any given coded character sequence will always result in the same sequence of bytes for a given UTF. The SCSU compression method, even though it is reversible, is not a UTF because the same string can map to very many different byte sequences, depending on the particular SCSU compressor. [[AF]]( For the formal definition of UTFs see Section 3.9, Unicode Encoding Forms in The Unicode Standard . For more information on encoding forms see UTR 17: Unicode Character Encoding Model . [[AF]]( The freely available open source project International Components for Unicode (ICU) has UTF conversion built into it. The latest version may be downloaded from the ICU Project GitHub site. Many other libraries may have built-in converters, so you may not have to write your own. [[AF]]( Q: Are there any byte sequences that are not generated by a UTF? How should I interpret them? None of the UTFs can generate every arbitrary byte sequence. For example, in UTF-8 every byte of the form 110xxxxx 2 must be followed with a byte of the form 10xxxxxx 2 . A sequence such as is illegal, and must never be generated. When faced with this illegal byte sequence while transforming or interpreting, a UTF-8 conformant process must treat the first byte 110xxxxx 2 as an illegal termination error: for example, either signaling an error, filtering the byte out, or representing the byte with a marker such as U+FFFD REPLACEMENT CHARACTER. In the latter two cases, it will continue processing at the second byte 0xxxxxxx 2 . A conformant process must not interpret illegal or ill-formed byte sequences as characters, however, it may take error recovery actions. No conformant process may use irregular byte sequences to encode out-of-band information. UTF-8 is most common on the web. UTF-16 is used by Java and Windows (.Net). UTF-8 and UTF-32 are used by Linux and various Unix systems. The conversions between all of them are algorithmically based, fast and lossless. This makes it easy to support data input or output in multiple formats, while using a particular UTF for internal storage or processing. [[AF]]( UTF-16 and UTF-32 use code units that are two and four bytes long respectively. For these UTFs , there are three sub-flavors: BE, LE and unmarked. The BE form uses big-endian byte serialization (most significant byte first), the LE form uses little-endian byte serialization (least significant byte first) and the unmarked form uses big-endian byte serialization by default, but may include a byte order mark at the beginning to indicate the actual byte serialization used. [[AF]]( Use UTF-8 . This preserves ASCII , but not Latin-1, because the characters 127 are different from Latin-1. UTF-8 uses the bytes in the ASCII only for ASCII characters. Therefore, it works well in any environment where ASCII characters have a significance as syntax characters, e.g. file name syntaxes, markup languages, etc., but where the all other characters may use arbitrary bytes. Use Java or C style escapes, of the form uXXXX or xXXXX. Thi