Character encoding
|
|
A character encoding is a code that pairs a set of natural language characters (such as an alphabet or syllabary) with a set of something else, such as numbers or electrical pulses. Common examples include Morse code, which encodes letters of the Latin alphabet as series of long and short depressions of a telegraph key; and ASCII, which encodes letters, numerals, and other symbols as both integers and 7-bit binary versions of those integers.
| Contents |
Character repertoire
In some contexts (especially computer storage and communication) it makes sense to distinguish a character repertoire, which is a full set of abstract characters that a system supports, from a coded character set or character encoding which specifies how to represent characters from that set using a number of integer codes.
In the early days of computing, most systems used only the character repertoire of the ASCII code. This was soon seen to be inadequate, and a number of ad-hoc methods were used to extend this. The need to support multiple writing systems, including the CJK family of scripts, required a far larger number of characters to be supported, and required a systematic approach to character encoding to be used, rather than the previous ad hoc approaches.
For example, the full repertoire of Unicode encompasses over 100,000 characters, each being assigned a unique integer code in the range 0 to hexadecimal 10FFFF (a little over 1.1 million, so not all integers in that range represent coded characters). Other common repertoires include ASCII and ISO 8859-1, which are identical to the first 128 and 256 coded characters of Unicode respectively.
Encoding forms and Encoding schemes
The term character encoding is sometimes overloaded to also mean how characters are represented as a specific sequence of bits. This involves an encoding form where the integer code is converted to a series of integer code values that facilitate storage in a system that uses fixed bit widths. For example, integers greater than 65535 will not fit in 16 bits, so the UTF-16 encoding form mandates that these integers be represented as a surrogate pair of integers that are less than 65536 and that are not assigned to characters (e.g., hex 10000 becomes the pair D800 DC00). An encoding scheme then converts code values to bit sequences, with attention given to things like platform-dependent byte order issues (e.g. D800 DC00 might become 00 D8 00 DC on an Intel x86 architecture). A character set or character map or code page shortcuts this process by directly mapping abstract characters to specific bit patterns. Unicode Technical Report #17 (http://www.unicode.org/reports/tr17/) explains this terminology in depth and provides further examples.
Since most applications use only a small subset of Unicode, encoding schemes like UTF-8 and UTF-16, and character maps like ASCII, provide efficient ways to represent Unicode characters in computer storage or communications using short binary words. Some of these simple text encodings use data compression techniques to represent a large repertoire with a smaller number of codes.
Popular character encodings
- ISO 646
- EBCDIC
- ISO 8859:
- DOS character sets:
- Windows character sets:
- KOI8-R, KOI8-U, KOI7
- ISCII
- VISCII
- Big5
- Guobiao
- ISO 2022, Shift-JIS, EUC
- Unicode (And subsets thereof, such as the 16-bit 'Basic Multilingual Plane'). See UTF-8
See also
- Variable-width encoding
- Code page (details vendor specific code pages)
- Chinese character encoding
- Mojibake
- Category:Character sets
External links
- Character sets registered by Internet Assigned Numbers Authority (http://www.iana.org/assignments/character-sets)
- Unicode Technical Report #17: Character Encoding Model (http://www.unicode.org/unicode/reports/tr17/)
- ICU Converter Explorer (http://www.ibm.com/software/globalization/icu/demo/converters)
- The Cyrillic Charset soup (http://czyborra.com/charsets/cyrillic.html)
- Early history of character set standardization (http://homepages.cwi.nl/~dik/english/codes/stand.html)
- Character Sets And Code Pages At The Push Of A Button (http://www.i18nguy.com/unicode/codepages.html)
- A complete introduction to Japanese character encodings (http://www.cs.mcgill.ca/~aelias4/encodings.html)de:Zeichencodierung
es:Codificación de caracteres fr:Codage_de_caractères gl:Codificación de caracteres ja:文字コード zh:字符集