Unicode®: Character Encodings
Unicode®: Character Encodings
Unicode®: Character Encodings
Unicode®
F.1 Introduction
The use of inconsistent character encodings (i.e., numeric values associated with charac-
ters) when developing global software products causes serious problems because comput-
ers process information using numbers. For example, the character “a” is converted to a
numeric value so that a computer can manipulate that piece of data. Many countries and
corporations have developed encoding systems that are incompatible with the encoding
systems of other countries and corporations. For example, the Microsoft Windows oper-
ating system assigns the value 0xC0 to the character “A with a grave accent,” while the Ap-
ple Macintosh operating system assigns the same value to an upside-down question mark.
This results in the misrepresentation and possible corruption of data.
In the absence of a universal character encoding standard, global software developers
had to localize their products extensively before distribution. Localization includes the
language translation and cultural adaptation of content. The process of localization usually
includes significant modifications to the source code (e.g., the conversion of numeric
values and the underlying assumptions made by programmers), which results in increased
costs and delays in releasing the software. For example, an English-speaking programmer
might design a global software product assuming that a single character can be represented
by one byte. However, when those products are localized in Asian markets, the pro-
grammer’s assumptions are no longer valid because there are many more Asian characters,
and therefore most of it, if not all, of the code needs to be rewritten. Localization is nec-
essary with each release of a version. By the time a software product is localized for a par-
ticular market, a newer version, which needs to be localized as well, can be ready for
distribution. As a result, it is cumbersome and costly to produce and distribute global soft-
ware products in a market where there is no universal character encoding standard.
In response to this situation, the Unicode Standard, an encoding standard that facil-
itates the production and distribution of software, was created. The Unicode Standard
F.2 Unicode Transformation Formats 1417
outlines a specification to produce consistent encoding of the world’s characters and sym-
bols. Software products which handle text encoded in the Unicode Standard need to be
localized, but the localization process is simpler and more efficient because the numeric
values need not be converted and the assumptions made by programmers about the char-
acter encoding are universal. The Unicode Standard is maintained by a non-profit organi-
zation called the Unicode Consortium, whose members include Apple, IBM, Microsoft,
Oracle, Sun Microsystems, Sybase and many others.
When the Consortium envisioned and developed the Unicode Standard, it wanted an
encoding system that was universal, efficient, uniform and unambiguous. A universal
encoding system encompasses all commonly used characters. An efficient encoding system
allows text files to be parsed quickly. A uniform encoding system assigns fixed values to all
characters. An unambiguous encoding system represents a given character in a consistent
manner. These four terms are referred to as the Unicode Standard design basis.
form that usually requires twice as much memory as UTF-16 encoded characters. The
major advantage of the fixed-width UTF-32 encoding form is that it expresses all charac-
ters uniformly, so it is easy to handle in arrays.
There are few guidelines that state when to use a particular encoding form. The best
encoding form to use depends on the computer system and business protocol, not on the
data itself. Typically, the UTF-8 encoding form should be used where computer systems
and business protocols require data to be handled in 8-bit units, particularly in legacy sys-
tems being upgraded, because it often simplifies changes to existing programs. For this
reason, UTF-8 has become the encoding form of choice on the Internet. Likewise, UTF-
16 is the encoding form of choice on Microsoft Windows applications. UTF-32 is likely
to become more widely used in the future as more characters are encoded with values
above FFFF hexadecimal. UTF-32 requires less sophisticated handling than UTF-16 in
the presence of surrogate pairs. Figure F.1 shows the different ways in which the three
encoding forms handle character encoding.
reading for anyone new to Unicode. It also includes a list of related links that can provide
the reader with additional information about Unicode. The How to Use this Site subsec-
tion contains information about using and navigating the site as well hyperlinks to addi-
tional resources. The FAQ subsection organizes frequently asked questions into several
topics. Each topic has a brief explanation that specifies what kinds of questions and
answers are provided. Readers unfamiliar with vocabulary terms used by the Unicode
Consortium can navigate to the Glossary of Unicode Terms subsection, which lists the Uni-
code terms and their definitions in alphabetical order.
The General Information section contains six subsections: Where is my Character, Display
Problems, Useful Resources, Unicode Enabled Products, Mail Lists and Conferences. The
main areas covered in this section include a link to the Unicode code charts (a complete
listing of code values) assembled by the Unicode Consortium as well as a detailed outline on
how to locate an encoded character in the code chart. The section also contains advice on
how to configure different operating systems and Web browsers so that the Unicode charac-
ters can be viewed properly. Moreover, from this section, the user can navigate to other sites
that provide information on such topics as, fonts, linguistics and other standards, such as the
Armenian Standards Page and the Chinese GB 18030 Encoding Standard.
The Consortium section consists of six subsections: Who we are, Our Members, How to
Join, Press Info, Policies & Positions and Contact Us. This section provides a list of the cur-
rent Unicode Consortium members as well as information on how to become a member.
Privileges for each member type—full, associate, specialist, individual and liaison—and
the fees assessed to each member are listed here.
The For Members Only section consists of two subsections: Member Resources and
Working Documents. These subsections are password protected—only consortium mem-
bers can access these links.
The Unicode Standard section consists of five subsections: Start Here, Latest Version,
Code Charts, Unicode Character Database and Unihan Database. This section describes the
updates applied to the latest version of the Unicode Standard as well as categorizing all the
defined encoding. The user can learn how the latest version has been modified to encom-
pass more features and capabilities. For example, one enhancement of Version 4.0 is that
it contains additional encoded characters.
The Key Specification section consists of five subsections: Unicode Collation (UCA),
Bidirectional Algorithm (Bidi), Normalization (NFC, NFD, ...), Locale Data (CLDR) and Scripts
Codes (ISO 15924). These subsections describe the key specifications and projects that are
related to Unicode.
The Technical Publication section consists of four subsections: Technical Reports & Stan-
dards, Technical Notes, Online Data Table and Updates & Errata. The Technical Reports &
Standards subsection contains reports and standards used to implement and develop Uni-
code standard. The Technical Notes subsection lists papers that Unicode users or implemen-
tors may be interested in. The Online Data Table subsection provides machine-readable tables
that are required to implement the Unicode standard. The Updates & Errata subsection lists
errata that are known to the current version and are to be fixed in the next version.
The Work in Progress section consists of six subsections: Calendar of Meetings, Unicode
Technical Committee, Meeting Minutes, Proposals for Public Review, Proposed Characters
and Submitting Proposals. This section presents the user with a catalog characters recently
included in the Unicode Standard scheme as well as characters being considered for inclu-
F.6 Using Unicode 1421
sion. A user who determines that a character has been overlooked can submit a written pro-
posal for the inclusion of that character. The Submitting Proposals subsection contains strict
guidelines that must be adhered to when submitting written proposals.
Fig. F.3 | Java application that uses Unicode encoding (Part 1 of 2.).
1422 Appendix F Unicode®
42
43 JLabel japaneseJLabel = new JLabel( "Unicode\u3078\u3087\u3045" +
44 "\u3053\u305D\u0021" );
45 japaneseJLabel.setToolTipText( "This is Japanese" );
46 add( japaneseJLabel );
47
48 JLabel portugueseJLabel = new JLabel( "\u0053\u00E9\u006A\u0061" +
49 "\u0020\u0042\u0065\u006D\u0076\u0069\u006E\u0064\u006F\u0020" +
50 "Unicode\u0021" );
51 portugueseJLabel.setToolTipText( "This is Portuguese" );
52 add( portugueseJLabel );
53
54 JLabel spanishJLabel = new JLabel( "\u0042\u0069\u0065\u006E" +
55 "\u0076\u0065\u006E\u0069\u0064\u0061\u0020\u0061\u0020" +
56 "Unicode\u0021" );
57 spanishJLabel.setToolTipText( "This is Spanish" );
58 add( spanishJLabel );
59 } // end UnicodeJFrame constructor
60 } // end class UnicodeJFrame
Fig. F.3 | Java application that uses Unicode encoding (Part 2 of 2.).
Arabic U+0600–U+06FF
Basic Latin U+0000–U+007F
Bengali (India) U+0980–U+09FF
Cherokee (Native America) U+13A0–U+13FF
CJK Unified Ideographs (East Asia) U+4E00–U+9FFF
Cyrillic (Russia and Eastern Europe) U+0400–U+04FF
Ethiopic U+1200–U+137F
Greek U+0370–U+03FF
Hangul Jamo (Korea) U+1100–U+11FF
Hebrew U+0590–U+05FF
Hiragana (Japan) U+3040–U+309F
Khmer (Cambodia) U+1780–U+17FF
Summary
• Before Unicode, software developers were plagued by the use of inconsistent character encoding
(i.e., numeric values for characters). Most countries and organizations had their own encoding
systems, which were incompatible.
• Localization of global software requires significant modifications to the source code, which re-
sults in increased costs and delays in releasing the product.
• Localization is necessary with each release of a version. By the time a software product is localized
for a particular market, a newer version, which needs to be localized as well, is ready for distribu-
tion. As a result, it is cumbersome and costly to produce and distribute global software products
in a market where there is no universal character encoding standard.
• The Unicode Consortium developed the Unicode Standard in response to the serious problems
created by multiple character encodings and the use of encodings.
• The Unicode Standard facilitates the production and distribution of localized software. It out-
lines a specification for the consistent encoding of the world’s characters and symbols.
• Software products which handle text encoded in the Unicode Standard need to be localized, but
the localization process is simpler and more efficient because the numeric values need not be con-
verted.
• The Unicode Standard is designed to be universal, efficient, uniform and unambiguous.
• A universal encoding system encompasses all commonly used characters; an efficient encoding
system parses text files easily; a uniform encoding system assigns fixed values to all characters; and
a unambiguous encoding system represents the same character for any given value.
• Unicode extends the limited ASCII character set to include all the major characters of the world.
• Unicode makes use of three Unicode Transformation Formats (UTF): UTF-8, UTF-16 and
UTF-32, each of which may be appropriate for use in different contexts.
• UTF-8 data consists of 8-bit bytes (sequences of one, two, three or four bytes depending on the
character being encoded) and is well suited for ASCII-based systems when there is a predomi-
nance of one-byte characters (ASCII represents characters as one-byte).
• UTF-8 is a variable-width encoding form that is more compact for text involving mostly Latin
characters and ASCII punctuation.
Terminology 1425
• UTF-16 is the default encoding form of the Unicode Standard. It is a variable-width encoding
form that uses 16-bit code units instead of bytes. Most characters are represented by a single 16-
bit unit, but some characters require surrogate pairs.
• UTF-32 is a 32-bit encoding form. The major advantage of the fixed-width encoding form is
that it uniformly expresses all characters, so that they are easy to handle in arrays and other uses.
• Characters are represented using glyphs—shapes, fonts and sizes for displaying characters.
• Code values are bit combinations that represent encoded characters. The Unicode notation for
a code value is U+yyyy in which U+ refers to the Unicode code values, as opposed to other hexa-
decimal values. The yyyy represents a four-digit hexadecimal number.
• Currently, the Unicode Standard provides code values for 96,382 character representations.
• An advantage of the Unicode Standard is its impact on the overall performance of the international
economy. Applications that conform to an encoding standard can be processed easily by computers.
• Another advantage of the Unicode Standard is its portability. Applications written in Unicode
can be easily transferred to different operating systems, databases and Web browsers. Most com-
panies currently support, or are planning to support Unicode.
• Numerous programming languages provide some level of support for the Unicode Standard.
• In Java programs, the \uyyyy escape sequence represents a character, where yyyy is the four-digit
hexadecimal code value. The \u0020 escape sequence is the universal encoding for the space char-
acter.
Terminology
\uyyyy escape sequence script
block surrogate
code value unambiguous (Unicode design basis)
diacritic Unicode Consortium
double-byte character set (DBCS) Unicode design basis
efficient (Unicode design basis) Unicode Standard
encode Unicode Transformation Format (UTF)
glyph uniform (Unicode design basis)
hexadecimal notation universal (Unicode design basis)
localization UTF-8
multibyte character set (MBCS) UTF-16
portability UTF-32
Self-Review Exercises
F.1 Fill in the blanks in each of the following.
a) Global software developers had to their products to a specific market be-
fore distribution.
b) The Unicode Standard is a(n) standard that facilitates the uniform produc-
tion and distribution of software products.
c) The four design basis that comprises the Unicode Standard are: ,
, and .
d) Characters are represented using .
e) Software that can execute on different operating systems is said to be .
F.2 State whether each of the following is true or false. If false, explain why.
a) The Unicode Standard encompasses all the world’s characters.
1426 Appendix F Unicode®
b) A Unicode code value is represented as U+yyyy, where yyyy represents a number in bina-
ry notation.
c) A diacritic is a character with a special mark that emphasizes an accent.
d) Unicode is portable.
e) When designing Java programs, a Unicode escape sequence is denoted by /uyyyy.
Exercises
F.3 Navigate to the Unicode Consortium Web site (www.unicode.org) and write the hexadecimal
code values for the following characters. In which block were they located?
a) Latin letter "Z."
b) Latin letter ‘n’ with the "tilde (~)."
c) Greek letter ‘delta.’
d) Mathematical operator "less than or equal to."
e) Punctuation symbol "open quote (“)."
F.4 Describe the Unicode Standard design basis.
F.5 Define the following terms:
a) code value.
b) surrogates.
c) Unicode Standard.
F.6 Define the following terms:
a) UTF-8.
b) UTF-16.
c) UTF-32.
F.7 Describe a scenario where it is optimal to store your data in UTF-16 format.
F.8 Using the Unicode Standard code values, write a Java program that prints your first and last
name. The program should print your name in all uppercase letters and in all lowercase letters. If you
know other languages, print your first and last name in those languages as well.