Character Model for the World Wide Web 1.0

3.1.1 Introduction

The glossary entry in [Unicode 3.0] gives:

"Character. (1) The smallest component of written language that has semantic values; refers to the abstract meaning and/or shape ..."

The word 'character' is used in many contexts, with different meanings. Human cultures have radically differing writing systems, leading to radically differing concepts of a character. Such wide variation in end user experience can, and often does, result in misunderstanding. This variation is sometimes mistakenly seen as the consequence of imperfect technology. Instead, it derives from the great flexibility and creativity of the human mind and the long tradition of writing as an important part of the human cultural heritage. The alphabetic approach used by scripts such as Latin, Cyrillic and Greek is only one of several possibilities.

The developers of W3C specifications, and the developers of software based on those specifications, are likely to be more familiar with usages of the term 'character' they have experienced and less familiar with the wide variety of usages in an international context. Furthermore, within a computing context, characters are often confused with related concepts, resulting in incomplete or inappropriate specifications and software.

This section examines some of these contexts, meanings and confusions.

3.1.2 Units of aural rendering

3.1.3 Units of visual rendering

Visual rendering introduces the notion of a glyph. Glyphs are defined by ISO/IEC 9541-1 [ISO/IEC 9541-1] as "a recognizable abstract graphic symbol which is independent of a specific design". There is not a one-to-one correspondence between characters and glyphs:

• A single character can be represented by multiple glyphs (each glyph is then part of the representation of that character). These glyphs may be physically separated from one another.

• A single glyph may represent a sequence of characters (this is the case with ligatures, among others).

• A character may be rendered with very different glyphs depending on the context.

• A single glyph may represent different characters (e.g. capital Latin A, capital Greek A and capital Cyrillic A).

Each glyph can be represented by a number of different glyph images; a set of glyph images makes up a font. Glyphs can be construed as the basic units of organization of the visual rendering of text, just as characters are the basic unit of organization of encoded text.

[S]  [I]  [C]  Specifications, software and content MUST NOT assume a one-to-one mapping between characters and units of displayed text.

See the appendix B Examples of Characters, Keystrokes and Glyphs for examples of the complexities of character to glyph mapping.

Some scripts, in particular Arabic and Hebrew, are written from right to left. Text including characters from these scripts can run in both directions and is therefore called bidirectional text. The Unicode Standard [Unicode] requires that characters be stored and interchanged in logical order, i.e. roughly corresponding to the order in which text is typed in via the keyboard (for a more detailed definition see [Unicode 3.0], Section 2.2). Logical ordering is important to ensure interoperability of data, and also benefits accessibility, searching, and collation. [S]  [I]  [C]  Protocols, data formats and APIs MUST store, interchange or process text data in logical order.

In the presence of bidirectional text, two possible selection modes must be considered. The first is logical selection mode, which selects all the characters logically located between the end-points of the user's mouse gesture. Here the user selects from between the first and second letters of the second word to the middle of the number. Logical selection looks like this:

Logical
Visual

Logical selection results in discontiguous visual ranges.

It is a consequence of the bidirectionality of the text that a single, continuous logical selection in memory results in a discontinuous selection appearing on the screen. This discontinuity, as well as the somewhat unintuitive behavior of the cursor, makes some users prefer a visual selection mode, which selects all the characters visually located between the end-points of the user's mouse gesture. With the same mouse gesture as before, we now obtain:

Logical
Visual

Visual selection results in discontiguous logical ranges.

In visual selection mode, as seen in the example above, a single visual selection range may result in two logical ranges, which have to be accommodated by protocols, APIs and implementations.

[S]  Specifications of protocols and APIs that involve selection of ranges SHOULD provide for discontiguous selections, at least to the extent necessary to support implementation of visual selection on screen on top of those protocols and APIs.

3.1.4 Units of input

In keyboard input, it is not always the case that keystrokes and input characters correspond one-to-one. A limited number of keys can fit on a keyboard. Some keyboards will generate multiple characters from a single keypress. In other cases ('dead keys') a key will generate no characters, but affect the results of subsequent keypresses. Many writing systems have far too many characters to fit on a keyboard and must rely on more complex input methods, which transform keystroke sequences into character sequences. Other languages may make it necessary to input some characters with special modifier keys. See B Examples of Characters, Keystrokes and Glyphs for examples of non-trivial input.

[S]  [I]  Specifications and software MUST NOT assume that a single keystroke results in a single character, nor that a single character can be input with a single keystroke (even with modifiers), nor that keyboards are the same all over the world.

3.1.5 Units of collation

A default collation order for all Unicode characters can be obtained from ISO/IEC 14651 [ISO/IEC 14651] or from Unicode Technical Report #10, the Unicode Collation Algorithm [UTR #10]. This default collation order can be used in conjunction with rules tailored for a particular locale to ensure a predictable ordering and comparison of strings, whatever characters they include.

3.1.6 Units of storage

Computer storage and communication rely on units of physical storage and information interchange, such as bits and bytes (8-bit units, also called octets). A frequent error in specifications and implementations is the equating of characters with units of physical storage. The mapping between characters and such units of storage is actually quite complex, and is discussed in the next section, 3.2 Digital Encoding of Characters.

[S]  [I]  [C]  Specifications, software and content MUST NOT assume a one-to-one relationship between characters and units of physical storage.

3.1.7 Summary

This section, 3.1 Perceptions of Characters, has discussed terms for units that do not necessarily overlap with the term 'character', such as phoneme, glyph, and collation unit. The next section, 3.2 Digital Encoding of Characters, lists terms that should be used rather than 'character' to precisely define units of encoding (code point, code unit, and byte).

[S]  When specifications use the term 'character' the specifications MUST define which meaning they intend. [S]  Specifications SHOULD avoid the use of the term 'character' if a more specific term is available.

3.6.1 Mandating a unique character encoding

Mandating a unique character encoding is simple, efficient, and robust. There is no need for specifying, producing, transmitting, and interpreting encoding tags. At the receiver, the character encoding will always be understood. There is also no ambiguity as to which character encoding to use if data is transferred non-electronically and later has to be converted back to a digital representation. Even when there is a need for compatibility with existing data, systems, protocols and applications, multiple character encodings can often be dealt with at the boundaries or outside a protocol, format, or API. The DOM [DOM Level 1] is an example of where this was done. The advantages of choosing a unique character encoding become more important the smaller the pieces of text used are and the closer to actual processing the specification is.

[S]  When a unique character encoding is mandated, the character encoding MUST be UTF-8, UTF-16 or UTF-32. [S]  If a unique character encoding is mandated and compatibility with US-ASCII is desired, UTF-8 (see [RFC 2279]) is RECOMMENDED. In other situations, such as for APIs, UTF-16 or UTF-32 may be more appropriate. Possible reasons for choosing one of these include efficiency of internal processing and interoperability with other processes.

3.6.2 Character encoding identification

The MIME Internet specification [MIME] provides a good example of a mechanism for character encoding identification. The MIME charset parameter definition is intended to supply sufficient information to uniquely decode the sequence of bytes of the received data into a sequence of characters. The values are drawn from the IANA charset registry [IANA].

[S]  Specifications SHOULD avoid using the terms 'character set' and 'charset' to refer to a character encoding, except when the latter is used to refer to the MIME charset parameter or its IANA-registered values. The term 'character encoding', or in specific cases the terms 'character encoding form' or 'character encoding scheme', are RECOMMENDED.

The IANA charset registry is the official list of names and aliases for character encodings on the Internet.

[S]  If the unique encoding approach is not taken, specifications SHOULD mandate the use of the IANA charset registry names, and in particular the names identified in the registry as 'MIME preferred names', to designate character encodings in protocols, data formats and APIs. [S]  [I]  [C]  Character encodings that are not in the IANA registry SHOULD NOT be used, except by private agreement. [S]  [I]  [C]  If an unregistered character encoding is used, the convention of using 'x-' at the beginning of the name MUST be followed. [I]  [C]  Content and software that label text data MUST use one of the names mandated by the appropriate specification (e.g. the XML specification when editing XML text) and SHOULD use the MIME preferred name of a character encoding to label data in that character encoding. [I]  [C]  An IANA-registered charset name MUST NOT be used to label text data in a character encoding other than the one identified in the IANA registration of that name.

[S]  If the unique encoding approach is not chosen, specifications MUST designate at least one of the UTF-8 and UTF-16 encoding forms of Unicode as admissible character encodings and SHOULD choose at least one of UTF-8 or UTF-16 as mandated encoding forms (encoding forms that MUST be supported by implementations of the specification). [S]  Specifications MAY define either UTF-8 or UTF-16 as a default encoding form (or both if they define suitable means of distinguishing them), but they MUST NOT use any other character encoding as a default. [S]  Specifications MUST NOT propose the use of heuristics to determine the encoding of data.

Examples of heuristics include the use of statistical analysis of byte (pattern) frequencies or character (pattern) frequencies. Heuristics are bad because they will not work consistently across different implementations. Well-defined instructions of how to unambiguously determine a character encoding, such as those given in XML 1.0 [XML 1.0], Appendix F, are not considered heuristics.

[I]  Receiving software MUST determine the encoding of data from available information according to appropriate specifications. [I]  When an IANA-registered charset name is recognized, receiving software MUST interpret the received data according to the encoding associated with the name in the IANA registry. [I]  When no charset is provided receiving software MUST adhere to the default character encoding(s) specified in the specification.

[I]  Receiving software MAY recognize as many character encodings and as many charset names and aliases for them as appropriate. A field-upgradeable mechanism may be appropriate for this purpose. Certain character encodings are more or less associated with certain languages (e.g. Shift_JIS with Japanese); trying to support a given language or set of customers may mean that certain character encodings have to be supported. The character encodings that need to be supported may change over time. This document does not give any advice on which character encoding may be appropriate or necessary for the support of any given language.

[I]  Software MUST completely implement the mechanisms for character encoding identification and SHOULD implement them in such a way that they are easy to use (for instance in HTTP servers).

[C]  Content MUST make use of available facilities for character encoding identification by always indicating character encoding; where the facilities offered for character encoding identification include defaults (e.g. in XML 1.0 [XML 1.0]), relying on such defaults is sufficient to satisfy this identification requirement.

Because of the layered Web architecture (e.g. formats used over protocols), there may be multiple and at times conflicting information about character encoding. [S]  Specifications MUST define conflict-resolution mechanisms (e.g. priorities) for cases where there is multiple or conflicting information about character encoding. [I]  [C]  Software and content MUST carefully follow conflict-resolution mechanisms where there is multiple or conflicting information about character encoding.

3.6.3 Private use code points

The Unicode Standard designates certain ranges of code points for private use: the Private Use Area (U+E000-F8FF) and planes 15 and 16 (U+F0000-FFFFD and U+100000-10FFFD). These code points are guaranteed to never be allocated to standard characters, and are available for use by private agreement between a producer and a recipient. However, their use on the Web is strongly discouraged, since private agreements do not scale on the Web. Code points from different private agreements may collide. Also a private agreement, and therefore the meaning of the code points, can quickly become lost.

[S]  Specifications MUST NOT define any assignments of private use code points. [S]  Conformance to a specification MUST NOT require the use of private use area characters. [S]  Specifications MUST NOT require the use of mechanisms for agreement on the use of private use code points. [S]  [I]  Specifications and implementations SHOULD NOT disallow the use of private use code points by private arrangement. As an example, XML does not disallow the use of private use code points.

[S]  Specifications MAY define markup to allow the transmission of symbols not in Unicode or to identify specific variants of Unicode characters.

4.1.1 Why do we need character normalization?

Solving the string matching problem involves normalization, which in a nutshell means bringing the two strings to be compared to a common, canonical encoding prior to performing binary matching. (For additional steps involved in string matching see 6 String Identity Matching.)

4.1.2 The choice of early uniform normalization

• Almost all legacy data as well as data created by current software is normalized (if using NFC).

• The number of Web components that generate or transform text is considerably smaller than the number of components that receive text and need to perform matching or other processes requiring normalized text.

• Current receiving components (browsers, XML parsers, etc.) implicitly assume early normalization by not performing or verifying normalization themselves. This is a vast legacy.

• Web components that generate and process text are in a much better position to do normalization than other components; in particular, they may be aware that they deal with a restricted repertoire only, which simplifies the process of normalization.

• Not all components of the Web that implement functions such as string matching can reasonably be expected to do normalization. This, in particular, applies to very small components and components in the lower layers of the architecture.

• Forward-compatibility issues can be dealt with more easily: less software needs to be updated, namely only the software that generates newly introduced characters.

• It improves matching in cases where the character encoding is partly undefined, such as URIs [RFC 2396] in which non-ASCII bytes have no defined meaning.

• It is a prerequisite for comparison of encrypted strings (see [CharReq], section 2.7).

The second axis is a choice of canonical encoding. This choice needs only be made if early normalization is chosen. With late normalization, the canonical encoding would be an internal matter of the smart compare function, which doesn't need any wide agreement or standardization.

By choosing a single canonical encoding, it is ensured that normalization is uniform throughout the web. Hence the two axes lead us to the name 'early uniform normalization'.

4.1.3 The choice of Normalization Form C

The Unicode Consortium provides four standard normalization forms (see Unicode Normalization Forms [UTR #15]). These forms differ in 1) whether they normalize towards decomposed characters (NFD, NFKD) or precomposed characters (NFC, NFKC) and 2) whether they normalize away compatibility distinctions (NFKD, NFKC) or not (NFD, NFC).

For use on the Web, it is important not to lose the so-called compatibility distinctions, which may be important (see [UXML] Chapter 4 for a discussion). The NFKD and NFKC normalization forms are therefore excluded. Among the remaining two forms, NFC has the advantage that almost all legacy data (if transcoded trivially, one-to-one) as well as data created by current software is already in this form; NFC also has a slight compactness advantage and a better match to user expectations with respect to the character vs. grapheme issue. This document therefore chooses NFC as the base for Web-related text normalization.

For a list of programming resources related to normalization, see D Resources for Normalization.

4.2.1 Unicode-normalized text

Text is, for the purposes of this specification, Unicode-normalized if it is in a Unicode encoding form and is in Unicode Normalization Form C, according to a version of Unicode Standard Annex #15: Unicode Normalization Forms [UTR #15] at least as recent as the oldest version of the Unicode Standard that contains all the characters actually present in the text, but no earlier than version 3.2 [Unicode 3.2].

4.2.2 Include-normalized text

Markup languages, style languages and programming languages often offer facilities for including a piece of text inside another. An include is an instance of a syntactic device specified in a language to include an entity at the position of the include, replacing the include itself. Examples of includes are entity references in XML, @import rules in CSS and the #include preprocessor statement in C/C++. Character escapes are a special case of includes where the included entity is predetermined by the language.

Text is include-normalized if:

1. the text is Unicode-normalized and does not contain any character escapes or includes whose expansion would cause the text to become no longer Unicode-normalized; or

2. the text is in a legacy encoding and, if it were transcoded to a Unicode encoding form by a normalizing transcoder, the resulting text would satisfy clause 1 above.

4.2.3 Fully-normalized text

Formal languages define constructs, which are identifiable pieces, occurring in instances of the language, such as comments, identifiers, element tags, processing instructions, runs of character data, etc. During the normal processing of include-normalized text, these various constructs may be moved, removed (e.g. removing comments) or merged (e.g. merging all the character data within an element as done by the string() function of XPath), creating opportunities for text to become denormalized. The software performing those operations then has to re-normalize the result, which is a burden. One way to avoid such denormalization is to make sure that the various important constructs never begin with a character such that appending that character to a normalized string can cause the string to become denormalized. A composing character is a character that is one or both of the following:

1. the second character in the canonical decomposition mapping of some primary composite (as defined in D3 of [UTR #15]), or

2. of non-zero canonical combining class (as defined in [Unicode]).

Please consult Appendix C Composing Characters for a discussion of composing characters, which are not exactly the same as Unicode combining characters.

Text is fully-normalized if:

1. the text is in a Unicode encoding form, is include-normalized and none of the constructs comprising the text begin with a composing character or a character escape representing a composing character; or

2. the text is in a legacy encoding and, if it were transcoded to a Unicode encoding form by a normalizing transcoder, the resulting text would satisfy clause 1 above.

Identification of the constructs that should be prohibited from beginning with a composing character (the relevant constructs) is language-dependent. As specified in 4.4 Responsibility for Normalization, it is the responsibility of the specification for a language to specify exactly what constitutes a relevant construct. This may be done by specifying important boundaries, taking into account which operations would benefit the most from being protected against denormalization. The relevant constructs are then defined as the spans of text between the boundaries. At a minimum, for those languages which have these notions, the important boundaries are entity (include) boundaries as well as the boundaries between most markup and character data. Many languages will benefit from defining more boundaries and therefore finer-grained full-normalization constructs.

4.3.1 General examples

The string suçon (U+0073 U+0075 U+00E7 U+006F U+006E) encoded in a Unicode encoding form, is Unicode-normalized, include-normalized and fully-normalized. The same string encoded in a legacy encoding for which there exists a normalizing transcoder would be both include-normalized and fully-normalized but not Unicode-normalized (since not in a Unicode encoding form).

In an XML or HTML context, the string suçon is also include-normalized, fully-normalized and, if encoded in a Unicode encoding form, Unicode-normalized. Expanding ç yields suçon as above, which contains no replaceable combining sequence.

The string suc¸on (U+0073 U+0075 U+0063 U+0327 U+006F U+006E), where U+0327 is the COMBINING CEDILLA, encoded in a Unicode encoding form, is not Unicode-normalized (since the combining sequence 'c¸' (U+0063 U+0327) should appear instead as the precomposed 'ç' (U+00E7)). As a consequence this string is neither include-normalized (since in a Unicode encoding form but not Unicode-normalized) nor fully-normalized (since not include-normalized). Note however that the string sub¸on (U+0073 U+0075 U+0062 U+0327 U+006F U+006E) in a Unicode encoding form is Unicode-normalized since there is no precomposed form of 'b' plus cedilla. It is also include-normalized and fully-normalized.

In plain text the string suçon is Unicode-normalized, since plain text doesn't recognize that ̧ represents a character in XML or HTML and considers it just a sequence of non-replaceable characters.

In an XML or HTML context, however, expanding ̧ yields the string suc¸on (U+0073 U+0075 U+0063 U+0327 U+006F U+006E) which is not Unicode-normalized ('c¸' is replaceable by 'ç'). As a consequence the string is neither include-normalized nor fully-normalized. As another example, if the entity reference &word-end; refers to an entity containing ¸on (U+0327 U+006F U+006E), then the string suc&word-end; is not include-normalized for the same reasons.

In an XML or HTML context, expanding ̧ in the string sub̧on yields the string sub¸on which is Unicode-normalized since there is no precomposed character for 'b cedilla' in NFC. This string is therefore also include-normalized. Similarly, the string sub&word-end; (with &word-end; as above) is include-normalized, for the same reasons.

In an XML or HTML context, the strings ¸on (U+0327 U+006F U+006E) and ̧on are not fully-normalized, as they begin with a composing character (after expansion of the character escape for the second). However, both are Unicode-normalized (if expressed in a Unicode encoding form) and include-normalized.

The following table consolidates the above examples. Normalized forms are indicated using 'Y', a hyphen means 'not normalized'.

4.3.2 Examples of XML in a Unicode encoding form

4.3.3 Examples of restrictions on the use of combining characters

Full-normalization prevents the markup of an isolated combining mark, for example for styling it differently from its base character (Benoi<span style='color: blue'>^</span>t, where '^' represents a combining circumflex). However, the equivalent effect can be achieved by assigning a class to the accents in an SVG font or using equivalent technology. View an example using SVG (SVG-enabled browsers only).

Full-normalization prevents the use of entities for expressing composing characters. This limitation can be circumvented by using character escapes or by using entities representing complete combining character sequences. With appropriate entity definitions, instead of A´, write Á (or better, use 'Á' directly).