Character Model for the World Wide Web 1.0

1.1 Goals and Scope

The goal of this document is to facilitate use of the Web by all people, regardless of their language, script, writing system, and cultural conventions, in accordance with the W3C goal of universal access. One basic prerequisite to achieve this goal is to be able to transmit and process the characters used around the world in a well-defined and well-understood way.

The main target audience of this document is W3C specification developers. This document defines conformance requirements for other W3C specifications. This document and parts of it can also be referenced from other W3C specifications.

Other audiences of this document include software developers, content developers, and authors of specifications outside the W3C. Software developers and content developers implement and use W3C specifications. This document defines some conformance requirements for software developers and content developers that implement and use W3C specifications. It also helps software developers and content developers to understand the character-related provisions in other W3C specifications.

The character model described in this document provides authors of specifications, software developers, and content developers with a common reference for consistent, interoperable text manipulation on the World Wide Web. Working together, these three groups can build a more international Web.

Topics addressed include encoding identification, early uniform normalization, string identity matching, string indexing, and URI conventions. Some introductory material on characters and character encodings is also provided.

Topics not addressed or barely touched include collation (sorting), fuzzy matching and language tagging. Some of these topics may be addressed in a future version of this specification.

At the core of the model is the Universal Character Set (UCS), defined jointly by The Unicode Standard [Unicode] and ISO/IEC 10646 [ISO/IEC 10646]. In this document, Unicode is used as a synonym for the Universal Character Set. The model will allow Web documents authored in the world's scripts (and on different platforms) to be exchanged, read, and searched by Web users around the world.

All W3C specifications must conform to this document (see section 2 Conformance). Authors of other specifications (for example, IETF specifications) are strongly encouraged to take guidance from it.

Since other W3C specifications will be based on some of the provisions of this document, without repeating them, software developers implementing W3C specifications must conform to these provisions.

1.2 Background

Starting with Internationalization of the Hypertext Markup Language [RFC 2070], the Web community has recognized the need for a character model for the World Wide Web. The first step towards building this model was the adoption of Unicode as the document character set for HTML.

The choice of Unicode was motivated by the fact that Unicode:

• is the only universal character repertoire available,

• covers the widest possible range,

• provides a way of referencing characters independent of the encoding of a resource,

• is being updated/completed carefully,

• is widely accepted and implemented by industry.

W3C adopted Unicode as the document character set for HTML in [HTML 4.0]. The same approach was later used for specifications such as XML 1.0 [XML 1.0] and CSS2 [CSS2]. Unicode now serves as a common reference for W3C specifications and applications.

The IETF has adopted some policies on the use of character sets on the Internet (see [RFC 2277]).

When data transfer on the Web remained mostly unidirectional (from server to browser), and where the main purpose was to render documents, the use of Unicode without specifying additional details was sufficient. However, the Web has grown:

• Data transfers among servers, proxies, and clients, in all directions, have increased.

• Non-ASCII characters [MIME] are being used in more and more places.

• Data transfers between different protocol/format elements (such as element/attribute names, URI components, and textual content) have increased.

• More and more APIs are defined, not just protocols and formats.

In short, the Web may be seen as a single, very large application (see [Nicol]), rather than as a collection of small independent applications.

While these developments strengthen the requirement that Unicode be the basis of a character model for the Web, they also create the need for additional specifications on the application of Unicode to the Web. Some aspects of Unicode that require additional specification for the Web include:

• Choice of encoding forms (UTF-8, UTF-16, UTF-32).

• Counting characters, measuring string length in the presence of variable-length encodings and combining characters).

• Duplicate encodings (e.g. precomposed vs decomposed).

• Use of control codes for various purposes (e.g. bidirectionality control, symmetric swapping, etc.).

It should be noted that such properties also exist in legacy encodings (where legacy encoding is taken to mean any character encoding not based on Unicode), and in many cases have been inherited by Unicode in one way or another from such legacy encodings.

The remainder of this document presents additional specifications and requirements to ensure an interoperable character model for the Web, taking into account earlier work (from W3C, ISO and IETF).

1.3 Terminology and Notation

3.1 Perceptions of Characters

3.1.3 Units of Visual Rendering

Visual rendering introduces the notion of a glyph. Glyphs are defined by ISO/IEC 9541-1 [ISO/IEC 9541-1] as "a recognizable abstract graphic symbol which is independent of a specific design". There is not a one-to-one correspondence between characters and glyphs:

• A single character can be represented by multiple glyphs (each glyph is then part of the representation of that character). These glyphs may be physically separated from one another.

• A single glyph may represent a sequence of characters (this is the case with ligatures, among others).

• A character may be rendered with very different glyphs depending on the context.

• A single glyph may represent different characters (e.g. capital Latin A, capital Greek A and capital Cyrillic A).

Each glyph can be represented by a number of different glyph images; a set of glyph images makes up a font. Glyphs can be construed as the basic units of organization of the visual rendering of text, just as characters are the basic unit of organization of encoded text.

[S] [I] Specifications and software MUST NOT assume a one-to-one mapping between character codes and units of displayed text.

See the appendix A Examples of Characters, Keystrokes and Glyphs for examples of the complexities of character to glyph mapping.

Some scripts, in particular Arabic and Hebrew, are written from right to left. Text including characters from these scripts can run in both directions and is therefore called bidirectional text (see example A.6 in Appendix A). The Unicode Standard [Unicode] requires that characters be stored and interchanged in logical order. [S] Protocols, data formats and APIs MUST store, interchange or process text data in logical order.

In the presence of bidirectional text, two possible selection modes must be considered. The first is logical selection mode, which selects all the characters logically located between the end-points of the user's mouse gesture. Here the user selects from between the first and second letters of the second word to the middle of the number. Logical selection looks like this:

In memory
On screen

It is a consequence of the bidirectionality of the text that a single, continuous logical selection in memory results in a discontinuous selection appearing on the screen. This discontinuity, as well as the somewhat unintuitive behavior of the cursor, makes some users prefer a visual selection mode, which selects all the characters visually located between the end-points of the user's mouse gesture. With the same mouse gesture as before, we now obtain:

In memory
On screen

In this mode, a single visual selection range results in two logical ranges, which have to be accommodated by protocols, APIs and implementations.

[S] Specifications of protocols and APIs that involve selection of ranges SHOULD provide for discontiguous selections, at least to the extent necessary to support implementation of visual selection on screen on top of those protocols and APIs.

3.2 Digital Encoding of Characters

Given the complexity of text encoding and the large variety of schemes for character encoding invented throughout the computer age, a more formal description of the encoding process is useful. The process of defining a text encoding can be described as follows (see [UTR #17] for a more detailed description):

1. A set of characters to be encoded is identified. The characters are pragmatically chosen to express text and to efficiently allow various text processes in one or more target languages. They may not correspond precisely to what users perceive as letters and other characters. The set of characters is called a repertoire.

2. Each character in the repertoire is then associated with a (mathematical, abstract) non-negative integer, the code point (also known as a character number or code position). The result, a mapping from the repertoire to the set of non-negative integers, is called a coded character set (CCS).

3. To enable use in computers, a suitable base datatype is identified (such as a byte, a 16-bit unit of storage or other) and a character encoding form (CEF) is used, which encodes the abstract integers of a CCS into sequences of the code units of the base datatype. The encoding form can be extremely simple (for instance, one which encodes the integers of the CCS into the natural representation of integers of the chosen datatype of the computing platform) or arbitrarily complex (a variable number of code units, where the value of each unit is a non-trivial function of the encoded integer).

4. To enable transmission or storage using byte-oriented devices, a serialization scheme or character encoding scheme (CES) is next used. A CES is a mapping of the code units of a CEF into well-defined sequences of bytes, taking into account the necessary specification of byte-order for multi-byte base datatypes and including in some cases switching schemes between the code units of multiple CESes (an example is ISO 2022). A CES, together with the CCSes it is used with, is identified by an IANA charset identifier. Given a sequence of bytes representing text and a charset identifier, one can in principle unambiguously recover the sequence of characters of the text.

In very simple cases, the whole encoding process can be collapsed to a single step, a trivial one-to-one mapping from characters to bytes; this is the case, for instance, for US-ASCII [MIME] and ISO-8859-1.

Text data is said to be in a Unicode encoding form if it is encoded in UTF-8, UTF-16 or UTF-32.

3.3 Transcoding

Transcoding is the process of converting text data from one Character Encoding Form to another. Transcoders work only at the level of character encoding and do not parse the text; consequently, they do not deal with character escapes such as numeric character references (see 3.7 Character Escaping) and do not adjust embedded character encoding information (for instance in an XML declaration or in an HTML meta element).

A normalizing transcoder is a transcoder that converts from a legacy encoding to a Unicode encoding form and ensures that the result is in Unicode Normalization Form C (see 4.2.1 Unicode-normalized Text). For most legacy encodings, it is possible to construct a normalizing transcoder; it is not possible to do so if the encoding's repertoire contains characters not in Unicode.

3.4 Strings

Byte string: A string viewed as a sequence of bytes representing characters in a particular encoding. This corresponds to a CES. As a definition for a string, this definition is most often useless, except when the textual nature is unimportant and the string is considered only as a piece of opaque data with a length in bytes. [S] Specifications in general SHOULD NOT define a string as a 'byte string'.

Code unit string: A string viewed as a sequence of code units representing characters in a particular encoding. This corresponds to a CEF. This definition is useful in APIs that expose a physical representation of string data. Example: For the DOM [DOM Level 1], UTF-16 was chosen based on widespread implementation practice.

Character string: A string viewed as a sequence of characters, each represented by a code point in Unicode [Unicode]. This is usually what programmers consider to be a string, although it may not match exactly what most users perceive as characters. This is the highest layer of abstraction that ensures interoperability with very low implementation effort. [S] The 'character string' definition of a string is generally the most useful and SHOULD be used by most specifications, following the examples of Production [2] of XML 1.0 [XML 1.0], the SGML declaration of HTML 4.0 [HTML 4.01], and the character model of RFC 2070 [RFC 2070].

EXAMPLE: Consider the string comprising the characters U+233B4 (a Chinese character meaning 'stump of tree'), U+2260 NOT EQUAL TO and U+0030 DIGIT ZERO, encoded in UTF-16 in big-endian byte order. The rows of the following table show the string viewed as a character string, code unit string and byte string, respectively:

Character string	U+233B4				U+2260		U+0030
Code unit string	D84C		DFB4		2260		0030
Byte string	D8	4C	DF	B4	22	60	00	30

3.5 Reference Processing Model

Many Internet protocols and data formats, most notably the very important Web formats HTML, CSS and XML, are based on text. In those formats, everything is text but the relevant specifications impose a structure on the text, giving meaning to certain constructs so as to obtain functionality in addition to that provided by plain text. HTML and XML are markup languages, defining entities entirely composed of text but with conventions allowing the separation of this text into markup and character data. Citing from the XML 1.0 specification [XML 1.0], section 2.4:

"Text consists of intermingled character data and markup. [...] All text that is not markup constitutes the character data of the document."

For the purposes of this section, the important aspect is that everything is text, that is, a sequence of characters.

Since its early days, the Web has seen the development of a Reference Processing Model, first described for HTML in RFC 2070 [RFC 2070]. This model was later embraced by XML and CSS. It is applicable to any data format or protocol that is text-based as described above. The essence of the Reference Processing Model is the use of Unicode as a common reference. Use of the Reference Processing Model by a specification does not, however, require that implementations actually use Unicode. The requirement is only that the implementations behave as if the processing took place as described by the Model.

A specification conforms to the Reference Processing Model if all of the following apply:

• [S ] Specifications MUST be defined in terms of Unicode characters, not bytes or glyphs.

• [S] Specifications SHOULD allow the use of the full range of Unicode code points from U+0 to U+0FFFF inclusive; any exceptions SHOULD be listed and justified; code points above U+10FFFF MUST NOT be used.

• [S] Specifications MAY allow use of any character encoding which can be transcoded to Unicode for its text entities.

• [S] Specifications MAY choose to disallow or deprecate some encodings and to make others mandatory. Independent of the actual encoding, the specified behavior MUST be the same as if the processing happened as follows:

  • The encoding of any text entity received by the application implementing the specification MUST be determined and the text entity MUST be interpreted as a sequence of Unicode characters - this MUST be equivalent to transcoding the entity to some Unicode encoding form, adjusting any character encoding label if necessary, and receiving it in that Unicode encoding form.

  • All processing MUST take place on this sequence of Unicode characters.

  • If text is output by the application, the sequence of Unicode characters MUST be encoded using an encoding chosen among those allowed by the specification.

• [S] If a specification is such that multiple text entities are involved (such as an XML document referring to external parsed entities), it MAY choose to allow these entities to be in different character encodings. In all cases, the Reference Processing Model MUST be applied to all entities.

[S] All specifications that involve text MUST specify processing according to the Reference Processing Model.

3.6 Choice and Identification of Character Encodings

Because encoded text cannot be interpreted and processed without knowing the encoding, it is vitally important that the character encoding (see 3.2 Digital Encoding of Characters) is known at all times and places where text is exchanged or processed. In what follows we use 'character encoding' to mean either CEF or CES depending on the context. When text transmitted as a byte stream is involved, for instance in a protocol, specification of a CES is required to ensure proper interpretation; in contexts such as an API, where the environment (typically the processor architecture) specifies the byte order of multibyte quantities, specification of a CEF suffices. [S] Specifications MUST either specify a unique encoding, or provide character encoding identification mechanisms such that the encoding of text can always be reliably identified. [S] When designing a new protocol, format or API, specifications SHOULD mandate a unique character encoding.

3.7 Character Escaping

In text-based protocols or formats where characters can be either part of character data or of markup (see 3.5 Reference Processing Model), it is often the case that certain characters are designated as having certain specific protocol/format functions in certain contexts (e.g. '<' and '&' serve as markup delimiters in HTML and XML). These syntax-significant characters cannot be used to represent themselves in text data in the same way as all other characters do. Also, often formats are represented in an encoding that does not allow to represent all characters directly.

To express syntax-significant or unrepresentable characters, a technique called escaping is used. This works by creating an additional syntactic construct, defining additional characters or defining character sequences that have special meaning. Escaping a character means expressing it using such a construct, appropriate to the format or protocol in which the character appears; expanding an escape (or unescaping) means replacing it with the character that it represents.

Certain guidelines apply to the way specifications define character escapes. [S] The guidelines in this document relating to the definition of character escapes MUST be followed when designing new W3C protocols and formats and SHOULD be followed as much as possible when revising existing protocols and formats.

• [S] Specifications MUST NOT invent a new escaping mechanism if an appropriate one already exists.

• [S] The number of different ways to escape a character SHOULD be minimized (ideally to one). [A well-known counter-example is that for historical reasons, both HTML and XML have redundant decimal (&#ddddd;) and hexadecimal (&#xhhhh;) escapes.]

• [S] Explicit end delimiters MUST be provided. Escapes such as \uABCD where the end delimiter is a space or any character other than [01-9A-F] SHOULD be avoided. These escapes are not clear visually, and can cause an editor to insert spurious line-breaks when word-wrapping on spaces. Forms like SPREAD's &UABCD; [SPREAD] or XML's &#xhhhh;, where the escape is explicitly terminated by a semicolon, are much better.

• [S] Whenever specifications define escapes that allow the representation of characters using a number the number SHOULD be in hexadecimal notation.

• [S] Escaped characters SHOULD be acceptable wherever unescaped characters are; this does not preclude that a syntax-significant character, when escaped, loses its significance in the syntax. In particular, escaped characters SHOULD be acceptable in identifiers and comments.

Certain guidelines apply to content developers, as well as to software that generates content:

• [I] [C] Escapes SHOULD be avoided when the characters to be expressed are representable in the character encoding of the document.

• [I] [C] Since character set standards usually list character numbers as hexadecimal, content SHOULD use the hexadecimal form of escapes when there is one.

4.1 Motivation

4.2 Definitions for W3C Text Normalization

The Unicode Consortium provides four standard normalization forms (see Unicode Normalization Forms [UTR #15]). For use on the Web, this document defines Web-related text normalization forms by picking the most appropriate of these, Unicode Normalization Form C (NFC), and additionally addressing the issues of legacy encodings, character escapes, includes, and character and markup boundaries..

4.3 Responsibility for Normalization

This section defines the responsibility for normalization, based on the goal of early uniform normalization.

Unless otherwise specified, the word 'normalization' in this section may refer to 'include-normalization' or 'full normalization', depending on which is most appropriate for the specification or implementation under consideration.

An operation is normalization-sensitive if its output(s) are different depending on the state of normalization of the input(s); if the output(s) are textual, they are deemed different only if they would remain different were they to be normalized. These operations are any that involve comparison of characters or character counting, as well as some other operations such as ‘delete first character’ or ‘delete last character’.

A text-processing component is a component that recognizes data as text. This specification does not specify the boundaries of a text-processing component, which may be as small as one line of code or as large as a complete application. A text-processing component may receive text, produce text, or both.

Certified text is text which satisfies at least one of the following conditions:

1. the text has been successfully validated for normalization, or

2. the source text-processing component is identified and is known to produce only normalized text.

Suspect text is text which is not certified.

[C] All text content on the Web MUST be in include-normalized form and SHOULD be in fully normalized form.

[S] Specifications of text-based formats and protocols MUST, as part of their syntax definition, require that the text be in normalized form.

[S] [I] A text-processing component that receives suspect text MUST NOT perform any normalization-sensitive operations unless it has first successfully validated the text for normalization, and MUST NOT normalize the suspect text. Private agreements MAY, however, be created within private systems which are not subject to these rules, but any externally observable results MUST be the same as if the rules had been obeyed.

[I] A text-processing component which modifies text and performs normalization-sensitive operations MUST behave as if normalization took place after each modification, so that any subsequent normalization-sensitive operations always behave as if they were dealing with normalized data.

EXAMPLE: If the 'z' is deleted from the (normalized) string cz¸ (where '¸' represents a combining cedilla, U+0327), normalization is necessary to turn the denormalized result c¸ into the properly normalized ç. Analogous cases exist for insertion and concatenation. If the software that deletes the 'z' later uses the string in a normalization-sensitive operation, it needs to normalize the string before this operation to ensure correctness; otherwise, normalization may be deferred until the data is exposed.

[S] Specifications of text-based languages and protocols SHOULD define precisely the construct boundaries necessary to obtain a complete definition of full normalization. These definitions MUST include at least the boundaries between markup and character data as well as entity boundaries (if the language has any include mechanism) and SHOULD include any other boundary that may create denormalization when instances of the language are processed.

[S] Specifications MUST document any security issues related to normalization.

[I] Implementations which transcode text data from a legacy encoding to a Unicode encoding form MUST use a normalizing transcoder.

[S] Specifications of API components (functions/methods) that perform operations that may produce unnormalized text output from normalized text input MUST define whether normalization is the responsibility of the caller or the callee. Specifications MAY make performing normalization optional for some API components; in this case the default SHOULD be that normalization is performed, and an explicit option SHOULD be used to switch normalization off. Specifications MUST NOT make the implementation of normalization optional.

[S] Specifications that define a mechanism (for example an API or a defining language) for producing a document SHOULD require that the final output of this mechanism be normalized. examples: DOM (load and) save, XSLT

NOTE: As an optimization, it is perfectly acceptable for a system to define the producer to be the actual producer (e.g. a small device) together with a remote component (e.g. a server serving as a kind of proxy) to which normalization is delegated. In such a case, the communications channel between the device and proxy server is considered to be internal to the system, not part of the Web. Only data normalized by the proxy server is to be exposed to the Web at large, as shown in the illustration below:

Illustration of a text producer defined as including a proxy.