UNICODE

AND
MULtILINGUAL
COMPUTING

Presented by:
K.HARITHA
V.MANASA
O7W11A1211 07W11A1218
Information technology
of
Vivekananda institute of engineering and
technology
 CONTENTS

 ABSTRACT
 INTRODUCTION
 SOFTWARE INTERNATIONALIZATION
 UNICODE CODED REPRESENTATION
 UNICODE IN THE SOLARIS 8 OPERATING ENVIRONMENT
 TECHICAL CONSIDERATIONS
 CONCLUSION

ABSTRACT
 Today's global economy demands global computing solutions.
Instant communications across continents--and computer platforms--characterize a
business world at work 24 hours a day, 7 days a week. The widespread use of the
Internet and e-commerce continue to create new international challenges.

More and more, users are demanding a computing

environment to suit their own linguistic and cultural needs. They want applications and
file formats they can share around the world, interfaces in their own language, and local
time and date displays. Essentially, users want to write and speak at the keyboard the
way they write and speak in the office.

The Solaris operating environment multilingual

framework (including multiple character sets and multiple cultural attributes) uses the
standard universal encoding codeset, Unicode (The Unicode Standard, Version 3.0).
Unicode is well-suited to applications such as multilingual databases, e-commerce, and
government research and reference.

INTRODUCTION
Unicode:
In most writing systems, keyboard input is converted into character codes, stored in
memory, and converted to glyphs in a particular font for display and printing. The
collection of characters and character codes form a codeset. To represent characters of
different languages, a different codeset is used.

Multilingual Computing:
"Multilingual" computing can mean.The movement from multilanguage to multiscript
to multilingual implies an increased level of complexity in the underlying operating
environment.

In a multilingual environment, a locale can support multiple scripts and multiple cultural
attributes, giving an application greater control over text manipulation.

Software Internationalization
Sun Microsystems defines the following levels at which an application can
support a customer's international needs:

 Internationalization
 Localization

Software internationalization is the process of designing and implementing software to

transparently manage different linguistic and cultural conventions without additional
modification.

Software localization is the process of adding language translation (including text

messages, icons, buttons, and so on), cultural data, and components (such as input
methods and spell checkers) to a product to meet regional market requirements.

Internationalization Framework:
A properly internationalized application separates language- and cultural-specific
information from the application code. The Solaris operating environment
internationalization framework achieves this with:

 Locales

 Localizable interface

  Codeset independence

A locale is the language and cultural data set by the user and dynamically loaded into
memory at run time. The locale settings are applied to the operating system and to
subsequent application launches.

Supporting the Unicode Standard:

Unicode (Universal Codeset) is a universal character encoding scheme developed and
promoted by the Unicode Consortium, a non-profit organization which includes Sun
Microsystems. The Unicode standard encompasses most alphabetic, ideographic, and
symbolic characters.

Benefits of Unicode:
Support for Unicode provides many benefits to application developers, including:

 Global source and binary.

 Support for mixed-script computing environments.
 Improved cross-platform data interoperability through a common codeset.
 Space-efficient encoding scheme for data storage.
 Reduced time-to-market for localized products.
 Expanded market access.

Unicode Coded Representations

In recent years, the Unicode Consortium and other related organizations have developed
different formats to represent and store a Unicode codeset. To represent characters from all
major languages in multibyte format, the ISO/IEC International Standard 10646-1
(commonly referred to as 10646) has defined the Universal Multiple-Octet Coded
Character Set (UCS) format. Character forms contained in the 10464 specifications are:

 Universal Coded Character Set-2 (UCS-2) also known as Basic Multilingual Plane
(BMP)--characters are encoded in two bytes on a single plane.

 Universal Coded Character Set-4 (UCS-4)--characters encoded in four bytes on

multiple planes and multiple groups.

 UCS Transformation Format 16-bit form (UTF-16)--extended variant of UCS-2 with

characters encoded in 2-4 bytes.
  UCS Transformation Format 8-bit form (UTF-8)--a transformation format using
characters encoded in 1-6 bytes.

UTF-8 encoding scheme:

Bits  Hex Min  Hex Max  UTF-8 Binary Encoding 

7  00000000  0000007F  0xxxxxxx 

11  00000080  000007FF  110xxxxx 10xxxxxx 

16  00000800  0000FFFF  1110xxxx 10xxxxxx 10xxxxxx 

21  00010000  001FFFFF  11110xxx 10xxxxxx 10xxxxxx 10xxxxxx 

26  00200000  03FFFFFF  111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 

31  04000000  7FFFFFFF  1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

10xxxxxx 

In addition to its backward compatibility with 7-bit ASCII, UTF-8 is a space-efficient

encoding scheme when the encoded data needs only one-byte or less (as for English and
other Roman character-based writing systems). Because UTF-8 stores one-byte data as one
byte, rather than, for example, the two bytes required by UTF-16, this can significantly
decrease the storage space required to hold large blocks of international data.

Because of its flexibility and compatibility with ASCII and UNIX, Unicode support of the
UTF-8 format is used in the Solaris operating environment. UTF-8 provides developers
with a format compatible with existing internationalized environments and an easy path
for Internet and legacy data interoperability. As a file system safe format, UTF-8 supports
one-byte unit I/O operations and can represent the Unicode formats UCS-2 and UCS-4.
Furthermore, UTF-8 fits well within the XPG internationalization framework.

Unicode in the Solaris 8 Operating

environment
The support of Unicode, Version 3.0 in the Solaris 8 Operating Environment's Unicode
locales has provided an enhanced framework for developing multiscript applications.
Properly internationalized applications require no changes to support the Unicode locales.
All internationalized CUI and GUI utilities and commands in the Solaris operating
environment are available in Unicode locales without modification.

Note -

To choose an input mode, press the Compose key and a two-letter code. For example, to
input text in Thai, press Compose+tt. Alternatively, click the status area and select an
input mode as shown in Figure 3-1. (To select the default English/European mode, press
Control+Space.)

Table 3-1 UTF-8 Input Mode two-letter codes

Language  Code 

Cyrillic  cc

Greek  gg

Thai  tt

Arabic  ar

Hebrew  hh

Unicode Hex  uh

Unicode Octal  uo

Lookup  ll

Japanese  ja

Korean  ko

Simplified Chinese  sc

Traditional Chinese  tc

English/European  Control+Space
Figure 3-1 UTF-8 Input Mode selection

To input text from a Lookup table, select the Lookup input mode. A lookup table with
all input modes and various symbol and technical codesets appears, as shown in
Figure 3-2.

The Arabic, Hebrew, and Thai input modes provide full complex text layout features,
including right-to-left display and context-sensitive character rendering. The Unicode octal
and hexadecimal code input modes generate Unicode characters from their octal and
hexadecimal equivalents, respectively. The Japanese, Korean, Simplified Chinese, and
Traditional Chinese input modes provide full native Asian input.
UTF-8 Table Lookup:
Asian input mode

For more information on each input method, refer to the chapter Overview of en_US.UTF-
8 Locale Support in the latest Solaris International Language Environments Guide, ATOK12
User's Guide, Wnn6 User's Guide, cs00 User's Guide, Korean Solaris User's Guide,
Simplified Chinese Solaris User's Guide, and Traditional Chinese Solaris User's Guide.

The Unciode locale supports various MIME character sets in dtmail, including various
Latin, Greek, Cyrillic, Thai, and Asian character sets. Some of the example character sets
are: ISO-8859-1 ~ 10, 13, 14, 15, UTF-8, UTF-7, UTF-16, UTF-16BE. With this support,
users can send and receive email messages encoded in MIME character sets from almost
any region in the world. Dtmail automatically decodes e-mail by recognizing the MIME
character set and content transfer encoding in the message. The sender specifies the MIME
character set for the recipient mail user agent.
Figure 3-4 multiple character sets in dtmail

Codeset Conversion
The Solaris operating environment locale supports enhanced code conversion among the
major codesets of several countries. Figure 3-5 shows the codeset conversions between
UTF-8 and many other codesets. Figure 3-5 Unicode codeset conversions

Codesets can be converted using the sdtconvtool utility or the iconv(1) command.
Sdtconvtool detects available iconv code conversions and presents them in an easy-to-
use format.
Figure 3-6 sdtconvtool for converting between codesets

Users can also add their own code conversions and use them in iconv(3) functions,
iconv(1) command line utilities, and sdtconvtool(1). For more information on user-
extensible, user-defined code conversions, refer to the geniconvtbl(1) and
geniconvtbl(4) man pages.

Developers can use iconv(3) to access the same functionality. This includes conversions
to and from UTF-8 and many ISO-standard codesets, including UCS-2, UCS-4, UTF-7,
UTF-16, KO18-R, Japanese EUC, Korean EUC, Simplified Chinese EUC, Traditional
Chinese EUC, GBK, PCK (Shift JIS), BIG5, Johap, ISO-2022-JP, ISO-2022-KR, and ISO-
2022-CN.

European Unicode Locales

In the Solaris 8 operating environment, five European Unicode locales offer the same level
of support as en_US.UTF-8 with modifications for language and cultural data.

The five European Unicode locales are:

 fr_FR.UTF-8 (French)

 de_DE.UTF-8 (German)

 it_IT.UTF-8 (Italian)
  es_ES.UTF-8 (Spanish)

 sv_SE.UTF-8 (Swedish)

The following additional five European locales support the Euro currency symbol and
monetary formatting conventions:

 fr_FR.UTF-8@euro (French with euro monetary convention)

 de_DE.UTF-8@euro (German with euro monetary convention)

 it_IT.UTF-8@euro (Italian with euro monetary convention)

 es_ES.UTF-8@euro (Spanish with euro monetary convention)

 sv_SE.UTF-8@euro (Swedish with euro monetary convention)

Note -

All Unicode locales (including en_US.UTF-8 and Asian Unicode locales) support input and
output of the new euro currency symbol.

3.4 Asian Unicode Locales

The Solaris 8 operating environment also supports four Unicode locales with the same
scope as en_US.UTF-8 and the European Unicode locales, with the necessary language
and cultural modifications:

 ja_JP.UTF-8 (Japanese)

 ko_KR.UTF-8 (Korean)

 zh_CN.UTF-8 (Simplified Chinese)

 zh_TW.UTF-8 (Traditional Chinese)

3.5 Unicode Font Resources

The Unicode Standard, Version 3.0 contains 49,194 characters from the world's scripts, with
over 25,000 ideographic characters for Chinese, Japanese, and Korean.To manage these
difficulties, the Solaris operating environment contains an output method combining
existing fonts to form a Unicode font set, instead of providing a single Unicode font. The
Solaris 8 operating environment supports the following range of scripts:

 English/European

 Greek, Turkish, Cyrillic

 Arabic, Hebrew, Thai

 Simplified Chinese, Traditional Chinese, Japanese, Korean

Font Resources
Properly internationalized applications require only a few changes to run properly in
the Solaris operating environment Unicode locales. The en_US.UTF-8 locale supports
the following set of font character sets as the Font Set:

 ISO 8859-1 (Latin-1)

 ISO 8859-2 (Latin-2)
 ISO 8859-4 (Latin-4)
 ISO 8859-5 (Latin/Cyrillic)
 ISO 8859-7 (Latin/Greek)
 ISO 8859-9 (Latin-5)
 ISO 8859-15 (Latin-9)
 ISO 8859-6 based one (Arabic)
 ISO 8859-8 (Hebrew)
 TIS 620-2533 based one (Thai)
 BIG5 (Traditional Chinese)
 GB 2312-1980 (Simplified Chinese)
 JIS X0201-1976, JIS X0208-1983 (Japanese)
 KS C 5601-1992 Annex 3 (Korean)

Setting Resource Definitions

To create a font set for an application, the resource definition should contain the
complete set of fonts supported by the Unicode locale. For example:

fs = XCreateFontSet(display,
"-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-iso8859-1,
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-iso8859-2,
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-iso8859-4,
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-iso8859-5,
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-iso8859-6,
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-iso8859-7,
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-iso8859-8,
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-iso8859-9,
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-iso8859-15,
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-big5-1,
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-gb2312.1980-0,
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-jisx0201.1976-0,
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-jisx0208.1983-0,
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-ksc5601.1992-3,
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-tis620.2533-0",
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-unicode-fontspecific",
&missing_ptr, &missing count, &def_string);

Or, more simply:

! This is an example XmNFontList definition for en_US.UTF-8 locale:

*fontList: \
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-iso8859-1; \
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-iso8859-2; \
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-iso8859-4; \
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-iso8859-5; \
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-iso8859-6; \
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-iso8859-7; \
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-iso8859-8; \
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-iso8859-9; \
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-iso8859-15; \
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-big5-1; \
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-gb2312.1980-0; \
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-jisx0201.1976-0; \
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-jisx0208.1983-0; \
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-ksc5601.1992-3; \
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-tis620.2533-0; \
-dt-interface system-medium-r-normal-s*utf*-*-*-*-*-*-*-*-unicode-fontspecifc:

Conclusions
What have you learned?
In this tutorial you have learned the very basic construction techniques of a
Unicode-based multilingual Web page. Through the use of XML, Unicode, and
Programming languages such as Perl, there are great possibilities for developing
Broad-based multilingual interactivity, as well as for organizing data and tabulating
Responses in many different languages. As the Unicode Standard Version 3.0 states,
Unicode "defines a consistent way of encoding multilingual text that enables the
Exchange of text data internationally and creates the foundation for global software"
The Unicode Standard Version 3.0, p. 1, 2000). We have merely scratched the surface
In demonstrating how this global software might be developed, but it will certainly
follow many of the principles.

Problem areas:
There are, nevertheless, still problem areas. As Michel Rodriguez points out in
"Character Encodings in XML and Perl" (see Resources), certain current applications
And databases do not readily accept Unicode input. There are various techniques
(Using appropriate encoding tables and mapping) for the required input and output
Conversions, but these can be cumbersome. Additionally, it will still take some time
Before Unicode fonts and XML-enabled browsers are standard worldwide. Regardless
Of these problems, Unicode-based multilingual development will soon begin to grow
Rapidly, providing programmers and Web developers with a tremendous number of
Possibilities in software applications that are truly global.