Tool

Assessing Translation A Manual for Requesters

TranslaTi
The health care

MORE THAN WORDS

PART I: To be com
Reques ter

For HealT
uester

on

QualiTy a
H
ment of an exis

Purpose: The primary purposes of this tool are to enable requesters to use the Translation Quality Assessment (TQA) Tool and to describe why this promising method of quality assessment is superior to other approaches now in use. Tool #6 is a practical how-to manual for requesters to initiate an evaluation of a translated text using the TQA Tool. It describes how TQA raters (professionals and/or academics with advanced language skills) use the tool to evaluate the quality of a translation and the recommendations they can make. This guide also acquaints requesters with the various methods and approaches used in evaluating translated materials, as well as those incorporated in the tool design. Also included are highlights about how the TQA Tool was tested and guidelines for selecting raters with whom to work.

decision mak er
(Name) rtment

pleted by Req

educaTion

ssessmen T

m aTerials
ed text.
Phone

Tool

requesting a

quality assess

ting translat

Title/Depa

Source Lan Tex t Type Tex t Title Target Aud Purpose of ience

TRANSLATIO
guage

N BRIEF

Deliver y Dat

English Health Educati on Material

Target Lan

guage

Health Educati on Material

Document

Priority Foc

us Area

2 Target Lan gua 1 Functional and 3 Non-Sp ecia 4 Specialized

QUALITY
ge

CRITERIA
quacy

Rank EACH from (1 being top 1 to 4 priority)

Tex tual Ade

lized Content

(Meaning)

PART II: To be com

Content and

Rater (Name) Phone Total Score

pleted by TQA

Terminology

Rater
Date Complet

ed d

Date Receive

ASSESSMEN
q Publish and
To be complet ed after evaluating tran slated text

T SUMMAR Y

/or use as is

AND RECO MMEN

Total Rating

Time

DATION

q Minor edit s

needed befo

q Major revi sion nee q Redo tran slation

re publishing* lishing*

ded before pub

Notes/ Rec ommende d Edits

q Translat ion will not be an effectiv options (e.g e communicat . create new ion strategy target languag for this text. e materials) Explore

other

Need: Generally speaking, those responsible for procuring translated materials in a health care setting often have little or no knowledge of the intricacies of translation. Consequently, poor quality translations are commonplace in health care organizations as well as in publicly-trusted sources of information like federal public health agencies and the Library of Medicine websites. Although health care organizations are aware of regulatory and accreditation compliance issues regarding translation and are becoming more aware of the impact of language barriers on operations and quality efforts, few have developed defined systems or processes for procuring or assessing the quality of translated materials. Further troubling is the fact that current approaches to translation quality are only somewhat applicable to (and therefore less than ideal for) health care organizations. Some approaches focus on the steps taken to produce a translated text or emphasize producing an English equivalent, while others focus on both the process and faithfulness to the original English text. A divergence of definitions of quality and a lack of consensus among experts also hinder progress in this area. In the end, health care organizations pay a lot of money for translated materials of dubious quality.

Using the translation QUality assessment tool a training manUal for reQUesters

Tool

With rapid growth in the LEP population, health care providers have a growing need for trustworthy and effective health information in the language of their patients. They need an efficient means of scrutinizing the quality of health care translations and assessment methods that go beyond a simple good or bad label. The TQA Tool offers health care providers evaluations with sufficient and detailed information in non-technical terms about the quality of a translation product, and if translation deficiencies exist, how they might affect intended communicative objectives. Who Should uSe ThiS Guide: This guide is meant for requesters, individuals, departments and organizations responsible for approving translations within their organizations. This tool explains how to complete Part I of the TQA Tool Cover Sheet, which is designed to orient a TQA rater to a specific rating assignment and establish quality criteria for the evaluation. Although raters with advanced linguistic skills are needed to apply the TQA Tool and render an opinion on translation quality, the evaluation process begins with a requester interested in assessing the quality of a specific translation product.

How to Use This Guide:

The Hablamos Juntos More Than Words Toolkit Series offers health care decision makers practical tools to become informed requesters and purchasers of non-English materials. A key message of the series is that translation quality requires a partnership between requesters and translators. Sections 1 and 2: About the TQA Tool Begin by reading Sections 1 and 2 to learn about the TQA Tool, how quality is defined and how the tool is constructed. Section 3: how to use the TQA Tool Next read Section 3 for a step-by-step review of the instructions for using the tool, how to complete Part I of the Cover Page and how to select raters with whom to work. Section 4: Finding TQA Raters Finally, Section 4 provides an overview of how to find and hire a qualified rater to apply the TQA Tool, including an overview of the training they should have and the fees raters are likely to charge for a quality assessment. Online training to become a TQA rater is also available through Hablamos Juntos for any professional and/or academic with advanced language skills.

Using the translation QUality assessment tool a training manUal for reQUesters

Tool

Section 1. General Characteristics of the TQA Tool

The TQA Tool, a four-page rating sheet, is a unique and ground-breaking approach to evaluation of translations. The tool, grounded in translation theory, builds on the experience of existing translation quality assessment approaches and is designed to overcome known limitations (e.g., arbitrary point systems, narrow evaluation foci). The tool draws on functionalist theory and textual and theoretical models that focus on the readers response, practical aspects and linguistic features of the translated text, and the relationship of the translation to the source text. Functionalist approaches recognize the role of extra-linguistic features (those not included within the realm of language) and promote the idea that the form (text type) and function or purpose and use of a target language text is of paramount importance in determining translation quality. Evaluations conducted by professionals with advanced language skills, trained to use the TQA Tool take into consideration various aspects emphasized in other current approaches to translation quality.
RATING INST
ctions only. on these instru should be based and evaluation Your decisions ries. RUCTIONS: translated text. onal catego review of the age and Functi the categories. ctions for the the Target LanguTally Shee ptor selection. each one of read the instru descri t h and score text given in ale for your 1. Carefully best fits the g at the Englis that t lookin to provide ration description decisions or target text withou 2. Check the you read the useful to make mended that they can be required, but 3. It is recom Component ents are not les or comm 4. Examp

The aim of the tool is to give requesters and health care decision makers a general overview of the quality of a translation product (i.e., the assessment purpose is summative).

1: Target Lang

uage

k Chec x bo one

curate e of sourc s inac difficult to read, borde use of the target langu er fectiv The structure ING ical the target). Non-S The amount of transf contain ry de is extremely d the RAT s. Ungrammat source nor pecialized n age text. ng) al. Ittext w rea inThe tions. Ve iency issue (neither the target langu eani riptio uc 5. No d language e orig language profic Content a sample of Desc t (M sort of mixe m th e instr considered nten the ns fro fie1.a th written in some extent that it cannot be purpose of the translation. d by tructure from the e viatio the ed Co . Speciaelements/s readability. dominates to d by th t be justified by er of lized Conte ializ ed de ot be justi tions ec fie essar y transf s its nt rrant cann source cannouc n-Sp from thesti and Terminology n instr ins some unnec unwa ation and affect at be ju 3: No rtant tions th nnot translatio comprehend. The text conta shows up in the transl ing, in most at ca d/or is hard to s impoand addi e language ssions result ns th an The text of the sourc ntain awkward expre additio or1.b al source text. The structure or co issions igin problems and gory om flects g of ble, there are or/and Cate ber ion re important al tex t. ions erstandin theis generally reada source text. the iss Num nslat d by target textm ning in the or respond to g, om isund ard Although the ire essar yea er from The tra ons and/ of the orig language that requ es of transf Problems/ awkw meanin some m in the target iti 1.c far casesisfrom unnec as it , and shad in the brief. ally written rend ehension ges in n shows translation s rly to texts origin inso chan fied for the ns. tio 3.a compr inal, ht nuance text reads simila type as those speci some issio ansla ig e orig The translated and text been tions. Tr or om in th tions. Sl se, audience all. uc e have Comments ditions di same purpo ained if existent at Ther ion instr 1.d g, ad are minimal t cont ns or ad lat expressions meanin conten omissio trans e 3.b ons in cts th ons, ents alterati y refle alterati Examples/ Comm Minor curatel rranted wa ion ac 3.c nslat thout un uately. The tra tions wi ed adeq er uc instr en rend be 3.d have

R INST

UCT

ginal sh ori Engli

IONS:

Category Number

revea Functional and ls translation Textua ation is nsible. The Adequacy The transl l age incomprehe mistakes. e langu ring on being age, spelling

Descriptio

Target Language

Check one box

Category Rating

Tr a n
PART
The hea

serious

I: To
lth car

be co

Score Value For

sla Tio n
Requ ester
qualit

Requ Title/

e dec ision
(Nam

mplet

Hea

es ter

maker

ed by

lTH

Qua

educa
nt of an

liTy

reques

Tion

asse

e)

ting a

m aT

ssm en
er ia ls

Depa

y ass

Tool

rtmen

essme

existin g

transl

ated tex

t.
e

Sourc Tex t Targe Purpo se of Type Tex t Total Score Title t Audie nce

TRAN
e Lang

SLAT

Health

Englis Educa

uage

ION

BRIEF

Deliv

Phon ery Da te

h terial Health
Targe t Lang uage

tion Ma

Docum

ent

Priori

Educa

ty Fo

tion Ma

cus Are

terial

Exam

ples /C

k Chec x bo one n Descriptio ut considering translated witho repaired with The text was not be ent. audience of the text. linguistic, etc.). Can function and d cont se, nce, (cultural, Phon ialize e the goals, purpo genre, need of the audie spec Disregard for n Total ation, but l of ogy ge purpose, textua Score for the transl inol riptio textual units, led know and audience on, needs of the audience, Desc 2.a Term ed purpose cient ons. and functi Date revisi to the intend some aspect of its Comp ent insuffi consideration lity, leted Cont d/or ASSE ent. text gives somet/s of it (e.g. level of forma SSME lized ific ogy an Date transl Thecont ated but it is ecia NT SU Recei spec . minol ized impor tant aspec ed audience, q Pu r requirtes effort ved 4: Sp MMAR ecial misses some ted. ons, etc.). Repaiconten ial ter of the intend be repaired with blish Total sp d spec fec derati and needs or2.b Y AN and /or Ratin Can ms an se (function) To be culturly af g Tim D RE translation. us al consi nce of y and/ use as comple q Mi e olog nora d of ter es to the goals, purpoand instructions for the serio COMM is t evalua min s/ig nor edi gory ting tran ted after ctions mman approximat ENDA t is no ts nee Cate ber ng ter ) arenes translated text could be, given the restri slated TION The good co ded bef m conten , persuasive it unaw tex t q involvi d ssive Nu ore pu als cts a as efficient as Major ative, expre akes ialize blishin Reve al needs and revisio tion: inform It fle not sted edits. e spec nt mist y.2.cre g* n nee purpose (func attends to cultur sugge eque but th ded bef 4.a lity). It also s the goals, inolog q Re us /fr rors, accomplishe ore pu level of forma do tra e term Serio blishin nslatio text accurately ed audience (including ical er of th g* n The translated ation and intend ition inolog needed. q Tra 4.b Notes rend or no edits term nslatio set for the transl the audience. Minor riate A few 2.d Edits /Recomme options n will no of prop t be c nd characteristics (e. and ap

omm

ents

2: Functiona
Category Number

l and Textu

al Adequacy

Check one box

2 Tar ITY get Lan CRITE guage RIA 1 Fu nction al and Tex tua 3 No l Adequ n-Spec acy ialized PART Conte 4 Sp II: To nt (M ecializ eaning be co ed Co ) mplet ntent and Ter ed by Rater minolo TQA (Nam Rater gy e)
Rank EA (1 bei CH from 1 ng top to priority 4 )

QUAL

4.

4.d Exam

ples /C

rate Accu subject. e to th ents omm

ed

Examples/ Comm

ents

g. cre

an ate new effective com target langu municatio age ma n terials strategy for ) this

tex t. Ex

plore

other

A Compositional Tool The TQA Tool uses four categories of assessment, with each component emphasizing a different aspect of translation quality. In this regard, the TQA Tool is a compositional tool designed to evaluate the four key aspects of any translation. These components include: Target Language an examination of the quality of the translations linguistic form (e.g., spelling, grammar, lexicon) Textual and Functional Adequacy a consideration of how well the translation achieves the goals, purpose and function of the text for its target audience Non-Specialized Content an assessment of whether the content accurately reflects the original text. Specialized Content an appraisal of translation accuracy of special terminology and/or specialized content.
Using the translation QUality assessment tool a training manUal for reQUesters

Tool

The aim of the tool is to give requesters and health care decision makers a general overview of the quality of a translation product (i.e., the assessment purpose is summative). Requesters receive from TQA raters a summary evaluation of the translations overall quality in the form of a recommendation about the adequacy of the translation. The object of assessment is the product, or the translated text. The goal is not to determine why the translator made a certain mistake, but rather what effects the mistake has on the text and its intended meaning. Although the TQA Tool has been tested specifically with health care education materials, it can also be used to evaluate other translation types. A Textual Approach The TQA Tool uses a textual approach to evaluation, meaning that texts are seen as whole units, comprised of smaller linguistic sub-units (words, headings, sentences, paragraphs). The basic premise of an evaluation is that texts are organized in a particular way to achieve a defined function (to educate, inform, explain) within a particular purpose for an intended audience (communicative context). In this regard, texts are the result of the writers assumptions about the audience, as well as the audiences contributions within a particular communicative context (usually in the form of previous knowledge, attitudes or beliefs). The translator must work to recreate the text for a new target audience, which is often different from the audience envisioned by the author of the original text, and for a new communicative context which is sometimes in conflict with that of the original (Colina 2003). Simply put, translated texts are not independent or objective entities. The same text can be translated in different ways depending on the communicative context of the target text or the assumptions made about the audience. In evaluating the quality of a translated text, raters are instructed to not only look at the linguistic sub-units, but also to how well the text achieves its overall communicative purpose. In other words, does an educational brochure work as such; does an ad sound like an ad and function like one? Finally, translation errors are not considered isolated linguistic or grammatical errors (e.g., agreement, conjugations, use of tense and mood with verbs, lexical choices) but instead should be evaluated based on the overall effect they have on the text and its communicative function. descriptive Statements for Classification Rather than using a system where points are assessed for each error, somewhat arbitrarily, the TQA Tool uses descriptive statements to classify texts into one of four assessment categories. Each descriptor defines the degree of compliance with specific criteria. Raters evaluate text by matching their assessment to a descriptive statement for each evaluation category. The goal is to choose the statement that best describes the text in each of area of evaluation. Descriptors reflect a four-point range, from unacceptable to ideal, for each component evaluated.

In evaluating the quality of a translated text, raters are instructed to not only look at the linguistic sub-units, but also to how well the text achieves its overall communicative purpose.

Requesters need an efficient means of scrutinizing the quality of health care translations and assessment methods that go beyond a simple good or bad label.

Using the translation QUality assessment tool a training manUal for reQUesters

Tool

6
RATING INSTRUCTIONS:
be based on these instructions Your decisions and evaluation should for the review of the translated text. 1. Carefully read the instructions categories. fits the text given in each one of the Language and Functional categories. 2. Check the description that best the English and score the Target the target text without looking at selection. 3. It is recommended that you read or to provide rationale for your descriptor they can be useful to make decisions or comments are not required, but 4. Examples only.

Target Language The text is: 1) extremely difficult to read (incomprehensible); 2) hard to comprehend, (includes elements/structure from the source text); 3) generally readable but with awkward expressions; or 4) reads similar to texts originally written in the target language. Textual and Functional Adequacy The text: 1) does not consider the goals, purpose, function and audience (cannot be repaired); 2) gives some consideration to the intended purpose and audience (repair will require considerable effort); 3) approximates goals, purpose, function and needs of the intended audience (can be repaired with edits); or 4) accurately accomplishes the goals, purpose, and function of the original (minor or no edits needed). Non-Specialized Content The text: 1) contains important unwarranted deviations from the original (very defective); 2) exhibits some misunderstanding of the original and/or the translation instruction; 3) minor alterations in meaning, additions or omissions; or 4) accurately reflects the content contained in the original.
Check one box

TQA Tool Rating Section

1: Target Language
Category Number Description Check one box

1.a

1.b 1.c

1.d

sible. The translation reveals serious bordering on being incomprehen mistakes. The translation is The text is extremely difficult to read, al use of the target language, spelling language proficiency issues. Ungrammatic The structure of source language (neither the source nor the target). written in some sort of mixed language language text. The amount of transfer be considered a sample of target dominates to the extent that it cannot translation. be justified by the purpose of the from the source cannot the transfer of elements/structure from text contains some unnecessary The text is hard to comprehend. The translation and affects its readability. source language shows up in the source text. The structure of the most awkward expressions resulting, in readable, there are problems and Although the target text is generally from the source text. cases, from unnecessary transfer language that respond to the to texts originally written in the target The translated text reads similarly in the brief. Problems/awkward type as those specified for the translation same purpose, audience and text at all. expressions are minimal if existent

Examples/Comments

2: Functional and Textual Adequacy

Category Number Description

2.a

2.b

2.c

2.d

The text was translated without considering function and audience of the text. etc.). Can not be repaired with Disregard for the goals, purpose, need of the audience, (cultural, linguistic, textual units, textual purpose, genre, revisions. for the translation, but the intended purpose and audience to The translated text gives some consideration of formality, some aspect of its function, needs of the audience, aspect/s of it (e.g. level misses some important requires effort. cultural considerations, etc.). Repair audience, but it is (function) and needs of the intended to the goals, purpose with The translated text approximates for the translation. Can be repaired the restrictions and instructions not as efficient as it could be, given suggested edits. persuasive) purpose (function: informative, expressive, s the goals, The translated text accurately accomplishe It also attends to cultural needs and audience (including level of formality). set for the translation and intended or no edits needed. characteristics of the audience. Minor

Examples/Comments

Specialized Content The text: 1) reveals an unawareness/ignorance of special terminology and/or insufficient knowledge of specialized content; 2) contains serious/ frequent mistakes involving terminology and/or specialized content; 3) has a few terminological errors, but the specialized content is not seriously affected; or 4) is accurate and appropriate. During the evaluation process, the translation is examined four times. Each time, a rater looks to match the text to the best description in each of the four categories. However, raters are instructed to first assess the translation for Target Language and Textual and Functional Adequacy without looking at the English original text. This enables raters to capture a first impression, as the target reader is likely to gain from reading the translation without reference to the source text. Raters are then asked to read the source text, contrasting the translation with the source text and then with their first impressions as a target reader. This step is vital as a measure of how well a translation anticipates and honors the communicative context of intended target audiences. Consequently, raters may need to adjust their first rating before continuing the assessment for the final two categories Non-Specialized Content and Specialized Content. With each rating component, raters emphasize a different aspect of translation quality during their assessment. Considering all four aspects raters are then ready to make a recommendation about the translation.

RATING INSTRUCTIONS:
5. Now read the English original

3: Non-Specialized Content
Category Number

(Meaning)
Description Check one box

3.a

3.b 3.c 3.d

from the original. It contains inaccurate important unwarranted deviations The translation reflects or contains by the instructions. Very defective and additions that cannot be justified renditions and/or important omissions comprehension of the original text. cannot be justified by the omissions or/and additions that have been some changes in meaning, There original and/or translation instructions. shows some misunderstanding of translation instructions. Translation Minor alterations in meaning, additions or omissions. insofar as it is required by the the content contained in the original, The translation accurately reflects Slight nuances and shades of meaning alterations, omissions or additions. instructions without unwarranted have been rendered adequately.

Examples/Comments

4: Specialized Content and Terminolog

Category Number

y
Description Check one box

4.a 4.b 4.c 4.d

Reveals unawareness /ignorance

of special terminology and/or insufficient

knowledge of specialized content.

Serious/frequent mistakes involving A few terminological errors, but the Accurate and appropriate rendition to the subject.

terminology and/or specialized content. affected. command of terms and content specific

specialized content is not seriously

of the terminology. It reflects a good

Examples/Comments

Using the translation QUality assessment tool a training manUal for reQUesters

Tool

Section 2. Concept of Quality Translations in health Care

Understanding what constitutes quality in a health care text that has been translated from one language to another was the first area of research while developing the TQA Tool. Through an extensive review of translation assessment methods, it became clear that most researchers and practitioners familiar with the literature on translation evaluation agree that there is a lack of assessment standards. The various methods for assessing translation quality fall into three basic categories: Procedural quality assurance systems Experience-based or anecdotal approaches Theoretical and research-based approaches Procedural quality assurance systems are those developed primarily by the translation industry and organizations recognized as the standard bearers in the field (e.g., ASTM International, ISO or International Organizations for Standardization), as well as those developed by private translation production firms. Experience-based or anecdotal approaches consist of point systems or marking and ad hoc scales (where points are deducted for major or minor errors) developed to measure quality for a specific professional organization or industry. Among these are the American Translators Association certification exams, used to assess translator competencies, and the SAE J2450 Translation Quality Metric system, which is used by the automotive industry. Theoretical and research-based approaches tend to focus on the user of a translation, evaluating reader response of the text, and closely examining the micro-level aspects of language. Each approach emphasizes an important aspect of quality and all enjoy some degree of success and applicability. Overarching is the premise that how a text is organized, down to the words chosen, is influenced by how that language achieves successful communication. However, words on a page, the more obvious linguistic or micro-level features of language, are only part of the picture. Assessing the quality of written language also requires taking into consideration the wider panorama of the communicative purpose of a text, the content, context and the intended audience. Written materials can be a valuable communication tool to help consumers take a more active role in their health care, but only if the patients are able to read and understand the information. With this in mind, translation quality needs to emphasize the relevance of a translation to the user. The TQA Tool is designed to measure quality by how well a translation meets the communicative purpose of the original text while anticipating and honoring the needs of the intended target language user (e.g., LEP patients). This definition of quality emphasizes the significance of creating translation products that accommodate the intended user while remaining loyal to the original meaning of the source text.

Definition of quality: Anticipate and honor the need of an intending user Source: Six Sigma Website

Using the translation QUality assessment tool a training manUal for reQUesters

Tool

Has the tool been tested?

Several rounds of testing in three languages Spanish, Chinese and Russian and involving 52 raters (translators, language professionals, and bilinguals) were conducted with the TQA Tool using publicly-available health care materials (e.g., CDC, Medline Plus En Espaol etc.). TQA Tool testing consisted of a pilot testing phase and a two-phase inter-rater reliability testing process. The pilot testing which included 22 raters and 9 texts is described elsewhere (Colina 2008). Language consultants, highly-skilled translators in each of the three languages participated in the pilot testing session and helped identify texts for a two-phase inter-rater reliability testing phase. The inter-rater reliability testing process included 30 raters, 3 consultants and 13 texts. Each round of testing was followed by a debriefing session with raters that included a structured survey about the rating experience. Quantitative data obtained from ratings using the tool and post-rating surveys were subjected to statistical analysis to determine inter-rater reliability. Qualitative data collected from raters in the post-rating surveys were also analyzed. As shown in the table below, the inter-rater reliability coefficients among the Spanish and Chinese raters respectively, are quite good and statistically significant. Benchmark testing included 12 raters: three Russian, three Chinese, and six Spanish. In the reliability testing session, 21 raters were included: six Russian, seven Chinese, and eight Spanish. As shown, the results of reliability testing for the Chinese raters achieved an acceptable inter-rater reliability coefficient although lower than in the Benchmark testing. Inconsistent and wide variability in ratings among the Russian raters, compared to the Spanish and the Chinese raters, led us to believe other unknown factors unrelated to the tool were present. Further testing for Russian raters and other languages is needed. Inter-rater reliability: Benchmark and Reliability Testing Results
Reliability Coefficient Benchmark Testing (12 Raters) Spanish Chinese Russian .953 .973 .128 Reliability Testing (21 Raters) .934 .780 .118

Several rounds of testing in three languages Spanish, Chinese and Russian and involving 52 raters (translators, language professionals, and bilinguals) were conducted with the TQA Tool using publicly-available health care materials (e.g., CDC, Medline Plus En Espaol etc.).

Using the translation QUality assessment tool a training manUal for reQUesters

Requesters begin a translation quality assessment by completing Part I of the TQA Tool Cover Page.
TranslaTion QualiTy assessmenT Tool
For HealTH educaTion m aTerials

PART I: To be completed by Requester

The health care decision maker requesting a quality assessment of an existing translated text.
Requester (Name) Title/Department Phone Delivery Date

TRANSLATION BRIEF
Source Language Target Language

English
Text Type Text Title Target Audience Purpose of Document

Health Education Material

Health Education Material

QUALITY CRITERIA
Priority Focus Area

2 Target Language 1 Functional and Textual Adequacy

Rank EACH from 1 to 4 (1 being top priority)

3 Non-Specialized Content (Meaning) 4 Specialized Content and Terminology

PART II: To be completed by TQA Rater

Rater (Name) Phone Total Score Date Completed Date Received Total Rating Time

ASSESSMENT SUMMARY AND RECOMMENDATION

q Publish and/or use as is q Minor edits needed before publishing*
To be completed after evaluating translated text

q Major revision needed before publishing* q Redo translation q Translation will not be an effective communication strategy for this text. Explore other options (e.g. create new target language materials)

Notes/Recommended Edits

Tool

Section 3. how to Complete Part i of the TQA Tool

The TQA Tool is used either after a translation project is complete (as part of the acceptance process) or to determine the quality of an existing target language text. Requesters begin a translation quality assessment by completing Part I of the TQA Tool Cover Page. This part of the Cover Page is used to formalize a request, set a delivery date and orient raters to the text to be evaluated. It is also used to prioritize each quality criteria.

Step 1: Provide contact information and establish a deadline for the quality review
Contact Information and Deadline
Requester (Name) Title/department Phone delivery date

Step 2: Complete the basic elements of a translation brief

The best translation evaluations are those where the rater has a detailed understanding of both the environment and communicative context (uses and purposes of a text) for which the source and target texts are used. Tool #3 in the More Than Words Toolkit Series introduces requesters to the practice of preparing a translation brief to accompany a translation request. The translation brief is a set of instructions that define the purpose and use of a source text and establish priorities to guide the translators in their preparation of the requested translation. Using Part I of the TQA Cover Page, requesters provide a translation brief orienting the rater to the text to be rated. This includes a brief description of the source text and those intended for the target text. As show below the translation brief for the TQA evaluation includes basic information about the source and target text, their respective purpose and use, and communicative context. Translation Brief Section TRANSlATioN BRieF
Source language Target language

English
Text Type Text Title Target Audience Purpose of document

Health Education Material

Health Education Material

Using the translation QUality assessment tool a training manUal for reQUesters

Tool

Source-Target language. Fill in the source language for the original text (usually English) and target language for the translated text Text Type. Describe the text type (brochure, checklist, consent form) Text Title. Write the title of each text Target Audience: Write a description of the source and target audience (the idealized readers with regard to background and assumed interest). For example, nationwide or regional audience, with general characteristics or specific attributes (e.g., gender, age, occupation). Purpose of Document. Write the respective communicative purposes for each text (e.g., to inform, educate, collect information, fulfill a legal requirement).

Step 3: establish quality criteria

The Quality Criteria section of Part I helps establish the requesters priority for the evaluation. Each component of the tool represents an important aspect of quality. Considering what each of the four assessment areas represents enables the requester to determine whether one focus area should be of greater priority than the others. During the development phase of the TQA tool, a priority ranking was established for health education materials. This priority is noted in the table below. Quality Criteria Ranking for Health Education Materials QuAliTY CRiTeRiA
Priority Focus Area

2 Target Language 1 Functional and Textual Adequacy

Rank EACH from 1 to 4 (1 being top priority)

3 Non-Specialized Content (Meaning) 4 Specialized Content and Terminology

By permitting the most relevant components to be given a higher priority, the TQA Tool provides flexibility in establishing the quality criteria of each text to be evaluated. For example, Specialized Content and Terminology could be given a higher priority for texts with highly-specialized content whereas Functional and Textual Adequacy might take priority for health materials developed for general audiences. The tool can also be used without assigning priority rankings (i.e., all components could be given equal weight). Raters with advanced language education and translation experience can also offer recommendations for ranking the criteria for a specific text. With flexibility to set quality criteria, requesters and raters can use the tool in unique settings. Note, however, that additional research is needed to determine empirically the importance of each quality criterion for different text types. These three easy steps initiate a translation quality assessment. When a rating is completed, raters summarize their assessment and make recommendations on Part II of the Cover Page.
Using the translation QUality assessment tool a training manUal for reQUesters

Tool

Part II Rating Results

Assessment Summary and Recommendation to be completed by TQA Raters Part II of the TQA Tool Cover Page is completed by raters, giving requesters a summary of the assessment with recommendations that can range from publish the text as is to the translation is beyond repair. For translations that do not meet the highest quality standard, but are salvageable, raters can recommend edits. Raters can also note whether, in their professional judgment, discrepancies exist between the source and target audiences too great to overcome with a translation. They should also provide a summary of their observations, with examples of problems found and the type of edits they recommended. This can include notes made during their rating and/or explanation of their evaluation. TQA Tool Cover Page Part ii
Rater (Name) Phone Total Score date Completed date Received Total Rating Time

Part II of the TQA Tool Cover Page is completed by raters, to provide requesters a summary of the assessment with recommendations that can range from publish the text as is to the translation is beyond repair.

ASSeSSMeNT SuMMARY ANd ReCoMMeNdATioN q Publish and/or use as is q Minor edits needed before publishing*
To be completed after evaluating translated text

q Major revision needed before publishing* q Redo translation q Translation will not be an effective communication strategy for this text. Explore other options (e.g. create new target language materials)

Notes/ Recommended edits

Section 4. A TQA Rater

Professionals or academics with advanced knowledge of the target language and experience in translation can learn to apply the TQA Tool by reading the training manual. The concepts and demarcations of the descriptors used in the TQA Tool are sufficient for raters to differentiate high-quality and low-quality translations. Some variability is expected in the ratings of mid-range quality translations. Raters should be able to provide examples of minor or major errors as well as describe their findings in sufficient detail to help requesters understand the rating results and the types of deficiencies found.
Using the translation QUality assessment tool a training manUal for reQUesters

10

Tool

Online training sessions are also available. These sessions review the underpinning of the tool, its design and how it is used to evaluate the quality of translated materials. All participants receive a training manual. A competency exam is also available to determine whether prospective raters understand and can apply the instructions as intended. Candidates successfully completing the training and online exam receive a certificate of completion.

Rater Profiles
Raters participating in the TQA Tool testing sessions were required to have a language proficiency equivalent to that of a native or near-native speaker in English and their second language (in this case, Spanish, Chinese or Russian). Raters were placed into one of three groups depending on their profile. Professional translators: Language professionals whose income is derived primarily from providing translation services. These raters had significant professional experience (five years minimum, most 12-20 years), membership in professional translation organizations and formal education in translation or related field. Although only two raters were ATA-certified, almost all were ATA-affiliates, if not members. language teachers: Individuals, including graduate students, whose main occupation is teaching foreign language courses at a university or other educational institution. These raters had some translation experience but did not rely on translation as their primary source of income. Bilinguals: Individuals who, while native or near-native, did not have significant formal learning or writing proficiency in their target language (as a translator, teacher or otherwise). Community interpreters (e.g., medical interpreters, legal assistants) with no translation experience were also included in this category. The most proficient and consistent raters were those classified as professional translators, followed by the language teachers, although translation experience strongly correlated with rating consistency. Bilinguals were least able to apply the rating tool consistently. Requesters are encouraged to use raters who have taken the TQA Tool online training and qualify as either a professional translator or language teacher with substantive translation experience.

how much does a TQA rating cost?

Payment for rating a translation is likely to vary based on word count and type of text. In each of the testing sessions, the amount of time raters took to assess a text was collected. Based on these data, the time taken to assess a text seemed to vary by rater. The hourly fees translators and language professionals charge for similar work also varies. Given these variables, requesters should expect to pay $35-50 for a text that is two to three pages in length, while longer texts will likely incur higher rates. Requests for corrections should be directed back to the original translator or negotiated separately with the rater.

Using the translation QUality assessment tool a training manUal for reQUesters

11

Tool

This Guide was Produced by hablamos Juntos Since 2001, Hablamos Juntos (We Speak Together), a Robert Wood Johnson Foundation-funded national initiative, has been studying language barriers in health care for patients who speak or understand little or no English. In our years of work, the fundamental lesson we have learned is that communicating across languages and cultures involves more than words. It requires recognition that the meaning of ideas and words is conveyed and extracted through the cultural lens of the interlocutors and that these can be vastly different in cross-cultural communication. Attention to these differences is essential in effective communication, whether in writing through translation or spoken through interpretation. The Hablamos Juntos More Than Words Toolkit Series brings together lessons learned from demonstration projects - eight years of working with nationallyrecognized health care leaders and language experts - and from original research on translation quality. It has been made possible through the contributions of many around the country, including language faculty, researchers, practicing interpreters and translators, and health care professionals dedicated to providing safe and quality health care to our diverse nation. Among those requiring special mention for the production of Tool #6 are Sonia Colina, Yi Yuan, Aracely Rosales, Gauhar Nguyen and all of the TQA Raters. For more information about Hablamos Juntos or to download the entire More Than Words Toolkit Series, visit www.HablamosJuntos.org. The More Than Words Toolkit Series is copyright protected and may not be modified or changed without expressed written consent of Hablamos Juntos. Tools may be used without permission as long as Hablamos Juntos is recognized as the source.

ReFeReNCeS
Colina, Sonia (2008). Translation quality evaluation; empirical evidence for a functionalist approach. In The Translator St Jerome Publishing Ltd (April) 14:1 (pp97-134) iSixSigma.com Website (2008). Quality Dictionary and Glossary. Accessed 2/12/2008 8:48 PM http://healthcare.isixsigma.com/dictionary/Definition_of_Quality-713.htm

Copyright 2009

Using the translation QUality assessment tool a training manUal for reQUesters

12

TranslaTion QualiTy assessmenT Tool

For HealTH educaTion m aTerials
PART I: To be completed by Requester
The health care decision maker requesting a quality assessment of an existing translated text.
Requester (Name) Title/Department Phone Delivery Date

TRANSLATION BRIEF
Source Language Target Language

English
Text Type Text Title Target Audience Purpose of Document

Health Education Material

Health Education Material

QUALITY CRITERIA
Priority Focus Area

2 Target Language 1 Functional and Textual Adequacy

Rank EACH from 1 to 4 (1 being top priority)

3 Non-Specialized Content (Meaning) 4 Specialized Content and Terminology

PART II: To be completed by TQA Rater

Rater (Name) Phone Total Score Date Completed Date Received Total Rating Time

ASSESSMENT SUMMARY AND RECOMMENDATION

q Publish and/or use as is q Minor edits needed before publishing*
To be completed after evaluating translated text

q Major revision needed before publishing* q Redo translation q Translation will not be an effective communication strategy for this text. Explore other options (e.g. create new target language materials)

Notes/Recommended Edits

Tally Sheet
Component Category Rating Score Value

Target Language Functional and Textual Adequacy Non-Specialized Content Specialized Content and Terminology Total Score

Comments

RATING INSTRUCTIONS:
1. Carefully read the instructions for the review of the translated text. Your decisions and evaluation should be based on these instructions only. 2. Check the description that best fits the text given in each one of the categories. 3. It is recommended that you read the target text without looking at the English and score the Target Language and Functional categories. 4. Examples or comments are not required, but they can be useful to make decisions or to provide rationale for your descriptor selection.

1: Target Language
Category Number Check one box

Description

1.a

The text is extremely difficult to read, bordering on being incomprehensible. The translation reveals serious language proficiency issues. Ungrammatical use of the target language, spelling mistakes. The translation is written in some sort of mixed language (neither the source nor the target). The structure of source language dominates to the extent that it cannot be considered a sample of target language text. The amount of transfer from the source cannot be justified by the purpose of the translation. The text is hard to comprehend. The text contains some unnecessary transfer of elements/structure from the source text. The structure of the source language shows up in the translation and affects its readability. Although the target text is generally readable, there are problems and awkward expressions resulting, in most cases, from unnecessary transfer from the source text. The translated text reads similarly to texts originally written in the target language that respond to the same purpose, audience and text type as those specified for the translation in the brief. Problems/awkward expressions are minimal if existent at all.

1.b 1.c

1.d

Examples/Comments

2: Functional and Textual Adequacy

Category Number Check one box

Description

2.a

Disregard for the goals, purpose, function and audience of the text. The text was translated without considering textual units, textual purpose, genre, need of the audience, (cultural, linguistic, etc.). Can not be repaired with revisions. The translated text gives some consideration to the intended purpose and audience for the translation, but misses some important aspect/s of it (e.g. level of formality, some aspect of its function, needs of the audience, cultural considerations, etc.). Repair requires effort. The translated text approximates to the goals, purpose (function) and needs of the intended audience, but it is not as efficient as it could be, given the restrictions and instructions for the translation. Can be repaired with suggested edits. The translated text accurately accomplishes the goals, purpose (function: informative, expressive, persuasive) set for the translation and intended audience (including level of formality). It also attends to cultural needs and characteristics of the audience. Minor or no edits needed.

2.b

2.c

2.d

Examples/Comments

RATING INSTRUCTIONS:
5. Now read the English original

3: Non-Specialized Content (Meaning)

Category Number Check one box

Description

3.a

The translation reflects or contains important unwarranted deviations from the original. It contains inaccurate renditions and/or important omissions and additions that cannot be justified by the instructions. Very defective comprehension of the original text. There have been some changes in meaning, omissions or/and additions that cannot be justified by the translation instructions. Translation shows some misunderstanding of original and/or translation instructions. Minor alterations in meaning, additions or omissions. The translation accurately reflects the content contained in the original, insofar as it is required by the instructions without unwarranted alterations, omissions or additions. Slight nuances and shades of meaning have been rendered adequately.

3.b 3.c 3.d

Examples/Comments

4: Specialized Content and Terminology

Category Number Check one box

Description

4.a 4.b 4.c 4.d

Reveals unawareness/ignorance of special terminology and/or insufficient knowledge of specialized content. Serious/frequent mistakes involving terminology and/or specialized content. A few terminological errors, but the specialized content is not seriously affected. Accurate and appropriate rendition of the terminology. It reflects a good command of terms and content specific to the subject.

Examples/Comments