International Language Testing Washback – standing the monster on its head

International Language Testing Washback – standing the monster on its head Thor May Brisbane 2013 I nt ernat ional language t est ing, and part icularly English language t est ing, is a m ult i- billion dollar business. For language st udent s, bot h em ploym ent chances and int ernat ional working or st udy visas are m ediat ed by such t est s. The t est s engage t he ingenuit y of m illions of st udent s in a st ruggle t o m ast er or out wit purpose designed language obst acle courses. I n t his life t ransit ion inst it ut ional language learners are assist ed by an arm y of specialist s who m ight or m ight not be skilled in t he process, rat her as corporat ions are assist ed by arm ies of lawyers and paralegals, or hospit als assist ed by an ent ourage of m edical professionals. At t he t op of t he assessm ent pyram id are m ult inat ional t est ing corporat ions, best known by t he nam es of t heir st andardized t est s, such as I ELTS, TOEI C, TOEFL, BULATS, TKT, Cam bridge ESOL m ain suit e, or G- TELP ( t here are m any ot her aspirant s) . I n som e ways t hese t est ing com panies can be t hought of as t he Big Phar m a corporat ions ( i.e. dr ug com panies) of t he educat ional world. Like Big Pharm a t hey are subj ect t o const ant challenges t o t heir et hics and reliabilit y from wit hin and w it hout , and like Big Pharm a t hey are rat her prone t o corrupt t he issues which t hey were designed t o assist w it h. I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 The possible corrupt ion of language learning by t he requirem ent s of t est ing is known as wash- back. Wash- back is not always m alignant . Language learning is t ypically a long, com plex and discour aging experience, especially in m ass educat ional inst it ut ions. The clear obj ect ive of a high- st akes exam at t he end of st udy can be powerfully m ot ivat ing. The explicit requirem ent s of ext ernal language t est s can organize t he m ult it udes of t eachers and inst it ut ions w ho have only a t enuous gr asp of product ive language t eaching processes. Those count ries com m it t ed t o t he m ass- skilling of t heir populat ions in foreign languages have also becom e aware t hat hom e- gr own diplom as of language achievem ent oft en convey lit t le reliable inform at ion about t he perform at ive abilit y of t heir graduat es in foreign languages. I nt ernat ional language t est s claim t o overcom e t his handicap. However, experience has repeat edly shown t hat t he int ernat ional t est s are also elusive guar ant ors. For exam ple, a level 6.5 score in I ELTS m ay be a poor predict or of success for st udent s vent uring int o English language t ert iary inst it ut ions overseas. Bot h t he learned m et hodologies for m ast ering such t est s, and t he act ual language skills w hich t hey t each m ay scarcely prepare t he new graduat e for real life success in a foreign language environm ent . There are oft en good argum ent s t hat t est wash- back int o schools has act ively inhibit ed t he acquisit ion of real language abilit y. The dilem m as of learning and t est ing j ust out lined have no easy solut ions. They engage difficult t o cont rol collisions of polit ics, profit s, aspirat ions and realized skills. The m ixt ure is differ ent in every count ry, it varies wit h social st rat a, and it varies wit h t he cult ure and st ruct ur e of every local college. That is, t here is unlik ely t o be any sim ple fix for wash- back in m ass educat ional inst it ut ions. 2 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 However t here can be st rat agem s t o bend t he process in local ways for m axim um language learning benefit . The analysis w hich follows is a t ent at ive at t em pt t o m anipulat e t he wash- back from an int ernat ional t est in a m anner which act ually assist s genuine language acquisit ion. The t est chosen for m anipulat ion is G- TELP, which is less w ell known t han m aj or players like I ELTS. Nevert heless t he discussion and principles engaged can apply anywhere. The present analysis is in fact an assem bly of four Mast ers level degree assignm ent s in a single sequence. This work was com plet ed in 2005 at t he Universit y of Newcast le, NSW. I t hought it would be useful t o collect t he assignm ent s here as a spur t o creat ive t hinking on a very im port ant t opic ( not as a crib for lazy degree candidat es! ) . I have ret ained t he original cover pages wit h t he assignm ent quest ions since t his seem s a useful device t o provoke a response from readers t hem selves. The opinions expressed are of course m y own, and are open t o fault or am endm ent . ______________________ 3 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Part I - Test Specification - Goal Evaluation From A Proficiency Test TABLE OF CONTENTS Statement of the Problem........................................................................... 11 Problems common to large scale standardized tests..............................13 Test Content............................................................................................... 16 Grammar.................................................................................................16 Listening .................................................................................................17 Speaking.................................................................................................18 Reading & Vocabulary ............................................................................19 Writing.....................................................................................................20 Types of text............................................................................................... 22 Listening .................................................................................................22 Speaking.................................................................................................22 Reading ..................................................................................................22 Writing.....................................................................................................22 Addressees of texts ................................................................................23 Lengths of texts ......................................................................................23 Topics .....................................................................................................23 Readability ..............................................................................................23 Structural Range.....................................................................................23 Vocabulary Range ..................................................................................24 Dialect and style .....................................................................................24 Speed of Processing...............................................................................24 Task Procedure, Timing and Medium......................................................... 24 Task Procedure ......................................................................................24 Number of Items .....................................................................................25 Timing .....................................................................................................26 Medium ...................................................................................................26 Organization ...........................................................................................26 Techniques.................................................................................................27 4 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Section 1 Grammar.................................................................................27 Section 2 Listening .................................................................................27 Section 3 Speaking.................................................................................28 Section 4 Reading ..................................................................................28 Section 5 Writing.....................................................................................28 Criterial levels of performance and Marking Scales ...................................28 Scoring Context ......................................................................................28 Scoring for the diagnostic purposes of the current test cycle ..................... 31 Grammar.................................................................................................31 Listening .................................................................................................31 Speaking.................................................................................................31 Reading ..................................................................................................32 Writing.....................................................................................................33 Test Construction Procedures....................................................................33 Sampling.................................................................................................33 Item writing and Moderation....................................................................34 Informal Trailling .....................................................................................34 Trialling and Analysis ..............................................................................34 Validation ................................................................................................35 Speaking - concurrent validity.................................................................35 Candidate Instructions................................................................................ 35 Teacher’s Handbook .................................................................................. 36 Rationale:................................................................................................36 Development and Validation: ..................................................................37 Description of the test: ............................................................................37 Sample items: .........................................................................................38 Section 1 Grammar .................................................................................... 38 Section 2 Listening ..................................................................................... 38 Section 3 Speaking .................................................................................... 39 Section 4 Reading ...................................................................................... 39 Section 5 Writing ........................................................................................ 39 Advice on preparing for the test ..............................................................39 Interpretation of test scores ....................................................................40 Training Materials: ..................................................................................40 5 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Test administration: ................................................................................41 Handbook for Candidates........................................................................... 41 How to prepare for the test .....................................................................42 What do your test scores mean? ............................................................42 What is in the test ? ................................................................................43 Sample test items: .................................................................................44 Section 1 Grammar .................................................................................... 44 Section 2 Listening ..................................................................................... 44 Section 3 Speaking .................................................................................... 44 Section 4 Reading ...................................................................................... 45 Section 5 Writing ........................................................................................ 45 APPENDIX 1 G-TELP Speaking Score Criteria ........................................45 G-TELP Proficiency Levels for Speaking ................................................46 APPENDIX 2 G-TELP Writing Score Criteria ............................................ 48 APPENDIX 3 G-TELP CONTENT SPECIFICATIONS FOR LEVEL 3......50 - Grammar ..............................................................................................50 - Listening ...............................................................................................51 - Reading and Vocabulary ......................................................................51 - Writing ..................................................................................................52 => Task Contents ...................................................................................52 - Speaking...............................................................................................53 Test Specification References..................................................................136 6 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Part II Test Construction TABLE OF CONTENTS Assigment topic & comments: Writing Test Items for listening, reading, writing, speaking, grammar / vocabulary 56 Listening Test............................................................................................. 58 Reading Test.............................................................................................. 62 Writing Test ................................................................................................ 67 Speaking Test ............................................................................................ 73 Grammar Test ............................................................................................ 76 Appendix 1 Listening Skills........................................................................83 Appendix 2 - Reading Test.........................................................................84 Appendix 3 - Speaking and Writing Tests - Picture Panel Cues for text construction ................................................................................................86 Appendix 4 - G-TELP Specifications for Levels 3 to 5, Grammar Tests .....87 Part III - PRACTICE ENGLISH LANGUAGE TEST Le ve l : I nt e r m e dia t e (based on the G-TELP international language test) TABLE OF CONTENTS Part I I I - PRACTI CE ENGLI SH LANGUAGE TEST ............................ 91 Test Instructions ....................................................................................... 91 1. Speaking Test ........................................................................................ 92 Instructions ............................................................................................... 92 Hints For Talking About The Pictures ....................................................... 92 Picture Panels Showing The History of Smallpox ..................................... 93 2. LISTENING TEST .................................................................................. 93 General Instructions.................................................................................. 93 2. LISTENING TEST - SECTION 1 ........................................................... 94 Recording : Telephone Duties for a Secretary .......................................... 94 7 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Part A : Multiple Choice Questions .......................................................... 94 Listening - Section 1, Part B - Dialogue Choice .........................................95 LISTENING TEST - SECTION B............................................................... 96 Recording : “Haven’t We Met Before?” ..................................................... 96 Listening Part A - Dialogue Completion .................................................... 96 Listening - Section 2, Part B : Multiple Choice Questions ........................ 96 3. Reading Test.......................................................................................... 97 General I nst ruct ions ................................................................. 97 Reading Section 1 - A Tourist Brochure ................................................... 98 Reading Section 1 - Multiple Choice Questions (Ideas ............................ 99 Reading Section 1 - Multiple Choice Questions (Vocabulary)................ 100 Reading Section 2 - An Interview.......................................................... 101 Reading Section 2 - Multiple Choice Questions (Ideas)......................... 102 Reading Section 2 - Multiple Choice Questions (Vocabulary)................ 102 4. Writing Test .......................................................................................... 103 Hints For Writing A Business E-mail ....................................................... 103 5. Grammar Test ...................................................................................... 105 Grammar - Section A (Multiple Choice Questions) ................................. 105 Grammar - Section B (Phrase Completion) ............................................ 108 Grammar - Section C (Sentence Correction) .......................................... 109 Pa r t I V – Pr a ct ica l Aspe ct s of La ngua ge Te st Const r uct ion TABLE OF CONTENTS 1. The notion of testing, and practical consequences .................... 112 2. Test Construction.......................................................................113 4. The Purpose of the Practice Test ..............................................115 5. The Application of the Practice Test applied to its purpose .......116 6. The Content of the Test ............................................................. 116 7. Test Specifications.....................................................................117 8. Instructions to Candidates ......................................................... 118 9. The Quality of Individual Test Items...........................................119 10. The Representativeness of the Sample...................................119 11. Reliability and Validity.............................................................. 120 12. Scoring Procedures .................................................................120 8 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 13. Aspects of Moderation ............................................................. 121 14. Summary Evaluation of the Test..............................................121 Appendix 1 : Error Summary of Kim Joon-woo’s Practice Test 123 1. Listening Test ........................................................................123 2. Reading Test .........................................................................123 Tester comment .........................................................................123 3. Grammar Test .......................................................................124 Tester comment : .......................................................................126 4. Writing Test............................................................................127 Tester comments : .....................................................................128 a) Formal grammatical errors : ................................................... 129 b) Register (the language of social marking).............................. 129 i) The erratic use of register ....................................................... 129 ii) Terms of address ..................................................................130 iii) Discourse etiquette............................................................... 130 c) Information Content................................................................ 131 5. Speaking Test .......................................................................132 Testee transcript ........................................................................132 Tester comment .........................................................................133 Dialect ........................................................................................ 133 Fluency ...................................................................................... 133 Formal grammatical errors ......................................................... 134 Lexical choice ............................................................................134 Appendix 2 Post-test e-mail from the Candidate ........................ 135 Appendix 3 - References .................................................................. 137 9 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Unive r sit y of N e w ca st le , N SW Assignm ent 1 : Language Test ing And Evaluat ion Topic : Writ e a set of t est specificat ions for a com prehensive ESL proficiency OR achievem ent t est Provide : 1) Test Specificat ions 2) I nst ruct ions t o Candidat es 3) Crit erial Levels of Perform ance Not e on Topic adapt at ion : The set t opic has been adapt ed in a nonconvent ional way. A proficiency t est widely used in Sout h Korea has been used as a m odel for init iat ing a t eacher - assist ed, st udent process of self- diagnosis and planning. The open purpose of t his t est fram ework developed here is t o provide English Maj or st udent s in a Korean universit y wit h inform at ion about t heir own skills relat ive t o an ext ernal pr oficiency t est com m only used by large em ployers in Sout h Korea. The t est design in t his proj ect is t herefore t hat of a proficiency t est , but it s funct ion is diagnost ic and ult im at ely m ot ivat ional. The underlying t eacher- purpose of t his t est is t o harness t he backwash of ext ernal t est m ot ivat ion by addressing it direct ly, giving st udent s clear inst rum ent al goals relat ive t o t heir own st rengt hs and weaknesses, t hen build on ( subvert ?) t his foundat ion int o genuine com m unicat ive language abilit ies. ________________________ 10 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Test Specification - Goal Evaluation From A Proficiency Test Statement of the Problem The open purpose of t his t est is t o provide English Maj or st udent s in a Korean universit y wit h inform at ion about t heir own skills relat ive t o an ext ernal proficiency t est com m only used by large em ployers in Sout h Korea. The t est design in t his proj ect is t herefore t hat of a proficiency t est , but it s funct ion is diagnost ic and ult im at ely m ot ivat ional. Because t his t est is expect ed t o be int egral in a t eaching program , t he sam ple present ed here will only be t he first of a series in a cycle, wit h each successive cycle increasing in com plexit y and lengt h. Each com plet ed t est cycle w ill be subj ect t o det ailed post - m ort em and crit ique by bot h t he st udent s and t he t eacher. The ext ernal t est in quest ion is t he Am erican G- TELP . The G- TELP claim s t o have a funct ional, t ask - orient ed approach and t o be crit erion referenced. Hence it s at t ract ion t o em ployers. I n fact t he G- TELP, like m ost of t he m aj or int ernat ional language t est s, has som e quit e st riking cult ural biases and som et im es dubious linguist ic cont ent , at least from t he viewpoint of a Korean learner. Nevert heless, m ost English learning in Sout h Korea is inst rum ent ally driven, and t he pressure above all is t o excel in t est s like G- TELP which will lead t o em ploym ent ( regardless of act ual L2 fluency) . Not e t hat t he real usage of English in Korean business frequent ly involves regional variet ies of English rat her t han Am erican English. Accept ing such variat ion is part icularly im port ant in spoken language, alt hough G- TELP m akes no such allowance. 11 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Tert iary foreign language t eachers in Korea and t heir em ployers com m only ignore t he int ernat ional t est m ast ery requirem ent s which are so im port ant t o t heir st udent s, unless t hey are specifically em ployed t o coach t hese ext ernal t est skills. I n cont rast , t he underlying t eacher - purpose of t his t est is t o harness t he backwash of ext ernal t est m ot ivat ion by addressing it direct ly, giving st udent s clear inst rum ent al goals relat ive t o t heir own st rengt hs and weaknesses, t hen build on ( subvert ?) t his foundat ion int o genuine com m unicat ive language abilit ies. Test s are norm ally seen as t ools t o be em ployed by t eachers, adm inist rat ions, inst it ut ions and em ployers t o cat egorize st udent s. Text books on t est ing m ay ask rhet orically " why do we have t est s?" ( Moskovsky 2005) , but t he " we" im plied t ypically excludes st udent s. I n effect t est s of all kinds ( not only language t est s) are inst rum ent s of social cont rol. Much of t he st igm a at t aching t o t est s and exam inat ions com es precisely from t his power funct ion. The t est specificat ion discussed in t his proj ect st ands t he norm al power equat ion on it s head. I t aim s t o help st udent s m ast er t he t est cult ure on t heir own t erm s. I n order t o m easure G- TELP abilit y, t he current t est m ust m irror t he general st ruct ure of t he G- TELP specificat ions. However, t he class t est will go beyond t hose specificat ions, especially in lat er cycles but no so dram at ically in t his sam ple, t o bring st udent s t o an im plicit awareness of how G- TELP st udy can be ext ended int o non- Am erican com m unicat ive environm ent s. The specificat ions of t he t est out lined in t his program assum es a level of approxim at ely 3 t o 4 on t he GTELP scale ( t hat is, roughly an int erm ediat e level of com pet ency in L2) . 12 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Problems common to large scale standardized tests The brief discussion below form s a necessary cont ext for t he specificat ion of t he current t est since t he design undert aken will be influenced by follow ing realit ies. Large scale t est s such as G- TELP suffer from m any deficiencies, which are widely discussed in t he lit erat ure ( e.g. Spolsky 1995) and m ay even be a hot polit ical issue in places like Sout h Korea. I ndeed, G- TELP it self, by aim ing t o be crit erion rat her t han norm r eferenced, t ries t o circum vent som e of t he m ost serious lim it at ions of norm referencing. The speaking/ list ening division of G- TELP is a credible at t em pt t o elicit m easurable holist ic perform ance, while t he writ ing division is som ewhat less convincing. Bot h depend t o a degree upon holist ic rat her t han it em - based m arking, wit h t he accom panying penalt y of lower int er - m arker reliabilit y. The process is advert ised as cost effect ive because local count ry m arkers are em ployed. That is non nat ive speaker m arkers are em ployed. G- TELP does not give reliabilit y st at ist ics for t his out com e or any ot her on it s web sit e. Alt hough descript or s are given for reading levels, no t est s appear t o be available in t his division. The gram m ar t est s are exact ly t he sam e kind of m ult iple choice t ypes and cat egories found in large scale norm referenced t est s such as TOEFL and TOEI C. From a language t eaching/ learning viewpoint , som e of t he m ost dam aging lim it at ions of current large scale st andardized t est s are : a) I t is in t he nat ur e of t he t est ing process t o em phasize discret e point accuracy over fluency or com m unicat ive success. This is t rue 13 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 even where evaluat ion appears t o be holist ic, since m arkers oft en respond im plicit ly t o discret e point elem ent s. b) The large int ernat ional t est s t ake no serious account of t he shift ing im port ance of fluency Vs accuracy in t he language acquisit ion process, or in different com m unicat ive cont ext s. Th is balance has been subj ect t o m uch professional debat e. The ( non dogm at ic) posit ion of t his writ er is t hat up t o around int erm ediat e level, fluency is m uch m ore im port ant t han accuracy, t hough t he balance will vary wit h t he act ivit y. That is, learners need t o experience com m unicat ive success and acquire t ract ion in using t he language. Nat ive speaker speech is, aft er all, lit t ered wit h violat ions of form al ’rules’. Depending upon learner goals, m ore at t ent ion m ay be paid t o accuracy in t he t arget language aft er int erm ediat e level. Wit h regard t o cont ext , t here is obviously ext rem e variat ion in t he need or desirabilit y for real- life accuracy. Vagueness, im pr ecision or even poor language cont rol can be beneficial ( as m any a polit ician, m any a Rom eo, and m any a foreigner get t ing int o scrapes knows very well) . c) The act ual analyses of linguist ic accuracy im plicit in m ost language t est s are naive ( I believe) . Taking synt ax for exam ple, t he easiest errors t o m easure are local violat ions such as t ense inflect ions or preposit ion choice. The m ost difficult errors bot h t o analyse and t o t est are global violat ions ( e.g. see Birt & Kiparsky 1972) . Global violat ions ( such as scope problem s, and confusions of assignm ent in t hem at ic relat ions) lead t o com m unicat ive br eak down because t he list ener can’t ext ract t he m essage reliably. Local violat ions rarely lead t o com m unicat ive breakdown. Som e local errors persist for years in t he speech of m any L2 users wit hout causing any serious handicap. Nevert heless, t he large published 14 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 t est s alm ost invariably t est for local synt act ic errors, which are easily am enable t o m ult iple choice t est ing. d) To t he ext ent t hat speaking is t est ed analyt ically at all, t est ing analysis nearly always focuses on segm ent al rat her t han nonsegm ent al phonology. For exam ple, t est s for m inim al pair discrim inat ion are fairly com m on. I n real com m unicat ive pr act ice, non- segm ent al problem s are probably m ore likely t han m inor phonem e confusion t o lead t o com m unicat ive breakdown and/ or social rej ect ion. This is especially t he case wit h learners from syllable t im ed languages t rying t o m ake t he t ransit ion t o a st ress t im ed language such as English. The G- TELP does m ake reference t o " a foreign int onat ion and rhyt hm " , but gives no guidance as t o what t his m ight act ually m ean, or how t est ers m ight evaluat e it . The widespread om issions of t est ers in t his area m ay well have a lot t o do wit h t heir general ignorance of t he whole field ( as well as, of course, t he ignorance of m any t eachers) . Since t he open focus of t he current diagnost ic t est is on ext ernal t est m ast ery, t he early cycles of t he diagnost ic process ( shown here) will reflect t he know n lim it at ions of t he ext ernal t est s. Those lim it at ions will only be m ade explicit t o part icipat ing st udent s at a lat er st age. ________________ 15 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Test Content The act ual t est cont ent specificat ions of G- TELP ( from t he G- TELP websit e) are given at t he end of t his sect ion. The cont ent specificat ions for t he diagnost ic cycle undert aken here are a not ional ext ract ion from t he m ore com prehensive Level 3 G- TELP specificat ion. Grammar " An exam inee at [ Level 3] is required t o dem onst rat e underst anding of basic gram m at ical st ruct ures in t wo- t o t hree- sent ence cont ext s" . This m ay be t he least useful part of t he G- TELP t est , especially given t he m ult iple choice form at . However, it is also t he part least likely t o be m isunderst ood by non- nat ive English m arkers and falls wit hin st udent expect at ions.. This first t est t raining cycle will follow G- TELP m ult iple choice form at . Lat er cycles will t est t he form s in a m ore genuinely cont ext ual way by posing com m unicat ive problem s; ( e.g. I am personally im pressed by t he way t hat Pim sleur audiolingual m at erials pose such problem s at lower levels in L1 for solut ion in L2. This is ext rem ely effect ive) . A sam ple will be t aken from t he fairly arbit rary select ion of st ruct ures chosen by G- TELP for Level 3, which is : * Verbs - past progressive - fut ure possible condit ional wit h will, would, and would have - present perfect 16 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 * Coordinat ors and Subordinat ors - and, but , or, nor, so - aft er, because, et c. Word form s - adj ect ives: equat ives, superlat ives Listening " An exam inee at [ Level 3] can underst and nat ive speakers of English who are speaking at a slower t han norm al rat e wit h pauses, repet it ions, and paraphrasing int ended for nonnat ive speakers" . The sam ple chosen for m odelling here will be : " .. direct ions for a procedure" Procedural direct ions have t he advant age of being easily adapt ed t o m any different levels of com plexit y, so t hat t he list ening t ask can be graduat ed in difficult y t o discrim inat e am ongst learner abilit ies. Not e t hat t he act ual t est form at for list ening in G- TELP poses difficult ies which ar e out of all proport ion t o t he language being evaluat ed. That is, six quest ions are pr esent ed orally prior t o a spoken passage, t he passage is t hen given, t hen t he six quest ions are asked again. Responses are in m ult iple choice form at , but t he quest ions are nowhere writ t en down. The quest ions relat e t o quit e specific inform at ion. This is m ore of a brut e m em ory t est t han a list ening t est . I n a self- t est wit h t he Level 3 G- TELP websit e sam ple, I found it alm ost im possible t o keep six explicit oral quest ions about a m eet ing in m ind w hile I list ened t o t he passage, t hen respond t o 17 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 queries about t hem . Even as a nat ive speaker I would have needed t o t ake not es. ( Long ago, as a dispat ch officer announcing depart ures at Sydney airport , I learned t hat t he list ening public has great difficult y processing even single, sim ple inform at ion it em s) . The short t erm language m em ory of an L2 speaker is generally m uch short er t han t hat of a nat ive speaker. This different ial needs t o be calibrat ed car efully in a list ening t est . The G- TELP m at erial shows lit t le evidence t hat t his has been done. Speaking The G- TELP Level 3 descript or st at es t hat " The Speaker is generally int elligible and gram m at ically accurat e in perform ing t he t asks. Speech m ay be int errupt ed by occasional pauses, false st art s, rephrasing, et c. Speech evidences som e consist ent pronunciat ion and st ress errors and a foreign int onat ion and rhyt hm , which only occasionally int erfere wit h com m unicat ion. " Assessm ent crit eria : " GST assesses six different areas such as gram m ar, vocabular y, pronunciat ion, cont ent , fluency, and int eract ion" . The act ual balance am ongst t hese crit eria is not specified. One m ight ant icipat e rat her different weight ings of evaluat ion crit eria t o be applied by nat ive and non- nat ive speaking t est ers. The full G- TELP offers a series of 11 speaking t asks w hich are said t o be graduat ed challenges from Level 5 ( elem ent ary) , t o Level 1 ( near nat ive speaker proficiency) . I n fact , m ost of t he challenges could be handled at several levels of proficiency wit h com m unicat ive success. One int ract able problem for all language t est s which elicit open- ended st udent perform ance is t hat som e individuals are nat urally econom ical in expression while som e are verbose. This can 18 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 easily lead t o false assessm ent , since t he t acit urn individual m ay follow inst ruct ions, but not reveal his full capacit ies t o t he exam iner. Also, it is not clear how t he G- TELP concept of " a foreign int onat ion and rhyt hm " is t o be evaluat ed. The sam ple chosen for m odelling here will be : " Narrat ing a st ory fr om pict ures. The speaker is able t o relat e a sim ple narrat ive illust rat ed in a series of pict ures. The speaker appropriat ely describes t he set t ing, people, and/ or obj ect s depict ed, and coherent ly relat es t he chronology of event s" . This t ask is act ually rat ed by G- TELP as t he last in Basic perform ance. Though it is t rue t hat it can be at t em pt ed at a m arginally basic level, clearly t he response can be elaborat ed t o any level. I t is a good choice for t he init ial diagnost ic cycle of t he current proj ect . Reading & Vocabulary G- TELP : " An exam inee at t his level can read sim plified and/ or sim ple aut hent ic t ext s of t he t ype found in t ext books writ t en for learners of English as a second or foreign language. ..... A person at t his level is also required t o dem onst rat e underst anding of select ed vocabulary in t he cont ext of t he reading t asks described" . I n fact , t he t asks nom inat ed by G- TELP descript ors are not t hose norm ally found in ESL t ext books, t hough it is t rue in Korea anyway, t hat lit t le else is read or writ t en in L2. I ndeed, as a t rainer of Korean graduat e t eachers of English I found t hat few read or wrot e anyt hing even in Korean ( L1) beyond t he bare m inim um required by 19 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 t heir em ploym ent . This m akes t he aut hent ic select ion of t ext ual m at erial beyond classroom cont ent difficult , except in ESP cont ext s. The quest ion of vocabulary recognit ion is also com plex, and from a t est ing point of view, problem at ic. List s for word recognit ion are rat her point less. Ext ract ing m eaning from cont ext is a skill or apt it ude which in any inst ance m ight or m ight not depend upon prior acquaint ance wit h t he word. Parry ( 1991) “ found unexpect edly t hat aft er even a short delay t he words had oft en been forgot t en. Moreover, it was oft en t he best guessers who were t he worst forget t ers. Good guessers were able t o perceive t he m ain lines of a t ext very quickly, fill in sem ant ic gaps left by unknown wor ds, and t hen show no vocabulary gain bet ween pret est and post t est .” Given t he preceding lim it at ions, for t his diagnost ic cycle as t ask has been select ed from t he G- TELP descript ors which will hopefully st rike som e chord wit h m ost of t he t arget st udent s : " ..descript ions of landm arks and places of int erest based on t ext s from t ourist guidebooks, t ravel m agazines, and brochures" . Writing G- TELP : " Gram m ar, vocabulary, organizat ion, logicalit y, and subst ance are assessed in t he following 11 t asks in t he range of t hree levels, Basic, I nt erm ediat e, and Advanced, at five levels" . As wit h speaking, t he writ ing division of G- TELP is graded int o eleven t ask sect ions, supposedly at escalat ing levels of sophist icat ion : 20 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Ba sic - Task 1 Sent ence const ruct ion( synt ax) / Task 2 Use of appropriat e synonym s & pronouns / Task 3 Sent ence lengt h consist ency / Task 4 A suit able exam ple / / I nt e r m e dia t e - Task 5 Organizat ion of t he body / Task 6 Variet y of expressions / Task 7 Validit y of t he subj ect / Adva nce d - Task 8 Clarit y of t he conclusion / Task 9 Logical coherence / Task 10 Originalit y of t he essay / Task 11 Developm ent of relat ed ideas. These t ask sect ions are no doubt a useful checklist for a writ er. An em ployer for cert ain occupat ions would cert ainly be int erest ed in a candidat e’s m anagem ent of t he cat egories. Unfort unat ely, t hey have not hing t o do wit h t he m ast ery of L2 per se, except ing perhaps Tasks 1 and 2; ( it is not clear t o m e t hat Task 3 is relevant t o writ ing skills at all) . A st udent who perform s poorly by t hese crit eria in L2 will alm ost cert ainly show t he sam e weaknesses in L1. This problem is generic t o all t he elem ent s of L2 t est ing, but becom es st ark in wr it ing skills. On t he ot her hand, from a t eaching & learning ( not a t est ing) perspect ive, t he ’problem ’ m ay be an opport unit y. Learning t o writ e well in L1 m ay be t he short est rout e t o skilled writ ing in L2, if account is t aken of discourse st yle differences. I n pract ice, t he Writ ing Grade Scale used by G- TELP is very broad and very vague : Poor, Fair, Fairly Good, Good. I nt erm ediat e writ ing skills are said t o : " Address t he writ ing t ask generally, but neglect s som e aspect s. Som e gram m at ical errors, but t he writ ing rem ains int elligible. Logic is unclear at som e point s. Lacks variet y in vocabulary and sent ence st ruct ure" . 21 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 For t he purpose of first cycle diagnost ic t est ing, a writ ing t opic t hat highlight s t he so- called int erm ediat e skills will be chosen. However, ’basic’ skills m ast ery will obviously for m part of t he assessm ent , while perform ance on t he supposedly advanced skills will be deem phasized. Types of text Listening - Pract ical direct ions t o follow a set of procedures in a new em ploym ent sit uat ion Speaking - A st udent conduct ed narrat ive of event s in a series of pict ure panels Reading - Travel brochures prepared for English speaking t ourist s in a Korean t ravel bureau Writing - Writ e an ext ended e- m ail t o a foreign businessm an who is visit ing your com pany and Sout h Korea for t he first t im e. Give clear direct ions about when and where he will be m et . Give a brief it inerary for his t wo day st ay in Korea. Explain his accom m odat ion. Give a lit t le cult ural background about your com pany and t he count ry. 22 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Addressees of texts - Young t ert iary educat ed Sout h Korean adult s in an em ploym ent cont ext . The writ ing segm ent requires t hem t o address anot her professional in a cross- cult ural cont ext . Lengths of texts - Text s or dialogues, up t o 500 words. The gram m ar segm ent deals wit h m ult iple choice sent ence fragm ent s. The writ t en response form at s for list ening and reading t asks are also m ult iple choice. Topics - Sit uat ions, event s and com m unicat ive dem ands relevant t o young t ert iary educat ed Sout h Korean adult s in an em ploym ent cont ext Readability - I nt erm ediat e, as conceived by t he Level 3 scale of G- TELP Structural Range - The t est const ruct or considers t his t o be fairly arbit rary, but for t he purpose of m irr oring t he G- TELP t est , evaluat ed st ruct ures will conform t o G- TELP specificat ions for Level 3 and below ( see t he Cont ent sect ion) 23 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Vocabulary Range - As found in t he G- TELP t est series, but wit h a part icular em phasis on Korean business em ploym ent cont ext s and local- t ovisit ing- foreigner int eract ion Dialect and style - Const rained by t he heavily Am erican orient at ion of t he G- TELP t est , but t ending t o relaxed form al, as found in a congenial business environm ent when Koreans deal wit h foreigners locally. Am erican idiom preferred ( for G- TELP purposes) , but som e allowance for ot her int ernat ional variet ies of English where m eaning is not im paired. Speed of Processing - I t is not considered t hat t his can or should be validly assessed at int erm ediat e level, unless delays seriously inhibit com m unicat ion. Short t erm m em ory dem ands in t he list ening t asks need t o be carefully calibrat ed. Task Procedure, Timing and Medium Task Procedure Five m aj or sect ions - gram m ar, speaking, list ening, reading, writ ing. Se ct ion 1 - Gr a m m a r - Com plet e a set of m ult iple choice it em s on select ed st ruct ures 24 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Se ct ion 2 - List e ning - One t ext ; list en t o t he passage on em ploym ent procedures, t hen respond t o spoken quest ions via a writ t en m ult iple choice form at . ( Alt hough G- TELP rat es procedural language at Level 2 - m ore advanced t han Level 3 - I feel t hat it s st rong cont ext ual fr am e can act ually m ake som e procedural inst ruct ions sim pler t han t he G- TELP nom inat ed Level 3 t asks) . Se ct ion 3 - Spe a k ing - generat e a coherent narrat ive of event s from a pict ure panel st ory. Apply appropriat e vocabulary, including linking words t o give t he t ext proper cohesion. Speak wit hin an int onat ion and phonem ic range t hat will be com prehensible t o nat ive English speakers not fam iliar wit h ’Konglish’ ( Korean English) . Se ct ion 4 - Re a ding - read ext ract s from an English language Korean t ravel brochure. Ext ract specific inform at ion, follow reasoning, and recognize som e inferences im plicit in t he t ext . Se ct ion 5 - W r it ing - an ext ended em ail wit h explicit infor m at ion for a foreigner visit ing Korea. Convey t im e, dat es, places, and an it inerary. Give a brief descript ion of t he local sit uat ion as it affect s a visit ing businessm an. Dem onst rat e a level of t exual organizat ion adequat e t o convey t he inform at ion wit hout am biguit y. Avoid form al errors t hat will seriously confuse t he m essage. Lengt h : adequat e t o do t he j ob. Number of Items Sect ion 1 - Gram m ar - 10 it em s Sect ion 2 - List ening - 5 it em s Sect ion 3 - Speaking - 5 t o 15 m inut es Sect ion 4 - Reading - 10 it em s 25 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Sect ion 5 - Writ ing - roughly 300 t o 500 words Timing Sect ion 1 - 20 m inut es Sect ion 2 - 15 m inut es Sect ion 3 - 15 m inut es Sect ion 4 - 30 m inut es Sect ion 5 - 30 m inut es Tot al - 1 hour, 40 m inut es, wit h t he speaking and list ening sect ions done at a different t im e t o t he ot her sect ions. Medium Pen and supplied paper, plus a cent rally cont rolled language laborat ory casset t e and casset t e recorder t o play t he list ening t ext and record t he pict ure panel narrat ive. There will be a separat e answer sheet and quest ion booklet . Organization - St udent s undert ake t he t est under supervision. - The gram m ar, reading and writ ing sect ions will be done in a group environm ent . - The speaking and list ening sect ions w ill be done in a language laborat ory, also in a group environm ent , at a different t im e t o t he gram m ar- reading- writ ing t est . - The list ening t est will be done before t he speaking t est . The list ening t ask w it h spoken quest ions will be available on t he t ape consoles for t en m inut es. 26 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 - The speaking t est will be done aft er t he writ ing t est . The pict ure panel st ory will be dist ribut ed wit h inst ruct ions. St udent s will be allowed up t o fift een m inut es t o orallly com pose and record a narrat ive relat ed t o t he pict ure panel st ory. Techniques Section 1 Grammar 1. Mult iple choice wit h one correct answer and t hree dist ract ors ( as per G- TELP; ot her form at s will be int roduced in lat er t est cycles) . Exam ple: 1. The KAL flight was delayed ___________ t he pilot becam e ill in Singapore a) but ; b) so; c) because Section 2 Listening List en t o t he j ob inst ruct ions. Answer each of t he five spoken quest ions. Select t he right answer for each quest ion from t hree choices. Exam ple : [ Spoken ] When ar e invoices collect ed from t he warehouse? [ Writ t en prom t ] a) im m ediat ely aft er lunch ; b) im m ediat ely you arrive at work; c) im m ediat ely aft er yest erday’s orders are finished 27 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Section 3 Speaking Make one st ory from t he cart oon pict ures you see here. Describe what you see, what happened, and why it happened. Say w hat happened next . You can t alk for five t o t en m inut es. Section 4 Reading Read t he t ravel brochure, t hen answer t he quest ions. Select t he answer which is m ost t rue from what t he brochure says. Each quest ion has t hree answer choices. Exam ple : a) Chungj u dam is t he m ain wat er supply for Seoul b) Chungj u dam is t he m ain dam in Sout h Korea c) Chungj u dam is t he m ain reason visit ors com e t o Chungj u Section 5 Writing Writ e an em ail t o a foreign businessm an visit ing your com pany. Tell him how he will be m et at I ncheon Airport . St at e t im es, dat es, places, and give him an it inerary. Give a brief descript ion of Chungj u and local inform at ion about your com pany. Criterial levels of performance and Marking Scales Scoring Context Crit erial levels in t his diagnost ic t est are calibrat ed approxim at ely t o t he Level 3 descript ors of t he G- TELP t est ( see t he Cont ent sect ion) . Since t he real purpose of t he t est is t o bring st udent s t o an underst anding of G- TELP t est crit eria, and t he lim it at ions of t hose 28 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 crit eria, act ual perform ance will be evaluat ed in post - t est discussion and analysis wit h t he st udent s. Furt her, t he sam ple of diagnost ic t asks is not ext ensive enough t o derive a valid evaluat ion of overall st udent abilit y. The final score of individual st udent s is not significan t in t his cont ext . The im port ant t hing is for st udent s t o arrive at t heir own underst anding of t he t echniques t hat G- TELP uses t o j udge t hem , and t o develop st rat egem s for m anaging G- TELP evaluat ions while im proving t heir real m ast ery of English; ( t his im plies a process which goes beyond t he scope of t his pr esent at ion exercise) . The G- TELP t est it self offers t his profile of achievem ent for t he speaking t ask ( st at ed as ’m ast ery’) : The level of m ast ery Separat e level scores for each of t he t hr ee levels Percent age profiles sum m arizing t he perform ances on each t ask Evaluat ions of t he perform ances in t he five Skill Areas across all 11 t asks: Cont ent , Gram m ar, Vocabulary, Pronunciat ion, Fluency, and I nt eract ion Descript ions of t he Skill Area Perform ance at t he exam inee’s level of m ast er. 29 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 The M e a ning of ’M a st e r y’ in G- TELP Thus, a score of 75% is t aken t o represent m ast ery at any given level. That score m ay be com pr ised of discret e point it em s ( as in t he m ult iple choice t est s) , or be based on im pressionist ic m ar king ( as for speaking and wr it ing t asks) . The balance of cr it er ial weight ing is probably different for each m arker in im pressionist ic m arking, but G- TELP claim s t hat " t he perform ance of t he exam inees is evaluat ed by local expert s in English as a foreign language, w ho have been t rained and cert ified by G- TELP Services t o assign obj ect ive and consist ent scores t o in - count ry nonnat ive speakers of English" . I nt er- rat er reliabilit y st at ist ics are not given. Com m ent : Nobody can assign a subj ect ive score obj ect ively. Even discret e point cr it er ia are, in t he end, assigned significance subj ect ively. The best t hat any language t est can do is t o arrive at som e kind of social consensus about crit eria, and st r ive for consist ency. 30 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Scoring for the diagnostic purposes of the current test cycle Grammar The gram m ar t est checks t he abilit y t o use several gram m at ical const ruct ions, ext ract ed from t he Level 3 G- TELP specificat ion. This is clearly only a sam ple, not a com prehensive t est of gram m at ical knowledge applying t o any level of English m ast ery. There are t en m ult iple choice quest ions. Each m ult iple choice quest ion will be wort h 1 m ark , wit h an overall score of 7 indicat ing ’m ast ery’ level. No half m arks will be awarded. Listening The list ening t est involves a single t ask wit h responses t o five m ult iple choice quest ions. This is clearly only a sam ple, not a com prehensive t est of list ening skills. Each m ult iple choice quest ion will be wort h 2 m arks, wit h an overall score of 8 indicat ing ’m ast ery’ level. No single m arks will be awarded. Speaking The speaking t est requires t he const ruct ion of a narrat ive based on a series of pict ure panels. This is clearly only a sam ple, not a com prehensive t est of speaking skills. The G- TELP m ast ery crit eria for t he pict ure narrat ive t ask is : 31 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Narrat ing a st ory from pict ures. The speaker is able t o relat e a sim ple narrat ive illust rat ed in a series of pict ures. The speaker appropriat ely describes t he set t ing, people, and/ or obj ect s depict ed, and coherent ly relat es t he chronology of event s. The G- TELP assesses speaking in six cat egories : Gram m ar / Vocabulary / Pronunciat ion / Fluency / Cont ent / I nt eract ion ( overall com prehensibilit y) . Descript ors for t hese cat egories are given in Appendix 1 and t he end of t his sect ion. For t his diagnost ic t est , Gram m ar & Vocabulary will be com bined int o a single cat egory, yielding a t ot al of five cat egories. The speaking t ask will be scored out of 10 m arks. Two m arks will be awarded for each of t he 5 cat egories above. That is , 2 m ar ks will be awarded in each cat egory when t he st udent perform s according t o t he Level 3 Speaking cr it eria in G- TELP ( see Appendix 2 below) . An overall score of 7 will indicat e Level 3 ’m ast ery’ for t he purposes of t he t est . Not e t hat t he scoring here is a form alit y. I t does not im ply t hat t he equal weight ing of cat egories necessarily applies in t rue com m unicat ive sit uat ions. The st udent diagnost ic purpose in t his cont ext is t o highlight areas t hat require at t ent ion. Reading The reading t est involves a single t ext , a t ravel brochure, w it h responses t o 10 m ult iple choice it em s. This is clearly only a sam ple, not a com prehensive t est of reading skills. Each m ult iple choice quest ion will be wort h 1 m ark , wit h an overall score of 7 indicat ing ’m ast ery’ level. No half m arks will be awarded. 32 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Writing The writ ing t est requires t he const ruct ion of a form al e- m ail, incorporat ing som e nom inat ed inform at ion. This is clearly only a sam ple, not a com prehensive t est of writ ing skills. The G- TELP assesses writ ing in four grades wit hin five cat egories. The grades are Poor / Fair / Fairly Good / Good. The cat egories are Gram m ar / Vocabulary / Organizat ion / Logic / Subst ance. Descript ors for t hese grades and cat egories are given in Appendix 2. The writ ing t ask will be awarded an overall score of 20, com prising four m arks for each cat egory. Poor = 1 m ark; Fair = 2 m arks; Fairly Good = 3 m arks; Good = 4 m arks. An overall score of 15 will indicat e ’m ast ery’ for t he purposes of t his diagnost ic t est . Not e t hat t he scoring here is a form alit y. I t does not im ply t hat t he equal weight ing of cat egories necessarily applies in t rue com m unicat ive sit uat ions. The st udent diagnost ic purpose in t his cont ext is t o highlight areas t hat require at t ent ion. Test Construction Procedures Sampling Text s, st ruct ures and pict ures for t he diagnost ic t est w ill be based on t he G- TELP fram ework which t his t est is seeking t o clarify for st udent s. However, t his cont ent w ill be given a local Korean 33 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 reference where possible. The underlying m essage of cont ent choice will be t hat English is an int ernat ional language first , and a nat ional language only incident ally. Item writing and Moderation I t em s will be writ t en according t o m odified G- TELP specificat ions. A key will be provided for m ult iple choice answers, and a list of m arking crit eria for holist ically m arked elem ent s. Using a checklist , it em s will be m oderat ed by a fellow nat ive speaker t eacher of English. Informal Trialling The t est will be t rialled by nat ive speaking Am erican co- workers of t he t est er ( if t heir cooperat ion can be secured) . Trialling and Analysis Trialling of t wo versions of t ext s and it em s will be carried out wit h a colleague’s English m aj or st udent s at Chungj u Nat ional Universit y ( if t heir cooperat ion can be secured) . Half t he class w ill com plet e a different version from t he ot her half. Any adm inist rat ion and scoring problem s will be not ed. I f possible, t he t est ed st udent s will be involved in a post - t est analysis of t he t est cont ent and design, as well as t heir own perform ance. This procedure is cent ral t o t he not ional purpose of t he t est . Basic st at ist ical analysis will be used t o select t he ’best ’ opt ions for inclusion fut ure versions of t he t est . A qualit at ive analysis will also 34 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 be done, drawing ext ensively upon a post - t est analysis wit h t est t akers. Validation Sect ions : Gram m ar, Speaking, List ening, Writ ing, Reading I t em s const ruct ed and select ed for t he final version of t he t est will be checked against t he m odified G- TELP specificat ions, as well as com parable it em s from t he act ual G- TELP t est . St udent s’ scores will be checked against t heir previous perform ances in ot her t est s and courses t hat involved an evaluat ion of gram m ar, list ening, speaking, reading or writ ing in English. Speaking - concurrent validity To check concurrent crit erion- relat ed validit y a random sam ple of st udent s will com plet e a 10 m inut e int erview and discussion, wit h t wo nat ive- speaking t eachers scoring. The scores achieved in t his perform ance will be com pared wit h t hose from t he full diagnost ic t rial. The at t ainm ent of a part icular form al validit y coefficient bet ween inform al t rialling and full diagnost ic applicat ion of t he t est is not im port ant at t his st age for m eet ing t he cent ral purpose of t he t est as a focus for st udent discussion and self- im provem ent . Candidate Instructions Unless underst anding t est ing inst ruct ions is form ally assessed, it seem s difficult t o j ust ify giving inst ruct ions in L2, especially at lower 35 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 levels. To do ot herwise is sim ply t o insert anot her unm easurable variable int o t he assessm ent process. That is, t he validit y of t he t est m ay be com prom ised. Nevert heless, it is com m on language t est ing pract ice t o give inst ruct ions in L2. The G- TELP sam ple t est sit e gives inst ruct ions part ly in English and part ly in Korean. The sam ple inst ruct ions in t his diagnost ic present at ion are in English, but will be t ranslat ed w here t he act ual applicat ion of t he t est is required. Teacher’s Handbook One handbook will be prepared as a sum m ary for t he current t est er, and for anyone who seeks t o develop t he m at erial for furt her applicat ion. A separat e handbook will be prepared for use by st udent s t o help in preparat ion for t he diagnost ic t est . The t est is a diagnost ic and analyt ic t ool for st udent s who want t o t o enhance t heir abilit y t o pass int ernat ional proficiency t est s such as G- TELP. I t is int ended for post - t est discussion by t est edst udent s. Lat er applicat ion of t he m at erial will show how it can form a foundat ion for t rue com m unicat ive language learning. Rationale: I nt ernat ionally st andardized t est s such as G- TELP have a huge backwash effect on t he learning behaviour of language st udent s in Sout h Korea, even quit e young st udent s. This backwash is oft en not beneficial, and t he form al syllabi of schools and universit ies are oft en not coordinat ed in any product ive way wit h t his underlying engine of t est ing m ot ivat ion. The aim of t he current diagnost ic t est 36 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 is t o m ake t hese relat ionships explicit , part icularly t o t he st udent s concerned, and involve t hem in a process which can harness t he t est ing cont ent of int ernat ional st andardized t est s as an inst rum ent for genuine language learning. Development and Validation: The t est was developed by carefully abst ract ing select ed elem ent s of t he int ernat ional G- TELP t est . That is, it is an inform al subset of t he G- TELP t est , and writ t en according t o slight ly m odified G- TELP specificat ions. The general cont ent and design of t he G- TELP and com parable t est s is fam iliar t o m ost Sout h Korean t ert iary st udent s who st udy English. Description of the test: The t est com es wit h five divisions : gram m ar, speaking, list ening, reading and writ ing. The gram m ar, reading and writ ing t est s will be done t oget her and t ake 1 hour, 20 m inut es. The speaking and list ening t est s will be done t oget her at a different t im e in a language laborat ory, and t ake 30 m inut es. The gr a m m a r t e st cont ains t en m ult iple choice quest ions. St udent s are allowed 20 m inut es t o com plet e t his. The spe a k ing t e st cont ains one t ask, const ruct ing a narrat ive from pict ures. The st udent narrat ive is recorded on t ape for lat er m arking and review . St udent s are allowed 15 m inut es t o com plet e t his. 37 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 The list e ning t e st involves list ening t o a short passage of inst ruct ions and answering five m ult iple choice quest ions. St udent s are allowed 15 m inut es t o com plet e t his. The r e a ding t e st involves reading a t ext and answering and answering t en m ult iple choice quest ions. St udent s are allowed 30 m inut es t o com plet e t his. The w r it ing t e st involves writ ing a for m al e- m ail cont aining cert ain given inform at ion. St udent s are allowed 30 m inut es t o com plet e t his. Sample items: Section 1 Grammar Exam ple: 1. The KAL flight was delayed ___________ t he pilot becam e ill in Singapore a) but ; b) so; c) because Section 2 Listening List en t o t he j ob inst ruct ions. Answer each of t he five spoken quest ions. Select t he right answer for each quest ion from t hree choices. Exam ple : [ Spoken ] When ar e invoices collect ed from t he warehouse? [ Writ t en prom pt ] a) im m ediat ely aft er lunch ; b) im m ediat ely you arrive at work; c) im m ediat ely aft er yest erday’s orders are finished 38 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Section 3 Speaking Make one st ory from t he cart oon pict ures you see here. Describe what you see, what happened, and why it happened. Say w hqt happened next . You can t alk for five t o t en m inut es. Section 4 Reading Read t he t ravel brochure, t hen answer t he quest ions. Select t he answer which is m ost t rue from what t he brochure says. Each quest ion has t hree answer choices. Exam ple : a) Chungj u dam is t he m ain wat er supply for Seoul b) Chungj u dam is t he m ain dam in Sout h Korea c) Chungj u dam is t he m ain reason visit ors com e t o Chungj u Section 5 Writing Writ e an em ail t o a foreign businessm an visit ing your com pany. Tell him how he will be m et at I ncheon Airport . St at e t im es, dat es, places, and give him an it inerary. Give a brief descript ion of Chungj u and local inform at ion about your com pany. Advice on preparing for the test St udent s should m ake t hem selves fam iliar wit h t he online version of t he G- TELP t est at ht t p: / / gt elp.co.kr/ e_gt elp/ gt elp/ e_gt elp04.asp . 39 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 They are likely t o already be fam iliar wit h t he t est ing pat t ern of m ult iple choice gram m ar it em s. • They should review t he gram m ar it em s for G- TELP level 3. • They can pract ice const ruct ing a narrat ive using pict ure cues, and list ening for specific inform at ion from a recorded t ext . • The class can st udy inform at ional brochures from businesses and governm ent depart m ent s, as well as office procedure m anuals. • They can pract ice writ ing form al e- m ails, part icular ly t o foreign businessm en who m ay not be fam iliar wit h condit ions in Sout h Korea. Interpretation of test scores This is a diagnost ic t est , int ended for review and discussion by t he t est ed st udent s t hem selves. The act ual scores are not very significant , except as a guide t o areas needing at t ent ion. The GTELP t est t akes a score of 75% in any area t o indicat e ’m ast ery’ at a part icular level from 1 t o 5. Level 5 is near nat ive speaker st andard. The diagnost ic t est is calibrat ed approxim at ely for st udent s at about G- TELP Level 3. Training Materials: This diagnost ic t est does not require a nat ive speaker, or even a t rained t eacher for adm inist rat ion. However, t he follow - up m arking, analysis and guided st udent discussion of t he cont ent and out com e does require t rained t eachers who are t horoughly fam iliar wit h st andardized t est s in Sout h Korea, wit h Korean educat ional norm s, and wit h Korean cult ural sensit ivit ies. 40 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Test administration: The quest ion booklet and answer sheet s are separat e. St udent s are not t o writ e in t he quest ion booklet . Quest ion booklet s should be collect ed from all st udent s at t he end of t he t est . However, since t he quest ion booklet s t hem selves will be analysed by t he st udent s in a post - t est analysis, t hey are not ext ernally confident ial in t he sam e way as norm al st andardized t est docum ent s. The t est should be adm inist ered wit hin t he specified t im e lim it s and under st andard t est condit ions. Answers should be writ t en in blue or black ink for legibilit y, not pencil. The t est will be done in norm al class t im e, eit her in a classroom wit h separat ed desks, or in t he case of t he speaking and list ening t est s, a language laborat ory. The t est ing at m osphere should be cooperat ive, but disciplined. St udent s should have a prior underst anding of t he purpose of t he t est , and be aware of t heir lat er role in analysing t he result s. Scoring of t he t est s should be done wit h reference t o t he key and rat ing scales provided. Where possible, writ ing and recorded speaking t est s should be cross- checked by anot her t eacher. All result s will ult im at ely be discussed and peer m oderat ed by t he st udent s t hem selves. Handbook for Candidates 1. This is a t est t o help you wit h ext ernal proficiency exam inat ions like G- TELP. You will learn from t his t est what your st rengt hs and weaknesses are in English. The t eacher can t hen design a program t o help you. 41 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 2. Aft er t he t eacher has m arked t he t est , he will discuss it wit h you st ep by st ep. I n t his way you will not only learn t o do t he G- TELP t est successfully, but also how t o use t est st udy t o really becom e a bet t er English speaker, reader and writ er. How to prepare for the test Check t he online version of t he G- TELP t est at ht t p: / / gt elp.co.kr/ e_gt elp/ gt elp/ e_gt elp04.asp . • Especially review t he gram m ar it em s for G- TELP Level 3. • Pract ice speaking by m aking up a st ory. Use pict ures or cart oons for ideas. • Pract ice list ening t o any recorded speaking t hat gives inform at ion like inst ruct ions or announcem ent s. Can you rem em ber what t he speaker said? • St udy inform at ional brochures from businesses and governm ent depart m ent s in English. Look at som e inst ruct ion m anuals. Can you underst and t his infor m at ion? • Pract ice writ ing form al e- m ails like you m ight have t o working in a com pany. Writ e e- m ails t o foreign businessm en who m ay not know about condit ions in Sout h Kor ea. What do your test scores mean? Don’t worry if you find som e part s of t he t est a bit hard. The m arks don’t m at t er t oo m uch. The whole idea is t o help you do bet t er in a real t est , and t o becom e a bet t er English speaker. The G- TELP t est it self t akes a score of 75% t o m ean ’m ast ery’ at som e level from 1 t o 5. Level 5 is near nat ive speaker st andard. 42 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 This t est you are doing now is set for st udent s at about G- TELP Level 3. What is in the test ? The t est com es wit h fiive divisions : gram m ar, speaking, list ening, reading and writ ing. The gram m ar, reading and writ ing t est s will be done t oget her and t ake 1 hour, 20 m inut es. The speaking and list ening t est s will be done t oget her at a different t im e in a language laborat ory, and t ake 30 m inut es. The gr a m m a r t e st cont ains t en m ult iple choice quest ions. St udent s are allowed 20 m inut es t o com plet e t his. The spe a k ing t e st cont ains one t ask, const ruct ing a narrat ive from pict ures. The st udent narrat ive is recorded on t ape for lat er m arking and review . St udent s are allowed 15 m inut es t o com plet e t his. The list e ning t e st involves list ening t o a short passage of inst ruct ions and answering five m ult iple choice quest ions. St udent s are allowed 15 m inut es t o com plet e t his. The r e a ding t e st involves reading a t ext and answering and answering t en m ult iple choice quest ions. St udent s are allowed 30 m inut es t o com plet e t his. The w r it ing t e st involves writ ing a for m al e- m ail cont aining cert ain given inform at ion. St udent s are allowed 30 m inut es t o com plet e t his. 43 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Sample test items: Section 1 Grammar Exam ple: 1. The KAL flight was delayed ___________ t he pilot becam e ill in Singapore a) but ; b) so; c) because Section 2 Listening List en t o t he j ob inst ruct ions. Answer each of t he five spoken quest ions. Select t he right answer for each quest ion from t hree choices. Exam ple : [ Spoken ] When ar e invoices collect ed from t he warehouse? [ Writ t en prom t ] a) im m ediat ely aft er lunch ; b) im m ediat ely you arrive at work; c) im m ediat ely aft er yest erday’s orders are finished Section 3 Speaking Make one st ory from t he cart oon pict ures you see here. Describe what you see, what happened, and why it happened. Say w hat happened next . You can t alk for five t o t en m inut es. 44 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Section 4 Reading Read t he t ravel brochure, t hen answer t he quest ions. Select t he answer which is m ost t rue from what t he brochure says. Each quest ion has t hree answer choices. Exam ple : a) Chungj u dam is t he m ain wat er supply for Seoul b) Chungj u dam is t he m ain dam in Sout h Korea c) Chungj u dam is t he m ain reason visit ors com e t o Chungj u Section 5 Writing Writ e an em ail t o a foreign businessm an visit ing your com pany. Tell him how he will be m et at I ncheon Airport . St at e t im es, dat es, places, and give him an it inerary. Give a brief descript ion of Chungj u and local inform at ion about your com pany. APPENDIX 1 G-TELP Speaking Score Criteria [ Mat erial in t hese appendices is t aken from t he G- TELP websit e ht t p: / / gt elp.co.kr / e_gt elp/ gt elp/ e_gt elp04.asp ] Gr a m m a r The speaker’s cont rol of basic sent ence st ruct ure Voca bula r y The speaker’s available vocabulary in t he perform ance of cert ain required speaking t asks 45 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Pr onuncia t ion The speaker’s cont rol over t he sound syst em Flue ncy The speaker’s flow of speech and degree of spont aneit y as well as facilit y in t he perform ance of t he requir ed t asks Cont e nt The t opics/ cont ext s about which t he speaker can speak I nt e r a ct ion The Overall Evaluat ion, which confirm s t he exam inee’s Mast ery Level, is based on com prehensibilit y, w hich is a m ore general assessm ent of t he speaker’s com m unicat ive abilit y, influenced t o som e ext ent by t h ot her, separat ely evaluat ed, skills. G-TELP Proficiency Levels for Speaking No Mast ery The speaker cannot perform or underst and t he t asks. Level 5 Beginning Speaking Skills The speaker is not com prehensible even in sim ple speech due t o frequent pauses, rephrasing, pronunciat ion errors, lim it ed vocabulary, and lack of gram m at ical cont rol. Level 4 Basic Speaking Skills 46 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 The speaker can successfully com plet e, and be underst ood overall, in perform ing t he t asks. Speech is slow and labored, and is char act erized by a st rong accent and frequent gram m at ical errors. Transm ission of t he m essage m ay be im peded frequent ly by phonem ic, st ress, and int onat ion problem s. Level 3 I nt erm ediat e Speaking Skills The Speaker is generally int elligible and gram m at ically accur at e in perform ing t he t asks. Speech m ay be int errupt ed by occasional pauses, false st art s, rephrasing, et c. Speech evidences som e consist ent pronunciat ion and st ress errors and a foreign int onat ion and rhyt hm , which only occasionally int erfere wit h com m unicat ion. Level 2 Advanced Speaking Skills The speaker is alm ost always int elligible in perform ing t he t asks. Speech is fair ly sm oot h and spont aneous alt hough having som e m ispronunciat ion of individual sounds. The speaker’s vocabulary is ext ensive and varied, exhibit ing good cont rol of t he st ruct ure of t he language. 47 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Level1 Aut hent ic Speaking Skills Level One is not evaluat ed by t he Speaking Test , alt hough m ast ery of Level Two indicat es t hat t he speaker is approaching t he acquisit ion of oral proficiency skills at t he nat ive speaker level. APPENDIX 2 G-TELP Writing Score Criteria Gr a m m a r 1 - Poor: The com posit ion shows v ery lit t le gram m at ical or st ruct ural cont rol. I t has errors t hat int erfere wit h m eaning. 2 - Fair: The com posit ion has som e gram m at ical errors, but t hese do not int erfere wit h m eaning. 3 - Fairly good: The com posit ion shows cont rol over basic st ruct ures, wit h at t em pt s t o use som e com plex st ruct ures. There are som e gram m at ical errors, but t hese do not int erfere wit h m eaning. 4 - Good: The com posit ion has cont rol over st ruct ures. Overall, t he writ ing is gram m at ically correct . The few errors rarely int erfere t h m eaning. Voca bula r y 1 - Poor: The com posit ion shows lim it ed vocabulary. Many words are used incorrect ly. 2 - Fair: The com posit ion shows lim it ed vocabulary. Word choice is fault y at t im es. However, overall m eaning is st ill conveyed. 48 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 3 - Fairly good: The com posit ion uses vocabulary appropriat ely and accurat ely, w it h occasional m isuse. I t shows t he writ er’s abilit y t o paraphrase. 4 - Good: The vocabulary is varied and ext ensive. The writ er rarely uses words incorrect ly, but when he/ she does, t he int ended m essage is com m unicat ed. Or ga niza t ion 1 - Poor: The com posit ion has no out w ard form s, int roduct ion, m ain discourse ( body) , or conclusion. I n cont ent , t he com posit ion falls short in each process. 2 - Fair: The com posit ion has out war d form s but , in cont ent , it does not have a definit e form in each process. 3 - Fairly good: The com posit ion has out ward form s, int roduct ion, m ain discourse ( body) , and conclusion, and it has a generally realized cont ent . 4 - Good: The com posit ion has a w ell- developed int roduct ion, m ain discourse ( body) , and conclusion. I t is writ t en in t he com plet e essay form . Logic 1 - Poor: The com posit ion has no generally developed logic. Only pieces of ideas have been present ed. 2 - Fair: The com posit ion has logic, but ideas oft en go ast ray and m iss t he logic. 3 - Fairly good: The com posit ion has a consist ent flow of logic in general but som et im es goes ast ray and m isses t he logic. 4 - Good: The com posit ion is consist ent ly logical and present s suit able logical exam ples. 49 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Subst a nce 1 - Poor: The cont ent is not appropriat e for t he t ask. The com posit ion has m any irrelevant inform at ion or incom prehensible expressions. 2 - Fair: The com posit ion addresses som e aspect s of t he writ ing t opic. I t has som e ir relevant inform at ion. 3 - Fairly good: The com posit ion adequat ely addresses t he writ ing t ask. I t provides relevant inform at ion, but lacks focus. 4 - Good: I t effect ively addresses t he w rit ing t ask. The com posit ion clearly and persuasively com m unicat es t o t he reader. APPENDIX 3 G-TELP CONTENT SPECIFICATIONS FOR LEVEL 3 ( ht t p: / / gt elp.co.kr / e_gt elp/ gt elp/ e_gt elp05.asp ) - Grammar An exam inee at t his level [ Level 3] is required t o dem onst rat e underst anding of basic gram m at ical st ruct ures in t wo- t o t hreesent ence cont ext s. St ruct ures t est ed at t his level include: * Verbs - past progressive - fut ure possible condit ional wit h will, would, and would have - present perfect * Coordinat ors and 50 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Subordinat ors - and, but , or, nor, so - aft er, because, et c. Word form s - adj ect ives: equat ives, superlat ives, com parat ives - Listening An exam inee at t his level can underst and nat ive speakers of English who are speaking at a slower t han norm al rat e wit h pauses, repet it ions, and paraphrasing int ended for nonnat ive speakers. Tasks at t his level include: * public announcem ent s which give sim ple inform at ion * explanat ions of t he physical funct ions or at t ribut es of fam iliar obj ect s * personal account s in which a personal event and it s chronology are described * direct ions for a procedure - Reading and Vocabulary An exam inee at t his level can read sim plified and/ or sim ple aut hent ic t ext s of t he t ype found in t ext books writ t en for learners of English as a second or foreign language. Tasks at t his level include: * descript ions of landm arks and places of int erest based on t ext s from t ourist guidebooks, t ravel m agazines, and brochures. * hist orical biographies based on t ext s fr om t ravel lit erat ure and hist ory t ext s 51 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 * inform al narrat ives in personal correspondence bet ween friends * announcem ent s concerning t he act ivit ies, m eet ings, and accom plishm ent s of organizat ions such as clubs and professional groups A person at t his level is also required t o dem onst rat e underst anding of select ed vocabulary in t he cont ext of t he r eading t asks described above. - Writing Gram m ar, vocabulary, organizat ion, logicalit y, and subst ance are assessed in t he following 11 t asks in t he range of t hree levels, Basic, I nt erm ediat e, and Advanced, at five levels. => Task Contents Ba sic Task 1 Sent ence const ruct ion( synt ax) Task 2 Use of appropriat e synonym s & pronouns Task 3 Sent ence lengt h consist ency Task 4 A suit able exam ple I nt e r m e dia t e Task 5 Organizat ion of t he body Task 6 Variet y of expressions Task 7 Validit y of t he subj ect Adva nce d Task 8 Clarit y of t he conclusion Task 9 Logical coherence Task 10 Originalit y of t he essay Task 11 Developm ent of relat ed ideas 52 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 - Speaking The cont ent of t he G- TELP Speaking Test s reflect s expect at ions in specific perform ance t asks in which t he exam inee is expect ed t o dem onst rat e com pr ehensible oral proficiency while progressing t hrough t he t hree levels. The t ask cont ent consist s of t he following everyday sit uat ions: Le ve l 4 - Ba sic Ta sk 1 Giving personal inform at ion. The speaker is able t o answer sim ple quest ions about self and fam ily, responding in words, phrases, or sim ple sent ences. Ta sk 2 Describing a fam iliar set t ing/ obj ect s. The speaker is able t o nam e and describe a fam iliar set t ing and t he obj ect s wit hin t hat set t ing. The speaker is able t o locat e and describe t he spat ial relat ionships am ong t he obj ect s in t he set t ing. He/ She m ay go beyond accurat e nam ing of what is illust rat ed t o describing som e physical charact erist ics such as size, shape, et c. Ta sk 3 Describing habit ual act ivit ies. The speaker is able t o nam e and describe personal act ivit ies so t hat t he descript ion is underst ood as a coherent sequence of event s. The speaker m ay go beyond a m ere list ing of event s or act ivit ies t o include som e elaborat ion of t hem . Ta sk 4 Narrat ing a st ory from pict ures. The speaker is able t o relat e a sim ple narrat ive illust rat ed in a series of pict ures. The speaker appropriat ely describes t he set t ing, people, and/ or obj ect s depict ed, and coherent ly relat es t he chronology of event s. 53 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Le ve l 3 - I nt e r m e dia t e Ta sk 5 Expressing and support ing an opinion. The speaker is able t o present a personal opinion and expr ess personal preferences, and t o subst ant iat e t hese opinions or preferences wit h at least t wo sim ple support ing det ails. Ta sk 6 Giving aut obiographical det ail about a place or event . The speaker is able t o describe a place or event in adequat e det ail. Ta sk 7 Responding t o request s for inform at ion about places of int erest . The speaker is able t o supply appropriat e and well- form ed quest ions and responses in a dialogue cent ering on request s for inform at ion. Ta sk 8 Discussing advant ages/ disadvant ages of t wo relat ed obj ect s. The speaker is able t o com pare and cont rast t he obj ect s by giving at least t wo advant ages and/ or disadvant ages of each. Le ve l 2 - Adva nce d Ta sk 9 Giving direct ions from a m ap. The speaker is able t o give direct ions for reaching a locat ion wit h clarit y and t horoughness so t hat t he rout e for reaching t he dest inat ion can be underst ood. Ta sk 1 0 Present ing a solut ion t o a specific problem . The speaker is able t o clearly st at e and express a posit ion on a t opic and present convincing argum ent s support ing such posit ion. The speaker is able t o r elat e an abst ract issue t o t he specifics of t he sit uat ion. Ta sk 1 1 54 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Present ing a solut ion t o a com plex hypot het ical problem . The speaker is able t o explain, wit h relevant and convincing det ail, how he/ she would solve t he problem ( s) associat ed wit h a hypot het ical sit uat ion. Asse sse d Ar e a s GST assesses six different areas such as gram m ar, vocabulary, pronunciat ion, cont ent , fluency, and int eract ion. 55 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Pa r t I I W r it ing Te st I t e m s for list e ning, r e a ding, w r it ing, spe a k ing, gr a m m a r / voca bula r y University of Newcastle, NSW Assignment 2 : Language Testing And Evaluation Topic : Write a set of between 20 and 30 test items for each of the following language skills : - listening, reading, writing, speaking, grammar / vocabulary . For each sample test prepare a list of ’DOs’ and ’DON’Ts’ Notes on Assignment Content : 1. I have followed the intention stated in the first assignment of roughly modeling test content and structure on the G-TELP test, as it is used in South Korea. However, in several sub-sections I have used a more innovative testing format. 2. Genuine test material of the kind dealt with here would be preceded by an analysis of sample language produced by target candidates for the test. This has not been done in the present case since the target candidates are more or less hypothetical. However, the general level is pitched around an intermediate level of English ability as conceived by the G-TELP test (Level 3 G-TELP). 3. I have preceded each set of test questions by a short commentary on the limitations of formal testing in the particular skill area. I feel this is necessary because although test constructors and writers such as Arthur Hughes (course text) do genuflect towards the limitations of testing, they generally 56 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 accept that the limitations they mention can be lived with, and their analysis overlooks a great deal. I am rather more skeptical of the whole process. 4. The purpose of a request for a list of DO’s and DON’Ts attaching to each skill item is not quite clear to me, (unless it is to check that I have read the text!). I do understand that there are design problems associated with test questions. The Hughes text is largely about spelling some of these out, and at various points lists of advice are given on choosing and constructing questions. Thus page 83 on gives advice on writing test construction (some of it poorly thought through in my view); page124 gives advice on speaking test construction, page 154 for reading test construction, page 164 for listening tests, page 174 for grammar, and page 180 for vocabulary. Since I have chosen to model the assignment tests on the G-TELP test as used in South Korea (see Assignment 1) , the general question formats have been largely selected for me. The assignment request for 30 questions on writing and speaking does not really fit the pattern employed by G-TELP. For those skills I have therefore constructed a set of cue questions to supplement the less explicit stimuli of a writing/ speaking topic and picture panels. 5. A recording for the testing content of the listening segment can be provided on request. A transcript of the recording is included in Appendix 3. _______________ 57 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Listening Test Tester’s note : In real life contexts, listening is usually supported by the discourse context and well as the physical situation. The listener can immediately clarify confusions, repeat elements to assist in recall, use body language and engage a variety of other stratagems and resources. the measure of listening success is in the listener’s responses (immediate or later behaviour). Information which is extensive, precise and complex is rarely delivered in a single monologue. In such contexts, speech is often used to clarify and support written material. This would almost certainly be the case with the "Telephone Duties for a Secretary" used in the recording transcripted in Appendix . For these reasons amongst others, genuine listening ability is difficult to measure reliably within the parameters of formal controlled test. It is envisaged that the testees in the context below would have access to the recording for replay for a fixed amount of time (say, 15 minutes). Extracting the information required for correct answers below would be difficult for some native speakers of English, depending upon their intelligence and education. Such variables will contaminate most second language tests also. I feel that the demands upon memory and focus required by the multiple choice style of the G-TELP test may be excessive in a listening test. However, since the test for this assignment is modeled on G-TELP I have mostly followed their format. Part B is my own innovation, but again may not be a very easy option for many testees. An easier testing option for lower level L2 testees would be to give them a transcript with missing elements for cloze completion. Marker’s note : The correct answer has been italicized in each question for the purposes of this assignment presentation. 58 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Recording : Telephone Duties for a Secretary [Appendix 1 ] Part A : Multiple Choice Questions Instructions : Choose the best answer for each question. The best answer is what you hear on the recording. The best answer is NOT what you guess from experience. Circle a, b or c in each of the following questions. 1. When you answer the telephone, you should a. speak kindly b. speak quietly c. speak clearly 2. The rules apply to a. all employees b. all secretaries c. all office staff 3. You give the caller an option a. of waiting or being called back b. of waiting or calling back c. of waiting or calling another number 4. Give progress reports a. every 30 minutes b. every 3 minutes c. every 30 seconds 5. When you return to the phone a. tell the caller to ring later b. thank the caller for waiting c. thank the caller for their call 59 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 6. You should take accurate messages a. to avoid confusion later b. to avoid criticism later c. to avoid calls later 7. Your record of a telephone message should include a. the time to call back b. the time of the call c. the time the call lasted 8. Tell the caller a. when the person they want is at work b. when the person they want is leaving c. when the person they want is expected 9. Offer the caller information a. if it is inappropriate b. if it is appropriate c. if it is accurate 10. Do not give the caller promises about a. when the call will be returned b. when the call will be reviewed c. when the call will be rerouted Part B - Dialogue Choice Instructions : a) Imagine that you are a manager’s secretary. You have to follow the procedures on the recording. b) From the list below, choose the ten best statements you can make. 1. "This is Jane Smith. How can I help you?" 60 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 2. "Cooperative Extension, Cook County Office, Jane speaking." 3. "Hello. Cooperative Extension Office, extension 594." 4. "Will you hold on, or will I call you back?" 5. "Hold the line please." 6. "The manager is busy now. Please call back later." 7. "You can give details when you talk to the manager." 8. "May I check that again? You want me to tell the manager you got a post card from someone called Sadie?" 9. "I don’t know anything about Sadie, sorry. However I will tell the manager you called." 10. "This morning the manager has been called out on an emergency. You can reach him at 137 5557 8352." 11. "This morning the manager is visiting his mother in hospital. You can reach him at 137 5528 1935." 12. "This morning you can reach the manager at our Orange County branch. The number is 137 5521 1689." 13. "Is there anyone else here who can help you?" 14. "I’m sorry. I can’t transfer you. You will have to ring the sales supervisor directly." 15. "That other person you mentioned works in the warehouse. I don’t know his number. Sorry." 16. "I will tell the manager to call you at 11 a.m. I’m sure he can help you." 17. "The manager should be here about 11 a.m. I will tell him about your call." 18. "The manager is sure to be in by 11 a.m. He will call you then." 19. "The person you need is Mr Jones in Sales. Would you like me to transfer you?" 20. " The person you need is in Sales. I will transfer you now." 21. "The person you need is in another department. Hang on a moment and I 61 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 will transfer you." 22. "You have the wrong extension sir. I’ll put you back to the switchboard." 23. " I don’t know why the call center directed you here sir. We can’t help you." 24. " I’m afraid Mr White can’t help you sir. If you can wait a few moments, I will find you the right person to speak to." 25. "If you like, I will transfer you to Mr Walters, who knows about this problem. His extension is 480." 26. "Just hang on while I put you through to Mr Walters." 27. " Call the main number again sir, and press 7 on the second beep." 28. "Hello; I want to make a booking for my boss." 29. "Hi, I’m Sarah. My boss needs an air ticket." 30. "Hello, my name is Sarah Winter from Agricultural Services. Can I make a booking please?" Reading Test [ see Appendix B for the text of the passage to be read] Tester's note on reading tests in a second language : Most reading tests, at bottom, are comprehension tests. However "comprehension" is only partially a learned skill, and only partially related to functional language knowledge. It is intimately related to the underlying aptitudes of the readers, to their learned or native capacity for inference, to their cultural presuppositions and the patterns of thinking supported by their foundation culture. In other words, any reading test is only incidentally a test of those language skills which are a necessary but not sufficient component for success. Of course, such a diffuse instrument may be suitable for the purposes of various end users. For example, both universities and employers require a certain amalgam of abilities. Nevertheless, the outcomes of such 62 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 tests may give only ambiguous information to both language learners and language teachers. Traditionally reading, or comprehension tests have been layered in sophistication by the use of questions which are purely factual, which require simple inference, or which require deduction from fairly complex arguments. A test which is trying to evaluate purely second language knowledge faces extreme difficulties here. Both habits and patterns of inference are greatly influenced by cultural background. Thus the safest and easiest questions are those which are fairly literally tied to the print on the page (although ability with these will not necessarily predict success in university courses etc. requiring critical evaluation). Texts can have other purely linguistic uses in language classes. For example students can be asked to identify certain word classes, or certain kinds of syntactic constructions, or certain kinds of language functions. Although this kind of exercise can be structured as a test, it is not obvious that the results of any test along these lines would convey useful information about the subject’s ability to decode textual information in a practical manner. The questions used in this assignment are mostly comprehension questions of a fairly literal type. Some vocabulary query items are added at the end. For the purposes of the assignment, the correct answers are italicized. Instructions Read the material from a KNTO tourist brochure. Choose the best answer in each multiple choice question. The best answer here is what the article says or implies, not what you think personally. In each question, circle a, b or c. 1. The quality tag on clothing tells you 63 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 a) that the item was made in South Korea b) that the item has been checked for faults c) that the item is a good buy 2. The tourist brochure advises you to check clothing for a) careful stitching b) whether it fits you c) whether it is fashionable 3. A good place to buy clothing is a) In front of Korea University b) In Itaewon shopping district c) Around Chongno-sam Ga 4. You should keep your receipt a) to prove that you bought the item if challenged b) to claim a tax refund c) to make it easy to return or exchange goods 5. Before you buy leather, you should a) condsider if you really need it b) consider the price c) consider how long it will last 6. A good place to find leather goods is a) hotel shops b) duty free shops c) department stores 7. Tourists prefer Itaewon because a) there are high quality goods at reasonable prices b) the shopkeepers speak English c) it is recommended by the tourist office 64 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 8. Custom goods can be ordered at a) leather factories b) some high class shops c) some shops owned by leather factories 9. When you are ordering custom items a) you are shown many designs to choose from b) it is best to take pictures of a design you like c) it is best to take professional advice 10. You should research the model number of electronic items a) before you buy anything b) after you see something you like c) before you come to Korea 11. With notebook computers a) the good models will suit everyone’s needs b) different computers are best for different kinds of people c) price is a good guide to quality 12. A five magapixel digital camera a) is necessary for professional standard photography b) is probably best for professional standard photograhy c) is recommended by the Tourist Office for professional standard photography 13. For digital still photography a) some camcorders can be used b) most camcorders can be used c) all camcorders can be used 14. Technomart sells a) only digital cameras b) goods of all kinds 65 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 c) all kinds of electronic goods 15. Youngsan Electronics Mart a) mostly sells used goods b) is famous for used goods c) contains both new and used goods 16. Shops show that they prefer cash sales a) by refusing most credit cards b) by offering discounts on cash purchases c) by offering gift vouchers with cash purchases 17. Cell phone chains are popular gifts because a) they are beautifully designed b) they are traditionally given c) they have a practical use 18. Myeongdong is less popular than Insadong to buy a) traditional cell phone chains b) trendy cell phone chains c) expensive cell phone chains 19. Insadong has a lot of a) art galleries b) tourists c) museums 20. Postcards are suitable souvenirs because a) they are easy to buy b) they are easy to carry c) they show many Korean themes 21. In this reading passage [paragraph 1], "on the fabric" means 66 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 a) printed on the garment you are buying b) about the material in the garment you are buying c) on the information tag about the garment you are buying. 22. In this reading passage [paragraph 1], "outlets" means a) places where water or other liquids come out of a pipe. b) ways people use to express their talents, or relieve tension c) places where things are sold 23. In this reading passage [paragraph 1], "case" means a) a kind of solid bag for carrying things in b) a job involving one patient or client, worked on by a professional person c) a circumstance, a situation 24. In this reading passage [paragraph 2], "custom" means a) the way people usually do things b) a kind of tax c) goods made for the special individual needs of a buyer 25. In this reading passage [paragraph 2], "sample" means a. an example of a particular product or design b. to try something out c. a small container of a patient’s urine taken by a doctor Writing Test [ see Appendix 3 for the picture story panel used to cue testees in the second testing option for this component] Tester’s Note : The elements which contribute to successful writing in any language may be partially catalogued, but their individual importance within certain tolerances is not constant. The total process includes a writer, a text, and readers. Each act of reading activates this tripolar relationship in a unique 67 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 way. Thus, one writer may write with apparent technical mastery but fail to engage readers, while another may seem clumsy but superbly effective. With the preceding in mind, it is not surprising that markers of academic writing show huge inconsistencies both in their own marking and compared to colleagues. Equally, ’testers’ of writing show very low inter-tester reliability unless the agreed criteria are so narrow and mechanical that the overall product of writing is not really evaluated at all. There are genres of writing where certain criteria must be met to qualify a piece as a candidate for acceptance. The poet writing sonnets must conform to the accepted pattern of sonnets. In a more contemporary context, industrial societies have given rise to a profession of ’technical writers’. Technical writers work within very tight specifications of style and content. They will be assessed accordingly. It may be no accident that such technical literature is widely considered to be almost unreadable, and only consulted under duress; (most people with industrial experience are aware of this, including technical writers themselves). The peer assessment of academic literature also leads to a conformity of style with covert criteria which are less concerned with genuine communication than prestige; (the public pretense is objectivity). The result is an infestation of literature which is scarcely read even by specialists in particular fields. Wherever material is subject to editing, whether by school teachers, news editors, fashion magazines or romance novel publishers, then the pattern of acceptable language is delimited. The restrictions on acceptable writing just discussed are not irrelevant to the ’testing’ of writing of second language learners. In addition to all the usual elements of style and content, the L2 writer is open to judgement on his or her technical ability to employ appropriate syntax and vocabulary, as well as manage register and the expected discourse organization. These may be assessed, as they are in speaking, listening and reading. Speaking and writing are so-called active skills in that memory is less prompted by an external text or discourse. On the other hand, the speaker or writer is free to employ vocabulary and structure suitable to their L2 mastery, while they can be overwhelmed by complexity in listening and reading. Unfortunately such 68 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 speaker/writer choice of complexity is not a reliable guide to mastery. Extremely able people may choose great economy of expression. There are two alternative sets of test content outlined below. The first alternative requires candidates to construct a business e-mail containing considerable detail. This is a practical task likely to be understood by most tertiary South Korean students. An extended set of cue questions is provided below. In practice most business e-mails are likely to be much shorter than the cue questions suggest. Most South Koreans do not do a lot of reading or writing, even in Korean, beyond the briefest employment or academic requirements. They prefer the telephone. E-mails I have seen (including some from employers) which should have detailed complex information, tended to be cryptic to the point of enigma, and often lacked even a signature. The second alternative writing test is lifted directly from the speaking test. It heavily cues the production of particular elements in the writing of candidates. The cues take the form of a series of picture panels, as well as a long list of elements which need to be communicated. Although such a process does indeed test whether the student has sufficient mastery to satisfy the cues (hence making marking easy), it adds up to a poor test of overall writing ability. This is because in a genuine writing situation the writer must identify and marshal such cues, or key points, himself. The picture panels used for cueing are taken from an excellent ESL book, "Storylines ... picture sequences for language practice" by Mark Fletcher and David Birt, page 36; published by Longman 1983. 69 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 1. First Writing Test Instructions Write an email to a foreign businessman who will your company. Tell him how he will be met at Incheon Airport. State times, dates, places, and give him an itinerary. This is his first visit to South Korea. Give a brief description of Chungju and local information about your company. Here is a list of questions to suggest some information which your e-mail could contain. You will need to organize your information. The questions below are not in any special order. 1. What is the proper greeting format to a stranger in a business e-mail? 2. Who are you and what is your role in the company? 3. What does the foreign businessman already know about his itinerary? 4. Are you going to TELL him his itinerary, or make suggestions? 5. Who will meet him at Incheon Airport? 6. What time will his flight be arriving? 7. What is the weather like in South Korea at this time of year, and what clothing should he bring? 8. How long should he expect passport and customs formalities to take? 9. Is it alright for him to bring product samples in his hand luggage into South Korea, or should they be packed separately? 10. Where will he be met in Incheon Airport? 11. Is there anything he needs to know about the layout and services of the airport if he is not met for some reason? 12. What is the cell phone number of the person who will meet him? 13. What does the person who will meet him look like? 70 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 14. How long will it take him to get from Incheon to Chungju? 15. What transport will be used for that trip? 16. Where will he be staying in Chungju? 17. What are company and accommodation contact numbers in Chungju? 18. How many days will his itinerary last? 19. What will he be doing on each day of his itinerary? 20. Will he be provided with a car, or a driver? 21. What seminars or other presentations (if any) will he be asked to give in South Korea? 22. Who will he be meeting during his stay? 23. Who speaks English in your company? 24. What are some useful facts he should know about your company? 25. What are some useful facts he should know about Chungju? 26. How far is Chungju from Seoul and what are the transport connections? 27. What are some special attractions of the Chungju area? 28. What educational and research facilities, and what libraries are available near Chungju? 29. What internet, telephone or other electronic services are available to visitors? 30. What banking services are available to visitors? 71 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 2. Alternative Writing Test [ based on material used in the Speaking Test] Part A Instructions Study the panel of eight pictures and diagrams. Use the panel to help you write a short history of smallpox vaccine. The marker must also find the information below in your writing. However, just answering the questions is not enough to make your essay. The order of the questions has been mixed up. 1. Where are reserve vaccination supplies of smallpox vaccine kept? 2. Who was the discoverer of smallpox vaccine? 3. Why was 1965 an important date in the history of smallpox eradication? 4. Where was smallpox most common before vaccinations began? 5. When was smallpox finally eradicated? 6. What are possible reasons that some countries were more effective than others in the early control of smallpox? 7. What actually is vaccination? 8. Which countries continued to be seriously affected by smallpox in 1900? 9. Where do we find early records of smallpox? 10. What kind of instruments are used for vaccination? 11. What do you think the WHO is? 12. What would make a vaccination program difficult to carry out in some countries? 13. When did the discoverer of smallpox vaccine live? 14. How much do you think villagers in remote African countries knew about the worldwide smallpox eradication campaign? 15. Which countries had most successfully controlled smallpox by 1900? 16. When were the last outbreaks of smallpox in South America, Asia and Africa? 17. Why was smallpox a problem? 18. Which countries do you think might take part in WHO decisions? 19. How many reserve vaccinations are kept in stock? 20. How might the successful smallpox eradication program have affected public opinion about medical science? 72 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Part B Instructions** Imagine that you are a journalist interviewing a doctor. The doctor is working on a village vaccination program in East Africa. Make interview questions from the five statements below. Then make an extra five interview questions of your own. 1. Ask how long she has worked in East Africa. 2. Ask about living conditions in the villages. 3. Ask her what is the main difficulty of her work. 4. Ask her what she likes about the job. 5. Ask her how the vaccination program is going. + questions 6, 7, 8, 9, 10 ____________________________ ** Editorial note : the exercises in Part B are taken directly from Fletcher & Birt 1983, p. 36, as referenced below.. Speaking Test [ see Appendix 3 for the picture story panel used to cue testees in this component] Tester notes : Except for the very special genres of speech making and private inner speech, speaking is normally done in a discourse context, assisted by both turn-taking cues and body language (unless it is a phone conversation). Intonation plays a central role, and there is an instant opportunity for requesting clarification and repetition. Thus although the test below is stimulated by a picture panel, it is a thoroughly unreal test of discourse performance. Candidates may be realistically more, or less, 73 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 articulate than the test suggests. As every university tutor knows, getting even an average native speaker to perform coherently in the the formal context of a tutorial is often extremely difficult. Probably the only genuine test of "speaking ability" is to tag someone electronically twenty-four hours a day for a week. That is scarcely amenable to a controlled testing format, even ignoring other practicalities. The questions below are provided essentially for the requirements of this assignment. Most speaking tests of the picture stimulation kind do not provide the additional cues of pre-formatted questions (the G-TELP test does not). It would be possible to create an interview format where the tester did provide some substitute for normal discourse exchange in the form of questions. Unless the questioning process was strictly structured, this would introduce new variables and hence reduce inter-rater reliability (probably one reason that large standardized tests like G-TELP tend to forego the interview format). Of course, asking a testee to speak without question cueing is a greater test of creativity, which is only partially related to the L2 situation. PART A Instructions : Study the panel of eight pictures and diagrams. Use the panel to help describe the history of smallpox vaccine. Here is some of the information the tester will be listening for. The questions below are only suggestions 1. When was smallpox finally eradicated? 2. Why was smallpox a problem? 3. Where do we find early records of smallpox? 4. Who was the discoverer of smallpox vaccine? 5. When did he live? 6. What actually is vaccination? 74 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 7. What kind of instruments are used for vaccination? 8. Where was smallpox most common before vaccinations began? 9. Which countries had most successfully controlled smallpox by 1900? 10.Which 11. countries continued to be seriously affected by smallpox in 1900? What are possible reasons that some countries were more effective than others in the early control of smallpox? 12. What would make a vaccination program difficult to carry out in some countries? 13. How might the successful smallpox eradication program have affected public opinion about medical science? 14. Why was 1965 an important date in the history of smallpox eradication? 15. What do you think the WHO is? 16. Which countries do you think might take part in WHO decisions? 17. How much do you think villagers in remote African countries knew about the worldwide smallpox eradication campaign? 18. When were the last outbreaks of smallpox in South America, Asia and Africa? 19. Where are reserve vaccination supplies of smallpox vaccine kept? 20. How many reserve vaccinations are kept in stock? Part B Instructions** Imagine that you are a journalist interviewing a doctor. The doctor is working on a village vaccination program in East Africa. Make interview questions from the five statements below. Then make an extra five interview questions of your own. 1. Ask how long she has worked in East Africa. 2. Ask about living conditions in the villages. 3. Ask her what is the main difficulty of her work. 4. Ask her what she likes about the job. 5. Ask her how the vaccination program is going. 75 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 + questions 6, 7, 8, 9, 10 ____________________________ ** Editorial note : the exercises in Part B are taken directly from Fletcher & Birt 1983, p. 36, as referenced above. Grammar Test [ see Appendix 4 for a set of G-TELP specifications relating to this segment ] Tester’s note : The underlying template for testing in this series of assignments is the G-TELP test (discussed in detail in Assignment 1). The GTELP test at intermediate level chooses to evaluate control of particular syntactic structures, in addition to those required at lower levels of the test. The relationship of these structures to any actual order of language acquisition is open to debate. Further, G-TELP testers give no indication that they distinguish between command of local elements (which are easy to test), and global factors (which influence meaning critically, but are rather more difficult to test reliably than the local variety). The G-TELP grammar questions are actually a rather unimaginative collection of multiple choice questions of the kind familiar from other large standardized tests like TOEIC. Nevertheless the questions below draw on the structures required by intermediate level GTELP, with an emphasis on those at Level 3. The tester is aware that a concentration of syntactically related structures, used contrastively in a single test can in itself cue and tutor the second language testee to reconsider answers. Although this might seem to detract from the validity of an achievement test, in the context of this test design, which is ultimately intended to coach students (see Assignment One) it is actually an advantage. 76 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Note that there is more than one possible answer to some questions in Part B. However the number of possible answers is small enough for markers competent in English to expect maximum inter-marker reliability. Questions, Part A Instructions The following items need a word or words to complete the sentence. From the four choices which follow each item, choose the best answer. Circle the letter of the correct choice. 1. In 1796 Edward Jenner ______________ a milkmaid when he realized that cowpox could immunize against smallpox. a. treated b. treating c. is treating d. was treating 2. If we take care, smallpox ____________ not kill large numbers of people again. a. would b. will have c. will d. have 3. If smallpox does not break out again, scientific cooperation __________ successfully eliminated the disease. a. will b. will have 77 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 c. would have d. would 4. If the WHO had not existed, there ____________ been no easy way to coordinate the anti-smallpox campaign. a. will b. will have c. would have d. had 5. Smallpox has now ceased to be a major health problem. a. is b. ____ [nothing] c. was d. has 6. Smallpox has been eliminated _________ doctors must continue to be alert. a. and b. or c. but d. so 7. Political organization _____ science are both needed to fight epidemics. a. and b. but c. or d. nor 8. Neither money ________ goodwill are enough to overcome major health problems. 78 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 a. and b. but c. nor d. with 9. Either a lot of luck ___________ intense research will be needed to beat the AIDS virus. a. and b. but c. nor d. or 10. Logical inquiry together _________ advanced technology is needed to solve many modern problems. a. through b. by c. with d. and 11. ______________ smallpox was eradicated, emergency supplies of vaccine were stored in Geneva and New Delhi. a. because b. after c. while d. however 12. Edward Jenner is famous ______________ he devised a vaccine to immunize against smallpox. a. although 79 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 b. when c. because d. after 13. Smallpox has apparently been eradicated _____________ we can never be sure there won’t be another outbreak somewhere. a. although b. when c. while d. after 14. Modern medical science has performed miracles. ____________, there are still many major medical challenges to overcome. a. and b. although c. while d. however 15. ____________ the anti-small pox campaign was underway, many city doctors went to remote regions to help. a. during b. although c. while d. because of 16. Smallpox caused many fatal epidemics _____________ it was eradicated. a. because b. before c. or d. during 80 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 17. Governments can always gamble that there will not be epidemics ________ not prepare for emergencies. a. but b. or c. and d. because of 18. Doctors can only treat the most urgent cases first, _____________ try to give quick help to all patients. a. and b. but c. or d. before 19. Helping remote communities is emotionally rewarding ___________ often costs doctors in lost income. a. although b. however c. and d. but 20. Mass vaccinations cannot be conducted often ___________ the high costs involved. a. when b. because c. because of d. whether 81 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Part B Instructions Join the words in columns A and C by writing a suitable word in column B. A 21 In medicine, prevention B C than trying to cure sick people is 22 Careful planning eventually costs 23 Smallpox was one of in suffering than ignoring medical risks successes of modern medicine the 24 In 1965, the antismallpox campaign was vaccination campaign ever attempted the 25 Persuading people to always very difficult cooperate 26 The smallpox vaccination campaign used as an example of international cooperation has often 27 A virus a kind of living organism 28 Defeating epidemics is difficult than curing a single patient 29 The defeat of experience for doctors epidemics is a/an 30 Not many diseases can and easily defeated by doctors be 82 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Appendix 1 Listening Skills Content : General Secretarial Duties - Management of the Telephone Source : http://www.ca.uky.edu/internal/Office%20Manual/13%20secretarial%20duties/ 13.htm - University of Kentucky County Extensions Office Transcript : Speak clearly, slowly and distinctly. How to answer and what to say - This applies to all secretaries and agents. "Cooperative Extension, Cook County Office, Jane speaking." Leaving the line - When you must leave the line to obtain information, it is courteous to give the caller the option of waiting or of being called back. You might say "Will you hold or shall I call you back?" If the caller waits, use the hold button. Give progress reports every 30 seconds and say "I’m still looking for _____." When you return to the line, thank the caller for waiting. May I take a message? - Take accurate messages to avoid misunderstandings later. Include all the information you have knowledge of: what may seem insignificant to you may be extremely important to the person receiving the message. Include date, time, your initials, the caller’s name, telephone number, the business or service represented and/or the information desired. Tell the caller when the person they want is expected or whether he/she can be reached at another location IF APPROPRIATE. Offer whatever information you can. Inquire whether anyone else can help them. Do not make promises to the caller regarding when the call will be returned. Do advise the caller when the person is expected to return or call in for messages. Say "I’ll give the message" or "I’ll tell ______ you called." Transferring calls - Tact, courtesy, and efficiency are essential when calls are 83 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 being transferred. Before transferring a call, explain to the caller that he/she is being transferred and ask him/her if he/she is are willing to be transferred. You might say, "_______ is responsible for touring information. May I transfer you to his/her extension?" Transferring tips: Provide the caller with number of the called party in case you are disconnected during the transfer process. If you are not sure about how to transfer a call, check instructions before taking action. Do not guess when transferring calls. If you are not certain to whom the call should be transferred, ask the calling party to wait a moment while you check. Consult with co-workers and find out all you can to avoid run around! Placing calls - Check the phone directory, number finder, or other lists before calling. Identify yourself and your organization immediately and state the purpose of your call. Remember time zones and avoid calling during lunch hours. Allow time for the party to answer. Leave clear, concise messages on answering machines or with answering services. At the minimum, leave your name, your organization, the date, the time, a phone number, and a brief message regarding the purpose of the call. Appendix 2 - Reading Test Source : Korea National Tourist Office - ’Things to Buy in Korea’ Clothing When purchasing clothing, consider the material, quality of the stitching, design, and price. Make sure to check for the quality tag (a tag with information on the fabric and washing instructions) on the inside of the garment to ensure that it has undergone quality control. Clothing is usually purchased in department stores, brand shops, outlets, Dongdaemun Market, Namdaemun Market, Myeongdong, Itaewon, and in front of Ehwa Women’s University. In the case of brand name goods, it is a good idea to wait for sales in January, April, July, and October (may be subject to change). Make sure to 84 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 keep your receipt in case you need to exchange or return something. Leather Goods Leather goods come in a wide variety- clothing, shoes, and bags. Before purchasing, consider the type of leather, quality of the dye, stitching, inner lining, and price. Good places to find leather goods are department stores, Namdaemun Market, Dongdaemun Market, Itaewon, and more. Tourists prefer markets at Itaewon because high quality goods are offered at reasonable prices. Many of the shops have their own factories, so custom goods can be ordered at some. When ordering a custom made item, you’re shown many samples of materials before choosing one- and it’s a good idea to take along pictures from magazines as well with designs you like. Electronics : Among items sought out by tourists, the most popular are notebook computers, digital cameras, camcorders, mp3 players, PDAs, and more. When making a purchase, you should research the name and model number of the product beforehand, and consider the following: For a notebook computer, consider the uses of the computer and whether or not it fits your lifestyle. Will you watch movies? Play games? Design Graphics? With your answers, select the processor and monitor accordingly. In the case of digital cameras, check for the megapixels and other functions. For the casual photographer, 3.0 megapixels is plenty, while 5.0 megapixels or more are recommended for professional use. With camcorders, you must also consider resolution and monitor size- some can even double as a digital camera, so check before you buy. Good places to purchase electronics are department stores, the Yongsan Electronics Market, discount marts, Technomart, and more. Of these, the Yongsan Electronics Market is the most famous- a good selection of goods for great prices. Computers, household appliances, and game software is available- there’s even a market for used goods. Keep in mind that most stores offer discounts for cash purchases. Cell Phone Chains Appreciated as a gift with actual use, cell phone chains range from cute and cheap plastic to delightful and more expensive handcrafted offerings. When purchasing as a gift, traditional Korean chains 85 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 are popular- they cost about 2~3 times more than regular chains, but many are beautifully designed with great workmanship. Trendy hand phone chains can usually be purchased in Myeongdong, Jongno, Apgujeong, Ehwa Women’s University, and more, while traditional chains can be found in Insadong. Postcards Postcards are economical and easy to transport, a perfectly convenient and appropriate souvenir. The most popular post cards have traditional Korean themed photographs or artwork, and can be purchased at sites with high tourist traffic such as Insadong, palaces, art galleries, and museums. Prices are from 500 won to 1000 won. Appendix 3 - Speaking and Writing Tests - Picture Panel Cues for text construction Source : "Storylines ... picture sequences for language practice" by Mark Fletcher and David Birt, page 36; published by Longman 1983. 86 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Appendix 4 - G-TELP Specifications for Levels 3 to 5, Grammar Tests Source : G-TELP website http://gtelp.co.kr/e_gtelp/gtelp/e_gtelp04.asp This is the G-TELP focus for structure (content extracted directly from GTELP literature) : Level 5 (lowest level) Ge ne r a l D e scr ipt ion This level assesses t he abilit y of t he t est t aker t o produce and recognize m inim al am ount s of English. The t est t aker has had lim it ed exposure t o English in t he classroom , and knows only basic elem ent s of t he language. - Gr a m m a r A t est t aker at t his level is exposed t o a variet y of basic gram m at ical st ruct ures, such as frequent ly used irregular verbs and 87 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 present t ense. He/ she is required t o dem onst rat e underst anding of basic gram m at ical st ruct ures in one- and t wo- sent ence cont ext s. St ruct ures t est ed at t his level include: * Verbs - present progressive - sim ple present - sim ple past - fut ure t ense wit h will and be going t o - const ruct ions w it h t here is/ t here are - const ruct ions w it h t here was/ t here were * Nouns - plurals - singular * Pronouns - personal pronouns: subj ect , possessive * Form at ion of Quest ions - yes/ no quest ions wit h t o be - inform at ion quest ions wit h t o be - short answers wit h t o be Level 4 descriptors Ge ne r a l D e scr ipt ion This level assesses t he abilit ies of a t est t aker t o handle basic com m unicat ive funct ions. The t est t aker is capable of basic form ulaic expressions such as t hose used in greet ings and int roduct ions, and is successful wit h basic com m unicat ive t ransact ions. Alt hough a t est t aker at t his level has had only lim it ed exposure t o English out side of t he classroom , he/ she is able t o use English in sim ple com m unicat ion wit h nat ive speakers in a narrow range of t asks. 88 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 - Gr a m m a r An exam inee at t his level is required t o dem onst rat e underst anding of basic gram m at ical st ruct ures in one- and- t wo sent ence cont ext s. St ruct ures t est ed at t his level include: * Verbs - be - com m ands - negat ives - const ruct ions w it h t here is/ t here are * Pronouns - personal pronouns: subj ect , obj ect , possessive * Form at ion of Quest ions - yes/ no quest ion w it h be, was, were, and do, does, did - short answers t o yes/ no quest ions wit h be, was, were, and do, does, did - quest ion- word quest ions ( Wh- words) Level 3 descriptors Ge ne r a l D e scr ipt ion This level assesses t he abilit ies of a t est t aker t o com m unicat e in English on an elem ent ary level. This per son has been exposed t o English m ainly in t he classroom wit h lit t le ext ended cont act wit h nat ive speakers. I t should be not ed, however, t hat while a person at t his level is not an independent user of t he language and is not expect ed t o cope wit h unm odified English, he/ she would be able t o funct ion wit hin a lim it ed range of aut hent ic English language sit uat ions. - Gr a m m a r An exam inee at t his level is required t o dem onst rat e underst anding 89 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 of basic gram m at ical st ruct ures in t wo- t o t hree- sent ence cont ext s. St ruct ures t est ed at t his level include: * Verbs - past progressive - fut ure possible condit ional wit h will, would, and would have - present perfect * Coordinat ors and Subordinat ors - and, but , or, nor, so - aft er, because, et c. Word form s - adj ect ives: equat ives, superlat ives, com parat ives ________________________ 90 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Part III - PRACTICE ENGLISH LANGUAGE TEST Le ve l : I nt e r m e dia t e (based on the G-TELP international language test) Test Instructions 1. This test is designed to give you practice in doing international language tests like G-TELP. The G-TELP test itself (http://www.gtelp.co.kr ) is an American test used in job hiring by many big companies in South Korea. 2. This test has five parts : Speaking, Listening, Reading, Writing and Grammar. 3. Because this is a practice test, we are not too worried about strict time limits. Take you time. Do your best. Be honest with yourself and the tester. The idea is to coach you to do better next time. "Cheating" won’t help you. 4. Don’t use a dictionary, don’t ask for help, and don’t do anything you couldn’t do in a real job testing situation. Tester : Thor May thormay@yahoo.com Department of English Chungju National University South Korea 91 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 1. Speaking Test Your Name : _______________ Your e-mail : ____________ Date : _____ Instructions 1. Study the panel of eight pictures and diagrams. 2. Use the picture panel and hints to help describe the history of smallpox vaccine. 3. The short hints below are only ideas; (they are not proper English sentences!). Make your own story. 4. Try to speak for several minutes (no more than 10 minutes !) 5. Record your story and send it to the tester** How to record your story : You can use a tape recorder, or record on an MP3 player, or on computer. **The tester : Thor May; e-mail : thormay@yahoo.com ; Department of English, Chungju National University, 123 Gyomdon-ri, Iryu-myon, Chungju, Chungbuk 380702, South Korea Hints For Talking About The Pictures 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. When eradicated? Why a problem? Where early records? Who discoverer ? When live Jenner ? What vaccination? What used for vaccination? Where most common before program? Which countries most successful 1900? Which countries affected 1900? What reasons some countries more effective? What make vaccination difficult ? How might success program affect public opinion? Why 1965 important ? What WHO is? Which countries WHO decisions? How much villagers Africa knew about campaign? When last outbreaks South America, Asia and Africa? Where reserve vaccination supplies ? 92 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head 20. Thor May 2013 How many vaccinations kept ? Picture Panels Showing The History of Smallpox 2. LISTENING TEST Your Name : _______________ Your e-mail : ____________ Date : _____ General Instructions : The listening test is in two sections. Each section has two parts. Section 1 The recording tells a secretary in an American office about how to use the telephone. 93 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 a) Listen carefully to the recording. You may re-play the recording to listen several times. b) Part A1 : Answer multiple choice questions about telephone duties. c) Part B1 : Choose the best answers a secretary can give to someone calling her office. Section 2 This is a conversation when a young man meets a woman in America a) Listen carefully to the recording. You may re-play the recording to listen several times. b) Part A2 : Fill in the gaps in the sentences. Only use words you hear in the recording. b) Part B2 : Answer multiple choice questions about the conversation. 2. LISTENING TEST - SECTION 1 Recording : Telephone Duties for a Secretary Part A : Multiple Choice Questions Instructions : 1. Choose the best answer for each question. 2. The best answer is what you hear on the recording. The best answer is NOT what you guess from experience. 3. Circle a, b or c in each of the following questions. 1. When you answer the telephone, you should a. speak kindly b. speak quietly c. speak clearly 2. The rules apply to a. all employees b. all secretaries c. all office staff 3. You give the caller an option a. of waiting or being called back b. of waiting or calling back c. of waiting or calling another number 94 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 4. Give progress reports a. every 30 minutes b. every 3 minutes c. every 30 seconds 5. When you return to the phone a. tell the caller to ring later b. thank the caller for waiting c. thank the caller for their call Listening - Section 1, Part B - Dialogue Choice Instructions : a) Imagine that you are a manager’s secretary. You have to answer the telephone in a certain way. b) Listen carefully to the "Telephone Duties for Secretaries". c) Now, what are the best answers you can give to a caller from the questions below ? d) Circle a, b or c in questions 1 to 5 below. 1a. "This is Jane Smith. How can I help you?" 1b. "Cooperative Extension, Cook County Office, Jane speaking." 1c. "Hello. Cooperative Extension Office, extension 594." 2a. "Will you hold on, or will I call you back?" 2b. "Hold the line please." 2c. "The manager is busy now. Please call back later." 3a. "You can give details when you talk to the manager." 3b. "May I check that again? You want me to tell the manager you got a post card from someone called Sadie?" 3c. "I don’t know anything about Sadie, sorry. However I will tell the manager you called." 4a. "This morning the manager has been called out on an emergency. You can reach him at 137 5557 8352." 4b. "This morning the manager is visiting his mother in hospital. You can reach him at 137 5528 1935." 4c. "This morning you can reach the manager at our Orange County branch. The number is 137 5521 1689." 5a. "Is there anyone else here who can help you?" 5b. "I’m sorry. I can’t transfer you. You will have to ring the sales supervisor directly." 5c. "That other person you mentioned works in the warehouse. I don’t know 95 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 his number. Sorry." LISTENING TEST - SECTION B Recording : “Haven’t We Met Before?” Listening Part A - Dialogue Completion Instructions : a) Listen to the recording carefully b) Fill in the gaps in the following sentences. Only use words you hear in the recording. 1. We ______________________ on campus last week 2. Let me guess. It’s Sherry, _______________________? 3. No, but you got the _________________ letter 4. You got it . . . and ____________________ on the fourth try. 5. Not bad. ___________________ what was your name? Listening - Section 2, Part B : Multiple Choice Questions Instructions : a) Choose the best answer for each question. b) The best answer is what you hear on the recording. The best answer is NOT what you guess from experience. c) Circle a, b or c in each question to show your choice. 1. The man a) thinks that he has the right qualifications to teach English as a second language b) wants to study to get the right qualifications to teach English as a second language b) isn’t sure if he has the right qualifications to teach English as a second language 96 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 2. What does the man want to study most at the moment? a) Marketing b) International business c) Accounting 3. The woman says a) she will miss her class b) she must hurry to get to her class c) her class began ten minutes ago 4. The man a) invites the woman to a dance b) thinks the woman is interested in dancing c) was wondering if there was a dance tonight 5. The woman a) is very interested in seeing the man again b) might be interested in seeing the man again c) doesn’t really want to see the man again 3. Reading Test Your Name : _______________ Your e-mail : ____________ Date : _____ General Instructions 1. The reading test is in two sections 2. Section 1 is advice taken from a South Korean tourist brochure 3. Section 2 is taken from an interview with a South Korean student who has moved to the United States 4. Both Section 1 and Section 2 contain multiple choice questions to check your understanding of ideas 5. Both Section 1 and Section 2 contain multiple choice vocabulary questions to check your understanding of words and phrases 97 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Reading Section 1 - A Tourist Brochure [Source : Korea National Tourist Office - ’Things to Buy in Korea’] Instructions : 1. Read the material from a KNTO tourist brochure. 2. Choose the best answer in each multiple choice question. The best answer here is what the article says or implies, not what you think personally. 3. In each question, circle a, b or c. 4. The underlined words are for vocabulary questions Things To Buy In South Korea Clothing When purchasing clothing, consider the material, quality of the stitching, design, and price. Make sure to check for the quality tag (a tag with information on the fabric and washing instructions) on the inside of the garment to ensure that it has undergone quality control. Clothing is usually purchased in department stores, brand shops, outlets, Dongdaemun Market, Namdaemun Market, Myeongdong, Itaewon, and in front of Ehwa Women’s University. In the case of brand name goods, it is a good idea to wait for sales in January, April, July, and October (may be subject to change). Make sure to keep your receipt in case you need to exchange or return something. Leather Goods Leather goods come in a wide variety- clothing, shoes, and bags. Before purchasing, consider the type of leather, quality of the dye, stitching, inner lining, and price. Good places to find leather goods are department stores, Namdaemun Market, Dongdaemun Market, Itaewon, and more. Tourists prefer markets at Itaewon because high quality goods are offered at reasonable prices. Many of the shops have their own factories, so custom goods can be ordered at some. When ordering a custom made item, you’re shown many samples of materials before choosing one- and it’s a good idea to take along pictures from magazines as well with designs you like. Electronics : Among items sought out by tourists, the most popular are notebook computers, digital cameras, camcorders, mp3 players, PDAs, and more. When making a purchase, you should research the name and model number of the product beforehand, and consider the following: For a notebook computer for example, consider the uses of the computer and whether or not it fits your lifestyle. 98 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Reading Section 1 - Multiple Choice Questions (Ideas) 1. The quality tag on clothing tells you a) that the item was made in South Korea b) that the item has been checked for faults c) that the item is a good buy 2. The tourist brochure advises you to check clothing for a) careful stitching b) whether it fits you c) whether it is fashionable 3. A good place to buy clothing is a) In front of Korea University b) In Itaewon shopping district c) Around Chongno-sam Ga 4. You should keep your receipt a) to prove that you bought the item if challenged b) to claim a tax refund c) to make it easy to return or exchange goods 5. Before you buy leather, you should a) condsider if you really need it b) consider the price c) consider how long it will last 6. A good place to find leather goods is a) hotel shops b) duty free shops c) department stores 7. Tourists prefer Itaewon because a) there are high quality goods at reasonable prices b) the shopkeepers speak English c) it is recommended by the tourist office 8. Custom goods can be ordered at a) leather factories b) some high class shops c) some shops owned by leather factories 99 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 9. When you are ordering custom items a) you are shown many designs to choose from b) it is best to take pictures of a design you like c) it is best to take professional advice 10. You should research the model number of electronic items a) before you buy anything b) after you see something you like c) before you come to Korea Reading Section 1 - Multiple Choice Questions (Vocabulary) 11. In this reading passage [paragraph 1], "on the fabric" means a) printed on the garment you are buying b) about the material in the garment you are buying c) on the information tag about the garment you are buying. 12. In this reading passage [paragraph 1], "outlets" means a) places where water or other liquids come out of a pipe. b) ways people use to express their talents, or relieve tension c) places where things are sold 13. In this reading passage [paragraph 1], "case" means a) a kind of solid bag for carrying things in b) a job involving one patient or client, worked on by a professional person c) a circumstance, a situation 14. In this reading passage [paragraph 2], "custom" means a) the way people usually do things b) a kind of tax c) goods made for the special individual needs of a buyer 15. In this reading passage [paragraph 2], "sample" means a. an example of a particular product or design b. to try something out c. a small container of a patient’s urine taken by a doctor 100 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Reading Section 2 - An Interview Instructions : 1. Read the Pacific University interview 2. Choose the best answer in each multiple choice question. The best answer here is what the interview says or implies, not what you think personally. 3. In each question, circle a, b or c. 4. The underlined words are for vocabulary questions Pacific University Interview [ http://mcel.pacificu.edu/as/students/korea/culture-interview.html ] In this interview Ben Bagley asks Theresa Han about the difference between Korean and American culture. Theresa is a teenager who recently moved to the United States so she has an excellent perspective for understanding the differences and similarities between these countries. [BAGLEY] This is Ben Bagley, and I’m going to interview Theresa Han about Korea. Could you introduce yourself? [HAN] My name is Theresa Han, I’m from South Korea, I’m 18 years old, and I’m a freshman in College. [BAGLEY] How long have you lived in America? [HAN] I think a little bit less than 3 years. [BAGLEY] Where did you live in Korea? [HAN] I lived in Pyoung Tek, It’s right below Oosan, where the American Air force is located. [BAGLEY] What were the people like where you lived? [HAN] They’re really busy. Fathers go to their work. Mothers if they have a job go to their work, and students go to school, so they don’t have enough time to communicate with each other, like time to spend together, because mostly students come home like 10:00pm-11:00pm. [BAGLEY] What did you do with your friends? [HAN] We mostly go to each other’s house, rent a movie or something, watch it, and do homework usually, because we have a whole bunch of homework. On the weekends we would go downtown; it’s kind of like a shopping mall. It’s a street. There are small restaurants, small clothes shops and all that stuff. It was kind of fun, but nothing special I think. 101 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Reading Section 2 - Multiple Choice Questions (Ideas) 1. Which of these statements is true ? a) Theresa spent most of her life in Korea b) Theresa spent most of her life in America c) Theresa spent about equal time in Korea and America 2. Which of these statements is true ? a) The American Air Force base is in Pyoung Tek b) The American Air Force base is below Oosan c) Theresa lived near The American Air Force base 3. Which of these statements is true ? a) Students in Pyoung Tek see a lot of their mother and father b) Students in Pyoung Tek are not good communicators c) Students in Pyoung Tek are too busy to communicate much with their families 4. Which of these statements is true ? a) Students visit each other’s houses both to watch movies and to do homework b) Students visit each other’s houses mostly to do homework c) Students visit each other’s houses mostly to watch movies 5. Which of these statements is true ? a) Downtown Pyoung Tek is a very exciting place b) Downtown Pyoung Tek has streets of small shops c) Downtown Pyoung Tek has a single street of small shops Reading Section 2 - Multiple Choice Questions (Vocabulary) 6. In this reading passage, "perspective" means a) skill b) reason c) viewpoint 7. In this reading passage, "similarities" means a) problems b) things rather like each other c) things the same 102 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 8. In this reading passage, "right below" means a) just a short distance south on the map b) a little less important c) a part of a town 9. In this reading passage, "have a whole bunch of" means a) have very important b) have a lot of c) have to do 10. In this reading passage, "all that stuff" means a) things like that b) things to do c) things we liked 4. Writing Test Your Name : _______________ Your e-mail : ____________ Date : _____ Instructions 1. Write a detailed email to a foreign businessman who will visit your company. 2. Tell him how he will be met at Incheon Airport**. State times, dates, places, and give him an itinerary. 3. This is his first visit to South Korea. 4. Give a brief description of Chungju and local information about your company. 5. ** If you don’t know South Korea well, or don’t know Chungju, you can use another country or city in your e-mail instead. 6. Below is a list of short hints. The hints suggest some information which your e-mail could contain. You don’t have to use all of them; (the hints are not proper English sentences !). Hints For Writing A Business E-mail 103 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 1. Proper greeting format to stranger in business e-mail? 2. Who you ? What your role ? 3. What foreign businessman already know => itinerary? 4. You TELL him itinerary, or just suggestions? 5. Who meet him Incheon ? 6. What time flight ? 7. What weather like South Korea this time ? Clothing ? 8. How long passport and customs ? 9. Alright product samples in hand luggage ? Packed separately? 10. Where met in Incheon ? 11. Airport layout, services if not met ? 12. What cell phone number person meet him? 13. What person meet him look like? 14. How long from Incheon => Chungju? 15. What transport that trip? 16. Where staying Chungju? 17. Company, accommodation contact numbers Chungju? 18. How many days itinerary ? 19. What doing each day ? 20. Provided with car, driver? 21. Seminars, presentations to give ? 22. Who meeting during stay? 23. Who speak English your company? 24. Useful facts about company? 25. Useful facts about Chungju? 26. How far Chungju => Seoul; what transport connections? 27. Special attractions Chungju area? 28. Educational, research facilities, libraries near Chungju? 29. Internet, telephone, other electronic services fo visitors? 30. Banking services for visitors? 31. Signature format 104 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 5. Grammar Test Your Name : _______________ Your e-mail : ____________ Date : _____ 1. The grammar test has 40 questions in three sections. 2. Section A asks you to complete multiple choice questions. 3. Section B asks you to join two phrases or sentences with a correct word. 4. Section C asks you to correct some sentences which may have grammatical errors. Grammar - Section A (Multiple Choice Questions) Instructions 1. Each sentence below needs a word or words to finish it. 2. Choose the best answer for the missing word in each sentence. 3. Circle the letter of the correct choice. 1. In 1796 Edward Jenner ______________ a milkmaid when he realized that cowpox could immunize against smallpox. a. treated b. treating c. is treating d. was treating 2. If we take care, smallpox ____________ not kill large numbers of people again. a. would b. will have c. will d. have 3. If smallpox does not break out again, scientific cooperation __________ successfully eliminated the disease. a. will b. will have c. would have d. would 105 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 4. If the WHO had not existed, there ____________ been no easy way to coordinate the anti-smallpox campaign. a. will b. will have c. would have d. had 5. Smallpox has now ceased to be a major health problem. a. is b. ____ [nothing] c. was d. has 6. Smallpox has been eliminated _________ doctors must continue to be alert. a. and b. or c. but d. so 7. Political organization _____ science are both needed to fight epidemics. a. and b. but c. or d. nor 8. Neither money ________ goodwill are enough to overcome major health problems. a. and b. but c. nor d. with 9. Either a lot of luck ___________ intense research will be needed to beat the AIDS virus. a. and b. but c. nor d. or 106 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 10. Logical inquiry together _________ advanced technology is needed to solve many modern problems. a. through b. by c. with d. and 11. ______________ smallpox was eradicated, emergency supplies of vaccine were stored in Geneva and New Delhi. a. because b. after c. while d. however 12. Edward Jenner is famous ______________ he devised a vaccine to immunize against smallpox. a. although b. when c. because d. after 13. Smallpox has apparently been eradicated _____________ we can never be sure there won’t be another outbreak somewhere. a. although b. when c. while d. after 14. Modern medical science has performed miracles. ____________, there are still many major medical challenges to overcome. a. and b. although c. while d. however 15. ____________ the anti-small pox campaign was underway, many city doctors went to remote regions to help. a. during b. although c. while d. because of 16. Smallpox caused many fatal epidemics _____________ it was eradicated. 107 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 a. because b. before c. or d. during 17. Governments can always gamble that there will not be epidemics ________ not prepare for emergencies. a. but b. or c. and d. because of 18. Doctors can only treat the most urgent cases first, _____________ try to give quick help to all patients. a. and b. but c. or d. before 19. Helping remote communities is emotionally rewarding ___________ often costs doctors in lost income. a. although b. however c. and d. but 20. Mass vaccinations cannot be conducted often ___________ the high costs involved. a. when b. because c. because of d. whether Grammar - Section B (Phrase Completion) Instructions 1. Join the words in columns A and C by writing a suitable word in column B. A 21 In medicine, prevention is B C than trying to cure sick people 108 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head 22 Careful planning eventually costs 23 Smallpox was one of the 24 In 1965, the antismallpox campaign was the 25 Persuading people to cooperate 26 The smallpox vaccination campaign has often 27 A virus 28 Defeating epidemics is 29 The defeat of epidemics is a/an 30 Not many diseases can be Thor May 2013 in suffering than ignoring medical risks successes of modern medicine vaccination campaign ever attempted always very difficult used as an example of international cooperation a kind of living organism difficult than curing a single patient experience for doctors and easily defeated by doctors Grammar - Section C (Sentence Correction) Instructions 1. Some of the sentences in Part C have incorrect grammar. 2. Write out the sentence in each question again. If there is a grammar problem, fix it. If there is no problem, don’t change anthing. 31. An active president has chosen our country. Answer : _________________________________________ 32. Escaped the professor from prison. Answer : _________________________________________ 33. The neighbours say both children he teases. Answer : _________________________________________ 34. Surprises me that he wears a wig. Answer : _________________________________________ 35. There was a very tall man here for you. 109 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Answer : _________________________________________ 36. She has been smoking, isn’t it ? Answer : _________________________________________ 37. I must can catch this train. Answer : _________________________________________ 38. Three packages sent yesterday. Answer : _________________________________________ 39. The secretary won’t be here today. Answer : _________________________________________ 40. Why we bow to each other ? Answer : _________________________________________ This is the end of the test. Thank you very much for trying. 110 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 University of Newcastle, NSW Assignment 3 : Topic : Do a trial test administration, collect the data and analyse the results. In about 1000 - 1200 words discuss practical aspects of language test construction, then critically analyse your own test considering the following : - purpose of the test - content of the test - test specifications - instructions for candidates - quality of individual test items (e.g. ambiguous or not) - representativeness of the sample - reliability and validity - scoring procedures - aspects of moderation - a summary evaluation of the test => Does your test measure the language ability it is designed to measure? 111 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Practical Aspects of Language Test Construction 1. The notion of testing, and practical consequences Most people undert ake scholast ic t est s, including language t est s, wit h t he ut m ost reluct ance. I n t he case of language t est s, t hey oft en have a st rong expect at ion of failure, and t hey fear t he consequences of failure. More oft en t han not , t heir expect at ions becom e a self- fulfilling prophecy. I f t he ult im at e success crit erion is som e level of useful funct ional abilit y in L2 as a foreign language ( t he st ory wit h im m igrant acquisit ion of English is different ) , t hen t he real failure rat e in count ries like Aust ralia probably st ands at over 90% ; ( e.g. see Asher’s est im at e for Am erica) . Most people enj oy playing gam es. These m ay be physical sport s, t radit ional indoor gam es of skill or luck, or m ore recent ly ext ended and ext rem ely elaborat e gam es t hrough elect ronic m edia such as t he int ernet . Many of t hese gam es involve ext ensive and ext ended learning. I t is charact erist ic of nearly all gam es t hat success is m easured in t he form of som e kind of t est ( it m ay be called a cont est or com pet it ion) . Players are rarely crushed or psychologically dam aged in gam e failur e. Rat her, t hey t end t o be spurred t o great er effort and event ually succeed. What is t he key difference bet ween academ ic language learning/ t est ing and gam e playing/ t est ing? I t is not com plexit y, and it is not durat ion. I t is not in t he t echnical cont ent of what is being learned or m anipulat ed. The key difference is socio- polit ical. That is, academ ic language learners and gam e players have ut t erly different percept ions of t heir roles in t he process, and t heir beliefs ar e confirm ed by an ext ensive cult ural infrast ruct ure of rewards and 112 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 punishm ent s. These ext end far beyond t he learning process it self, and frequent ly have no int r insic t ie t o it . I t is t his role percept ion which causes psychological dam age, and it is t he role percept ion which is as t he heart of t he overwhelm ing public failure of academ ic foreign language t eaching. The language t est in it s various form s is a principal inst rum ent of socio- polit ical punishm ent . 2. Test Construction Language t eachers and language t est ers are generally wellint ent ioned groups of people. My t hirt y years of cont act wit h t hese folk has led m e t o t he general conclusion t hat m ost of t hem have an ext rem ely sim ple idea of what a nat ural language is com prised of, what m ent al processes are involved in it s generat ion, and how it is best t aught t o a learner. This does not m ean t hat t hey are bad language t eachers. I f t hey can keep learners engaged in t he t arget language t hey have a useful role, even if t hey don’t know what is really going on. However, it is fair t o say t hat nearly all language t est s will be const ruct ed as a com prom ise bet ween t eacher folk wisdom and t he polit ical requirem ent s of a workplace or cult ure. Crit eria like validit y and reliabilit y w ill be ult im at ely grounded in t he convent ional wisdom of language pedagogy as it exist s wit hin a local polit ical cult ur e; ( t hus conclusions could vary som ewhat bet ween, say, Aust r alia and Sout h Korea) . That is t he t est er m ay m axim ise obj ect ive procedures and st rive for consist ency. However, t he underlying decisions about what quest ions t o ask and answer will always be subj ect ive ( t his is t rue of all science) . The Pract ice Test of t his assignm ent has been const ruct ed in t he convent ional m anner, based in it s st ruct ure ( but not it s purpose) on 113 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 an int ernat ional crit erion- referenced public language t est , t he GTELP. 3 . Com pa r ison of t he Pr a ct ice Te st a nd G- TELP : G- TELP Proficiency t est for em ploym ent PRACTI CE TEST Diagnost ic t est of current proficiency, int ended t o encourage focused st udy Mixt ure of direct and indirect Mixt ure of direct and indirect m easurem ent m easurem ent Discret e point ex cept for speaking & Discret e point ex cept for speaking & writ ing t est s; non- discret e elem ent s writ ing t est s; non- discret e elem ent s int egrat ive int egrat ive G- TELP Crit erion referenced ( supposedly) PRACTI CE TEST ‘ Negot iable’ : scoring is not com piled for t he ov erall t est . I ndividual elem ent s are discussed and explored, post - t est , wit h t he t est ee The t est has been m oderat ed by a body The t est has been designed and of nat ive and non- nat ive t est ing ex ecut ed by a single t est er on a personnel, using t est result s from a m inim um num ber of st udent s large populat ion of int ernat ional st udent s The t est is conduct ed under cont rolled For logist ical reasons t he t est has been t est condit ions self- adm inist ered by t est ees at a locat ion rem ot e from t he t est er Claim s t o be m ainly obj ect ive; Discret e point elem ent s are m ost ly subj ect ive j udgem ent s subj ect t o obj ect ively m ark ed, subj ect t o som e m oderat ion wit h ot her t est ers cult ural int erpret at ion. Subj ect ive j udgem ent s are discussed post - t est wit h t he t est ee Measurem ent scale : ordinal scales 1 t o A com prehensive m easurem ent scale is 5 for speaking, list ening, reading ( & irrelevant t o t he purpose of t he Pract ice vocabulary) , writ ing and gram m ar. 5 on Test . The t est cont ent is m odelled on t he scale represent s beginner. I nt erval subset t ak en from Lev el 3 G- TELP scale t est scores are convert ed t o an ( int erm ediat e level) . 114 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 ordinal m ast ery m at rix. “ Mast ery ” is a score of 75% or gr eat er in all skill areas; [ t his crit erion is apparent ly arbit rary] Validit y : G- TELP claim s face validit y The sam ple of t est ees has been t oo based on t he cont inued pat ronage of sm all t o est ablish any kind of form al large Korean com panies. Neit her validit y. The t est er is dubious t hat t rue cont ent validit y or concurrent validit y const ruct validit y can be est ablished for are publicly quant ified however. There any language t est : t here are t oo m any seem t o be no published st udies of t he int eract ing variables. predict ive validit y of t he t est . Reliabilit y : G- TELP publishes no - The sam ple of t est ees has been t oo coefficient s of reliabilit y. However, as a sm all t o est ablish any m easure of large professional t est ing organizat ion form al reliabilit y. - Wit h a single scorer one w ould expect t hat t heir t est design t here is no issue of int er - scorer would be closely m onit ored for bot h reliabilit y form al validit y and form al reliabilit y - Som e of t he elicit at ion t asks ( e.g. writ ing) disclosed problem s t hat are inherent ly int ract able for t est m easurem ent , and hence resist ant t o m easures of validit y and r eliabilit y. 4. The Purpose of the Practice Test The Pract ice Test is a diagnost ic t est of current proficiency, int ended t o encourage focused st udy. This purpose falls out side t he norm al range of form al t est specificat ions recognized in t ext books. However, in non- English speaking count ries worldwide t her e is a m aj or publishing and coaching indust ry m aking use of sam ples and im it at ions from int ernat ional st andard t est s like TEOFL and I ELTS. I n net effect , t hese big t est s are vehicles t o j ust ify t he coaching indust ry. The Pract ice Test is an at t em pt t o harness t his phenom ena. 115 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 5. The Application of the Practice Test applied to its purpose Twelve individuals m ade som e prom ise t o t ake part in an init ial t rial of t he Pract ice Test . I n act ualit y only one part icipat ed, at a rem ot e locat ion under condit ions of self- adm inist rat ion. The wit hdrawals were influenced by local condit ions ( exam s m ainly) , but also by a percept ion t hat t he Pract ice Test was long and t im e- consum ing. Lengt h is a dilem m a in all t est ing, and becom es acut e where volunt ary part icipat ion is expect ed. The Korean individual who did part icipat e found t he exercise, and a follow up analysis of his perform ance t o be very useful. See t he em ail in Appendix 2. This was an indicat ion t hat t he Pract ice Test had succeeded in it s explicit purpose as a “ diagnost ic t est of current proficiency, int ended t o encourage focused st udy” . 6. The Content of the Test The brief was t o com pose a “ com prehensive ESL t est ” . As t his t est er underst ands t he expression, a com prehensive ESL t est is im possible in principle. No language t est is able t o t est L2 com prehensively. The Pract ice Test aim ed t o be a useful ESL t est . Useful t hat is t o bot h t he t est ee( s) and t he t est er. I n order t o achieve t his aim it had t o be short enough t o encourage part icipat ion, and focused enough t o show up a pat t ern of weakness where one exist ed. One out com e was t hat it was perhaps t oo long t o encourage part icipat ion under volunt ary condit ions. 116 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 The focus, which was effect ively t hat of G- TELP level 3, was indeed fine enough t o show up som e pat t erns of weakness in t he L2 usage of t he part icipat ing t est ee. For exam ple, by rest rict ing it self t o a fairly narrow band of synt act ic const ruct ions and probing t hese in som e dept h, it becam e clear t hat t he t est ee had a problem wit h t he logical cont ent of English conj unct ions. See Appendix 1 for a det ailed exam inat ion of errors and infelicit ies, followed by t est er com m ent s. The gram m ar t est also included a sm all select ion from Burt & Kiparsky’s collect ion of ‘ goofs’ ( The Gooficon’, 1972) , wit h inst ruct ions t o correct t hem . An open- ended exercise like t his is anat hem a t o t he design of form al st andardized t est s ( t he out com es are t oo variable) , but are quit e useful for diagnost ic purposes. The Writ ing and Speaking sect ions of t he Pract ice Test could not in t heir nat ure be discret e point checks on L2 usage. The t est ee’s perform ance in t hese sect ions did not show serious problem s wit h t he elem ent s norm ally t arget ed in language t est s. However t here were ser ious discourse problem s which could render t he t est ee linguist ically unfit in, for exam ple, a business environm ent . The kind of discourse problem s ident ified could probably not be reliably t est ed for in a form al, m ass t est ing environm ent . I n ot her words, t he discourse problem s showed up t he lim it at ions of t he whole t est ing process. See Appendix 1 for an explicit analysis. 7. Test Specifications The specificat ions of t he Pract ice Test were broadly set by t he GTELP specificat ions. Since G- TELP is a very professional undert aking, t he specificat ions w ere not likely t o be badly ast ray. However t he 117 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 adapt at ion of G- TELP specificat ions t o a new diagnost ic and incent ive- generat ing purpose was experim ent al. Som e sm all deviat ions from t he G- TELP quest ion pat t erning ( e.g. t he Gooficon ent ries) were clearly out side of st andard specificat ions, but t hought t o be helpful. Wit h only a single t est ee, post - evaluat ion is difficult , but t here seem t o have been no m aj or problem s. 8. Instructions to Candidates I nst ruct ions t o candidat es t ook t hree form s. First ly t here was a Candidat e I nst ruct ion Booklet . This sum m arized inst ruct ions in t he act ual t est and provided som e pre- t est exam ples. Candidat es were also referred t o t he G- TELP web sit e. The second level of inst ruct ion was w it hin t he t est it self, before each set of t est it em s. The t est er t ried t o m ake t hese inst ruct ions as direct and unam biguous as possible. The single t est ee who finally part icipat ed report ed no problem s wit h t he inst ruct ions, alt hough he did om it a w hole sect ion of t he list ening t est . I underst and t hat t his was an oversight . The t hird level of inst ruct ion was inform al and pre- t est . I t was m ore general in nat ure t han t he t est - int ernal inst ruct ions, and was necessary since m ost pot ent ial candidat es were part icipat ing rem ot ely and had t o be reached by e- m ail. Technical inst ruct ions on how t o handle downloads, e- m ail at t achm ent s, and ( especially) recording were necessary. I kept it sim ple, but I believe t hat one t echno- allergic individual, 500km away, was scared off by t he m echanics of part icipat ing elect ronically. 118 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 9. The Quality of Individual Test Items The t est er would have liked bot h explicit feedback and st at ist ical evidence on t he qualit y of individual t est it em s. The single part icipant did not r eport any problem s. Post - t est , t he t est er did discover t hat one m ult iple choice it em had been m isprint ed, rendering it invalid. Since no overall t est score was being com piled, t his was not crit ical. 10. The Representativeness of the Sample The sam ple of English language em bedded in t he Pract ice Test is a t iny fragm ent of possible English language cont ent , and of t he m any kinds of pat t erns found in syst em s of nat ural language. I t does not cont ain pat t erns or cont ent which are deviant from general st andard English, but cannot claim t o represent t he t ot alit y of t hose pat t erns in any com prehensive m anner. The t est er has already argued t hat a com prehensive language t est is im possible in principle. However, a useful language t est is possible, and t he sam ple of language in t he Pract ice Test has alr eady served a useful funct ion of checking, elicit at ion and m ot ivat ion, if we are t o believe feedback from t he individual t est ed. The t est er feels from his own experience t hat t he single candidat e’s English has a great deal in com m on wit h t hat of m any ot her st udent s in East Asia, especially Sout h Korea. However, t he present cont ext offers no st at ist ical base for est ablishing t his. 119 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 11. Reliability and Validity The com parison bet ween t he G- TELP t est and t he Pract ice Test earlier in t his paper em phasized t hat t he Pract ice Test , applied t o one individual, offer ed no st at ist ical base for calculat ing any kind of t echnical reliabilit y or validit y. To t he ext ent t hat t he Pract ice Test m irrors t he G- TELP t est it is likely t o be defensible. However t he Pract ice Test is not assessing any overall com pet ence or achievem ent in L2. I t is at t em pt ing t o pinpoint and diagnose problem areas for at t ent ion by st udent s and t heir t ut ors. I n t hose t erm s, we would need t he largest possible sam ple of part icipat ing st udent s, t est ing over a period of t im e under closely observed condit ions, t o draw any credible conclusions about t he real value ( reliabilit y and validit y) of t he Pract ice Test . 12. Scoring Procedures The discret e it em s in t he Pract ice Test could be scored m ore or less obj ect ively, and we could expect t he sam e kind of int er - m arker consist ency w hich is found in t est s like G- TELP. The evaluat ion of fact ors such as discourse et iquet t e, regist er and discourse cont ent w hich t his t est er has at t em pt ed on t he single t est paper ( Appendix 1) , could probably not be scored in a fair num erical m anner wit h any consist ency. Furt her, such an evaluat ion would be beyond t he com pet ence of m ost non- nat ive speaking t eachers of English in count ries such as Sout h Korea wit hout ext ensive reeducat ion. For t his r eason alone, t he Pr act ice Test could not be scored on a num erical scale of language achievem ent . 120 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 The int ernat ional sub- cult ure of form al language t est ing would probably have t o dism iss t he Pract ice Test as a valid exercise because of it s inher ent inabilit y t o score t he kind of feat ur es m ent ioned in t he previous paragraph. I f t est ing is about num bers, one can underst and t his. Pedagogically, t he rej ect ion would be nonsense. The problem is wit h t he t est ing concept and it s crit eria of value. 13. Aspects of Moderation The discret e point elem ent s in t he Pract ice Test would not r equire m oderat ion unless ext ensive use of t he t est showed up an am biguit y in cert ain quest ions. The qualit at ive discourse feat ures m ent ioned in t he previous sect ion would need ext ensive m oderat ion if m ore t han one t est er used t he docum ent , and even t hen num erical scoring could be dubious. Given t he purpose of t he Pract ice Test , t he real ‘ moderat ion’ would com e when t he t est er changed hat s t o becom e advisor, and t he t est ee( s) becam e individual learners negot iat ing and com ing t o underst and t heir pr oblem s in post - t est consult at ion. The init ial applicat ion of t he Pract ice Test wit h one t est ee in a rem ot e locat ion leaves lit t le scope for post - t est m oderat ion of course. 14. Summary Evaluation of the Test The Pract ice Test , as based on G- TELP, was t ent at ively pit ched at GTELP’s concept of an int erm ediat e st andard of English language 121 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 acquisit ion. G- TELP provides descript ors for it s scales ( see earlier assignm ent papers) , but does not j ust ify t hem in any rigorous way. The t est er has 30 years of experience in assessing how ESL learners are likely t o perform in a variet y of environm ent s – social, academ ic and occupat ional. The crit eria applied in such subj ect ive assessm ent are t oo various, subt le and largely sub- conscious t o easily m ake explicit ; ( t his is charact erist ic of all kinds of expert knowledge) . However, life experience gives a fair am ount of confidence in t his kind of assessm ent ( t hough surprises are always possible) . One kind of evaluat ion of t he Pract ice Test would be t o ask how it squared wit h t he t est er’s subj ect ive assessm ent of t he candidat e’s L2 abilit ies. I f we t ake t he Pract ice Test in G- TELP num erical scoring m ode, I t hink t hat in som e sect ions, especially t he gram m ar sect ion, it would have badly underest im at ed t he candidat e. For exam ple, t he gram m ar score would have predict ed t he candidat e t o be incapable of handling t ert iary st udy in English wit hout great difficult y. I have t aught t he candidat e at graduat e level in English, and alt hough he didn’t find it easy, he proved perfect ly able t o handle t he cont ent . One of his graduat e essays is forwarded for com parison. Taken on it s own t erm s as a diagnost ic and m ot ivat ional t ool, t he single applicat ion of t he Pract ice Test seem s t o have been fairly successful. I t showed up som e issues t hat could be usefully addressed by t he t est ee and his t eachers. I n alert ing t he t est ee t o pot ent ial problem s, it has apparent ly m ot ivat ed him t o look for m ore im aginat ive solut ions t han his prior, narrowly academ ic experience of English. 122 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Appendix 1 : Error Summary of Kim Joon-woo’s Practice Test Thor May, 2005 Exam ples of errors and infelicit ous usage for each sect ion of t he Pract ice Test are past ed direct ly int o t his docum ent . The problem s ident ified in each sect ion are t hen followed by a t est er com m ent . St udent responses are in bold t ype. Correct ions are highlight ed in yellow. 1. Listening Test (part B not done) = > No errors in Part A 2. Reading Test 1. The qualit y t ag on clot hing t ells you a) t hat t he it em was m ade in Sout h Korea b) t hat t he it em has been checked for fault s c) t ha t t he it e m is a good buy Tester comment I n general t he t est ee had no problem int erpret ing bot h fact ual and inferent ial inform at ion in t he reading t est . The error recorded above 123 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 m ay have st em m ed from inferring from t he word ‘ quali t y’, rat her t han st udying t he cont ext carefully. 3. Grammar Test 2. If we take care, smallpox ____________ not kill large numbers of people again. a . w ould b. will have c. will d. have 5. Sm allpox has now ceased t o be a m aj or healt h problem . [ Test const ruct ion error] a. is b. _ _ _ _ [ not hing] c. was d. has 10. Logical inquiry t oget her _________ advanced t echnology is needed t o solve m any m odern problem s. a . t hr ough b. by c. wit h d. and 17. Governm ent s can always gam ble t hat t here will not be 124 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 epidem ics ________ not prepare for em ergencies. a . but b. or c. and d. because of 18. Doct ors can only t reat t he m ost urgent cases first , _____________ t ry t o give quick help t o all pat ient s. a . a nd b. but c. or d. before 19. Helping rem ot e com m unit ies is em ot ionally rewarding ___________ oft en cost s doct ors in lost incom e. a . a lt hough b. however c. and d. but 20. Mass vaccinat ions cannot be conduct ed oft en ___________ t he high cost s involved. a . w he n b. because c. because of d. whet her 125 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Grammar Section B A 21 I n m edicine, B I m por t a nt t han t rying t o cure sick people prevent ion is xx 22 Careful planning H ighe r event ually cost s 23 Sm allpox was one of t he C xx I ll in suffering t han ignoring m edical risks successes of m odern m edicine Xx ? Tester comment : The st ruct ures t est ed in t he gram m ar sect ion of t he Pract ice Test are t aken direct ly from Level 3, G- TELP. Level 3 is claim ed t o indicat e int erm ediat e m ast ery of English. The part icular choice of st ruct ures is not explained by t he G- TELP com m it t ee. I f it is supposed t o indicat e som e universal st age in language acquisit ion, t hen I t hink t hey have a problem . However t he G- TELP t est s overall, covering all levels, m ust clearly draw fr om a m enu of language st ruct ures which seem significant , and divide t hem up for dist ribut ion across t he various level t est s. I f t his is t he case, t hen t he correlat ion bet w een st ruct ural m ast ery at a part icular level ( as checked by m ult iple choice quest ions) and funct ional m ast ery in live language cont ext s is probably quit e arbit rary. There is t hen a problem of deciding what ‘ int erm ediat e’ m ast ery actually refers t o. The t est ee in t he Pr act ice Test is living evidence ( I t hink) t hat a problem wit h t he gr am m ar t est at Level 3 G- TELP is a poor indicat or 126 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 of funct ional m ast ery in t he t arget language. I f we t ook his perform ance on t hese gram m ar quest ions as final evidence he would be relegat ed t o Lev el 2 G- TELP, which is basic social English. I know from ext ended cont act t hat t his t est ee is perfect ly capable of handling a t ert iary level course in an English speaking count ry. His is bet t er equipped linguist ically t han a large num ber of individuals who undert ake t hat st ep. To m anage t ert iary level courses, a st udent m inim ally needs t o cont rol a language at about 3 on a scale of 5 ( ASLPR, or Am erican Foreign Service scale) . The focused select ion of it em s at each level of G- TELP does have diagnost ic value, and t hat is t rue in t he present case. The t est ee has som e problem s wit h condit ionals, w it h t he logic im plicit in conj unct ions, and w it h t he choice of com parat ive form s. At his st age of learning, a good t eacher should be able t o provide useful explanat ion and coaching in all of t hese it em s. 4. Writing Test Dear, Jackson Hello, Mr. Jackson. I ’m Joonoo, a chancellor of ChungJu Nat ’l Univ. The reason I ’m sending you t his e- m ail is t o inform you about pick up t im e, dat e, m eet ing place…t hings like t hat . You are going t o m eet one of our st aff at m ain gat e of I ncheon Airport t om or row. Your flight will arrive at 2p.m . and one of our st aff will pick you up 127 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 as soon as you show yourself up in t he arrival gat e. Don’t worry, m ost of st aff speak fluent English. By t he way, m y m an will guide you t o a very fancy car which is very expensive. And it will t ake one hour t o get you t o Chungj u. ChungJu is very beaut iful cit y which is locat ed in m iddle part of Sout h Korea. Chungj u is t he hom e of unique cult ures and art s w it h abundant nat ional cult ural t reasures, nat ural beaut ies and hist oric t hings. Cent ral Tower, and Goguryeo Monum ent are t he cent ers of Jungwon cult ure, boast ing it s unique cult ure and long hist ory. I f you com e t o com e t o our college, I ’ll t ell you det ails about our cont ract ion and t ravel inform at ion. Sincerely, Chancellor. Kim , Joonoo Tester comments : Grammatical errors apart, most of the issues raised below are rarely or never tested in any formal way. Indeed, many of them are never taught. The problems of mass testing for things like register and discourse etiquette may be insurmountable, especially where large norm-referenced tests and statistical reliability are paramount. Yet the illocutionary effect of most spoken and written communication is far more likely to be corrupted by non-tested, non-taught elements than by formally recognized errors . 128 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 a) Formal grammatical errors : - Overall m anagem ent of t he form al gram m ar by t he t est ee is quit e good, and would m at ch t hat of m any nat ive English users, except for a couple of char act erist ic L2 problem s. - the writer has an unstable control of the determiners a and the . They are omitted in a number of places in the e-mail, and once a is inserted inappropriately. These elements are often amongst the last mastered by ESL learners (quite often never mastered) : they are unstressed in speech, and their application is frequently subtle. The subject & topic particles in Korean, and the object particle, have some functions which are analogous to English determiners, but the match is very approximate. b) Register (the language of social marking) The short e-mail by this testee contains a fair number of register violations which could be serious in a business environment. i) The erratic use of register The errat ic use of regist er is t ypical of Korean L2 users of English ( including Korean professors of English) . Korean it self m akes const ant use of regist er by m anipulat ing t he form al set of honorific m arkings on verbs. This is a highly st ruct ured process. Korean learners of English, and t heir Korean t eachers, t end t o believe t hat because regist er is not rigidly m arked in English, it doesn’t exist . Thus t he cont rol of English m odals ( for exam ple) is oft en poor. ESL 129 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 t ext s in Korea frequent ly fail t o indicat e whet her words and expressions should be used in a form al or inform al cont ext . ii) Terms of address (the special register rules for addressing other people) These are ext rem ely im port ant wit hin t he Korean cult ural and linguist ic cont ext . Howev er, t hey are hardly t aught for English usage, and alm ost universally m isused. There is no underst anding t hat t he honorifics ‘ Mr, Mrs, Miss’ always go wit h a surnam e in second person, but never wit h a given nam e. Korean colleagues, greet ing m e inform ally in English, alm ost always say ‘ Hello May’. Korean st udent s alm ost always address a foreign t eacher as “ Miss Susan” or “ Mr Pet er” ; ( I t ell t hem t hat t his is slave t alk from old Am erican cot t on plant at ions and Brit ish colonies ! Foreign t eachers find it quaint , and rarely correct t he problem ) . The sa lut a t ion opening t he t est ee’s e- m ail shows considerable confusion over nam ing t erm s and honorifics, and so does t he st ruct ure of t he signa t ur e block ( w it h j ob t it le preceding t he signat ure) . The Korean convent ions are quit e different here, and clearly no one has t aught t he t est ee t he st andard English pat t erns. iii) Discourse etiquette ( what t o appropriat ely raise in a t ext , and how t o express it ) This is som et hing w hich I have never seen form ally t est ed anywhere. However, it can underm ine t he whole effect of a com m unicat ion. For exam ple, it m ay be appropriat e in a Korean business e- m ail ( I don’t know) t o say : “ m y m an will guide you t o a very fancy car which is 130 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 very expensive” . I n English t he whole m essage is reduced t o farce, or worse. The t est ee’s descript ion of cult ural at t ract ions is sim ilarly inappropriat e, but so t ypical for East Asia t hat I have highlight ed it in blue; ( it m ight have been lift ed from a t ourist brochure) . c) Information Content The general purpose of the kind the business e-mail requested in the Practice Test is to enable a recipient to confirm known facts, check for unknown facts, and provide the kind of information necessary for forward planning. At least, that is the convention in Western business practice. By West ern st andards, t he t est ee’s e- m ail ( in m y view) is a very weak docum ent . We know t hat Mr Jackson will be m et by an English speaker. A flight arrival t im e is given, as well as a ( very incorrect ) t ravel t im e t o Chungj u. The rest is blah. Mr Jackson could j ust ifiably feel exasperat ed, and his exasperat ion is shared on a daily basis by foreign businessm en and ot her visit ors wit h engagem ent s in East Asia. Mr Jackson has no way t o plan his it inerary in det ail, no way t o check for alt ernat ives, and no one t o call if t hings go wrong ( as t hey very oft en do) . The kind of failure I have j ust indicat ed is, again, never t est ed form ally in Korea, and t he backwash consequence is t hat it is no part of any curriculum , let alone a language curriculum . Yet t his kind of failure is a const ant and m aj or cause of frict ion in relat ions bet ween Koreans ( as well as Chinese and Japanese) and t he wider world. 131 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 The problem has a great deal t o do wit h cult ure, and t he organizat ion of East Asian wor kplaces. The Korean “ vice chancellor” of t he t est ee’s e- m ail would expect his subordinat es t o t ake care of all act ual det ails, and it m ight never occur t o him t hat Mr Jackson would want t o m ake personal j udgem ent s about such m inor m at t ers. He would also believe inst inct ively ( in accordance wit h his cult ure) t hat revealing t he absolut e m inim um of inform at ion was t he best way t o m aint ain som e cont rol over t he foreigner’s act ions. ( I have worked in East Asia for six years. The pat t ern is very, very clear) . 5. Speaking Test Testee transcript “ Thousands of years ago, so m any people were dead by suffering from fat al infect ious disease called sm allpax. Many people around t he world were suffering painful disast er because of t hat . But in 1797, alm ost at t he end of t he eight eent h cent ury, a vaccine for t he infect ious disease was discovered by a doct or nam ed Edward Jenner. From t hat m om ent doct ors from all around t he world gat hered t heir power t o dest roy t his bad kind of disease. The vaccine was delivered t o t he infect ed in pat ient s t hrough inj ect ion and soon t he disease was concurred ( conquered ?) by cooperat ion of doct ors from all around t he world. I t was t he vict ory for t he m edical science. Deat h rat e of sm allpax was dram at ically decreased t o sm all num bers and overall sit uat ions were im proved by 1900. Tim e passed by. I n 1965 WHO, t he World Healt h Organizat ion, decided t o provide a cure t o all around t he world t hroughout whole part of t he count ries. So, m any pat ient s who lived in isolat ed areas like Africa or Asia could get vaccine inj ect ion. Aft er act ive m ovem ent of t he 132 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 t ruer ( WHO ?) proj ect , vaccine for 200 m illion vaccinat ions was st ored in Geneva and New Delhi.” Tester comment Not e : Problem segm ent s have been highlight ed in yellow. Dialect This speaker uses a slight ly Korean- m odified form of Am erican English speech. I t is consist ent , and close enough t o t he st andard t o be accept able in m ost social sit uat ions. Those m ispronunciat ions which do occur ( e.g. conquered, Geneva) are likely t o be t he consequence of sight - reading vocabulary which t he speaker cannot recall having heard before. Fluency The speaker is quit e fluent . Because t he t est was self - adm inist ered t he t est er has no w ay of checking if t he t est ee worked from his own not es ( a list of cues was pr ovided) . The pauses, breaks and anacolut hon which are charact erist ic of spont aneous speech by bot h nat ive and L2 speakers are st rikingly absent . The int onat ion is m arkedly flat ( shows lit t le pit ch variat ion) , and t his is a com m on propert y of reading aloud. On t he ot her hand, t he use of repeat ing phrases ( underlined in t he t ranscript ) is one of t he st rongest differ ent iat ors bet ween spoken and writ t en t ext s. The repet it ion is t ypically a product of real- t im e m ent al processing, and evidence of it s use in speech has underpinned m uch of m y own doct oral research on form ulaic 133 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 ut t erance ( e.g. see m y paper “ Post supposit ion and Past iche Talk” at ht t p: / / t horm ay.net / lxesl/ t ech2.ht m l ) . Formal grammatical errors Much inform al spoken speech by nat ive speakers cont ains a significant num ber of gram m at ical slips, but t hey t end t o be qualit at ively different and m ore random t han errors by L2 speakers. The t est ee in t he pr esent cont ext is speaking in a form al and selfconscious m ode. Those errors which do occur are t herefore likely t o be a consist ent problem in his repert oire, rat her t han sim ple slips. Predict ably, t he speaker’s m ost obvious gram m at ical problem ( and which also showed up in ot her sect ions of t he Pract ice Test ) is in t he use of det erm iners. Fort unat ely, t his kind of error rarely has global consequences for t he m eaning of an ut t erance, alt hough it m ay t rigger negat ive social j udgem ent s. Lexical choice I nappropriat e lexical choice is t he fact or which m ost st rongly m arks t he t est ee as a non- nat ive speaker. ( The sam e problem was evident in t he Writ ing Test segm ent of t his Pract ice Test ) . Examples : /1. so many people / 2. this bad kind of disease / 3. throughout whole part of the countries / 4. After active movement / These phrases are quit e accept able in ot her cont ext s ( except for # 3 ) . That is t he nub of t he problem . This kind of t hing is difficult t o t each ( t here are count less such phrases) , and difficult t o t est . The L2 learner can only really m ast er t his issue t hrough cont inued ext ensive exposure t o t he t arget language ( especially t hrough reading) . 134 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Appendix 2 Post-test e-mail from the Candidate Hello, Thor This is Joonoo. Teacher, how have you been up t o lat ely? I hope you are well also...^ ^ ; ; My sum m er vacat ion has j ust st art ed^ ^ ; ; ( from t oday t o sept em ber) I read your com m ent s about m y t est r esult . I t was so specific t han I expect ed.... I didn’t know t hat I m issed som e part s in t he t est ..... I checked m any errors of m ine.....what a sham e.....^ ^ : ; I t hink I have t o read m ore books( of course in English version) And I realized t hat I ’m lack of pract ice in speaking and writ ing English language This G- TELP t yped t est m ade m e st udy English harder t han m y past days^ ^ ; ; I t hink it was a good and wort hwhile t est for m e t o check m y English abilit y m ore specifically.... Teacher, I really want t o use English language like nat ive speakers. When I speak English in front of som e foreigners, I feel t hat I ’m using different st yle of English...w hich is very st range....^ ^ ; ; Near fut ure.....or soon....I ’m going t o t ake a nat ional t est for t eacher candidat es...and..if I pass t he t est , I will be a m id or high school English t eacher. But I t hink I ’m in very serious sit uat ion...... For bet t er writ ing skill, I t hink ’r eading a lot ’and ’writ ing a lot ’ is t he best way....and I ’m ready t o do t hat ........but I have no confidence in speaking part ..........since I have no nat ive par t ner t hat I could t alk w it h....or t alk t o 135 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Prof. Thor, I ’m 25 years old t his year, do you see any possibilit y t hat I could reach nat ive speaker level?( in English) .......I t hought t hat I ’m quit e good at English but now....I don’t t hink so....hm m ... I t hink I have t o st udy harder......... Happy always........ from your st udent Joonoo References - Test Specification [ hyperlinks list ed here were current at 19 April 2005] AWEMAP - A Worldwide ELT EFL ESL EAL LEP ESOL Assessm ent Scales and Test s Mapping Proj ect ht t p: / / www .geocit ies.com / esolscale/ index.ht m l?2005 10 Birt , Marina & Carol Kiparsky The Gooficon, pub. Newbury House 1972 G- TELP websit e ht t p: / / gt elp.co.kr/ e_gt elp/ gt elp/ e_gt elp04.asp Hughes, Art hur Test ing for Language Teachers ( 2nd edit ion) , pub. UK: CUP 2003 Kit ao, Kenj i and S. Kat hleen Kit ao Language Test ing Resource Page ht t p: / / ilc2.doshisha.ac.j p/ users/ kkit ao/ online/ www/ t est .ht m Moskovsky, Christ o Newcast le Universit y, NSW; MATESOL course not es and t est ing sam ple, 2005 Parry, K., " Building vocabulary t hrough academ ic reading" . TESOL Quart erly, 25 ( 4) , 629- 653. 1991 136 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Pim sleur Audio- lingual Courses ht t p: / / www.pim sleurapproach.com / learn- chinesecant onese.asp? ; ht t p: / / www.pim sleurdirect .com / languages/ sam ple/ / Power, Ted " Language t est ing & m et hods of assessm ent - What is t est reliabilit y and validit y? What aspect s are m ost im port ant for t he t eacher/ t est er?" ht t p: / / www.bt int ernet .com / ~ t ed.power/ esl0 736.ht m l Spolsky, Bernard Measured Words, pub. UK: OUP 1995 ____________________________ References - Part IV : Practical Aspects of Language Test Construction Asher, Jam es 2004 " TPR Aft er Fort y Years" ht t p: / / www.t pr world.com / j apan- ar t icle.ht m l Birt , Marina & Carol Kiparsky The Gooficon, pub. Newbury House 1972 G- TELP websit e ht t p: / / gt elp.co.kr/ e_gt elp/ gt elp/ e_gt elp04.asp Hughes, Art hur Test ing for Language Teachers ( 2nd edit ion) , pub. UK: CUP 2003 Kit ao, Kenj i and S. Kat hleen Kit ao Language Test ing Resource Page ht t p: / / ilc2.doshisha.ac.j p/ users/ kkit ao/ online/ ww w/ t est .ht m Power, Ted " Language t est ing & m et hods of assessm ent - What is t est reliabilit y and validit y? What aspect s are m ost im port ant for t he t eacher/ t est er?" ht t p: / / www.bt int ernet .com / ~ t ed.power/ esl0 736.ht m l Spolsky, Bernard Measured Words, pub. UK: OUP 1995 _____________________________________________ 137 I nt ernat ional Language Test ing Washback– st anding t he m onst er on it s head Thor May 2013 Pr ofe ssion a l bio: Thor May’s PhD dissert at ion, Language Tangle, dealt wit h language t eaching product ivit y. Thor has been t eachin g English t o nonnat ive speak ers, t raining t eachers and lect uring linguist ics, since 1976. This work has t ak en him t o sev en count ries in Oceania and East Asia, m ost ly wit h t ert iary st udent s, but wit h a couple of det ours t o t each secondary st udent s and youn g children. He has t rained t eachers in Aust ralia, Fij i and Sout h Korea. I n an earlier life, prior t o becom ing a t eacher, he had a decade of drift ing t hrough unskilled j obs in Aust ralia, New Zealand and finally England ( aft er backpacking across Asia in 1972 ) . cont act : ht t p: / / t horm ay .net t horm ay AT yahoo.com All opinions expressed in t his paper are ent irely t hose of t he aut hor, w ho has no aim t o influence, proselyt ize or persuade ot hers t o a point of view. He is pleased if his writ ing generat es reflect ion in readers, eit her for or against t he sent im ent of t he argum ent . “ I nt ernat ional Language Test ing Washback – st anding t he m onst er on it s head " © copyright ed t o Thor May; all right s reserved 2013 138