Inputlog 6.0: Pause and fluency analysis.

Presentation Inputlog 6.0: Pause and fluency analysis. Reference: Van Waes, L., & Leijten, M. (2014). Inputlog 6.0: Pause and fluency analysis. Paper presented at the Keystroke logging training school, Antwerp. 15/04/2014 Pauses: technical Perspectives Pauses in keystroke logging A technical and theoretical perspective  technical (logging) writer process instrumental definitions threshold key in/out  aggregation   text  accuracy   measurement error filtering theoretical framework Accuracy Pause distribution Interkey intervals pausing or flight time the latency between pressing the previous key and the actual key  video registration: 25 pictures/sec ~40 ms  keystroke logging: +7/‐8 ms Pausei = Pi – Pi-1 i i‐1 P P R R action or dwell time latency between pressing a key and the release of the same key Actioni = Pi – Ri i‐1 P Using the testing procedure introduced by Morgan et al. (2013) and Fried (2012) we evaluated the accuracy of the Inputlog data under different logging conditions. Results are comparable for Inputlog/Scriptlog and Morgan et al.'s findings for the program Recording User Input (RUI) or TypingSuite. i R PP Training school on keystroke logging | Antwerp R logarithmic scale 1 15/04/2014 Audacity test Number of pauses at different text levels Threshold  speech > 200 ms  eye‐tracking > 30 – 50 ms  writing > 200 ‐ 3000 ms 90 80 70 60 50 25 20 15 10 5 0 ‐5 ‐10 ‐15 ‐20 ‐25 40 30 20 10 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 00 all The AD (Average deviation) of a unit of 30 keystrokes is 0 ms The AOD (Average absolute Deviation) is 4.3 ms Total pause time at different text levels Pause rules /* Between words 90% 80% 70% 60% 50% 40% 30% 20% 10% 00% all >200 >500 >1000 >2000 Training school on keystroke logging | Antwerp >200 >500 >1000 >2000 General analysis: location identification Priority 3 */ //W1 (IsSentenceReadingMark{MarkAfterWords})+ . !IsSentenceReadingMark {Reset}; //W4 ((IsAlphaNumeric|IsWithinWordChar) {MarkBeforeWords}).(IsAlphaNumeric|IsWithinWordChar)* . IsWordReadingMark {NotifyWord, MarkAfterWords, Reset}; //W2 IsWordReadingMark {MarkAfterWords, Reset}; //W3 (IsAlphaNumeric|IsWithinWordChar)* . (IsTab|IsSpace) {NotifyWord, MarkAfterWords} . ((IsTab|IsSpace) {MarkBeforeWords})* . (!IsTab,!IsSpace) {Reset}; //W7 (IsAlphaNumeric|IsWithinWordChar)* . (IsTab|IsSpace)+ . (IsBindingCharacter {MarkWithinWords})+ . ((!IsTab,!IsSpace) {Reset} | (IsTab|IsSpace) {NotifyWord, MarkAfterWords}) . ((IsTab|IsSpace) {MarkBeforeWords})* . (!IsTab,!IsSpace) {Reset}; //W5 (IsAlphaNumeric|IsWithinWordChar)* . IsCtrlBackSpace {NotifyWord, MarkAfterWords, T h i s SPACE i s SPACE a SPACE t e s t . SPACE N e 640 249 250 109 140 281 109 110 140 468 2480 78 250 187 655 452 1701 390 Before sentence Within words Within words Within words After words Before words Within words After words Before words After words Before words Within words Within words Within words After sentence After sentence Before sentence Within words LSHIFT T h i s SPACE i s SPACE a SPACE t e s t LSHIFT LSHIFT LSHIFT LSHIFT LSHIFT LSHIFT LSHIFT LSHIFT LSHIFT LSHIFT . SPACE LSHIFT N 640 250 249 250 109 140 281 109 110 140 468 2480 78 250 187 655 500 31 31 31 47 31 31 32 31 0 452 1701 187 Before words Within words Within words Within words Within words After words Before words Within words After words Before words After words Before words Within words Within words Within words After sentence After sentence After sentence After sentence After sentence After sentence After sentence After sentence After sentence After sentence After sentence After sentence Before sentence Within words 2 15/04/2014 Pause aggregation Mean pause length at different intervals Median Transition times 3,00 x 120 x 120 x 120 x 140 180 x 120 x 120 . 500 x 200 180 120 9,0 > 0 sec 8,0 > 2 sec 2,50 7,0 2,00 before sent. 5,0 sec after sent. pause length (sec) 6,0 after sent. after before word word 1,50 4,0 3,0 between word between sentence 320 880 1,00 2,0 1,0 0,50 0,0 within word > word sentence paragraph 0,00 threshold Before words: developmental aspects D^D Pauses before-after syntactic boundaries a^_a a^a ,^_a a_^a .^_a D^a ,_^a a^D a^. a^, ._^a Pauses before-after syntactic boundaries 6000,0 Stromqvist: before word > discriminating factor 6000,0 5000,0 5000,0 4000,0 milliseconds milliseconds 4000,0 all 3000,0 2000,0 all > 1 sec 3000,0 > 2 sec 2000,0 1000,0 1000,0 0,0 before after word level before after sentence level before after 0,0 paragraph level before after word level before after sentence level before after paragraph level EJ case Training school on keystroke logging | Antwerp 3 15/04/2014 Before word: length & frequency Before words: length * frequency Differences between persons (n=20 | CI 95%) SnodWrite [Torrance, Nottbush et al. | Tessa Nuyts & Veerle Staels (MPC masters)]  short words  long words (Snodgrass) low low high high Differences within digraphs (median | n=20) Differences within digraphs (n=20 | CI 95%) Pausing time 250 correct typing error 200 188 150 140 100 50 a f w i j k e n d e^ v r i e n d e l i j k h e i d 0 el ke nd he ie ei en er ij ng ek ge jk de re ro te li it nt a f w i j k e nd e^ v r i e nde l i j k h e i d Training school on keystroke logging | Antwerp 4 15/04/2014 Within words: developmental discrimination Copy task: subtasks Copy tasks (digraphs): repetition healthy elderly students MCI & dementia MCI & dementia healthy elderly healthy elderly students students MCI & dementia Copy tasks (digraphs): frequency Copy tasks (digraphs): hand combination MCI & dementia MCI & dementia healthy elderly healthy elderly students students Pauses: writer characteristics  personal characteristics e.g. age, disorders  technical expertise e.g. touch typist, left handed  language expertise e.g. L1/L2  genre expertise e.g. knowledge telling/transforming/crafting Training school on keystroke logging | Antwerp 5 15/04/2014 Pauses: text Pauses: process  characteristics  level: textual and linguistic hierarchy  letter – word – clause ‐ sentence ‐ paragraph e.g. bigram: frequency, syllable boundaries,morphological, correctness  framing devices remark: linguistic analysis      duration of process fragmentation complexity subprocesses: e.g. planning vs. revision, distribution collaboration What happens during a pause? • interpretation of context • combination with eye‐tracking • combination with thinking aloud protocols  intertextuality  task & genre  translation  internal and external sources  analogue and digital sources e.g. text length e.g. genre characteristics  down‐time Leijten, M., Van Waes, L., Schriver, S., & Hayes, J.R. (2014). Writing in the workplace: Constructing documents using multiple digital sources. Journal of Writing Research, 5(3), 285‐337 | PDF Reading during writing - scheme Fluency A multi-dimensional perspective 28 64 8 8 56 36 84 90 81 84 19 16 19 22 81 78 16 10 Fluency in L1 vs L2       participants proficiency in L2 two tasks time on task observation analysis 68 EFR level C1 L1 + L2 expository (knowledge telling) 10 min max Inputlog 5.0 ~ [230 000 observations] GLM repeated measures Study is used as a stepping stone to further define fluency. Training school on keystroke logging | Antwerp 6 15/04/2014 Indicators for Fluency Percentage of Pause Time Pauses per level (ms - log reconverted) 80  product  process        1000 η2= .689 70 891 900 60 characters per min pauses: length | number * [level] * [thresholds] percentage of pause time * [thresholds] P‐Bursts: length | number * [thresholds] intervals absolute and personal optimum … 829 significant at all levels 800 700 η2= .425 50 600 40 L1 η2= .245 377 400 30 308 η2= .152 300 20 200 130 150 100 10  ratio process / product L1 L2 500 L2 0 within words 0 >200 ms more than 400 variables Process graph > 500 ms >1000 ms Process: absolute benchmark 70000 between words between sentences >2000 ms Interval benchmarking 100 1. Theoretical optimum 3000 90 60000 80 2. Personal optimum 2500 50000 70 60 2000 40000 50 L1 1500 40 L2 30000 30 1000 20000 20 10 500 10000 0 1 0 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 Training school on keystroke logging | Antwerp 23 2 3 4 5 6 7 8 9 10 perc. ~ opt. 0,38 0,48 0,41 0,59 0,52 0,31 0,61 0,28 0,09 0,27 char. per min char per interval 38:20 min 150 193 164 235 207 122 244 113 38 108 576 738 630 900 792 468 936 432 144 414 400 400 400 400 400 400 10 intervals theor. optimum 10 min 400 400 400 400 22 7 15/04/2014 Process: absolute benchmark Interval benchmarking 100 Personal percentage of personal optimum 2. Personal optimum      90 80 70 60 50 divide process in 10 equal intervals allocate number of characters produced in each interval recalculate to characters per min define personal optimum calculate proportion L1 40 L2 30 20 100 90 80 70 define personal optimum 60 •divide process in periods of 10 sec •calculate characters produced within this period •calculate moving average (3 periods) * 6 •identify maximum 50 L1 40 L2 30 20 258 10 36 42 43 40 41 35 50 44 10 0 0 1 2 3 4 5 6 7 8 9 10 Pause variance L1: pers.optimum vs. stdev 52 24 8 23 38 23 18 1 2 3 4 5 6 Optimum 25,0 Interval Total Strokes Strokes/min % Optimum 15,0 0:00:00 257 315 0,84 10,0 0:00:49 274 336 0,89 0:01:38 212 260 0:02:27 232 0:03:16 20,0 Window Size Optimum Location 09:00 - 09:30 Total Strokes Strokes/min % Optimum 936 625 0,79 0:01:29 1114 744 0,94 284 0,76 0:02:59 1046 699 0,88 215 264 0,70 0:04:29 1043 697 0,88 0:04:05 225 276 0,73 0:05:59 1069 714 0,90 0:04:54 215 264 0,70 0:07:29 1038 693 0,87 30,0 0:05:43 210 257 0,68 0:08:59 1086 725 0,91 25,0 0:06:32 212 260 0,69 0:10:29 1026 685 0,86 0:07:21 170 209 0,56 0:11:59 1027 686 0,86 0:13:29 1088 727 0,92 40 50 60 70 80 90 100 L2: pers.optimum vs. stdev 45,0 54% 40,0 35,0 20,0 15,0 Interval 3 0,69 0,0 30 10 794 0:00:00 5,0 20 9 Period Size 0:10 30,0 10 8 Task Optimum Optimum 376 35,0 0 7 Personal Optimum (L1) 62% 40,0 sd 26 New Fluency analysis module: xml-output 45,0 sd 32 10,0 5,0 0,0 0,0 10,0 20,0 30,0 40,0 50,0 60,0 70,0 80,0 90,0 Training school on keystroke logging | Antwerp 8 15/04/2014 Task optimum Identification of optimum period Absolute optimum: 400 p/m New Fluency analysis module: Graph Person 1 vs. Person 2: Absolute optimum Person 1 vs. Person 2: personal  L1  L2 Training school on keystroke logging | Antwerp  Person 1 ‐ L1  Person 2 ‐ L1  Person 2 ‐ L2  Person 1 ‐ L1  Person 2 ‐ L1  Person 2 ‐ L2 9 15/04/2014 Principal Component Analysis Person 1 vs. Person 2: Task optimum  Person 1 ‐ L1  Person 2 ‐ L1  Person 2 ‐ L2 Exploration of main components: converted selected L1 variables to Z‐Scores checked correlations (singularity) built components iteratively applied varimax rotation Result: 4 components based on 10 variables explaining 84 % of variance Fluency components 1 Product mean number of characters per min ‐ in process ‐ in product ‐ per .10 intervals ‐ per P‐bursts 2 Process variance stdev. of characters per min ‐ per .10 interval ‐ per .10 interval ~ personal optimum 3 Revision mean number of characters ‐ product vs. product ratio ‐ length of R‐burst 4 Pausing mean pause length between words (> 200 ms) proportion of pause time (> 2000 ms) Summary Number of pauses A pause … is not a pause …. is not a pause …. Pauses are dependent on:  technical characteristics  textual characteristics  personal characteristics Training school on keystroke logging | Antwerp Pauses during writing: Victoria Johansson Pia Gustafson, Johan Frid 10‐year‐olds 2323 13‐year‐olds 2993 15‐year‐olds 1704 17‐year‐olds 1460 Adults (younger group) 2230 Adults (older group) Total number of pauses 2596 13429 10 15/04/2014 Writing time and pausing time Production Keystrokes in final text 6000 5000 5000 4000 4000 3000 3000 2000 2000 1000 1000 0 10‐year‐olds 13‐year‐olds 15‐year‐olds Total number of words Keystrokes in linear and final texts 6000 5000 4000 3000 2000 1000 0 10‐year‐olds 13‐year‐olds 15‐year‐olds Number of keystrokes Linear text 17‐year‐olds Univyoung Number of keystrokes Final text Training school on keystroke logging | Antwerp Univold 17‐year‐olds Univyoung Univold Number of keystrokes in linear text 0 10‐year‐olds 13‐year‐olds 15‐year‐olds 17‐year‐olds Univyoung Univold Pauses – when, where and why? Coding pauses • They could be anywhere, but they’re not! • In connection with larger syntactic units. (Natural point to evaluate and plan further.) Matsuhashi 1981; Spellman‐Miller 2006; van Waes & Schellens 2003; van Hell et al 2008. • When we need to (we don’t have to think about the reader). On‐line constraint of speech is lifted. • What to write next? What have we written so far? Detect spelling errors? • If we believe that pauses in language production reflect cognitive effort (Goldman‐Eisler 1968; Matsuhashi 1981; Chanquoy et al 1996) this will give (some) insight into where more effort is needed. • We will also know more about developmental aspects: do pauses occur in different contexts depending on the writer’s age? • …which will indicate that different things require effort depending on age (and education, and skills, and cognitive development…) 11 15/04/2014 wj07fBEW200 Category Before After text initial start word <START> Coding criteria <4.98>Det <DELETE4>Många fuskade ju i filmen och för att förbättra det Pause time following editing It <DELETE4>Many cheated of course in the film and to improve that • Syntactic‐morphologic context • The context before the pause determines the pause categorization (following Spelman‐Miller’s (2006) idea of Potential Completion Point. • Higher syntactic/morphologic unit has priority over a lower (e.g. phrase boundary instead of word boundary if possible; clause boundary instead of phrase boundary if possible). • We also coded for the unit that came before and after the pause. <5.21>så tyckerjag att<LEFT7> <RIGHT7> man ska So Ithink that<LEFT7> <RIGHT7> one should <6.56>försöka sätta dom en och en eller utspritt. Det är och ganska bra ifall amn<DELETE3>man te – clause phrase boundary clause phrase medial word phrase word medial word word fragment fragment Percentage pause me following edi ng try to seat them one by one or spread them out. It is and quite good if oyu<DELETE3>you e <2.23>x sätter br<DELETE>ara ett par elever i nä<DELETE> ågonstan<DELETE>s <DELETE2>ns eller då e 10‐year‐olds 13‐year‐olds .g. place js<DELETE>ust a couple of pupils n<DELETE>somew<DELETE>h <DELETE2> where or then o <2.00><DELETE>att alla dom som sak<DELETE2>ka göra that all those who aer<DELETE2>are doing <2.05><DELETE111> <11.38>Det var en gansa<DELETE>ka dåö<DELETE>lig öärare<DELETE6>lärare som inte märkte hans fusklapp han he<DELETE2>hade i pennfacket eller vad det vad. <CR> 15‐year‐olds word medial 17‐year‐olds word editing fragment phrase phrase boundary clause editing boundary Univyoung Univold editing clause It was a quie<DELETE>te baf<DELETE>d reacher<DELETE6>teacher who didn’t notice his crib he he<DELETE2>had in his pencil box or what it what.<CR> Clause initial‐Clause‐Clause Phrasefinal‐phrase‐editing 0,1 Clause final‐clause‐phrase 0,05 0,06 0,09 0,045 0,05 0,08 0,04 0,07 0,035 0,04 10‐year‐olds 0,06 10‐year‐olds 13‐year‐olds 15‐year‐olds 0,05 0,04 15‐year‐olds 17‐year‐olds 17‐year‐olds Adults (younger) Adults (younger) Adulds (older) 10‐year‐olds 0,03 13‐year‐olds 0,03 13‐year‐olds 15‐year‐olds 0,025 17‐year‐olds 0,02 Adults (younger) Adulds (older) 0,02 Adulds (older) 0,03 0,015 0,02 0,01 0,01 0,01 0,005 0 0 1 The cat climbs the tree <PAUSE> and then she catches the bird. Training school on keystroke logging | Antwerp 0 1 The red cat with the long white tail<PAUSE><DELETE> 1 Yesterday I saw a very cute cat <PAUSE>with white paws. 12