Table 1. Definitions of the accuracy error types in the MQM model The comprehension test for the quasi-experiment was designed to cover both factual and inference questions and consisted of 15 items. These items were related to the content of the presented videos and covered a range of difficulty levels. More specifically, the test included true- or-false and multiple-choice questions. The questionnaire was initially developed in English by the researchers to ensure that the questions matched the presented information in its original language. Considering that most participants had low to intermediate levels of English proficiency. the test was translated by the researchers into Arabic and presented to the participants in thei native language. Google Forms was used to present both the videos and the comprehension test. Table 2. Evaluation design For the comprehension test, the participants (n = 62) were randomly selected through convenience sampling. The sample was then stratified using a shorter version of the Cambridge syeneral English Test (i.e., consisting of 25 questions) to eliminate any significant differences in inglish proficiency between the participants of the control and experimental groups. The articipants were met online via Zoom. Those in the control group were given a form that included he selected MOOC videos with human-translated subtitles, whereas those in the experimental roup were given another form that included the same videos with machine-translated subtitles. ‘he participants in both groups completed the same comprehension test. Table 3. Kolmogorov—Smirnov test of normality The data collected through the comprehension test were then analyzed statistically using SPSS version 28.0. A Kolmogorov—Smirnov normality test (Table three) indicated that the scores of the experimental group were not normally distributed. Thus, an independent samples t-test was considered inapplicable. Alternatively, a Mann—Whitney U test was implemented in this study to compare the mean ranks rather than medians. Results evaluators. The data presented in Table four reveal that the two instances of untranslated-segment errors had the highest agreement percentage (100%) among the evaluators. Terminology errors had the second-highest agreement percentage (42%). Mistranslation and addition errors, however, had low agreement percentages of 26% and 14%, respectively. The evaluators did not agree on any omission errors. Overall, the agreement percentage across all error types was approximately 27%. An error annotation of each video subtitle was carried out by more than one evaluator to limit the subjectivity of manual evaluation by offering multiple assessments of the same texts. However, considering that manual evaluations are highly subjective (Klubicka et al., 2018), an additional analysis was carried out by the researchers to measure interrater agreement among the Table 4. Frequencies and percentages of agreement on identified accuracy errors
