Developing a Questionnaire to Investigate Second Language Learners

Available online at www.sciencedirect.com System 37 (2009) 70–81 www.elsevier.com/locate/system Developing a questionnaire to investigate second language learners’ preferences for two types of form-focused instruction Nina Spada *, Khaled Barkaoui, Colette Peters, Margaret So, Antonella Valeo Department of Curriculum Teaching and Learning, Modern Language Centre, Ontario Institute for Studies in Education/University of Toronto, 252 Bloor Street West, Room 10-253, Toronto, Ontario, Canada M5S 1V6 Received 21 April 2008; received in revised form 24 June 2008; accepted 24 June 2008 Abstract In this paper, we report on the process of developing and validating a questionnaire to explore second language learners’ preferences for two types of form-focused instruction (FFI): isolated and integrated FFI. Three types of evidence regarding the validity of the questionnaire were collected: content, reliability and construct. Content validity was obtained from expert judges and Cronbach’s alpha was used to measure the internal consistency reliability of the isolated and integrated FFI scales. A principal components analysis was used to assess the construct validity of isolated and integrated FFI. The results indicated that the questionnaire was both a valid and reliable measure of isolated and integrated FFI. We conclude by discussing some of the ways in which the questionnaire has evolved since its use in this study, how it has been used in related research and plans for its implementation in future studies. Ó 2008 Elsevier Ltd. All rights reserved. Keywords: Form-focussed instruction; Grammar; Learners’ instructional preferences; Questionnaire development 1. Introduction Research in second language learning suggests that learners’ and teachers’ preferences for instructional approaches warrant investigation. Learning may be negatively affected when learners’ expectations are not matched by the reality of the classroom (Horwitz, 1987) and learner attitudes play a strong role in determining the success of innovations in instructional practices (Savignon and Wang, 2003). A mismatch between instructors’ and learners’ attitudes may also lead to conflict that can adversely affect learning (Schulz, 2001; Spada and Gass, 1986; Wesche, 1981; Yorio, 1986). * Corresponding author. Tel.: +1 416 978 0270; fax: +1 416 926 4769. E-mail addresses: nspada@oise.utoronto.ca (N. Spada), kbarkaoui@oise.utoronto.ca (K. Barkaoui), cpeters@oise.utoronto.ca (C. Peters), mso@oiseutoronto.ca (M. So), avaleo@oise.utoronto.ca (A. Valeo). 0346-251X/$ - see front matter Ó 2008 Elsevier Ltd. All rights reserved. doi:10.1016/j.system.2008.06.002 N. Spada et al. / System 37 (2009) 70–81 71 In this paper we describe the process of developing and validating a questionnaire to explore second language (L2) learners’ preferences for isolated and integrated form-focused (FFI) instruction. We begin by defining isolated and integrated FFI. Next, we discuss current conceptualizations of validity, in particular questionnaire validity and its importance in the field of L2 research. This is followed by a description of how we developed and validated the questionnaire used in this study. 2. Literature review and research rationale 2.1. Isolated and integrated form-focussed instruction In the instructed second language acquisition (SLA) literature, there is consensus that instruction is most effective if it includes attention to both form and meaning. There is less agreement, however, as to when it is most beneficial to draw learners’ attention to form in communicative and content-based instruction. We have chosen to use the terms isolated and integrated to describe two approaches to drawing learners’ attention to language form in second language instruction. Isolated FFI is provided separately from the communicative use of language. That is, isolated FFI may be used in preparation for a communicative activity or after a communicative activity in which students have experienced difficulty with a particular language feature. In isolated FFI, the focus on language form is separated from the communicative or content-based portions of the lesson. Integrated FFI on the other hand, is provided to learners within ongoing communicative or content-based instruction. The overall focus remains on the exchange of meaning: attention to form is brief, and may consist of instruction and/or feedback from the teacher. Although the form focus occurs within a communicative activity, the language features in focus may have been anticipated and planned for by the teacher or they may occur incidentally in the course of ongoing interaction (see Spada and Lightbown, 2008 for a discussion of isolated/integrated FFI). There is theoretical and empirical support for both isolated and integrated FFI. Theoretical support for isolated FFI can be found in the work of DeKeyser (1998), who argues on the basis of skill acquisition theory that [grammar] should be taught explicitly, to achieve a maximum of understanding, and then should be followed by some exercises to anchor it solidly in the students’ consciousness, in declarative form, so that it is easy to keep in mind during communicative exercises. (58) Support for integrated FFI is expressed in a number of theoretical constructs in SLA. This includes Long’s, (1996) ‘revised interaction hypothesis’, Lyster’s, (1998) ‘negotiation of form’ and Swain and Lapkin’s, (2002) ‘meta-talk’. All three constructs are based on the assumption that if learners’ attention is drawn to form within communicative practice, they will have the opportunity to make form-meaning connections and receive information about language form right at the time when they need to express messages. Empirical support for integrated FFI comes from several studies investigating different ways in which learners’ attention has been drawn to form in communicative and content-based classrooms (Doughty and Varela, 1998; Ellis et al., 2001; Harley, 1998; Lyster, 1994; Spada and Lightbown, 1993; Williams and Evans, 1998). While some research has been carried out to investigate second and foreign language learners’ views about grammar instruction and corrective feedback, none of them has specifically investigated preferences for isolated or integrated FFI. Schulz (1996, 2001) has explored learners’ preferences for explicit versus implicit corrective feedback and deductive versus inductive instruction with ESL and EFL learners. More recently, Loewen et al. (2009) investigated several underlying factors regarding learners’ beliefs about grammar instruction and corrective feedback, but to our knowledge, none has investigated when in the pedagogical sequence learners and teachers prefer to focus on form – i.e., during or separately from communicative practice. Thus, it became clear that we would need to design a questionnaire for our purposes. Below, we describe the process of how we developed, validated, and implemented a questionnaire to explore learners’ preferences for FFI but first we discuss some general issues related to the importance of validation in questionnaire-based research. 72 N. Spada et al. / System 37 (2009) 70–81 2.2. Questionnaire validation Before making any claims based on questionnaire data, one needs to present theoretical and empirical evidence supporting the adequacy and appropriateness of the inferences one draws from such data (Brown, 2001; Dörnyei, 2003; Litwin, 1995; Hatch and Lazaraton, 1991; Messick, 1989; Oppenheim, 1992). This is referred to as the process of questionnaire validation, ‘the process of building a case – articulating an argument and collecting evidence’ in support of the particular interpretations and inferences one makes based on questionnaire data (or any other measure) (Bachman, 2004, p. 262; cf. Messick, 1989; Weir, 2005). Such evidence can and should be collected at different stages of the research process and can take different forms (Brown, 2001; Weir, 2005). Few L2 studies, however, have collected and reported validation data for the questionnaires they use (e.g., Block, 1998; Brown et al., 2001; Petric and Czarl, 2003; Sakui and Gaies, 1999; Wintergerst et al., 2001). To support conclusions drawn from questionnaire studies, one needs to make explicit the links between empirical data and the conclusions drawn and, most importantly, to collect and report theoretical and empirical evidence about the quality of one’s data collection instrument. It is important to describe how and why items are constructed, how they are trialed and revised, and how they performed (Brown, 2001; Dörnyei, 2003; Hatch and Lazaraton, 1991; Litwin, 1995; Oppenheim, 1992). In addition, it is important to demonstrate that the questionnaire measures the intended construct and nothing else (i.e., validity evidence) and that it does so consistently (i.e., reliability evidence). As Brown (2001) emphasizes, the more persuasive the validity evidence collected and presented, the more readers will view the inferences and conclusions drawn from questionnaire data as valid. Below we describe the process of developing our questionnaire. We then describe the participants in the study and present three types of evidence concerning its validity: content-related, construct-related, and reliability evidence. 3. Methodology 3.1. Developing the item pool We began the development of our questionnaire by considering how integrated and isolated grammar instruction could be characterized from a learner’s point of view. Following the advice of Dörnyei (2003), items were succinctly worded, with each item containing one construct. Items were written in pairs, with each integrated item in parallel with an isolated item. The original item bank contained 44 items. These items were developed and categorized into 8 groups along two dimensions: (a) whether they are isolated or integrated, and (b) whether they relate to effectiveness, preferences or beliefs.1 Through extensive discussion within the project team, which included the primary researcher and Ph.D. students with experience in English instruction both in second and foreign language settings, the items were drafted, scrutinized and re-worded several times. Care was taken to ensure that the language used in the questionnaire was suitable for ESL students at an intermediate level of proficiency or above. In our efforts to accomplish this, the items went through many reiterations that typically involved paring them down, clarifying the intent, and removing any unnecessary and/or complex phrases. For example the original phrasing of item 20 on the questionnaire was ‘I find it helpful if the instructor teaches a grammar point on its own before or after reading a text’. The first concern about this item was the before or after phrase; it was thought that it created unnecessary confusion by distracting the learners with an ‘or’ choice. Thus, it was decided to revise the item to ‘I find it helpful if the instructor teaches a grammar point before reading a text’. While this revision took care of the before or after problem, it was felt that the item could still be shortened and that the emphasis should be shifted from the instructor to the learner 1 Initially we had intended to explore whether learners’ opinions would differ depending on whether they were being asked about their preferences for isolated or integrated FFI, or whether they thought that one approach was more effective than the other. We did not find any differences and thus, did not continue to pursue this any further. 73 N. Spada et al. / System 37 (2009) 70–81 – our intended audience. This led to another revision resulting in the final version ‘I find it helpful to learn a grammar point before I read it in a text’. Care was also taken to ensure that the item was measuring the construct intended (i.e., isolated or integrated) and not something else. We avoided using the words ‘isolated’ and ‘integrated’ because they are technical terms and could be open to misinterpretation. Instead, we used phrases such as ‘learning grammar by itself’ and ‘learning grammar within communicative activities’. We also struggled with the different meanings of verbs and recognized that ‘I can pay attention to grammar while reading or listening to a passage’ was different from ‘I can learn grammar while reading or listening to a passage’. We settled with the latter in this case believing that the verb ‘learn’ directed the learner to think more about the outcomes of integrated FFI rather than the process ‘can pay attention to’. Table 1 shows how four of the questionnaire items evolved during the item writing and revision process. Despite a concerted effort to carefully craft each item, one is never totally confident that the choices made are the right ones. This is where piloting and reliability testing of the items can increase confidence levels. To allow for a reasonable administration time, the final number of items was reduced from 44 to 20. The remaining 20 items (10 isolated and 10 integrated) fell into the categories of student preferences for isolated/ integrated instruction, their opinions about the effectiveness of both, with two remaining items that focused exclusively on corrective feedback. Respondents were asked to respond to each item using a 5-point Likert scale with 5 indicating ‘Strongly Agree’ and 1 indicating ‘Strongly Disagree’. The questionnaire also contained a short section on the participants’ language background and age, and an open-ended question inviting participants to provide any further comments on their preferences for learning grammar (see the Appendix for the questionnaire). 3.2. Participants We collected data from two groups of participants: 12 expert judges who provided judgments of the content validity of the questionnaire and 314 ESL students who provided responses to the final version of the questionnaire. 3.2.1. Expert judges Data were collected from the 12 expert judges as part of a formal expert review in order to explore the content validity of the isolated (ISO) and integrated (INT) constructs as presented in the questionnaire items (for the categorization of items as ISO or INT, see the note at the bottom of the first page of the Questionnaire in the Appendix). The judges considered as ‘content experts’ (Henk and McKenna, 2004, p. 211) included twelve students pursuing Masters and Doctoral degrees in second language education. Virtually all of the graduate Table 1 Development and revisions of four questionnaire items. Itema First version Second version Final version 3 ISO I find it easier to concentrate on the grammar point if the instructor teaches it on its own I find it easier to concentrate on the grammar point if the instructor teaches it on its own I find it easier to learn grammar when the instructor teaches it by itself 14 INT I find it helpful when the instructor teaches a grammar point found in that text I find it helpful when the instructor teaches a grammar point while we read the text I find it helpful when the instructor teaches grammar while we read a text 16 INT I can pay attention to a grammar point being taught while I am also thinking I can pay attention to grammar while reading or listening to a passage I can learn grammar while reading or listening to a passage 20 ISO I find it helpful if the instructor teaches a grammar point on its own before or after reading a text I find it helpful if the instructor teaches a grammar point before reading a text I find it helpful to learn a grammar point before I read it in a text a Item number in the final version of the questionnaire (Appendix). ISO = Isolated, INT = Integrated. 74 N. Spada et al. / System 37 (2009) 70–81 students had considerable experience as teachers of second or foreign languages. They were given explanations and definitions of the terms isolated and integrated before they were asked to categorize each item on the questionnaire into these categories. On the questionnaire itself, the items were not labeled as isolated or integrated and were presented in random order. The expert participants were also given a ‘Not Sure’ option. Results of the expert review are presented below in Section 4.1. 3.2.2. Questionnaire respondents The questionnaire respondents were 314 ESL students who came from two different programs: 210 students from a community-based ESL program, and 104 students from a university EAP program. Both are communicatively oriented programs with some differences. The community-based ESL program has a settlement focus with a topic-based syllabus and is publicly funded, while the university EAP program has an academic focus with a skills-based syllabus and is privately funded. Students in both programs were from a range of ages and proficiency levels, with the majority at the intermediate level. They came from a variety of language groups with clusters in the EAP program of Korean, Chinese and Japanese, and in the ESL program of Spanish, Chinese and a variety of other languages. The questionnaires were administered during classroom time, taking no more than 20 min to complete. The students’ individual responses to each item on the questionnaire were entered into a database. To conduct the various analyses described below, we excluded respondents with missing data on any of the 20 questionnaire items, resulting in a sample of 294 students out of the original sample of 314. 3.2.3. Data analyses We adopted three main strategies to collect information about the validity of the questionnaire. First, to examine the content validity of the isolated (ISO) and integrated (INT) constructs as presented in the questionnaire, we conducted the formal expert review of all the items, as just described above. Second, to assess the reliability of the integrated and isolated subscales, we used Cronbach’s Alpha index. This index indicates the degree to which a scale is internally consistent, or reliable (Brown, 2001). We also computed itemremainder correlations and Alpha if item deleted for each subscale. Our main strategy to empirically demonstrate construct-related validity, i.e., that our questionnaire is measuring what we claim it is measuring, was to perform a principal components analysis (PCA) on the questionnaire items. PCA is a statistical technique often used in questionnaire development to determine if groups of items go together to form a component or a construct (Brown et al., 2001; Gorsuch, 1983; Stevens, 2002; Tabachnick and Fidell, 1996; Wintergerst et al., 2001). 4. Results 4.1. Content validity As noted above, 12 expert judges reviewed the 20 questionnaire items to assess their content validity. We then calculated the percentage agreement among the expert judges as to whether an item was INT or ISO and decided to keep only those items that achieved 70% agreement or higher. The results of the content judgment are presented in the last column of Table 2. Agreement on the 20 items tended to be high, with 100% agreement obtained for 10-items, 92% agreement for 4 items, and 83% agreement for three items. Three items failed to meet the 70% agreement criterion. Item 11 achieved 58% agreement, while the pair of items concerning corrective feedback (items 4 and 15) achieved an agreement of 25%. Evidence for content-related validity was collected after, rather than before, the administration of the questionnaire to the ESL students, and, as a result, the three items were not excluded from the final version of the questionnaire. Not surprisingly, when the internal consistency reliability of the questionnaire items was later explored, these two corrective feedback items were among those excluded, due to their negative effect on the internal consistency of their respective scales. In addition, the correlations between these items and their respective scales were very low. 75 N. Spada et al. / System 37 (2009) 70–81 Table 2 Descriptive statistics, percentage agreement and reliability of isolated and integrated subscales. Integrated Scale: Alpha = 0.69 2. I believe my grammar will improve quickly if I communicate using English 4. I like the teacher to correct my mistakes as soon as I make them 5. I prefer lessons that focus on communication and teach grammar only when necessary 7. I like learning grammar by using language 8. I can learn grammar during reading or speaking activities 12. I prefer to learn grammar as I work on different skills and activities 14. I find it helpful when the instructor teaches grammar while we read a text 16. I can learn grammar while reading or listening to a passage 18. I like learning grammar during speaking, writing, listening or reading activities 19. Doing communicative activities is the best way to learn to use English more accurately Isolated Scale: Alpha = 0.63 1. I like to know exactly which grammar point I am studying 3. I find it easier to learn grammar when the instructor teaches it by itself 6. I like learning grammar by seeing the explanation and doing practice exercises 9. I like lessons that focus only on teaching grammar 10. Doing grammar exercises is the best way to learn to use English more accurately 11. I find it hard to learn grammar through reading or listening activities 13. I like learning grammar by itself 15. I like the teacher to correct my mistakes after an activity is completed 17. I believe my English will improve quickly if I study and practice grammar 20. I find it helpful to learn a grammar point before I read it in a text a Mean SD Mode 4.33 0.96 5 4.66 0.68 3.04 Skew Corrected itemtotal correlation Cronbach’s alpha if item deleted Percentage agreementa 1.39 0.350 0.669 100 5 1.61 0.145 0.697 25.00 1.35 3 0.01 0.189 0.712 83.33 4.44 3.99 0.84 1.08 5 5 1.69 0.76 0.332 0.537 0.674 0.633 100 100 3.79 1.15 5 0.65 0.405 0.659 100 3.80 1.09 4 0.66 0.332 0.674 83.33 3.60 1.09 4 0.39 0.495 0.641 91.66 4.49 0.78 5 1.62 0.435 0.659 100 4.09 0.99 5 0.97 0.361 0.668 100 4.38 0.88 5 1.32 0.333 0.607 100 3.72 1.19 5 0.70 0.279 0.615 91.66 4.44 0.82 5 1.32 0.413 0.596 83.33 2.68 1.29 3 0.33 0.445 0.574 100 3.68 1.25 5 0.64 0.401 0.586 100 2.82 1.27 2 0.12 0.165 0.643 58.33 2.53 4.08 1.22 1.23 2 5 0.45 1.13 0.152 0.224 0.644 0.629 100 25 4.16 1.03 5 1.19 0.369 0.597 100 3.89 1.03 5 0.55 0.333 0.604 91.66 Percentage agreement among the 12 expert judges with the classification of an item as INT or ISO. 4.2. Reliability Table 2 reports the item-remainder correlations and Cronbach’s alpha for each subscale. Also included in Table 2 are Cronbach’s Alphas if item deleted for both integrated and isolated subscales. Alpha measures the internal consistency of the scale, meaning how well items in a scale vary together in a sample. Alpha if item deleted indicates the effect of removing an item on the overall alpha coefficient for that scale. 76 N. Spada et al. / System 37 (2009) 70–81 The 10-item isolated scale achieved an alpha coefficient of 0.63, while the 10-item integrated scale achieved an alpha of 0.69. We considered two criteria when evaluating each item, alpha if item deleted and item-remainder correlation. We decided to keep items if the item-remainder correlation for an item was equal to or greater than 0.30 and if an item’s deletion failed to increase alpha for its respective subscale by at least 0.01. Examination of individual items in Table 2 indicates that one integrated item (item 5) and two isolated items (items 11 and 13) failed to meet both criteria. Cronbach’s Alpha if item deleted indicates that deleting these items increases the internal consistency reliability of their respective scales. In addition, corrected item-total correlations between these items and their respective scales were very low. A decision was made to exclude these items from further analysis. Excluding these three items resulted in a 9-item integrated scale with an alpha of 0.71 and an 8-item isolated scale with an alpha of 0.66. Although it is difficult to know why these three items were not as reliable as the others, it is possible that the isolated items created problems for the learners because of the expression ‘by itself’ in number 13 (‘I like learning grammar by itself’) and item 11 (‘I find it hard to learn grammar through reading or listening activities’) may not have been sufficiently clear for students to understand what was intended. The integrated item number 5 (‘I prefer lessons that focus on communication and teach grammar only when necessary’) may have been problematic because of the use of the word ‘only’ which may have been perceived by the students as too ‘extreme’. 4.3. Construct-related validity As noted above, we performed a principal components analysis (PCA) on the questionnaire items to assess whether the questionnaire is measuring what we claim it is measuring, i.e., construct-related validity. Only the 17 items that demonstrated acceptable levels of reliability were included in this analysis. Because the items were measured on a five-point ordinal scale, we computed polychoric correlations, instead of Pearson product-moment correlations, for the 17 items, as recommended by Jöreskog and Sörbom (1989) for ordinal data. Polychoric correlations assume underlying unobservable continuous variables and are recommended because for items with few response options, Pearson product-moment correlations tend to be underestimated (Mislevy, 1986). We then performed PCA on the resulting polychoric correlation matrix using the computer program LISREL8.30 (Jöreskog and Sörbom, 1999). A sizable number of correlations were higher than 0.32 as recommended by Tabachnick and Fidell (1996) for running PCA. We first performed PCA on the correlation matrix to estimate the number of components. We then used three criteria to determine the number of components to rotate: Kaiser’s rule of keeping components with eigenvalues greater than 1, the scree plot of eigenvalues, and the interpretability of the component solution (Tabachnick and Fidell, 1996). Four components had eigenvalues greater than 1 and explained 47% of the item variance. The number of components to retain for rotation to an interpretable solution was established by considering the scree plot, which showed a big drop after the first component, a small drop after the second component and even smaller one after the third component. The scree plot supported a multi-component solution despite the big drop between the first component and the rest of the components. The first component explained only 26.65% of the item variance. Consequently, we considered two- and three-component solutions to find a meaningful solution. We used the Principle Component extraction technique and Varimax and Promax rotation procedures for two- and three-component solutions. Following Tabachnick and Fidell (1996), an item was considered to load on a component if the absolute value of the loading was equal to or greater than 0.30. The three-component solution resulted in few items loading higher than 0.30 on the third component (items 14, 15, and 16). These items had a loading of 0.30 or higher on one of the first components as well. A two-component solution indicated that: (a) item 2 had higher than 0.30 loadings on both components and (b) item 4 (integrated) and item 15 (isolated) loaded on what was interpreted as isolated and integrated components, respectively (i.e., they loaded on the opposite component). As indicated above, items 4 and 15 were the pair of corrective feedback items that caused the most difficulty in the expert review of 77 N. Spada et al. / System 37 (2009) 70–81 Table 3 Principal components loadings for PCA extraction and Varimax rotation for 14 items. Component Integrated Unique Variance Isolated 6. I like learning grammar by seeing the explanation and doing practice exercises 17. I believe my English will improve quickly if I study and practice grammar 10. Doing grammar exercises is the best way to learn to use English more accurately 1. I like to know exactly which grammar point I am studying 9. I like lessons that focus only on teaching grammar 20. I find it helpful to learn a grammar point before I read it in a text 3. I find it easier to learn grammar when the instructor teaches it by itself 0.232 0.746 0.389 0.231 0.587 0.602 0.016 0.570 0.675 0.170 0.199 0.147 0.033 0.547 0.538 0.459 0.404 0.672 0.671 0.768 0.836 8. I can learn grammar during reading or speaking activities 18. I like learning grammar during speaking, writing, listening or reading activities 16. I can learn grammar while reading or listening to a passage 12. I prefer to learn grammar as I work on different skills and activities 7. I like learning grammar by using language 19. Doing communicative activities is the best way to learn to use English more accurately 14. I find it helpful when the instructor teaches grammar while we read a text 0.756 0.659 0.022 0.257 0.428 0.500 0.645 0.539 0.494 0.454 0.050 0.051 0.054 0.061 0.582 0.707 0.754 0.790 0.416 0.204 0.785 Table 4 Relationship between number of items and the internal consistency of the subscales. Subscale 4 Items 7 Items 11 Items 14 Items 21 Items Integrated Isolated 0.54 0.52 0.70 0.67 0.78 0.75 0.82 0.80 0.88 0.86 the questionnaire. It is not clear why item 2, ‘‘I believe my grammar will improve quickly if I communicate using English” was problematic although the use of the adverb ‘quickly’ may have led to the split in learner beliefs. A decision was made to drop items 2, 4, and 15 and a final two-component PCA was run on the remaining 14 items using a Varimax rotation. The two-components in the final solution explained 43.35% of the item variance: The first component accounted for 26.94% of the variance (eigenvalue = 3.77), the second component accounted for 16.41% (eigenvalue = 2.30). Table 3 presents the loadings for those questionnaire items (14 items) that had loadings of 0.30 or higher on no more than one component. The items are ordered in terms of the absolute value of their loadings on the components. The two-component solution is interpreted as follows: (a) component 1: isolated, and (b) component 2: integrated. The final scale consists of an equal number of items (7) in each subscale. The integrated subscale has a Cronbach’s alpha of 0.70 and the isolated subscale achieved an alpha of 0.67. These reliabilities give an indication of the extent to which the items on each scale are coherent. We computed Spearman–Brown prophecy formula to assess the effect of increasing or reducing the number of items on the internal consistency of each subscale. Table 4 reports estimates of what the internal consistency reliability of each of the subscales would be if each included 4, 11, 14, or 21 items. Table 4 shows that including four or seven more items of the same quality would improve the internal consistency of each subscale considerably. Future researchers who choose to use this questionnaire may want to consider adding new items of the 78 N. Spada et al. / System 37 (2009) 70–81 same quality to the current 14 item version to enhance its internal consistency. This is discussed in more detail below. 5. Discussion and conclusions The main finding from this research is that the final 14 item version of the questionnaire is a reliable measure and can be used in other contexts for both research and pedagogical purposes. The analyses to determine the reliability and validity of the questionnaire revealed its success in measuring the intended constructs: learner preferences for isolated and integrated FFI. As expected, some items were discarded due to low levels of internal consistency on both the isolated and integrated scales and these deletions led to an increase in the overall reliability of the questionnaire. Furthermore, the principal component analyses resulted in a two-component solution confirming the isolated and integrated factors. Even though there were only seven items in each subscale the integrated scale obtained a Cronbach’s alpha of 0.70 and the isolated scale 0.67 which are respectable values for a new questionnaire with a small number of items. Of course, potential users of this questionnaire will also need to collect and report evidence about the quality of the data they obtain using it because validity and reliability are relative and context dependent (Bachman, 2004; Weir, 2005). Such studies will contribute to further evaluation and improvement of the questionnaire. Indeed, since the validation study reported here, the questionnaire has been considerably revised based on its administration with a new sample of students in an EFL context. Preliminary analyses of the new version indicate even higher levels of reliability and confirmation of the constructs (Spada and dos Santos Lima, 2008). One of the goals in carrying out this research was to describe the process of the development and validation of a questionnaire for use in L2 research. Even though the results indicate some success in measuring the intended constructs, there are many difficulties and challenges involved in defining and measuring preferences for different types of second/foreign language instruction. Support for this comes from qualitative data we have collected about teachers’ preferences within the same research project, in which they expressed a need for more specific information about the instructional context that individual items referred to, suggesting that it may be helpful to create a specific frame of reference to help respondents make their decisions. We attempted to do this in a related study in which students (and their teachers) participated in 2-hour lessons: one designed to teach a specific target form via integrated FFI and the other via isolated FFI. Immediately after the instruction, teachers and learners were asked to provide feedback on both lessons (Spada et al., 2008). This enabled learners and teachers to indicate their preferences for one type of instruction over the other within a clearly defined context. Interestingly, the findings from that study are similar to those obtained in this study – that learners expressed positive opinions about both integrated and isolated FFI (see Spada et al. 2008 for details). The positive results of this validation study have led not only to the development of an expanded version of the learner questionnaire but also to a parallel questionnaire for teachers, both with more items to measure the isolated and integrated constructs. An online version of the teacher questionnaire has also been developed to facilitate widespread use. Administering these questionnaires to other groups of learners and teachers will permit us to explore whether teachers and learners in varying instructional contexts express similar or different preferences for and opinions about integrated and isolated FFI and how these compare with the findings reported here. The next phase of our research includes a quasi-experimental study to investigate the effects of isolated and integrated FFI on different aspects of L2 knowledge and use. In addition to exploring whether these different instructional approaches lead to differences in L2 learning, we will also examine whether students with a particular preference for learning (i.e., isolated or integrated) progress in similar or different ways depending on whether the instruction they receive is more (or less) compatible with their learning preferences. Our development of a questionnaire that has been shown to be a valid measure of learner opinions about isolated and integrated FFI gives us the confidence to move forward in our work. N. Spada et al. / System 37 (2009) 70–81 Appendix. Student Questionnaire 79 80 N. Spada et al. / System 37 (2009) 70–81 References Bachman, L.F., 2004. Statistical Analyses for Language Assessment. Cambridge University Press, Cambridge. Block, B., 1998. Exploring interpretations of questionnaire items. System 26, 403–425. Brown, J.D., 2001. Using Surveys in Language Programs. Cambridge University Press, Cambridge. Brown, J.D., Cunha, M.I.A., Frota, S.F.N., Ferreira, A.B.F., 2001. The development and validation of a Portuguese version of the motivated strategies for learning questionnaire. In: Dorneyi, Z., Schmidt, R. (Eds.), Motivation and Second Language Acquisition. University of Hawaii at Manoa, Second Language Teaching and Curriculum Center, pp. 257–280. DeKeyser, R., 1998. Beyond focus on form: cognitive perspectives on learning and practising second language grammar. In: Doughty, C., Williams, J. (Eds.), Focus on Form in Classroom Second Language Acquisition. Cambridge University Press, New York, pp. 42–63. Dörnyei, Z., 2003. Questionnaires in Second Language Research: Construction, Administration and Processing. Lawrence Erlbaum, Mahwah, NJ. N. Spada et al. / System 37 (2009) 70–81 81 Doughty, C., Varela, E., 1998. Communicative focus on form. In: Doughty, C., Williams, J. (Eds.), Focus on Form in Classroom SLA. Cambridge University Press, Cambridge, pp. 114–138. Ellis, R., Basturkmen, H., Loewen, S., 2001. Learner uptake in communicative ESL lessons. Language Learning 51, 281–318. Gorsuch, R.L., 1983. Factor Analysis, second ed. Lawrence Erlbaum, Hillsdale, NJ. Harley, B., 1998. The role of focus-on-form tasks in promoting child L2 acquisition. In: Doughty, C., Williams, J. (Eds.), Focus on Form in Classroom Second Language Acquisition. Cambridge University Press, Cambridge, pp. 156–174. Hatch, E., Lazaraton, A., 1991. The Research Manual. Design and Statistics for Applied Linguistics. Heinle, Boston, MA. Henk, W.A., McKenna, M.C., 2004. Developing affective instrumentation for use in literacy research. In: Duke, N.K., Mallette, M.H. (Eds.), Literacy Research Methodologies. The Guildford Press, New York, pp. 197–226. Horwitz, E.K., 1987. Surveying student beliefs about language learning. In: Wenden, Anita L. (Ed.), Learner Strategies in Language Learning. Prentice-Hall, NJ, pp. 119–129. Jöreskog, K.G., Sörbom, D., 1989. LISREL 7: A Guide to the Program and Applications, second ed. SPSS, Chicago. Jöreskog, K.G., Sörbom, D., 1999. LISREL 8. Scientific Software, Chicago. Litwin, M.S., 1995. How to Measure Survey Reliability and Validity. Sage, Thousand Oaks, CA. Loewen, S., Li, S., Fei, F., Thompson, A., Nakatsukasa, K., Ahn, S., Chen, X., 2009. L2 learners’ beliefs about grammar instruction and error correction. Modern Language Journal 93 (1), 91–104. Long, M.H., 1996. The role of the linguistic environment in second language acquisition. In: Ritchie, W., Bhatia, T. (Eds.), Handbook of Second Language Acquisition. Academic Press, San Diego, pp. 413–468. Lyster, R., 1994. The effect of functional-analytic teaching on aspects of French immersion learners’ sociolinguistic competence. Applied Linguistics 15, 263–287. Lyster, R., 1998. Negotiation of form, recasts, and explicit correction in relation to error types and learner repair in immersion classrooms. Language Learning 48, 183–218. Messick, S., 1989. Validity. In: Linn, R.L. (Ed.), Educational Measurement, third ed. American Council on Education, Macmillan Publishing Company, New York, pp. 13–103. Mislevy, R.J., 1986. Recent developments in the factor analysis of categorical variables. Journal of Educational Statistics 11, 3–31. Oppenheim, A.N., 1992. Questionnaire Design and Attitude Measurement. Prentice-Hall, London. Petric, B., Czarl, B., 2003. Validating a writing strategy questionnaire. System 31, 187–215. Sakui, K., Gaies, S.J., 1999. Investigating Japanese learners beliefs about language learning. System 27, 473–492. Savignon, S.J., Wang, C., 2003. Communicative language teaching in EFL contexts: learner attitudes and perceptions. International Review of Applied Linguistics 41, 223–249. Schulz, R., 1996. Focus on form in the foreign language classroom: students’ and teachers’ views on error correction and the role of grammar. Foreign Language Annals 29 (3), 343–364. Schulz, R., 2001. Cultural differences in student and teacher perceptions concerning the role of grammar instruction and corrective feedback: USA – Colombia. Modern Language Journal 85 (2), 244–258. Spada, N., dos Santos Lima, M., (2008). Integrated and isolated FFI: Views from ESL and EFL teachers and learners. Paper presented at the Canadian Association of Applied Linguistics. Vancouver, June, 2008. Spada, N., Gass, S., 1986. The role of learner opinions regarding classroom instruction: a review of research. In Papers in Applied Linguistics 1 (2), 11–23. Spada, N., Lightbown, P.M., 1993. Instruction and the development of questions in L2 classrooms. Studies in Second Language Acquisition 15, 205–224. Spada, N., Lightbown, P.M., 2008. Form-focused instruction: Isolated or integrated? TESOL Quarterly 42, 181–208. Spada, N., Peters, C., So, M., Valeo, A., 2008. The case for integrated or isolated form-focused instruction: views from teachers and learners. In: Manuscript. OISE University. Stevens, J.P., 2002. Applied Multivariate Statistics for the Social Sciences. Lawrence Erlbaum, Mahwah, NJ. Swain, M., Lapkin, S., 2002. Talking it through: two French immersion learners’ responses to reformulation. International Journal of Educational Research 37, 285–304. Tabachnick, B.G., Fidell, L.S., 1996. Using Multivariate Statistics, third ed. Harper and Row, New York, NY. Weir, C.J., 2005. Language Testing and Validity Evidence. Palgrave, London. Wesche, M.B., 1981. Language aptitude and measures in streaming, matching students with methods, and diagnosis of learning problems. In: Diller, K. (Ed.), Individual Differences and Universals in Language Learning Aptitude. Newbury House, Rowley, MA, pp. 119– 139. Williams, J., Evans, J., 1998. What kind of focus and on which forms? In: Doughty, C., Williams, J. (Eds.), Focus on Form in Classroom Second Language Acquisition. Cambridge University Press, Cambridge, pp. 139–155. Wintergerst, A.C., DeCapua, A., Itzen, R.C., 2001. The construct validity of one learning styles instrument. System 29, 385–403. Yorio, C., 1986. Consumerism in second language learning and teaching. The Canadian Modern Language Review 42, 668–687.