ICLS2012 Proceedings Vol 2 2012

10th International Conference of the Learning Sciences
The Future of Learning
Proceedings
Volume 2 - Short Papers, Symposia, & Abstracts
July 2 nd - 6 th, 2012
July 2-6, 2012 Sydney, Australia www.isls.org/icls2012
SPONSORED BY: HOSTED BY:
Centre for Research on Computer Supported Learning & Cognition (CoCo)
ICLS 2012 Conference Proceedings Volume 2 Short Papers, Symposia, and Abstracts
10th International Conference of the Learning Sciences The University of Sydney, Sydney, NSW, AUSTRALIA July 2nd 6th, 2012, Sydney
THE FUTURE OF LEARNING
Editors: Jan van Aalst, Kate Thompson, Michael J. Jacobson, and Peter Reimann
Title:
Cite as:
Editors:
ISBN: 978-0-578-10704-2
2012, INTERNATIONAL SOCIETY OF THE LEARNING SCIENCES [ISLS] Copyright 2012 International Society of the Learning Sciences, Inc. Rights reserved
van Aalst, J., Thompson, K., Jacobson, M. J., & Reimann, P. (Eds.) (2012). The Future of Learning: Proceedings of the 10th International Conference of the Learning Sciences (ICLS 2012) Volume 2, Short Papers, Symposia, and Abstracts. International Society of the Learning Sciences: Sydney, NSW, AUSTRALIA
The Future of Learning: Proceedings of the 10th International Conference of the Learning Sciences (ICLS 2012) Volume 2, Short Papers, Symposia, and Abstracts
Jan van Aalst, Kate Thompson, Michael J. Jacobson, and Peter Reimann
Published by: International Society of the Learning Sciences (ISLS) Proceedings printed by: LuLu http://www.lulu.com Proceedings distributed by: LuLu and Amazon Book Cover Design: Dorian Peters Book Cover Photo: Christine Bree
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the International Society of the Learning Sciences. The International Society of the Learning Sciences is not responsible for the use which might be made of the information contained in this book. http://www.isls.org
OUR SPONSORS
ICLS2012 would like to thank our sponsors:

NSW Trade & Investment The Centre for Research on Computer Supported Learning and Cognition (CoCo) Smart Services Collaborative Research Centre
The Early Career workshop and Doctoral Consortium would particularly like to thank the following for their support:
National Science Foundation The International Society of the Learning Sciences Asia-Pacific Society of Computers in Education
We would also like to thank the following for their generosity in donating their time, skills and information to the conference.
Inquirium
ICLS2012
Volume 2: Short Papers, Symposia, and Abstracts
ORGANISERS AND COMMITTEES CONFERENCE CHAIRS

Michael J Jacobson | The University of Wouter Van Joolingen | University of
Twente, the Netherlands

Anindito Aditomo | The University of
Sydney, Australia Peter Reimann | The University of Sydney, Australia CONFERENCE ADVISORY BOARD
Shaaron
Sydney, Australia EARLY CAREER WORKSHOP

Tom Moher | University of Illinois, United
States
Anne Newstead | The University of Sydney,
Ainsworth | University of Nottingham, UK Michael Baker | Telecom ParisTech, Paris, France Katerine Bielaczyc | Nanyang Technological University, Singapore Paul Chandler | Wollongong University, Australia Susan Goldman | University of Illinois, United States Kai Hakkarainnen | University of Helsinki, Finland Yasmin Kafai | University of Pennsylvania, United States Paul Kirschner | Open University of the Netherlands, the Netherlands Marcia Linn | University of California, United States Ric Lowe | Curtin University, Australia Naomi Miyake | University of Tokyo, Japan Stella Vosniadou | University of Athens (Greece) and University of Adelaide (Australia) Uri Wilensky | Northwestern University, United States James Pellegrino | University of Illinois, United States
Australia
Tak-Wai Chan | National Central Taiwan
University, Taiwan PRE-CONFERENCE WORKSHOPS AND SPECIAL SESSIONS

Lina Markauskaite | The University of
Sydney, Australia
Dan Suthers | University of Hawaii, United
States
Carol Chan | The University of Hong Kong,
Hong Kong
Nikol Rummel | Ruhr-Universitat Bochum,
Germany PAPER REVIEW COORDINATION

Peter Reimann | The University of Sydney,
Australia
Peter Freebody | The University of Sydney,
Australia Kyza | Cyprus University of Technology, Cyprus Ton de Jong | University of Twente, the Netherlands Nick Kelly | The University of Sydney, Australia
Eleni
PROCEEDINGS
Jan van Aalst | The University of Hong
Kong, Hong Kong

Brian J. Reiser | Northwestern University,
United States
Cindy Hmelo-Silver | Rutgers University,
United States
Kate Thompson | The University of Sydney,
LOGISTICS AND ADMINISTRATION SUPPORT Sadhbh Warren | The University of Sydney, Australia Sadhana Puntambekar | The International Society of the Learning Sciences Dana Gnesdilow | The International Society of the Learning Sciences
Australia
Stella Wen Tian | The University of Hong
Kong, Hong Kong DOCTORAL CONSORTIUM

Susan Yoon | University of Pennsylvania,
United States
ISLS
Page vi
ICLS2012
PAPER REVIEWERS Abrahamson, Dor, University of California, United States Aditomo, Anindito,, The University of Sydney, Australia Ahn, June, University of Maryland, United States Ainsworth, Shaaron, University of Nottingham, United Kingdom Alac, Morana, University of California, United States Alonzo, Alicia, Michigan State University, United States Alterman, Rick, Brandeis University, United States Andriessen, Jerry, Utrecht University, The Netherlands Ares, Nancy, University of Rochester, United States Arnseth, Hans Christian, University of Oslo, Norway Arthur-Kelly, Michael, University of Newcastle, Australia Asterhan, Christa, Hebrew University of Jerusalem, Israel Avramides, Katerina, London Knowledge Lab, United Kingdom Bagley, Elizabeth, University of Illinois, United States Bairral, Marcelo, Federal Rural University of Rio de Janeiro, Brazil Baker, Michael, CNRS - Telecom ParisTech, France Baker, Ryan, Worcester Polytechnic Institute, United Kingdom Barnes, Jacqueline, Indiana University, United States Barth-Cohen, Lauren, University of California, United States Belland, Brian, Utah State University, United States Berland, Leema, University of Texas, United States Bers, Marina, Tufts University, United States Bielaczyc, Katerine, Singapore National Institute of Education, Singapore Booker, Angela, University of California, United States Brett, Clare, University of Toronto, Canada Brian, Reiser, Northwestern University, United States Buckingham, Brandy, Northwestern University, United States Burke, Quinn, University of Pennsylvania, United States Calabrese Barton, Angela, Michigan State University, United States Carmela, Aprea, Swiss Federal Institute for Vocational Education and Training, Switzerland Carpendale, Jeremy, Simon Fraser University, Canada Castro-Alonso, Cris, University of New South Wales, Australia Cesar, Collazos, Universidad del Cauca, Colombia Chan, Carol, The University of Hong Kong, Hong Kong Chang, Yi-Hsing, Southern Taiwan University, Taiwan Charoenying, Timothy, University of California, United States Chase, Catherine, Carnegie Mellon University, United States Chen, Ying-Chih, University of Minnesota, United States Childs, Joshua, University of Pittsburgh, United States Ching, Cynthia Carter, University of California, United States Chinn, Clark, Rutgers University, United States Christophe, Reffay, ENS Cachan, France Chye, Stefanie, Singapore National Institute of Education, Singapore Clegg, Tamara, University of Maryland, United States Cooper, Benny, University of California, United States Correia, Ana-Paula, Iowa State University, United States Crain, Rhiannon, Cornell University, United States Cress, Ulrike, Knowledge Media Research Center, Germany D'Angelo, Cynthia, University of Wisconsin, United States Damsa, Crina, Utrecht University, The Netherlands Davis, Pryce, Northwestern University, United States De Jong, Ton, University of Twente, The Netherlands de Vries, Erica, University of Grenoble, France Dillenbourg, Pierre, Swiss Federal Institute of Technology, Lausanne, Switzerland
ISLS
Page vii
ICLS2012
Drago, Kathryn, University of Michigan, United States Dugan, Therese, University of Washington, United States Duncan, Ravit, Rutgers University, United States Durga, Shree, University of Wisconsin, United States Dyke, Gregory, Carnegie Mellon University, United States Earnest, Darrell, University of California, United States Ehret, Christian, Vanderbilt University, United States Eisenberg, Michael, University of Colorado, United States Elen, Jan, K.U.Leuven, Belgium Engelmann, Tanja, Knowledge Media Research Center, Germany Engle, Randi A., University of California, United States Enyedy, Noel, University of California, United States Erkens, Gijsbert, Utrecht University, The Netherlands Erlandson, Benjamin, Arizona State University, United States Ertl, Bernhard, Universitt der Bundeswehr, Germany Fatos, Xhafa, Universitat Politecnica de Catalunya, Spain Femke, Kirschner, Utrecht University, The Netherlands Fields, Deborah, University of California, United States Fischer, Frank, University of Munich, Germany Fishman, Barry, University of Michigan, United States Forest, Dominique, Universitat de Brest, France Freebody, Peter, The University of Sydney, Australia Frode, Guriybe, University of Bergen, Germany Furtak, Erin Marie, University of Colorado, United States Gegenfurtner, Andreas, Technical University of Munich, Germany Gerofsky, Susan, University of British Columbia, Canada Gillen, Julia, Lancaster University, United Kingdom Ginns, Paul, The University of Sydney, Australia Goldman, Susan R, University of Illinois, United States
Goldman, Ricki, New York University, United States Gomez, Kimberley, University of Pittsburgh, United States Gomez, Kimberley, University of California, United States Gonzalez, Carlos, Pontificia Universidad Catolica, Chile Gresalfi, Melissa, Indiana University, United States Groza, Gabriela, University of Illinois, United States Gruson, Brigitte, IUFM de Bretagne, France Gutierrez, Jose, University of California, United States Hajime, Shirouzu, Chukyo University, Japan Hakkarainen, Kai, University of Turku, Finland Halverson, Erica, University Of Wisconsin, United States Halverson, Richard, University of Wisconsin, United States Hawi, Nazir, Notre Dame University, Lebanon Herman, Phillip, University of Pittsburgh, United States Herold, David, Hong Kong Polytechnic University, Hong Kong Herrmann, Thomas, University of Bochum, Germany Hershkovitz, Arnon, Worcester Polytechnic Institute, United Kingdom Hesse, Friedrich, Knowledge Media Research Center, Germany Hickey, Daniel, Indiana University, United States Hmelo-Silver, Cindy, Rutgers University, United States Hoadley, Christopher, Penn State University, United States Holbert, Nathan, Northwestern University, United States Hong, Huang-Yao, National Chengchi University, Taiwan Hora, Matthew, University of Wisconsin, United States Horn, Ilana, Vanderbilt University, United States Howard, Sarah, University of Wollongong, Australia Hulshof, Casper, University of Utrecht, The Netherlands Inge, Molenaar, University of Amsterdam, The Netherlands Ingerman, ke, University of Gothenburg, Sweden Jackson, Kara, McGill University, Canada
Page viii
ISLS
ICLS2012
Jacobson, Michael, The University of Sydney, Australia Jamaludin, Azilawati, National Institute of Education, Singapore Jay, Tim, University of Bristol, United Kingdom Jochen, Rick, Saarland University, Germany Joiner, Richard, University of Bath, United Kingdom Jurow, A. Susan, University of Colorado, United States Kafai, Yasmin, University of Pennsylvania, United States Kanselaar, Gellof, Utrecht University, The Netherlands Kapur, Manu, National Institute of Education, Singapore Keifert, Danielle, Northwestern University, United States Kimmerle, Joachim, University of Tuebingen, Germany King Chen, Jennifer, University of California, United States Kirschner, Paul, Open University, The Netherlands Kollffel, Bas, University of Twente, The Netherlands Kollar, Ingo, University of Munich, Germany Koschmann, Timothy, Southern Illinois University, United States Kraemer, Nicole, University Duisburg-Essen, Germany Krauskopf, Karsten, Knowledge Media Research Center, Germany Kwon, Samuel, Learning Point Associates, United States Kyza, Eleni, Cyprus University of Technology, Cyprus Lai, Jiang, Katholieke University of Leuven, Belgium Lakkala, Minna, University of Helsinki, Finland Langer-Osuna, Jennifer, University of Miami, United States Law, Nancy, University of Hong Kong, Hong Kong Lazonder, Ard, University of Twente, The Netherlands Leary, Heather, University of Colorado, United States Lee, Kyungmee, University of Toronto, Canada Lee, Chien Sing, National Central University, China Lee, Tiffany, University of Washington, United States Lee, Victor, Utah State University, United States
Lehrer, Rich, Vanderbilt University, United States Lei, Jing, Syracuse University, United States Liao, Chang-Yen, National Central University, China Liesbeth, Kester, Open University of the Netherlands, The Netherlands Limon, Margarita, University Autonoma of Madrid, Spain Lin, Hsien-Ta, National Chengchi University, Taiwan Lindgren, Robb, University of Central Florida, United States Lindwall, Oskar, University of Gotheburg, Sweden Liu, Shiyu, University of Minnesota, United States Liu, Ru-De, Beijing Normal University, China Lone, Dirckinck-Holmfeld, Aalborg University, Denmark Lonn, Steven, University of Michigan, United States Looi, Chee Kit, National Institute of Education, Singapore Louca, Loucas T., European University, Cyprus Lowe, Ric, Curtin University, Australia Lozano, Maritza, University of California, United States Luckin, Rose, London Knowledge Lab, United Kingdom Ludvigsen, Sten, University of Oslo, Norway Lund, Kristine, CNRS, France Magnifico, Alecia Marie, University of Illinois United States Magno, Carlo, De La Salle University, Philippines Manches, Andrew, University of Nottingham, United Kingdom Martell, Sandra, National Science Foundation, United States Martin, Crystle, University of Wisconsin, United States McElhaney, Kevin, University of California, United States McGee, Steven, The Learning Partnership, Canada McKenney, Susan, Open University of the Netherlands, The Netherlands Medina, Richard, University of Hawaii, United States Mercier, Emma, Durham University, United Kingdom Miyake, Naomi, the University of Tokyo, Japan Moher, Tom, University of Illinois, United States
Page ix
ISLS
ICLS2012
Moore, Joyce, University of Iowa, United States Mor, Yishay, Open University, United Kingdom Morch, Anders, University of Oslo, Norway Moskaliuk, Johannes, Knowledge Media Research Center, Germany Munoz-Baell, Irma M, University of Alicante, Spain Myint Swe, Khine, Bahrain Teachers College, Bahrain Nckles, Matthias, University of Freiburg, Germany Naidoo, Jayaluxmi, University of KwaZuluNatal, South Africa Nicolau, Juan L., University of Alicante, Spain Noroozi, Omid, Wageningen University, The Netherlands O'Malley, Claire, University of Nottingham, United Kingdom O'Neill, Kevin, Simon Fraser University, Canada Oeberst, Aileen, Knowledge Media Research Center, Germany Olimpo, Jeffrey, University of Maryland, United States Opfermann, Maria, University of DuisburgEssen, Germany Orrill, Chandra, University of Massachusetts, United States Oshima, Jun, Shizuoka University, Japan Oztok, Murat, University of Toronto, Canada Paavola, Sami, University of Helsinki, Finland Papademetri-Kachrimani, Chrystalla, European Univesrity Cyprus, Cyprus Parisio, Martin, The University of Sydney, Australia Pea, Roy, Stanford University, United States Pedaste, Margus, University of Tartu, Estonia Penuel, William, University of Colorado, United States Pluta, William, Rutgers, The State University of New Jersey, United States Polman, Joe, University of Missouri, United States Potgieter, Marietjie, University of Pretoria, South Africa Prins, Frans, Utrecht University, The Netherlands Radinsky, Josh, University of Illinois, United States Raes, Annelies, Ghent University, Belgium Ranney, Michael, University of California, United States Reber, Rolf, University of Bergen, Norway Reimann, Peter, The University of Sydney, Australia
ISLS
Renninger, K. Ann, Swarthmore College, United States Rivet, Ann, Columbia University, United States Rogotneva, Elena, Tomsk Polytechnic University, Russia Rose, Carolyn, Carnegie Mellon University, United States Rummel, Nikol, University of Freiburg, Germany Rusman, Ellen, Open University of the Netherlands, The Netherlands Ryan, Stephanie, University of Illinois, United States Saleh, Asmalina, Indiana University, United States Sandoval, William, University of California, United States Sayre, Eleanor, Kansas State University, United States Schnotz, Wolfgang, University of KoblenzLandau, Germany Schoerning, Emily, University of Iowa, United States Schwarz, Baruch, Hebrew University, Israel Schwendimann, Beat, The University of Sydney, Australia Scott Curwood, Jen, University of Wisconsin, United States Seifert, Colleen, Univ. of Michigan, United States Sensevy, Grard, University of Western Brittany, France Shapiro, Amy, University of Massachusetts, United States Shelton, Brett, Utah State University, United States Singer, Susan, Carleton College, United States Slof, Bert, Utrecht University, The Netherlands Smith, Debbie, Clemson University, United States Solomou, Maria, Indiana University, United States Spada, Hans, University of Freiburg, Germany Specht, Marcus, Open University, The Netherlands Stager, Sarah, Pennsylvania State University, United States Stevens, Reed, Northwestern University, United States Stieff, Mike, University of Illinois, United States Strijbos, Jan-Willem, Ludwig-MaximiliansUniversity Mnchen, Germany Sudol-DeLyser, Leigh Ann, Carnegie Mellon University, United States
Page x
ICLS2012
Suthers, Daniel, University of Hawaii, United States Tan, Seng-Chee, National Institute of Education, Singapore Taylor, Martin, University of Texas, United States Tchounikine, Pierre, University of Grenoble, France Tee, Meng Yew, University of Malaya, Malaysia Thayer, Alexander, University of Washington, United States Thompson, Kate, The University of Sydney, Australia Tok, kran, Pamukkale University, Turkey Trausan-Matu, Stefan, University "Politehnica" of Bucharest, Romania Trninic, Dragan, University of California, United States Tung, I-Pei (Vicky), McGill University, Canada Turns, Jennifer, University of Washington, United States Tzialli, Dora, European University Cyprus, Cyprus van Aalst, Jan, The University of Hong Kong, Hong Kong van Amelsvoort, Marije, Tilburg University, The Netherlands van Es, Beth, University of California, United States van Oers, Bert, University Amsterdam, The Netherlands van Oostendorp, Herre, Utrecht University, The Netherlands Vandewaetere, Mieke, K.U. Leuven, Belgium Vanover, Charles, University of South Florida, United States Vatrapu, Ravi, Copenhagen Business School, Denmark Venkataswamy, Arjun, Community Links High School, United States Virnes, Marjo, University of Eastern Finland, Finland Volker, Wulf, University of Siegen, Germany Walker, Richard, The University of Sydney, Australia Walkington, Candace, University of Wisconsin, United States
Wang, Tsungjuang, National Taipei University of Technology, Taiwan Wardrip, Peter, Northwestern University, United States Warren, Scott, Indiana University, United States Wecker, Christof, University of Munich, Germany Wee, Juan Dee, National Institute of Education, Singapore Weinberger, Armin, Saarland University, Germany Weinstock, Michael, Ben-Gurion University of the Negev, Israel Wessel, Daniel, Knowledge Media Research Center, Germany Wessner, Martin, Fraunhofer IESE, Germany White, Tobin, University of California, United States Wichmann, Astrid, Ruhr University Bochum, Germany Williams, Robert, Lawrence University, United States Wodzicki, Katrin, Knowledge Media Research Center, Germany Wong, Lung-Hsiang, National Institute of Education, Singapore Wouters, Pieter, Utrecht University, The Netherlands Wu, Hsin-Kai, National Taiwan Normal University, Taiwan Yael, Kali, University of Haifa, Israel Yeo, Ai Choo Jennifer, National Institute of Education, Singapore Yilmaz, Kaya, Marmara University, Turkey Young, Michael, University of Connecticut, United States Zagal, Jose, DePaul University, United States Zahn, Carmen, Knowledge Media Research Center, Germany Zannetou, Marilyn, EUC European University of Cyprus, Cyprus Zhang, Jianwei, University at Albany, United States Zingaro, Daniel, University of Toronto, Canada Zywica, Jolene, University of Pittsburgh, United States
ISLS
Page xi
ICLS2012
ABOUT ISLS
ISLS is a professional society dedicated to the interdisciplinary empirical investigation of learning as it exists in real-world settings and how learning may be facilitated both with and without technology. ISLS sponsors two professional conferences, held in alternate years. The International Conference of the Learning Sciences (ICLS), first held in 1991 and held biannually since 1996, covers the entire field of the learning sciences. http://www.isls.org/
ABOUT "COCO" AT THE UNIVERSITY OF SYDNEY
ICLS 2012 is hosted by the Centre for Research on Computer Supported Learning and Cognition (CoCo) at the University of Sydney. CoCo's mission is to contribute to theory and research in the field of the learning sciences in order to discover how innovative learning technologies and pedagogical approaches can enhance formal and informal learning. CoCo is a University of Sydney Research Centre operating within the Faculty of Education and Social Work. http://sydney.edu.au/edsw/coco
ISLS
Page 12
ICLS2012
PREFACE
Michael J. Jacobson, Peter Reimann, and Jan van Aalst The international and interdisciplinary field of the learning sciences brings together researchers from the fields of cognitive science, educational research, psychology, computer science, artificial intelligence, anthropology, neuroscience, and other fields to study learning in a wide variety of formal and informal contexts. This field emerged in the late 1980s and early 1990s, with the first International Conference of the Learning Sciences (ICLS) being held in 1991 at Northwestern University in Evanston, Illinois, USA. Subsequent meetings of ICLS were held again in Evanston, USA (1996), and then in Atlanta, USA (1998), Ann Arbor, USA (2000), Seattle, USA (2004), Santa Monica, USA (2004), and Bloomington, USA (2006). The first ICLS to be held outside of North America was in Utrecht, the Netherlands (2008), and then back to the USA in Chicago (2010). The ICLS 2012 in Sydney is the first hosting of the conference in the Asia-Pacific region. Papers for this conference were submitted in November 2011, and then went through a process of peer review. Full papers and symposia submissions received three anonymous reviews with a member of the Program Committee summarizing the reviews and making a recommendation. Short paper poster submissions went through the same process, except with two anonymous reviews. See Table 1 for a summary of the conference paper statistics. Table 1. Paper submissions, acceptance, and rejection rates.
Full papers 264 Accepted as Full Papers: 65 (25%) Accepted as Short Paper: 54 (20%) Accepted as Poster: 42 (16%) Rejected: 103 (39%) Short Papers 79 Accepted: 15 (19%) Accepted as poster: 11 (14%) Rejected: 53 (67%) Posters 76 Accepted: 37 (49%) Rejected: 39 (51%) Symposia 27 Accepted: 18 (67%) Rejected: 9 (33%) Total Submissions: 446
The final papers included in the ICLS 2012 Proceedings are for 18 symposia, 60 full papers, 61 short papers, and 62 poster papers, as well as abstracts for the four keynote talks and three special sessions, including the invited Presidential Session on The Future of Learning, and nine workshops. The themes reflected in the papers and presentations at ICLS 2012 cover a wide range of issues and research areas. Some papers deal with long standing theoretical issues, such as conceptual change and knowledge transfer, whereas other papers report on new learning research in conventional subject areas such as science, mathematics, and literacy. Research is also reported on newer knowledge areas such as complex systems, as well as more recent
ISLS Page xiii
ICLS2012
perspectives on pedagogy and learning such as knowledge building, inquiry, and productive failure. A number of learning sciences research projects internationally are also exploring ways in which innovative environments for learning may be designed with new technologies such virtual and game environments, modeling and visualization systems, robotics, collaboration technologies, mobile and hand held devices, and educational data mining and learning analytics. Emerging areas in learning sciences research that perhaps may lead to newer perspectives and directions in our understanding of learning include creatively, identity, and embodied cognition. The field continues to engage the broad issues of contributing to and impacting policy and practice more generally. Overall, the research presented in the proceedings of ICSL 2012 contributes numerous perspectives on the conference theme of the future of learning, both as trajectories of research that have been maturing over a number of years and bold new perspectives that promise to shape new trajectories for the future. Making this conference possible, we thank the hard work and the countless hours put in by our international Conference Advisory Board and the various conference subcommittees. We would like to thank all those who assisted with the review process. The Conference is also fortunate to have a number of financial sponsors whose support contributes to conference events: NSW Trade & Investment, Smart Services Collaborative Research Centre, United States National Science Foundation, The International Society of the Learning Sciences, Asia-Pacific Society of Computers in Education, Inquirium, LLC, and The Centre for Research on Computer Supported Learning and Cognition. Any conference is but a snapshot in the dynamic process of articulating and vetting scientifically principled ideas and approaches. That the International Conference of the Learning Sciences has now had its 10th meeting and has entered its third decade are exciting milestones that this proceedings helps to document. We close by reflecting that just as a great movie is about the journey of discovery and development that the characters experience, so might our research field be a journey of discovery and development to more deeply understand how people learn now and as the future unfolds, knowing learning itself as a core essence of life and our humanity.
ISLS
Page xiv
ICLS2012
CONTENTS - VOLUME 2
KEYNOTES
Envisioning the Next Generation Classroom and the Next Generation of Learning Technologies
Janet L. Kolodner
Productive Failure
Manu Kapur
2 3 4
Lifelong Learning as a Driver for Designing Pervasive Technologies

Judy Kay
Classroom Orchestration: Interweaving Digital and Physical Workflows

Pierre Dillenbourg
INVITED SPECIAL SESSIONS

Invited Presidential Session: The Future of Learning and the Learning Sciences
Susan R. Goldman, Louis M. Gomez, Naomi Miyake, Nichole D. Pinkard, Nikol Rummel
Invited Panel Sponsored by the ISLS Membership Committee: Linking Learning Sciences Communities across the World
Nancy Law, Baohui Zhang, Konstantin . Kolin, Paulo Blikstein, Miguel Nussbaum, Michael Jacobson
SYMPOSIA
Hybrid Spaces for Science Education
Ole Smrdal, Jim Slotta, Ingeborg Krange, Tom Moher, Francesco Novellis, Alessandro Gnoli, Brenda Lopez Silva, Michelle Lui, Alfredo Jornet, Cecilie F. Jahreie
Building Upon What Is Already There: The Role of Prior Knowledge, Background Information, and Scaffolding in Inquiry Learning
Christof Wecker, Ard Lazonder, Jennifer Chiu, Cheryl Madeira, Jim Slotta, Yvonne Mulder, Ton de Jong, Alexander Rachel, Hartmut Wiesner, Peter Reimann
17
Assessing Interests in the Service of Supporting Personalized Learning Through Networked 25 Resources
Brigid J. Barron, Robert B. W. Ely, Mary D. Ainley, K. Ann Renninger
Supporting Teachers in Capturing and Analyzing Learning Data in the Technology-Rich Classroom
Peter Reimann, Friedrich Hesse, Gabriele Cerniak, Rose Luckin, Ravi Vatrapu, Susan Bull, Matthew Johnson, Wolfgang Halb, Wilfrid Utz, Michal Kossowski
33
Gathering Evidence of Scientific Argumentation Practices: From Pre-Kindergarten to High 41 School

Tiffany Lee, Hiroki Oura, Giovanna Scalone, Kari Shutt, John Bransford, Andrew Shouse, Katie Van Horne, Nancy Vye, Maureen Munn, Randy Knuth
Engaging Middle School-Aged Students in Classroom Science and Mathematics: Implications for Design and Research
Kimberley Pressick-Kilborn, Melissa Gresalfi, K. Ann Renninger, Jessica Bachrach, Nicole Shechtman, Britte Cheng, Patrik Lundh, Gucci Trinidad, Richard Walker
49
Robot Facilitation as Dynamic Support for Collaborative Learning

Naomi Miyake, Sandra Okita, Carolyn Rose
57
Embedded Phenomena for Knowledge Communities: Supporting complex practices and interactions within a community of inquiry in the elementary science classroom Rebecca Cober, Cresencia Fong, Alessandro Gnoli, Brenda Lpez Silva, Michelle Lui, Cheryl
64
ISLS
Page xv
ICLS2012
Madeira, Colin McCann, Tom Moher, Jim Slotta, Mike Tissenbaum
Critical Aspects in Learning with Technologies

Ching Sing Chai, Huang Yao Hong, Hwee Ling Koh, Naomi Miyake, Chwee Beng Lee, Choon Lang Quek, Peter Reimann
72
Interactive Surfaces and Spaces: A Learning Sciences Agenda

Michael A. Evans, Jochen Rick, Michael Horn, Chia Shen, Emma Mercier, James McNaughton, Steve Higgins, Mike Tissenbaum, Michelle Lui, James D. Slotta
78
Instructional Approaches to Promote Conceptual Change

Stella Vosniadou, Naomi Miyake, Marcia Linn, Stellan Ohlsson, David D. Cosejo, Michael J. Jacobson, Doug Clark
86
Everyday Interactions and Activities: Field Studies of Early Learning Across Settings
Reed Stevens, Lauren Penney, Danielle Keifert, Pryce Davis, Siri Mehus, RichLehrer
91 99
You're It! Body, Action, and Object in STEM Learning

Dor Abrahamson, Carmen Petrick ,David DeLiema, Mina C. JohnsonGlenberg, David Birchfield, Tatyana Koziupa, Caroline Savio-Ramos, Julie Cruse, Robb Lindgren, Cameron Fadjo, John B. Black, Michael Eisenberg
The Future of Assessment: Measuring Science Reasoning and Inquiry Skills Using Simulations and Immersive Environments
Jodi Davenport, Edys Quellmalz, Jody Clarke-Midura, Chris Dede, Janice Gobert, Kenneth Koedinger, Marty McCall, Michael Timms
110
Scripting Science Inquiry Learning in CSCL Classrooms

Annelies Raes, Tammy Schellens, Bram De Wever, Ingo Kollar, Christof Wecker, Frank Fischer, Mike Tissenbaum, Jim Slotta, Vanessa L. Peters, Nancy Songer
118
Learning as Identity Formation: Implications for Design, Research, and Practice

Rick Jochen, Ben DeVane, Tamara Clegg, Vanessa Peters, Nancy Songer, Susan R Goldman, Cindy Hmelo-Silver
126
Building Bridges between Learning Analytics, Educational Data Mining and Core Learning 134 Sciences Perspectives
Paulo Blikstein, Marcelo Worsley, Bruce Sherin, Ryan Baker, Matthew Berland, Taylor Martin, Ido Roll, Vincent Aleven, Ken Koedinger, Arnon Hershkovitz
The Use of Game Design, Social Learning Networks, and Everyday Expertise to Engage Youth with Contemporary Science
Philip Bell, Leah Bricker, Katie Van Horne, Theresa Horstman, Nichole Pinkard
142
SHORT PAPERS
Developing Primary Students' Argumentation Skills in Inquiry-Based Mathematics Classrooms
Jill Fielding-Wells, Katie Makar
149
Language Formality, Learning Environments, and Student Achievment

Emily Schoerning, Brian Hand
154
Memetic Processes as Conceptual Framework for Idea Improvement in Knowledge Building 157
Karsten Krauskopf, Johanna Bertram, Ya Ping (Amy) Hsiao, Stefan Huber, Katherine Panciera, Nicole Strfling, Astrid Wichmann, Jan van Aalst
Unraveling Idea Development in Discourse Trajectories

Iassen Halatchliyski, Aileen Oeberst, Martina Bientzle, Franziska Bokhorst, Jan van Aalst
162
Investigating Emergent Dynamics to Understand Interactions in Small Professional Development Groups

Susan Yoon
167
What a Long Strange Trip It's Been: A Comparison of Authors, Abstracts, and References in 172 the 1991 and 2010 ICLS Proceedings
ISLS Page xvi
ICLS2012
Victor Lee, Lei Ye, Mimi Recker
Scaffolding Collaborative Sensemaking during Critique of Explanations in TechnologyEnhanced Science Curriculum

Elissa Sato, Vanessa Svihla
177
Collaborative Reading Comprehension with Communication Robots as Learning Partner

Jun Oshima, Ritsuko Oshima, Naomi Miyake
182
Multi-Touch Technology to Support Multiple Levels of Collaborative Learning in the Classroom

Emma Mercier, Steve Higgins, Elizabeth Burd, Andrew Joyce-Gibbons
187
The SunBay Digital Mathematics Project: An Infrastructural and Capacity-Based Approach 192 to Improving Mathematics Teaching and Learning at Scale
Charles Vanover, George Roy, Zafer Unal, Vivian Fueyo, Phil Vahey
Measuring Sophistication of Epistemic Beliefs Using Rasch Analysis

Johan van Strien, Monique Bijker, Saskia Brand-Gruwel, Henny P. A. Boshuizen
197 202
Establishing a Mathematical Practice in a Middle School Classroom

Marta Kobiela, Rich Lehrer
Redesigning Classroom Learning Spaces: When Technology Meets Pedagogy and When They Clash
Elizabeth Charles, Nathaniel Lasry, Chris Whittaker
207
Bridging Design and Practice: Towards a Model-based Collaborative Inquiry Science Learning Environment
Daner Sun, Chee-kit Looi, Baohui Zhang
212
Identity and Digital Media Production in the College Classroom

Erica Halverson, Michelle Bass
217
Mathematics Learning in a Racial Context: Unpacking Students' Reasoning about "Asians are Good at Math"
Niral Shah
222
Dimensions of Scaffolding in Technology-Mediated Discovery Learning Environments

Mark J. W. Lee, Barney Dalgarno
227
How a Power Game Shapes Expressing Opinions in a Chat and in an Argument Graph during a Debate: A Case Study
Gaelle Molinari, Kristine Lund
232
Using the Learner-Generated Drawing Strategy: How Much Instructional Support Is Useful? 237
Annett Schmeck, Luisa Amelie Friedrich, Maria Opfermann, Detlev Leutner
Unpacking traces of collaboration from multimodal data of collaborative concept mapping at 241 a tabletop
Roberto Martinez Maldonado, Judy Kay, Kalina Yacef, Beat Schwendimann
Motivation in the Learning Sciences: Connecting to Practice by Design

Phillip Herman, Peter Wardrip, Jolene Zywica
246
Supporting Student Learning using Conversational Agents in a Teachable Agent Environment

James Segedy, John Kinnebrew, Gautam Biswas
251
Epistemic Mentoring in Virtual and Face-to-Face Environments

Elizabeth Bagley, David Williamson Shaffer
256
Two Models of Authenticity: Signature Pedagogy, Problem Based Learning, and Cultural Context
261
ISLS
Page xvii
ICLS2012
Moshe Krakowski
New Evidence on Productive Failure - Building on Students' Prior Knowledge is Key!

Katharina Westermann, Nikol Rummel
266
Explanandum and visibility condition change children's gesture profiles during explanation: 271 implications for learning?
Audrey Mazur-Palandre, Kristine Lund
Bidirectional artifact analysis: A method for analyzing creative processes

Alecia Marie Magnifico, Erica Rosenfeld Halverson
276
Learning from the Folly of Others: Learning to Self-Correct by Monitoring the Reasoning of 281 Projective Pedagogical Agents
Sandra Okita, Azadeh Jamalian
Understanding Influence in Collaborative Group Work: The Importance of Artifacts

Lesley Dookie, Indigo Esmonde
286
Theoretical Issues: Indicators of Decentralized and Centralized Causality as a Gauge for Students Understanding of Complex Systems
Lauren Barth-Cohen
291
Engaging in Design: Reflections of Young Paper Engineers

Rob Rouse, Richard Lehrer
296 301 306
Using Critical Reading Tools to Facilitate the Learning of Argumentation Skills

Jingyan Lu, Nancy Law
Changing Explanations about Sand Dune Movement

Lauren Barth-Cohen
High School Students' Epistemic Engagement in Producing Documentaries about Public Science Concerns
David J. DeLiema, Jarod N. Kawasaki, William A. Sandoval
311
Improving Middle School Students' Understanding of Core Science Ideas Using Coherent Curriculum
Joseph Krajcik, Sung-Youn Choi, Namsoo Shin, LeeAnn M. Sutherland
316
Physical Activity Data Use by Technoathletes: Examples of Collection, Inscription, and Identification
Victor Lee, Joel Drake
321
Locating the Development of Interest: Tools for Studying the Mutual Constitution of Persons and Cultural Practices in Places
William Penuel, John H. Falk, Lynn D. Dierking, Ben Kirshner, Julie Haun-Frank, Adam J. York
326
What to Look for and What to Do: Novice Teachers' Abilities for Noticing and Responding 331 to Their Students' In-Class Inquiry
Loucas Louca, Thea Skoulia, Dora Tzialli
Developing a Technology-Enhanced Scientific Inquiry Curriculum in a Primary School: Outcomes of the School-Based Support
Yau-yuen Yeung, Zhihong Wan, Winnie Wing-mui So
336
Towards Teaching Analytics: Repertory Grids for Formative Assessment

Ravi Vatrapu, Peter Reimann, Abid Hussain
341
The Teacher's Balance between Structure and Flexibility in the Technology-Enhanced Collaborative Inquiry Setting
Marjut Viilo, Pirita Seitamaa-Hakkarainen, Kai Hakkarainen
346
ISLS
Page xviii
ICLS2012
Scaffolding and Assessing Knowledge Building among Chinese Tertiary Students Using E351 portfolios
Chunlin Lei, Carol K.K. Chan
Contribution of Motivational Orientations to Student Outcomes in a Discovery-Based Program of Game Design Learning
Rebecca Reynolds, Ming Ming Chiu
356
The Knowledge connections Analzer

Jan van Aalst, Carol Chan, Stella W. Tian, Christopher Teplovs, Yuan Y. Chan, Wing-San Wan
361
Effects of Argumentation Scaffolds and Problem Representation on Students' Solutions and 366 Argumentation Quality in Physics
Carina M. Rebello, Eleanor Sayre, N. Sanjay Rebello
Inevitable Breakdowns in Putting Argumentation into Practice

Baruch B. Schwarz, Nitza Shahar
371
Multiple Solutions and Their Diverse Justifications to the Service of Learning in Early Geometrical Problem Solving
Naomi Prusak, Rina Hershkowitz, Baruch Schwarz
376
ChemVLab+: Evaluating a Virtual Lab Tutor for High School Chemistry

Jodi Davenport, Anna Rafferty, Michael Timms, David Yaron, Michael Karabinos
381
Legitimate Peripheral Participation in Academic Communities of Practice How Newcomers' Learning is Supported in Student Councils
Julia Eberle, Karsten Stegmann, Frank Fischer
386
Characterizing Collaboration with Metapragmatics: Using PreK Virtual Manipulatives on a 391 Multi-Touch Tabletop
Michael Evans, Kamala Russell, Michaela Hnizda, David McNeill
Development and Validation of a Scale to Place Students along a Learning Progression

Namsoo Shin , Shawn Y. Stevens
396 401 406 411
Learners as Informants of Educational Game Design

Beaumie Kim, Lynde Tan, Mi Song Kim
The Role of Gesture in Solving Spatial Problems in STEM

Matthew Lira, Mike Stieff, Stephanie Scopelitis
Audience Effects: A Bidirectional Artifact Analysis of Adolescents' Creative Writing

Alecia Marie Magnifico
The Use of Text and Process Mining Techniques to Study the Impact of Feedback on Students' Writing Processes
Rafael Calvo, Anindito Aditomo, Vilaythong Southavilay, Kalina Yacef
416
Variation in Fifth Grade Students' Propensities for Managing Uncertainty during Collaborative Engineering Projects
Michelle Jordan
424
Breeding Birds to Learn about Artificial Selection: Two Birds with One Stone?
Aditi Wagh, Uri Wilensky
426
Tracing Ideas and Participation in an Asynchronous Online Discussion across Individual and 431 Group Levels over Time
Alyssa Friend Wise, Ying-Ting Hsiao, Farshid Marbouti, Yuting Zhao
Scaffolding a Knowledge Community for High School Physics

Mike Tissenbaum, Jim Slotta
436 441
Examining System Dynamics Models Together: Using Variation Theory to Identify
ISLS
Page xix
ICLS2012
Learning Opportunities in Online Collaboration

Anindito Aditomo, Kate Thompson, Peter Reimann
Effects of Computer-Supported Collaboration Scripts on Domain-Specific and DomainGeneral Learning Outcomes: A Meta-Analysis
Freydis Vogel, Ingo Kollar, Frank Fischer
446
POSTERS
Computer Input Capabilities that Stimulate Diagramming and Improved Inferential Reasoning in Low-performing Students
Sharon Oviatt, Kumi Hodge, Andrea Miller
451
Alternative Spaces for Engagement: Performance and conversation in the Connected Classroom
Rachel Perry, Matthew Kearney
453
The Comparison of The Reinvestment of Collaborative Asynchronous Discourse Observed 455 by Two Main Actors of Pre-Service Teacher Education
Stephane Allaire
Effectiveness of Combining Worked Examples And Deliberate Practice for High School Geometry
Mariya Pachman, John Sweller, Slava Kalyuga
457
Can Technology-Based Gaze Replays of Experts Model Diagnostic Performance of Novices? A Test in Medical Education
Marko Seppnen, Andreas Gegenfurtner
459
Facilitating Teachers' Integrated Technological Pedagogical Content Knowledge

Hsin-Yi Chang
461
Use of A CSCW Platform by Trainers and Trainees. Trace Analysis: Multimodal Analysis Vs Data Mining approach
Jean Simon, Henri Ralambondrainy
463
Social Network Analysis for Knowledge Building: Establishment of Indicators for Collective Knowledge Advancement
Jun Oshima, Yoshiaki Matsuzawa, Ritsuko Oshima, Carol Chan, Jan van Aalst
465
Advancing Understanding Using Nonaka's Model of Knowledge Creation and ProblemBased Learning
Meng Yew Tee, Shuh Shing Lee
467
The Idea Manager: A Tool to Scaffold Students in Documenting, Sorting, And Distinguishing Ideas During Science Inquiry
Camillia Matuk, Kevin McElhaney, Jennifer King Chen, David Miller, Jonathan Lim-Breitbart, Marcia Linn
469
Teacher Education Students' Research Training And E-Research: Current Perspectives And 471 Potential for Development
Carlos Gonzlez
Promoting Teacher Candidates' Awareness for Teaching Dilemmas: A Field Experiment

Elisabeth Wegner, Iris Kaufmann, Matthias Nckles
473
The Candy Factory Game: An Educational iPad Game for Middle School AlgebraReadiness
Michael Evans, Anderson Norton, Kirby Deater-Deckard, Mido Chang
475
Evaluating Claims in Popular Science Media: Nature of Science Versus Dynamic Epistemological Knowledge
Pryce Davis, Rosemary S. Russ
477
ISLS
Page xx
ICLS2012
Adolescent Profiles of Knowledge And Epistemic Beliefs in the Context of Reading Multiple Texts
Leila Ferguson, Ivar Brten, Helge I. Strms, istein Anmarkrud
479
Improving Americans' Modest Global Warming Knowledge in the Light of RTMD (Reinforced Theistic Manifest Destiny) Theory
Michael Ranney, Dav Clark, Daniel Reinholz, Sarah Cohen
481
Improving Students' Scientific Reasoning Skills via Virtual Experiments and Worked Examples
Shiyu Liu, Keisha Varma
483
Synthesizing Quantitative Predictors for Interaction in an Asynchronous Online Course

Daniel Zingaro, Murat Oztok, Jim Hewitt
485
Engines of Representation: Processing Raw Student Data into Useable Student Information 487
Suzanne Rhodes, Halverson Richard
The Role of Technologies in Facilitating Collaborative Engagement

Suparna Sinha, Karlyn Adams, Toni Kempler- Rogat, Cindy Hmelo-Silver
489
Integrating Insights from Critical Race and Queer Theories with Cultural-Historical Learning Theory
Indigo Esmonde, Miwa Takeuchi, Lesley Dookie
491
Iterative Technology-based Design with Deaf/Hard of Hearing Populations: Working with Teachers to Build a Better Educational Game
Brett Shelton, Mary Ann Parlin, Jon Scoresby, Vonda Jump, Claudia Pagliaro
493
Enhancing Students' Understanding of What Ideas Are for Knowledge Building

Chieh-Hsin Chiu, Huang-Yao Hong
495
Teacher-Education Students' Beliefs in Science Teaching in a Knowledge Building Environment

Chih-Hsuan Chang, Huang-Yao Hong
497
Investigating Teacher Change in Creating Classroom Knowledge Building Communities

Katerine Bielaczyc, Sunhee Paik, John Ow
409 501
Transformative Experiences and Biological Evolution: Facilitating Deep Engagement

Benjamin Heddy, Gale Sinatra
Argument-based Inquiry and Students with Disabilities: Improving Critical Thinking Skills 503 and Science Understanding
Jonte Taylor, William Therrien, Brian Hand
Fostering Teachers' Use of Talk Moves to Promote Productive Participation in Scientific Practices
William Penuel, Yves Beauvineau, Angela Haydel DeBarger, Savitha Moorthy
505
Inquiry in the Kindergarten Science: Helping Kindergarten Teachers to Implement Inquiry507 Based Teaching
T. L. Loucas, Dora Tzialli, Constantinos Constantinou
Metacognizing Across Self And Socio Dialectics

Azilawati Jamaludin, David Hung
509
Combining Knowledge And Interaction Perspectives to Decipher Learning During A Clinical Interview
Shulamit Kapon
511
Asking for Too Much Too Early? Promoting Mechanistic Reasoning in Early Childhood Science And Mathematics Education
513
ISLS
Page xxi
ICLS2012
Loucas Louca, Chrystalla Papademetri-Kachrimani
SMILE - Smartphones in A University Learning Environment: A Classroom Response System

Celia Kaendler, Linus Feiten, Katrin Weber, Michael Wiedmann, Manuel Buehrer, Sebastian Sester, Bernd Becker
515
Stability and Change in Achievement Goals and Transfer

Erkka Laine, Andreas Gegenfurtner
517
Commercial Development of e-Learning Materials for Science and Mathematics Subjects in 519 Hong Kong: Preliminary Evaluation
Yau-yuen Yeung, Irene Chung-man Lam
Relationships between Representational Characteristics, Students' Education Levels, and Beliefs of Models
Silvia Wen-Yu Lee, Hsin-Yi Chang, Hsin-Kai Wu
521
Matchballs - A Casual Game for Learning and Domain Ontology Enrichment

Sabrina Ziebarth, Nils Malzahn, H. Ulrich Hoppe
523
Becoming a Writer: Examining Preschoolers' Interactions, Modes of Participation, and Use 525 of Resources at the Writing Center
Amy Gillespie, Deborah Rowe
Fifth and Seventh Graders' Patterns of Understanding About Cells and Heredity in a Technology-Enhanced Curriculum
Dante Cisterna, Michelle Williams, Joi Merritt
527
Ontological Stances and Systems Thinking in the Favela and Asfalto

Izabel Duarte Olson
529
Challenges of Teaching through Web-Based Inquiry: A Longitudinal Case Study of A Veteran High School Teacher
Eleni Kyza, Iolie Nicolaidou
531
Creating and Sustaining Online Communities of Practice for Science Teachers' Professional 533 Development: Overcoming the Barriers
Yiannis Georgiou, Eleni A. Kyza, Andri I. Ioannou
Reasoning About Mechanism: Children's Explanations of Pop-ups

Mayumi Shinohara, Rob Rouse
535 537
Resources for Reasoning: Use of Video and Text in a CSCL Environment

Cindy Hmelo-Silver, Carolyn Maher, Marjory Palius, Robert Sigley
Weaving Together Parts to Achieve A Whole: Gestural Activity for the Coordination of Information in the Teaching and Learning of Chemistry
Stephanie Scopelitis, Mike Stieff
539
Participatory Learning and Assessment in e-Learning Contexts

Daniel Hickey, Andrea Strackeljahn, Rebecca Itow
541
Irreducible Complexity: How Do Causal Bayes Nets Theories of Human Causal Inference Inform the Design of a Virtual Ecosystem?
M. Shane Tutwiler, Tina Grotzer
543
From Tacit Knowing to Explicit Explanation: Mining Student Designs for Evidence of Systems Thinking
Melissa Gresalfi, Leon Gordon, Sinem Siyahhan
545
The Authority of Ideas: How Students Become Influential in Linguistically Heterogeneous Small Group Discussions
Jennifer Langer-Osuna
547
ISLS
Page xxii
ICLS2012
The Power of Improvement Networks to Transform Educational Inquiry: A Preliminary Exploration

Jonathan Dolle, Peter Wardrip, Jennifer Russell, Louis Gomez, Anthony Bryk
549
Understanding Teachers' Cultural Models about Technology

Jen Scott Curwood
551
Design-Based Implementation Research of Spreadable Educational Practices within the Participatory Learning and Assessment Network (PLAnet)
Rebecca Itow, Daniel Hickey
553
Picture-Based Science Attitudes Assessment

Emily Johnson, Amy Bolling, Robb Lindgren
555
Connecting Visitors to Exhibits through Design: Exploring United States census data with CoCensus
Jessica Roberts, Leilah Lyons, Joshua Radinsky, Francesco Cafaro
557
Methods of analysis for identifying patterns of problem solving processes in a computersupported collaborative environment
Shannon Kennedy-Clark, Kate Thompson
559
An Eye For Detail: Techniques For Using Eye Tracker Data to Explore Learning in Computer-Mediated Environments
Marcelo Worsley, Paulo Blikstein
561
Finding the Common Thread: Learners' Intuitive Knowledge of General Patterns that Apply 563 Across Domains
Hillary Swanson
A Case Study of P2PU: New Models for Open and Peer-Focused Learning
Monica Resendes, Stian Haklev
565
UltraLite Collaboration: A Low-Cost Toolkit to Promote Collaborative Learning in the Classroom

Claire Rosenbaum, Paulo Blikstein, Patti Schank, Ken Rafanan, Jeremy Roschelle
567
Learning to Graph: A Comparison Study of Using Probe or Draw Tools in a Web-Based Learning Environment
Libby Gerard, Amber Zertuche, Marcia Linn
569
Bifocal Biology: Combining Physical and Virtual Labs to Support Inquiry in Biological Systems
Tamar Fuhrmann, Daniel Greene, Shima Salehi, Paulo Blikstein
571
Design-Based Research in Practice: A Technology-Based Classroom Experiment that Explores How Students Use Virtual Manipulatives to Order Groups of Fractions
Maria Mendiburo, Gautam Biswas, Ted Hasselbring
573
Learning by Collaborative Design and Evaluation

Shannon Kennedy-Clark, Vilma Galstaun, Martin Parisio, Kate Anderson
575
WORKSHOP ABSTRACTS
Bifocal Modeling: Combining Virtual and Physical Experiments in Real Time Using Lowcost Sensors and Open-source Computer Modeling
Paulo Blikstein, Tamar Fuhrmann, Daniel Greene, Shima Salehi, Claire Rosenbaum, Marcelo Worsley
577
Classroom Orchestration: Moving Beyond Current Understanding of the Field

Yannis Dimitriadis, Pierre Dillenbourg, Miguel Nussbaum, Chee-Kit Looi, Jeremy Roschelle
579
Re-thinking Roles in Future Learning: Teachers as Curriculum Media Designers and Students as Teacher Collaborators
581
ISLS
Page xxiii
ICLS2012
Eric Hamilton, Gina Chaves, Wendy Chaves, Israel Ramirez
Teachers as Designers of Technology-Enhanced Learning Materials

Yael Kali, Susan McKenney
582 584
Inducing Learners Creativeness and Encouraging Learning Across Disciplines

Moseli Mafa, Palesa Khotso, Mamotena Jankie, Makomosela Qhobela
Developing a Competitive Educational Research Proposal for NSFs Division of Research on Learning
Sandra Toro Martell, Janet L. Kolodner
586
Digital Ecosystems for Collaborative Learning: Embedding Personal and Collaborative Devices to Support Classrooms of the Future
Roberto Martinez, James Slotta, Pierre Dillenbourg, Andrew Clayphan, Mike Tissenbaum, Beat Schwendimann and Chirstopher Ackad
588
Tightening Research-Practice Connections: Applying Insights and Strategies during Design 590 Charrettes
Susan McKenney, Kimberley Gomez, Brian Reiser
Analyzing Collaborative Learning at Multiple Levels

Gerry Stahl, Keith Sawyer, Heisawn Jeong, Dan Suthers
592 593
AUTHOR INDEX
ISLS
Page xxiv
ICLS2012
Volume 2: Keynotes
Envisioning the Next Generation Classroom and the Next Generation of Learning Technologies
Keynote 1 by Janet L. Kolodner Regents Professor, Georgia Institute of Technology, United States jkolodne@nsf.gov
Abstract
Back in 1990, when I founded The Journal of the Learning Sciences, I expressed the hope, in my opening editorial, that what we learned about learning as learning scientists would make a real difference in the world. That hasnt happened as fast as I was looking forward to. On the other hand, we know a lot more about learning and promoting learning some two decades later, and in the past few years, weve seen a proliferation of new technologies that may make it easier to transition what we have been learning about learning into real use. In this talk, I will present some of the promise I see in what our community has been doing over the past two decades, some opportunities I see for increasing and transforming the opportunities people have for learning, and some of the challenges to making real-world contributions through our research. Some of those challenges have to do with systems in place in the world, and some have to do with how we organize ourselves as a community of researchers. The vision I will present is drawn from my personal experiences as Editor in Chief of JLS for 18 years, as a pedagogy designer and curriculum creator (who co-authored a wonderful science curriculum that is not selling as well as wed like), as a developer of learning technologies (who could never manage to get any of them used), and most recently, as a program officer at the US National Science Foundation. Note, however, that I wont give away any NSF secrets, and my talk will not in any way represent plans for support of learning sciences and learning technologies at NSF.
ISLS
ICLS2012
Volume 2: Keynotes
Productive Failure
Keynote 2 by Manu Kapur Associate Professor of Curriculum, Teaching & Learning, National Institute of Education, Singapore manu.kapur@nie.edu.sg
Abstract
I will describe my program of research on productive failure (PF), which is now into its ninth year. PF is a learning design that leverages students formal as well as intuitive ideas about concepts they have yet to learn to design learning. PF provides opportunities for students to design solutions to complex, novel problems even though they may fail to produce canonical solutions in the process. This failure however can potentially be the locus of deep learning as it affords students the opportunity to collaboratively generate, elaborate, critique, and refine their representations and solution strategiesa process that is germane for learning. I will share empirical evidence from a series of experimental and quasi-experimental studies with 7th-11th graders in Singapore and Indian mathematics classrooms, and discuss implications for theory and practice.
ISLS
ICLS2012
Volume 2: Keynotes
Lifelong Learning as a Driver for Designing Pervasive Technologies

Keynote 3 by Professor Judy Kay University of Sydney, Australia Judy.Kay@sydney.edu.au
Abstract
This talk presents a vision for lifelong learning as a driver for designing pervasive technologies. It does this via case studies for challenging long-term learning goals associated with health and wellness, collaboration to learn and learning to collaborate. One strand of that vision involves the new learning interfaces across each learner's personal digital ecosystem of devices, ranging from mobiles, to desktops and embedded interactive surfaces on walls and tables. A second strand concerns the huge amounts of data that these devices can, and do, capture about learners. This takes diverse forms, including personal information, learning data and digital footprints. There is a huge and growing amount of this data. It lives across the personal digital ecosystem, on personal devices and in the cloud. The talk illustrates the design of technologies to give this data to the learner at three levels. One concerns learner control over the capture and use of their data. Another involves data mining to transform it into new insights for the learner, their teachers and facilitators, and for education researchers. The final one is the design of interfaces such as in Open Learner Models to scaffold the learner's metacognitive activities of self-monitoring, self-reflection and planning. This keynote is kindly sponsored by NSW Trade & Investment.
ISLS
ICLS2012
Volume 2: Keynotes
Classroom Orchestration: Interweaving Digital and Physical Workflows

Keynote 4 by Professor Pierre Dillenbourg EPFL Switzerland pierre.dillenbourg@epfl.ch
Abstract
Why should students login in an educational tool? The problems that occur at login time may take 2 or 3 minutes, which wastes 5% of the lesson time. Do the functionalities enabled by the login compensate this loss by 5% additional learning gains? This anecdote illustrates the practical flavor of 'classroom orchestration'. I will argue that learning sciences should not only investigate how people learn but also how teachers manage multiple constraints such as time, discipline, physical space, etc... what Nussbaum called the 'logistics' of schools. Some principles emerged for reducing orchestration load: flexibility, control, visibility, physicality and minimalism. They constitute what I refer to as 'modest computing'. But, how to turn this set of anecdotes, examples or principles into the beginning of a theory? One step forward consists in modeling a classroom as two semi-mirrored workflows, a digital and a physical, connected by a certain number of contact nodes. For instance, our experience is that paper-based interfaces are effective orchestration tools, probably because they implement these dual workflows. These findings are close to Hutchins' analysis of the relevance of paper cards in the information flows of an aircraft cockpit. The complexity we face is that, in the context of a classroom, the pilot is alone and he has to serve food to all passengers.
ISLS
ICLS2012
Volume 2: Special Sessions
Invited Presidential Session: The Future of Learning and the Learning Sciences
Chair: Susan R. Goldman, University of Illinois at Chicago, sgoldman@uic.edu Louis M. Gomez, UCLA, Carnegie Foundation for the Advancement of Teaching, lmgomez@ucla.edu Naomi Miyake, School of education, University of Tokyo, Professor, Consortium of Renovating Education of the Future, Deputy Director, nmiyake@p.u-tokyo.ac.jp Nichole D. Pinkard, College of Computing and Digital Media, DePaul University, nicholepinkard@gmail.com Nikol Rummel, Institute of Educational Research, Ruhr-Universitt Bochum, nikol.rummel@rub.de
Introduction
In keeping with the ICLS 2012 Conference Theme the Future of Learning and the Learning Sciences, this invited session features four representatives of the Learning Sciences community who will present their visions of the future. The panelists span several generations, countries, and areas of interest. Participants are Louis Gomez, Naomi Miyaki, Nichole Pinkard, and Nikol Rummel. Following their presentations, the panelists and audience will engage in a dialogue about the future. Brief summaries of each presentation are provided. Susan R. Goldman, President of ISLS, will chair the session.
Improvement? The next Wave in the Learning Sciences

Louis M. Gomez Professor of Education, UCLA Senior Fellow, Carnegie Foundation for the Advancement of Teaching lmgomez@ucla.edu Abstract: From the beginning, we, in the learning sciences, have sought to steer a course of scholarship that would guide our efforts so that they would amount to something important in the theater of practice. We have used tools, like design, engineering and implementation and novel social arrangements, like communities of practice to build our base of expertise and our knowledge claims that we matter, to practice. The intentional focus on artifacts and collaborative arrangements has led to notable successes. Yet the problems of practice still loom large. We need to identify ways to work that will accelerate our progress on the enduring problems that vex our ability to provide learning environments that work that work for large populations for diverse learners. I will suggest that Educational Improvement Science might be one way to accelerate our progress. In our conception, Educational Improvement Science draws directly on the ideas of quality improvement pioneered by W. Edwards Demming and Walter A. Shewhart, among others. Their ideas have revolutionized fields from manufacturing to healthcare. We also draw on Douglas Englebart's insights about energizing, within diverse collaboratives, to learn together. Only time will tell if these ideas will gain traction in the next wave of learning sciences. Their promise merits our consideration.
Renovating education of the future: Three scenarios

Naomi Miyake School of education, University of Tokyo, Professor Consortium of Renovating Education of the Future, Deputy Director nmiyake@p.u-tokyo.ac.jp Abstract: I discuss three potential solutions to the issue of sharing with educational practitioners and students the new knowledge being generated by the learning sciences community. The solutions reflect things that are doable now as well as things that will be doable in the foreseeable future. Restructure basic understanding of how collaborations increase understanding of all participants. Engage teachers and researchers in collaborative analysis of their collaborative learning processes. This makes more transparent to all the nature of learning and understanding. Create a dependable and sustainable cloud to enrich classrooms with professional wisdom. Remotely operated robots provide us with new tools for all of the above. They can be mixed among the students in collaborative classes, to let us experiment on what kinds of relation-making foster the learning abilities among the kids. They could work to solicit and record utterances and behaviors of the learners. They would also form a reasonable interface for the people with professional wisdom to answer questions coming from teachers and learners, guiding them toward real-world problems.
ISLS
ICLS2012
Connecting Learning Through Learning Pathways

Nichole D. Pinkard, PhD Associate Professor College of Computing and Digital Media DePaul University nicholepinkard@gmail.com Abstract: A young person's learning journey will be guided by role-driven learning pathways that link learning across formal and informal spaces. Youth will be required to demonstrate, through artifacts, how learning experiences across home, school, after school and community context contribute to leveling up towards mastery of a set of core standards. Classroom teachers will be charged with managing youths' overall learning pathways ensuring youth are moving forward on their learning journey through participation in the recommend (via pattern analysis) set of learning experiences. Youth will also interact virtually and face-to-face with mentors and peers in variable-size learning groups based upon content, context, and learning dispositions as opposed to the current structure of 30 youth to a class.
Exploring new spaces without reinventing the wheel

Nikol Rummel, Institute of Educational Research Ruhr-Universitt Bochum nikol.rummel@rub.de Abstract: When working on a textbook chapter on model learning in 2004, I read many of Banduras publications. Also, I got to read a number of articles on mirror neurons. The authors of those articles in their discussion took up ideas that Bandura had been writing about decades before (of course without having any neurophysiological data to support his argumentation). I was astonished to find that none of the papers on mirror neurons that I read, cited Bandura. I am using this example to illustrate a point I want to make in my contribution to this panel: I believe that exciting new spaces to explore lay ahead of us as researchers in the learning sciences. New technologies (Web 2.0) afford new forms of learning and enable new kinds of interaction among learners; new methods (e.g. eye tracking) allow us to answer new types of questions; new research foci (e.g. questions concerning emotional and motivational aspects of learning) broaden the scope of our findings. However, in the light of the above example, I would like to caution us to avoid wearing blinders or in other words to avoid reinventing the wheel when exploring those new spaces. We must make sure to carefully take into account prior work as well as work from other disciplines if it is (or as it becomes) relevant to our research.
ISLS
ICLS2012
Invited Panel Sponsored by the ISLS Membership Committee: Linking Learning Sciences Communities across the World
Chair: Nancy Law, University of Hong Kong, Hong Kong, nlaw@hku.hk Baohui Zhang, Nanjing University, China, baohui.zhang@nju.edu.cn Konstantin . Kolin, Institute of Informatics Problems, Russian Academy of Sciences, Russia, kolinkk@mail.ru Paulo Blikstein, Stanford University, USA, paulob@stanford.edu Miguel Nussbaum, Pontificia Universidad Catlica de Chile, Chile, mn@ing.puc.cl Michael Jacobson, The University of Sydney, Australia, michael.jacobson@sydney.edu.au
Introduction
One of the goals of ISLS is to expand its membership across a broader geographical and cultural representation. In 2007, members from North American and Western Europe accounts for 89% of all ISLS membership (362). This has grown over the years to 603 in 2011. Another satisfying aspect of this increase in membership is its expansion to regions that were under-represented in our membership. Members residing outside of North America and Western Europe grew from 55 in 2007 to 184 in 2011, an increase of 235%! While this is an encouraging trend, the increase is mainly due to rise in membership from Asia, while the other regions are still seriously under-represented. It is our view that the field of Learning Sciences can benefit from more interactions of our membership with researchers and practitioners in allied disciplines who may not know about or identify themselves as learning scientists. The goal of organizing this symposium is to provide an opportunity for ISLS members to learn more about Learning Sciences related research in different countries around the world, particularly from countries outside of North American and Western Europe, and to explore ways in which ISLS can contribute to enriching the field of Learning Sciences through expanding the international representation of its membership.
Learning Sciences Oriented Research Communities in China

Baohui Zhang Nanjing University, China baohui.zhang@nju.edu.cn Abstract: The panelist will present the preliminary findings from a study conducted in Mainland China by him and his colleagues to identify the key Learning Sciences related organizations, communities, and research agendas being pursued in China. In addition to conducting a comprehensive literature review of learning sciences related publications, empirical data have also been collected using email surveys, phone interviews and face-to-face semi-structured interviews. The study finds that while there are a number of research communities that conduct research on learning, such as the communities of science educators and educational psychology researchers, there is an active academic community that also refers to its field of study as learning sciences. This community started around 1986 and has its own research conferences and publication venues. This community does engage in academic exchange with international scholars, but has evolved separately and independently from the mainstream ISLS community. This presentation will highlight the research foci, as well as dominant theoretical and methodological orientations adopted by members of this learning sciences community in Mainland China. Possibilities and implications for building links between this Chinese learning sciences community and ISLS will also be discussed.
Russian Concept of Advanced Education
Konstantin . Kolin Institute of Informatics Problems, Russian Academy of Sciences kolinkk@mail.ru Abstract: The idea of Advanced Education was formally launched in Russia in 1996 in the 2nd International Congress of UNESCO's "Education and Informatics". The essence of this concept is to reconstruct the content and methodology of the educational process at all levels of the education system so that it is able to prepare the populace for life in the Global Information Society. This is of strategic importance in improving the nation's intellectual potential as the basic human capacity required for successful social adaptation for the 21st century is different from traditional education. One main direction in the formation of Advanced Education in Russia is to develop direct links between universities and institutes of academic science. For this purpose, in Russian universities, specialized scientific and educational centers (SEC) are established to serve as agents for the introduction of new knowledge into the education system. While this is potentially a very effective form of
ISLS
ICLS2012
integration of academic research and education, funding is a major obstacle. The second obstacle is that the State Educational Standards reflect traditional approaches to education and does not meet the primary objectives of the concept of Advanced Education. Connections between the Russian concept of Advanced Education and the Learning Science will be discussed.
Brazilian education at crossroads

Paulo Blikstein Stanford University, School of Education and (by courtesy) Computer Science Department paulob@stanford.edu Abstract: Brazilian education is at a crossroads. The country is experiencing speedy economic development and education is seen as the only hope to make it sustainable. Large government projects in the last 15 years managed to bring almost all children to school. The challenge now is how to improve quality in a deeply unequal system. Educational research in Brazil has been historically connected to continental philosophy and critical theory, especially due to the work of Paulo Freire. Although highly influential in academic circles and in teacher preparation programs, this approach has been confronted in recent years by researchers and policymakers as too theoretical and ideological. The public debate that has ensued has focused on methodological and philosophical controversies between economists and educators, but has paid little attention to the development of a multidisciplinary area equivalent to the Learning Sciences (LS). The development of LS, in the United States and elsewhere, has allowed for a third way which has made possible fast development of technological platforms and new theoretical advances, outside of the traditional realm of education and econometric research. In this presentation, I will discuss how Latin America, and especially Brazil, could be the next frontier in the LS revolution.
Making Progress
Miguel Nussbaum Pontificia Universidad Catlica de Chile mn@ing.puc.cl Abstract: Chile is a country where the schools of Education are still in its infancy. A recent national test of graduates of the last year promotion of teachers showed that 42% failed in mastering the contents of what is expected of 8th grade students and only 8% showed an excellent command of the contents. The government, being aware of this fact, sponsored four years ago two centres in the two main Chilean universities to foster research in education. However the research is mainly focused on policy and its scientific productivity is scarce.
The Learning Sciences and Technology in Australia and New Zealand

Michael Jacobson The University of Sydney michael.jacobson@sydney.edu.au Abstract: Researchers at a number of Australian and New Zealand universities have long had a strong national and international profile in a number of fields aligned with the learning sciences, such as artificial intelligence and education (AI & Ed) research, information and communication technologies (ICT), e-Learning, science education, and so on. There has also been a recent Digital Education Revolution, which is a nationally funded initiative to provide a laptop to all 9th grade students in Australia, in conjunction with teacher professional development to support innovative teaching and learning with technology. In my remarks, Ill discuss ways in which we have been promoting learning sciences related research and degree programs at the Centre for Research on Computer Supported Learning and Cognition that have also involved building broader networks of collaborations with other researchers and academics in our region in affiliated areas.
ISLS
ICLS2012
Volume 2: Symposia
Hybrid spaces for science learning: New demands and opportunities for research
1
Ole Smrdal1, Jim Slotta1,2, Tom Moher3, Michelle Lui2, Alfredo Jornet1 The University of Oslo; 2The University of Toronto; 3University of Illinois at Chicago
Abstract: Hybrid spaces for science learning refers to the merging of real and virtual worlds to produce new environments and visualizations where physical and digital objects co-exist and interact in real time. Learning science within hybrid spaces can be a fun, engaging, and reflective experience. Further, hybrid spaces are inherently social, facilitating dialogue and social exchange, as well as the construction of knowledge, paralleling the nature of contemporary science. This symposium brings together several research programs that address learning across contexts, that span classroom activities, museum visits, and engaging, embodied experience of science phenomena. We include an international set of presenters from Canada, USA and Norway, each engaged in design and empirical investigations of designs that blends conceptual learning with the development of inquiry skills and epistemological knowledge. Each paper presents the research context, method of design and evaluation, research progress, and science learning outcomes.
Introduction to Symposium
The various disciplines of science are rapidly evolving, in part because of the infusion of new technologies and media practices, in a trend sometimes referred to as science 2.0 (Nisbet & Mooney, 2007; Bell et al., 2007). Cutting edge technologies are increasingly used to represent data in new forms, and by means of new analytic tools, such as 3D visualizations, simulations. Social and aggregative media (e.g., Facebook, YouTube, or wikis) have also entered into the scientific realm, creating new opportunities for collaboration and exchange across disciplines and other contexts. Indeed, scientists exploring the cosmos, the oceans, and the human genome have seized upon the advantages of large, shared datasets and open lab books. While the Internet and other advanced technologies have revolutionized many facets of civilization including science - the typical science classroom is still dominated by traditional instruction: sequences of curricular units, presented didactically, that rely on previously learned concepts, formalized testing, and teacher-student hierarchies (Krajcik et al., 2008). Our symposium represents a collection of learning science projects that are investigating how new technologies and pedagogical approaches can expand the horizons of science education beyond the space and time restrictions of the classroom, leading to powerful new ways of learning and instruction. Hybrid spaces refer to the merging of real and virtual worlds to produce new environments for science learning. Students manipulating a physical device in a museum setting can produce changes within a digital space that, in turn, influence subsequent curricular activities. Students social interactions online can result in changes to their physical classroom environment, or to an aggregate representation gathered from many students and classrooms through activities in an informal setting. Observations or other data collected using mobile or ubiquitous environments can provide a source of user contributed content for curriculum. Students could create large shared repositories (i.e., similar to flickr) of science-related content, including social and semantic metadata that become a resource for a wide community of learners. Social networks can be used to support special interest groups or to coordinate complex pedagogical designs. Tangible and physical computing elements (e.g., multi-touch surfaces, Arduino-controlled objects, or the Microsoft Kinect) can be integrated, allowing direct or embodied manipulation of ideas. We define hybrid spaces along physical, semantic and social variables that can be used as indices for the design of new learning experiences that cut across conventional contexts. We explore three aspects of such spaces: 1) The use of mixed reality for learning, where physical and digital objects (e.g., visualizations and simulations of science phenomena) co-exist and interact in real time. 2) Opportunities gained from digital representations of knowledge and dialogue about science that correspond spatially with student activity in physical environments, such as classrooms and science centers. 3) Understanding how the integration of informal and playful learning experiences in spaces outside the classroom (e.g. field trips to museums and science centers) can foster deeper conceptual and epistemological learning for students in the K-12 classroom. Design has long been recognized in the learning sciences as a crucial dimension of intellectual activity, and many papers have been written about design-oriented methodologies (Brown, 1992; Hoadley. 2002). Recently, there has been an important inquiry into the nature of our design processes, which are said to capture as much of our fields learning about learning as any scientific outcomes from empirical studies. At the 2011 meeting of Computer Supported Collaborative Learning (CSCL) in Hong Kong, a new book series was
ISLS
ICLS2012
Volume 2: Symposia
announced (Chris Hoadley and Naomi Miyake, editors) that will encourage the unpacking and deep discussion of our designs and our design process. This perspective is particularly relevant to research of hybrid environments, which entail dramatically new forms of learning. Several years can be required just to conceptualize our environments and the kind of learning that we wish to engender, before we can conduct careful studies of learning outcomes. By way of analogy, it is challenging to develop a musical score for an instrument that has not yet been designed. Thus, we find ourselves developing the instrument first, as well as the kinds of music the tonalities, melodies, and harmonies that that may be suited to it. There is clearly a cyclical set of dependencies in advancing such complex pedagogical, technological and social forms of learning, in which the media are deeply part of the message. Thus, while the papers all refer to learning environments that have been developed and tested to some extent, they are all still in active design mode, and the authors have been asked to focus on their design processes and elements. We have gathered three distinct research groups one from Canada, one from USA, and one from Norway (with two presentations from the latter) who have done empirical research into notions of spaces as an affordance and facilitation for interaction with science phenomena. These projects have designed and investigated learning trajectories in such spaces, paying attention to activities that span virtual and physical spaces. Our discussant, Wolff-Michael Roth, who is not affiliated with any of these research programs, will synthesize the ideas and advances in this work and lead a discussion amongst participants.
Paper 1: Science Hub: A digital medium for supporting collective science inquiry in hybrid spaces
1
Ole Smrdal1, Jim Slotta2, and Ingeborg Krange1. InterMedia, The University of Oslo; 2The University of Toronto, correspondence: ole.smordal@intermedia.uio.no
Across the science disciplines, the normal practices and expectations are changing in stride with the emergence of new technologies, media practices, and methodological affordances (NSF Cyberlearning Report, 2008). Bell et al. (2007) argue that the new data-intensive nature of 21st century science has moved scientific practices away from individual or even small groups of scientists working with individual databases and computational simulations, toward the use of cutting edge technologies to represent and analyze data in new forms, including powerful visualizations, GIS and satellite mapping, and new experimental tools and methods in every discipline. Social and aggregative media have entered the scientific realm, creating new opportunities for collaboration and exchange across disciplines and contexts, leading some to coin the term, science 2.0 (e,g., Nisbet & Mooney, 2007). The exciting transitions occurring in science have important implications for science education. Our research will investigate a curriculum that responds to the changing complexion of energy both as a science phenomena and a global socio political challenge in the 21st century in such a way that actually capitalize on students digital skills, making science more engaging and at the same time preparing students more deeply for careers in science related disciplines. We address variations and transitions between instructional contexts (Dierking et al. 2003; Rennie, 2007). In particular we address how learning in classrooms and informal settings (e.g., museums and science centers) can be mutually supportive. This paper reports on our design of a digital medium called The Science Hub (SciHub) to support student activities across formal and inform contexts, using rich, interactive media in museums, Web pages at home, and their own smart phones out on the street. The digital infrastructure is important for interconnecting these various forms of interaction, for adding a semantic layer of accessible metadata (including social networking information and patterns of use), and for enabling the design of powerful new forms of aggregated representation, real-time feedback, and pedagogical scripting. This paper will focus on two aspects of the SciHub. First, we present our own design process, which has embodied Science 2.0, occurring across various physical and virtual contexts. We discuss our goals, constraints, guiding principles and technology systems. Second, we present a complex sequence of curriculum activities designed in conjunction with the Norwegian Museum of Science, Technology and Medicine, that engages students at home, in school and at the museum in a variety of forms of learning, including embodied activities, collective inquiry, the creation of shared multi modal resources, and the use of rich visual representations and simulations. The SciHub is being developed to support complex patterns of inquiry that must respond to the unique contexts (both formal and informal) and configurations (individual/small group/whole class interactions) in which learning takes place (Lemke, 2000). This includes learning activities that are physically and temporally distributed, that can and should be described on multiple levels, and that engage users in emergent (i.e., not predictable, a priori) forms of learning and knowledge advancement. The description and coordination of these factors is often referred to as a script, which includes the timing and sequencing of activities, planned
ISLS
10
ICLS2012
Volume 2: Symposia
moments for student reflection, and roles for students and teachers (Dillenbourg & Jermann, 2007, Hakkinen & Makitalo-Siegl, 2007). We regard scripts as a means for specifying and making available structures, procedures, collaboration patterns, roles, materiality and resources of 21st century scientific practices, and making them relevant and engaging for learners. By embedding such scripts within a technology-enhanced environment, we can scaffold their various elements, capture the products of interactions, and provide timely feedback or input to students and teachers. Such feedback can provide participants with a visual sense of the current state of the activity, including aggregated representations from small groups, all students in a class, or everyone who has ever interacted with the script. They can provide teachers with insight into students understandings of curriculum topics, and support classroom orchestration through evidence-based decisions (Dillenbourg, Jarvela & Fischer, 2009). The design language of SciHub references concepts from theatrical performances, such as role, actor, script, stage, and performance. In SciHub, students contribute to aggregated products that change over time, with the products themselves serving as resources in the activity. Students cooperatively produce artifacts and experiences, challenging our notion of assessment, and adding important new dimensions of collaboration, reflection, revision and engagement. Designing research to evaluate the impacts of such activities - in terms of student learning as well as a more overarching aspect of progress for the community - has been an interesting challenge. We must consider new forms of electronic discourse that are related to building upon peers ideas, connecting artifacts, revising documents, and interacting with physical and tangible media. Emerging notions of social and embodied learning have inspired new ways of measuring progress, capturing participation and evaluating the impact of our innovations. Our first enactments of SciHub supported contextualization of simulations of science phenomena. Students inquiry can be structured, or more open-ended, including a wide range of digital interactions such as logs, notes, video recordings, that are available as a resource for subsequent learning activities. Scaffolds and prompts are implemented using a system intelligence, where agents operate on content in databases, state information, proximity awareness and other parameters. SciHub activities serve to connect formal and informal contexts, with a focus on experience-based learning that takes place in the museum, and how this can be related to conceptual learning in the classroom. SciHub supports a wide range of student inputs, and can infer from sensors in a spatial environment (i.e. who is present, types of activity, location, body gestures, gaze, tangibles, mobile devices, etc.).
Paper 2: Embodied science practices in hybrid spaces

Tom Moher, Francesco Novellis, Brenda Lopez Silva, and Alessandro Gnoli University of Illinois at Chicago, correspondence: moher@uic.edu In the Embedded Phenomena framework (Moher, 2006; 2008), simulated spatial phenomena are mapped onto the physical space of self-contained classrooms as subjects of collaborative inquiry. In prior designs employing this framework for inquiry in seismology (RoomQuake), astronomy (HelioRoom), and ecology (WallCology), access to and interaction with the simulated phenomena have been restricted to stationary portals (computers) strategically situated within classrooms. While these designs have leveraged embodiment in their utilization of learners locations for perspective-taking, distributed collaboration, and enactment of analogical solutions to mathematics problems (Antle, 2009), the digital and natural worlds have remained largely parallel realities rather than hybrid spaces. AquaRoom (Novellis & Moher, 2011) extends the Embedded Phenomena paradigm by introducing hybrid artifacts that richly problematize physical aspects of science practices (Reiser, 2004). AquaRoom is based on the conceit that the classroom is a small town beneath which runs a network of coherent aquifers. Children are challenged to recommend a location for a new chemical plant to be constructed within the town, with the goal of minimizing the impact of potential effluents on underground water supplies. To inform their decision, they were organized into groups to conduct a hydro-geological survey. Using the standard practice of injecting dye tracers into potential aquifer sites, they collect water samples in adjacent areas and use a photospectrometer to analyze the sample for presence of the dye. These analyses allow them to construct a map of the topography, direction and rate of flow of the aquifers under their town.
Figure 1. Portable drilling unit.
Figure 2. Dye and sample tubes.
Figure 3. Spectro-photometer.
ISLS
11
ICLS2012
Volume 2: Symposia
AquaRoom blends a collection of information-carrying tangible artifacts and analogical (but information free) props within the learning environment. Dye injection and sample retrieval are modeled through the use of a portable drilling unit (Fig. 1) that includes a non-functional Ethernet cable taped to a suction cup which, when applied to the classroom floor, provides a simulated conduit for liquid flow to and from underground aquifers. Dyes are stored, and samples collected, in test tubes to which are attached special electronic caps (iButtons), each of which contains a unique identifier (Fig. 2). Students connect the test tubes to iButton readers attached to the drilling units to effect injection and retrieval, carrying samples (in opaque collection test tubes) to a simulated photo-spectrometer (also connected to an iButton reader) to look for traces of the injected dyes (Fig. 3). Students mark up a large, shared paper grid on the wall to reflect the results of their investigations. The design choices in AquaRoom impose physical demands analogous to those required in actual hydrogeological practice, including the selection and recording of injection and retrieval sites, physical travel to those sites, management of field instruments, collection and management of samples, use of laboratory instruments for analysis of samples, and inscription of findings (Duschl et al., 2007). In these ways, AquaRoom represents a more seamless model (Ishii & Ullmer, 1997) of a hybrid space than prior Embedded Phenomena, with physical actions in the real world yielding scientifically appropriate and meaningful effects in the virtual environment. In two enactments of AquaRoom in fourth grade classrooms, we have focused on childrens responses to the hybrid artifact designs, their ability to utilize the environments affordances to conduct their investigation, their relative sense of presence within the physical and virtual worlds, and their development of understandings of effective spatial strategies for injection and sampling in the context of aquifer mapping. In our first pilot (Novellis & Moher, 2011), fidelity of representation became a prominent thread of discussion, particularly around the issue of whether correct placement of the suction cup on the floor was necessary. (It wasnt: students self-reported their locations using a coordinate system based on the rooms ceiling tiles.) While opinion split on the necessity of the artifact, there was a strong consensus regarding its utility, with students citing embodied reinforcement of coordinate positions, opportunities for participation within investigative groups, and play value associated with a willing suspension of disbelief (Coleridge, 1817) in support of its use. In a recent enactment, researchers shadowed the working groups to track students practices surrounding the use of the artifacts. Students accurately placed and reported their injection/sampling locations in all but three of 176 observed cases; surprisingly, artifact fidelity did not become a subject of classroom discourse, though about a quarter of the class manifested skepticism about suction cups during post-unit interviews. Note that the students accuracy in self-reporting their locations suggest the viability of a low-cost alternative to indoor location tracking instrumentation, at least for coarsely grained coordinate systems such as those used here. The handling of samples was similarly consistent, with only five of 217 collected samples failing to be subjected to spectroanalysis. Across both enactments, students showed a strong ability to transcribe analysis results to the shared map of the town, accurately determining the aquifer topographies. Flow direction and rate (measured using stopwatches) were more challenging, but differences of opinion created opportunities for discussion of methods for resolving observational discrepancies and the practice of repeated measurements. In post-unit interviews, students almost universally were able to articulate an effective (concentric circle) inject-and-sample strategy. Analysis of student responses to the question In your own words, please describe what weve been doing in the AquaRoom unit showed an overwhelming predominance of domain-oriented (e.g., town, drill, chemical plant) vs. implementation-based (e.g., computer, suction cup, classroom) terms.
Paper 3: Designing Immersive Environments for Collective Inquiry

Michelle Lui, James D. Slotta Ontario Institute for Studies in Education, University of Toronto, Canada, correspondence: michelle.lui@utoronto.ca This research explores how physical space, when augmented with a system of networked tablets, interactive whiteboards, and a web-based learning environment, can support face-to-face interactions as well as asynchronous knowledge co-construction. We present EvoRoom as a case study of an immersive simulation for collective inquiry. The notion of immersive simulations is not a new one, but the use of such media for conceptual learning, including the design of interactions amongst students, peers, teachers and materials, is still in nascent stages. We discuss our research objectives and our design process, as well as the outcomes from a study evaluating student perceptions and behavior within the immersive, physical simulation. Theoretical Perspective: Recently, educators have advocated the approach of forming knowledge communities in classrooms where members are given the responsibility to generate ideas and build on each others ideas within an emergent, collective knowledge base (Bereiter & Scardamalia, 2003). This approach is well suited for todays science students, who must now develop collaboration and communication skills in
ISLS 12
ICLS2012
Volume 2: Symposia
addition to learning scientific content and procedures. In our own prior work, we have applied Web 2.0 collaboration technologies in support of knowledge communities, connecting students with peers to co-develop a shared knowledge base (Peters & Slotta, 2010). We have also advanced the concept of a smart classroom, where the physical environment (e.g., walls, furniture, etc.) is deeply infused with a range of digital tools and media, scaffolding students in complex pedagogical designs, through different roles and responsibilities (Lui, Tissenbaum & Slotta, 2011; Slotta, 2010). In response to prior research in virtual worlds, such as River City (Dede, 2009) and Second Life, we now seek a layer of immersivity to transform our smart classroom into a mixed-reality immersive environment. While such virtual worlds may cultivate students sense of ownership and play, they are fundamentally limited in terms of bodily-kinesthetic interactions, as users act through avatars via screen-based interactions (Birchfield et al., 2008). We posit that hybrid environments hold the potential to transform typical learning activities into more meaningful, whole-body experiences, promoting collaboration and a sense of knowledge community. Recent research from the Learning Sciences and Human-Computer Interaction (HCI) have advanced the notion of embodied learning, where the human mind and body are viewed as inseparable (Garbarini & Adenzato, 2004). New technologies have led to a surge of interest in augmented physical environments for hands-on and participatory learning. Examples of projects that incorporate ideas of immersion and embodiment include the Cave Automated Visualization Environment (Cruz-Neira, Sandin, & DeFanti, 1993), SMALLab (Birchfield et al., 2008) and Ambient Wood (Rogers & Price, 2004). Inspired by these prior efforts, we seek to create powerful experiences for groups of co-located students to share and develop meaning through embodied interactions with one another and their surroundings, working a knowledge community. We added functionality for immersive simulations to our smart classroom infrastructure, where the room itself is converted into a rich simulation, and conceptual content is embedded using ubiquitous technology that supports students as they engage in carefully designed learning activities. Our research focuses on how such media and activities can support new forms of learning and instruction. Method: In January 2011, we began a co-design partnership with a high-school biology teacher to create our immersive simulation. We considered important design elements and outlined our overall strategy. Our first design decision was the selection of an accessible topic that would allow students to gain enhanced conceptual depth and where their embodied experience would provide distinct opportunities for perception, reflection, or integration of their scientific understandings. We required a topic that was sufficiently challenging and broad enough in scope to engage students for several sessions, including introductory (i.e., non-immersive) activities. We arrived at the topics of biodiversity and evolution, which are notoriously challenging topics for teachers and students alike, and have many characteristics that would support our development of robust, engaging materials and interactions.
Figure 1. The EvoRoom immersive environment.
Figure 2. The smart classroom layout
As the context for students scientific inquiry within our immersive environment, we selected the rainforest ecosystem of Borneo and Sumatra. The varied species of the rainforests offer a rich and complex ecosystem from which biodiversity may be studied, and the disappearance of the land bridges that connected Borneo and Sumatra approximately 10,000 years ago offer an interesting phenomenon through which to study evolution. This theme allowed us to develop rich, interactive media, showing various species of flora and fauna, and their interdependencies, which students can directly observe in the context of the simulation (see Figure 1). In order to support such an experience, the room is set up with multiple versions of the simulation. On each of the two long walls of the room, three projection displays are connected to form a 5-meter wide projected surface. These two wide, immersive displays on the facing walls provide a sense of enclosure. On the front wall of the room, serving to bridge these two wide displays, a single projected display provides higher-level symbolic and social information, as a resource for inquiry (see Figure 2). For all student inquiry, we developed specialized applications for tablet and laptop computers, which provided a field guide, collected student observations, and coordinated log-ins and group membership within the room. We also created specialized applications for an interactive white board to support the teachers orchestration of activities within the room.
ISLS
13
ICLS2012
Volume 2: Symposia
To support a sense of identity, each student is given a role of ecology expert for a certain species. They are asked to work as field researchers in specialized teams (e.g., primate ecologist team), with clearly delineated tasks and scientific agenda. In the biodiversity unit, students work collectively to create a food web of the ecosystem each bringing to the activity their expert knowledge of their species. To promote connections to the rainforest environment, students create scientific scenarios in an introductory activity, predicting how changing one factor in the ecosystem could impact the long-term biodiversity of the rainforest. When students enter the room to find the rainforest simulated with their choice of perturbations, such that the students effectively build their own immersive environment. This helps students build identity with the materials and take ownership of their own learning. In the evolution unit, which occurred several weeks after the biodiversity unit, students returned to the same smart classroom environment, retaining their specialty research teams, to conduct a field visit and survey. The teacher coordinated the activity, advancing the simulation through several historical time periods. One of the large walls was programmed to depict prehistoric Sumatra, and the other to depict Borneo in the same time period (e.g., 2 million years ago). Originally, since the two regions were connected by a land bridge, their species are identical. When the teacher advances the simulation to a more recent time period, students observe that the species have diverged! The teacher guides student observations, highlighting historical and climatological records, and drawing attention to important features of each environment,. Findings and Discussion: In our first classroom trial, we examined student behavior and perceptions of the immersive environment using questionnaires given pre- and post-activity as well as video analysis of the activity. Students were highly engaged by the use of the tablet computer supports as well as by the immersivity of the smart classroom. Students made special note of the interactive white board application that collected and shared ideas in real-time, which allowed them to gain powerful insights about evolution. We refined the activities, and are presently enacting the biodiversity and evolution environments as part of a fully integrated biology curriculum. We have added pedagogical supports as well as visualizations to represent the community knowledge base for the duration of the curriculum. The goal is for materials to become more visible (i.e., visually rich) yet less intrusive, responding intelligently to student inputs, as well as capturing the collective wisdom of the classroom community as a resource for all participants in subsequent inquiry activities. We will report on student learning gains, interaction patterns (ie, between peers, and with elements of the environment) and next steps for research.
Paper 4: Epistemic artefacts for inquiry in hybrid spaces

Alfredo Jornet, Cecilie F. Jahreie & Ingeborg Krange. InterMedia, The University of Oslo, correspondence: alfredog@intermedia.uio.no In this paper, we report on our research on the relation between physical experience and conceptual understanding, as mediated by tangible artefacts and visual representations. We explore students meaning making as they solve disciplinary problems of their science curriculum in an inquiry-based learning environment that involves experimenting with both tangible phenomena and digital models. Drawing on perspectives on visual representations as social practice (Roschelle & Clancey, 1992; Roth & McGinn, 1998), and on socio-cultural approaches to learning and cognition (Vygotsky, 1962; Wertsch, 1998), we describe semiotic mediation as a useful framework for informing how actual (phenomenological, social, material) experience relates to the development of disciplinary knowledge. The aim of the study is two-fold. On the one hand, we aim to investigate how tangible objects and visual representations become meaningful resources in students problem solving along an inquiry-based sequence of disciplinary activities. On the other hand, we aim to inform further development of a technology-enhanced learning environment that integrates tangible and digital experiences. Emerging mobile and ubiquitous computing technologies have opened new possibilities for designing inquiry-based learning environments in which students can engage in embodied experimentation with simulated or augmented phenomena (Moher, 2006). In these environments, both tangible and digital objects are combined as to constitute hybrid spaces where the material and the symbolic become explicitly linked. In these spaces, both digital and tangible artefacts may become objects of, as well as resources for, inquiry. Thus, the boundaries between what can be conceived as a mere artefact and what can be conceived as a representation become blurred. In this sense, scholars have pointed out the need for theoretical accounts of the relations between physicality and representations in hybrid learning spaces (Hornecker & Buur, 2006; Price, 2008). We adopt the term epistemic artefacts (Scardamalia & Bereiter, 2006) in order to refer to any object, whether tangible or digital, that may serve for the advancement of discussion and coordination in a problem solving activity. While there has been much research on visual representations as deployed in traditional desktop computers, such as multiple linked representations (Ainsworth, 2006) or interactive simulations and models (van der Meij & de Jong, 2006), few studies have investigated the role of visual representations in linking physical experiences with conceptual knowledge, or the role of physical artefacts and experience in understanding visual
ISLS
14
ICLS2012
Volume 2: Symposia
representations. In contrast to traditional views of visual representations as carrying meanings that are to be decoded by mental information processing, a growing body of literature points to the role of visual representations as resources for coordinating social conduct (Roschelle & Clancey, 1992; Roth & McGinn, 1998). From these perspectives, meanings are established through social, situated, material interaction, in semiotic processes by which a sign comes to refer to an object for the agents participating within given activities. This semiotic process is grounded in social interaction and is anchored to material experience. In order to investigate how both tangible and digital objects become epistemic artefacts through social interactions, we draw on video recordings gathered from a pilot study of a multidisciplinary project that aims to develop technology-enhanced solutions for linking experiences at the school and the museum (Jahreie & Krange, 2011; Jornet & Jahreie, 2011). Using Interaction Analysis (Jordan & Henderson, 1995), we follow a group of students and their teacher as they move through a set of activities that go from direct experimentation with physical artifacts to activities involving interaction with digital models. Along the trajectory, we have observed how the meanings of the different objects and representations emerge and evolve as students together make sense of the situations they confront. The students co-construct shared empirical grounds by physically and verbally pointing and particularizing salient features of their material contexts. The students and their teacher make these categories the topic of their discussion and bring them relevant to different discourses, moving from an everyday language to more disciplinary talk. The emotional and motivational components, together with the institutional aspects of the contexts and tasks, seem to influence the kind of discourses the students adopt. The material environment, including both digital and tangible objects, as well as their own talk and bodily gestures and positions, become a complex set of semiotic resources in constant development, and which are in reciprocal relation to the participants positions and dispositions in regard to the activities at hand. However, although both tangible objects and representational inscriptions acquire their functions and meanings through similar grounding processes, they appear to have different semiotic potentials as resources for inquiry. Implications of our findings for the design of hybrid inquiry-based learning environments are discussed.
References
Ainsworth, S. (2006). DeFT: A conceptual framework for considering learning with multiple representations. Learning and Instruction, 16(3), 183-198. doi: 10.1016/j.learninstruc.2006.03.001 Antle, A.N. (2009). Embodied child computer interaction: why embodiment matters. Interactions 16, 2 (Mar./Apr. 2009). ACM, New York, NY, 27-30. Bell, G., Gray, J. & Szalay, A. (2007) Petascale Computational Systems, arXiv:cs/0701165v1 [cs.DB]. Bereiter, C., & Scardamalia, M. (2003). Learning to work creatively with knowledge. In E. De Corte, L. Verschaffel, N. Entwistle, & J. van Merrinboer (Eds.), Powerful learning environments: Unraveling basic components and dimensions (pp. 55-68). Oxford, UK: Elsevier Science. Birchfield, D., Thornburg, H., Megowan-Romanowicz, M. C., Hatton, S., Mechtley, B., Dolgov, I., & Burleson, W. (2008). Embodiment, Multimodality, and Composition: Convergent Themes across HCI and Education for MR Learning Environments. Advances in Human-Computer Interaction, 2008, 120. Brown, A. L. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. The Journal of the Learning Sciences, 2(2), 141-178. Coleridge, S.T. (1817) Biographia literaria. Cruz-Neira, C., Sandin, D., & DeFanti, T. A. (1993). Surround-screen projection-based virtual reality: the design and implementation of the CAVE. In SIGGRAPH '93: Proceedings of the 20th annual conference on Computer graphics and interactive techniques, (pp. 135-142). ACM Press. Dede, C. (2009). Immersive interfaces for engagement and learning. Science, 323, 6669. Dierking, L.D., Falk, J.H., Rennie, L., Anderson, D., & Ellenbogen, K. (2003). Policy statement of the "Informal Science Education" Ad Hoc Committee. J. of Research in Science Teaching, 40(2), 108-111. Dillenbourg, P., & Jermann, P. (2007). Designing Integrative Scripts. In F. Fischer, I. Kollar, H. Mandl & J. r. M. Haake (Eds.) Scripting Computer-Supported Collaborative Learning (Vol. 6, pp. 275-301): Springer. Dillenbourg, P., Jrvel, S., & Fischer, F. (2009). The Evolution of Research on Computer-Supported Collaborative Learning. In N. Balacheff, S. Ludvigsen, T. Jong, A. Lazonder & S. Barnes (Eds.) Technology-Enhanced Learning. (pp. 3-19): Springer Netherlands. Duschl, R.A., Schweingruber, H.A., and Shouse, A.W. (2007). Taking science to school: Learning and teaching science in grades K-8. National Academy Press. Garbarini, F., & Adenzato, M. (2004). At the root of embodied cognition: Cognitive science meets neurophysiology. Brain and Cognition, 56(1), 100106. Hkkinen, P., & Mkitalo-Siegl, K. (2007). Educational Perspectives on Scripting CSCL. In F. Fischer, I. Kollar, H. Mandl & J. M. Haake (Eds.) Scripting Computer-Supported Collaborative Learning. (Vol. 6, pp. 263271): Springer US. Hoadley, C. (2002). Creating context: Design-based research in creating and understanding cscl. In Stahl, G., editor, Computer support for collaborative learning 2002, (pp. 453-462), Mahwah, NJ: Erlbaum.
ISLS
15
ICLS2012
Volume 2: Symposia
Hornecker, E., & Buur, J. (2006). Getting a grip on tangible interaction: a framework on physical space and social interaction. Paper presented at the SIGCHI conference on Human Factors in computing systems, Montreal, Qubec, Canada. Ishii, H. & Ullmer, B. (1997). Tangible Bits: Towards Seamless Interfaces between People, Bits and Atoms. Proceeding ACM Conference on Human Factors in Computing Systems (Atlanta, GA, March 22-27, 1997). CHI 97. ACM Press, 234-241. Jahreie, C. F., & Krange, I. (2011). Learning in Science Education Across School and Science Museums Design and Development Work in a Multiprofessional Group Nordic Journal of Digital Literacy. Jordan, B., & Henderson, A. (1995). Interaction analysis: Foundations and practice. The Journal of the Learning Sciences, 4(1), 39103. Jornet, A., & Jahreie, C. (2011). Designing for immersive learning environments across schools and science museums. Multi-professional conceptualisations of space. Paper presented at the ReLIVE11: Researching learning in immersive virtual environments, Milton Keynes (UK). Krajcik, J., Slotta, J., McNeill, K. L. & Reiser, B. (2008). Designing learning environments to support students constructing coherent understandings. In Kali, Y., Linn, M. C., & Roseman, J. E. (Eds.) Designing coherent science education. (pp.39-64). NY, NY: Teacher College Press. Lemke, J. L. (2000). Across the Scales of Time: Artifacts, Activities, and Meanings in Ecosocial Systems. Mind, Culture, and Activity, 7(4), 273-290. Lui, M., Tissenbaum, M., & Slotta, J. D. (2011). Scripting collaborative learning in smart classrooms: Towards building knowledge communities. Proceedings of the 9th International Conference on ComputerSupported Collaborative Learning (CSCL) Volume 1, (pp. 430-437). International Society of the Learning Sciences (ISLS). Moher, T. (2006). Embedded Phenomena: Supporting Science Learning with Classroom-sized Distributed Simulations. Proceedings ACM Conference on Human Factors in Computing Systems (Montreal, Canada, April 22-27, 2006). CHI 2006. ACM Press, 691-700. Moher, T. (2008). Learning and participation in a persistent whole-classroom seismology simulation. In Proceedings ICLS, Vol. 2 (Utrech, Netherlands, June 23 - 28, 2008). ICLS 2008. ISLS, 82-90. Nisbet, M. C. & Mooney C. (2007) Framing Science. Science: 316 (5821). Novellis, F. & Moher, T. (2011). AquaRoom: Designing Tangibles and Props to Engage Young Learners in a Full Body Learning Experience. Proceedings 10th international Conference on Interaction Design and Children (Ann Arbor, MI, June 20 - 23, 2011). IDC '11. ACM Press, 90-98. NSF Task Force on Cyberlearning. (2008). Fostering learning in the networked world: The cyberlearning opportunity and challenge, (NSF 08-204). Arlington, VA: National Science Foundation. Peters, V. L., & Slotta, J. D. (2010). Scaffolding knowledge communities in the classroom: New opportunities in the Web 2.0 era. In M. J. Jacobson & P. Reimann (Eds.), Designs for learning environments of the future: International perspectives from the learning sciences (pp. 205-232). Secaucus, NJ: Springer. Price, S. (2008). A representation approach to conceptualizing tangible learning environments. Paper presented at the 2nd International Conference on Tangible and Embedded Interaction, Bonn, Germany. Reiser, B. J. (2004). Scaffolding complex learning: The mechanisms of structuring and problematizing student work. Journal of the Learning Sciences, 13, 273304. Rennie, L. J. (2007) Learning science outside of school. In Abell S. K. & Lederman N. G. (eds.) Handbook of research on science education. Lawrence Erlbaum Associates Ltd. Rogers, Y., & Price, S. (2004). Extending and augmenting scientific enquiry through pervasive learning environments. Children, Youth and Environments, 14(2), 6783. Roschelle, J., & Clancey, W. J. (1992). Learning as Social and Neural. Educational Psychologist, 27(4), 435453. doi: 10.1207/s15326985ep2704_3 Roth, W.-M., & McGinn, M. K. (1998). Inscriptions: Toward a Theory of Representing as Social Practice. Review of Educational Research, 68(1), 35-59. Scardamalia, M., & Bereiter, C. (2006). Knowledge building: Theory, pedagogy, and technology. In R. K. Sawyer (Ed.), Cambridge Handbook of the Learning Sciences. New York: Cambridge University Press. Slotta, J. D. (2010). Evolving the classrooms of the future: The interplay of pedagogy, technology and community. In K. Mkitalo-Siegl, F. Kaplan, J. Zottmann, & F. Fischer (Eds.), Classroom of the Future: Orchestrating collaborative spaces (pp. 215-242). Rotterdam: Sense. van der Meij, J., & de Jong, T. (2006). Supporting students' learning with multiple representations in a dynamic simulation-based learning environment. Learning and Instruction, 16(3), 199-212. doi: 10.1016/j.learninstruc.2006.03.007 Vygotsky, L. S. (1962). Thought and language. Cambridge, Mass.: MIT Press. Wang, F., & Hannafin, M. J. (2005). Design-based research and technology-enhanced learning environments. Educational Technology Research and Development, 53(4), 5-23. Wertsch, J. V. (1998). Mind as action. New York: Oxford University Press.
ISLS
16
ICLS2012
Volume 2: Symposia
Building Upon What Is Already There: The Role of Prior Knowledge, Background Information, and Scaffolding in Inquiry Learning
Christof Wecker, University of Munich, Department of Psychology, Leopoldstr. 13, D-80802 Mnchen, Germany, christof.wecker@psy.lmu.de Ard W. Lazonder, University of Twente, Department of Instructional Technology, P.O. Box 217, 7500 AE Enschede, The Netherlands, a.w.lazonder@utwente.nl Jennifer L. Chiu, University of Virginia, 313 Bavaro Hall, Curry School of Education, Charlottesville, VA 22903, jlchiu@virginia.edu Cheryl Madeira, James D. Slotta, University of Toronto, 252 Bloor St., West., Toronto, Ontario, Canada, M5R2X2 Email: cheryl.madeira@utoronto.ca, jslotta@oise.utoronto.ca Yvonne G. Mulder, Ton de Jong, University of Twente, Department of Instructional Technology, P.O. Box 217, 7500 AE Enschede, The Netherlands Email: y.g.mulder@utwente.nl, a.j.m.dejong@utwente.nl Alexander Rachel, Hartmut Wiesner, University of Munich, Department of Physics, Theresienstr. 37, D-80333 Mnchen, Germany Email: alexander.rachel@physik.uni-muenchen.de, hartmut.wiesner@physik.uni-muenchen.de Eva Heran-Drr, University of Bamberg, Marcushaus, Markusplatz 3, D-96047 Bamberg, Germany, eva.herandoerr@uni-bamberg.de Frank Fischer, University of Munich, Department of Psychology, Leopoldstr. 13, D-80802 Mnchen, Germany, frank.fischer@psy.lmu.de Discussant: Peter Reimann, University of Sydney, Sydney NSW 2006, Australia, peter.reimann@sydney.edu.au Abstract: Prior knowledge is one of the most important factors for learning. During the iterative cycles of inquiry learning, learners prior domain knowledge is modified, refined, and further developed, provided that learners act upon self-assessments of their understanding and that they can actually think of appropriate hypotheses. Furthermore, knowledge about inquiry strategies influences the quality of the learners inquiry activities, and the lack thereof requires compensatory support. This symposium brings together recent work about the role of prior knowledge for inquiry learning and ways to compensate for the lack of it. The four papers focus on the role of learners self-assessment of their current understanding for their subsequent inquiry activities, on the gradual refinement of their knowledge on the basis of reflection, and on prior presentation of theoretical background information and concurrent presentation of inquiry strategies as ways to compensate for lack of prior theoretical knowledge and strategy knowledge, respectively. One of the most powerful factors that influence learning is what learners already know (Dochy, Segers & Buehl, 1999). This rather general finding applies to inquiry learning as well (see, e. g., Gijlers & de Jong, 2005). This symposium brings together recent work about the ways prior knowledge influences inquiry learning and ways to compensate for the lack of it. Inquiry learning has been characterized as an iterative process in which knowledge is modified, refined, and thereby further developed (e.g., White & Frederiksen, 1998). Students generate knowledge by asking questions and stating hypotheses, conducting experiments, and drawing conclusions, and each of these cycles is informed by the knowledge the learners have acquired during the previous ones. In this line of reasoning, student learning in subsequent cycles can benefit from accurate metacognitive judgments about the appropriateness of their current knowledge. In fact, interventions that target learners self-assessment of their understanding can foster knowledge about the domain content (White & Frederiksen, 1998). Within the framework of Scientific Discovery as Dual Search (SDDS, Klahr & Dunbar, 1988; van Joolingen & de Jong, 1997), learners knowledge can be characterized by attributes of hypotheses within the socalled hypothesis space. This space contains all possible hypotheses about a given class of phenomena. A learners knowledge is constituted by the information for each hypothesis he or she can think of whether it is considered worthwhile testing, has or has not been tested so far, and if it has been tested, whether it has been rejected or retained for further consideration (cf. van Joolingen & de Jong, 1997; Gijlers & de Jong, 2005). The set of hypotheses a learner can think of, i.e. the so-called learner hypothesis space, is determined by the learners knowledge of variables and relations (van Joolingen & de Jong, 1997) and constitutes an important limiting factor during iteratively progressing inquiry learning because learners cannot discover the right hypothesis if it is not within their search space. If learners hypothesis spaces are too limited, remedial intervention such as providing background information is required.
ISLS
17
ICLS2012
Volume 2: Symposia
Another kind of knowledge that is relevant during the cycles of inquiry learning is knowledge about inquiry strategies. While generating hypotheses, designing and conducting experiments, making observations and recording data, and drawing conclusions are activities that operate upon the content of the domain, they are part of inquiry strategies. Accordingly, learners with better knowledge about inquiry strategies are more likely to be successful during their inquiry activities and therefore should acquire more domain knowledge. Conversely, learners lacking sufficient levels of knowledge about inquiry strategies need to be scaffolded appropriately in order to be able to conduct fruitful inquiry activities and learn successfully about the domain. The papers in this symposium cover this array of aspects of the role of prior knowledge during inquiry learning. Jennifer Chiu investigated the effects of learners self-assessments of their understanding during a WISE unit about chemical reactions. Cheryl Madeira and Jim Slotta followed the recommendation to apply the iterative character of inquiry learning to teacher training (White & Frederiksen, 1998) and investigated the iterative refinement of teachers pedagogical content knowledge under the conditions of practical enactment and peer exchange. Alexander Rachel, Christof Wecker, Eva Heran-Drr, Hartmut Wiesner and Frank Fischer investigated ways to compensate for elementary school students limited hypothesis spaces that do not contain assumptions about theoretical entities related to magnetism. Yvonne Mulder, Ard Lazonder and Ton de Jong focused on scaffolding that may compensate for lacking knowledge about inquiry strategies. They studied the effects of heuristic worked examples demonstrating how to gradually develop an equation that specifies a relationship among a set of variables. The series of the four paper presentations will be complemented by the presentation of a discussant. This role has been taken over by Peter Reimann who is an eminent scholar in the learning sciences and has conducted and published research about inquiry learning himself. He will identify the major achievements and unresolved issues of the four papers. This will provide the basis for an open discussion with the audience.
Paper 1: Student Self-Assessment of Knowledge Integration in a TechnologyEnhanced Chemistry Lesson

Jennifer L. Chiu Supporting self-assessment can help students learn across many domains (e. g., Scardamalia & Bereiter, 1991). Self-assessment skills such as questioning or judging ones own understanding and making decisions based on those assessments are especially important for inquiry science learning in technology-enhanced environments (Quintana, Zhang & Krajcik, 2005; White & Frederiksen, 1998). Although students benefit from accurate selfassessment, research demonstrates that students have difficulty evaluating their own performance (Dunning, Heath & Suls, 2004). Many factors contribute to learners overestimating or underestimating their understanding, such as the nature of the assessment task, subject-matter knowledge, the learning environment, and motivation. Students may be able to assess their understanding of facts or simple recall items accurately, but overestimate their understanding on open-ended or explanation items (Zoller, Fastow, Lubezky & Tsaparlis, 1999). Even if students can accurately assess themselves, they may or may not go back to improve their understanding. This study explored how high school chemistry students assessed their own understanding in inquiry settings using criteria based on the knowledge integration (KI) framework. The KI perspective calls for conscientious, intentional learning and focuses on the connections among scientific ideas (Linn & Eylon, 2006). Engaging in knowledge integration encourages students to elicit prior knowledge, add new, normative ideas to these existing frameworks, develop criteria to examine new ideas and links that they have formed to their prior knowledge, and evaluate their understanding to sort and distinguish more productive and relevant ideas from less productive ideas. Traditional instruction tends to focus on eliciting and adding ideas, often leaving out critical processes of supporting students to examine, evaluate and refine their understanding (Linn & Eylon, 2011). The KI perspective can particularly help chemistry learners because many students struggle to make connections among concepts at molecular, observable, and symbolic levels. For example, students view equations of chemical reactions, such as CH4 + 2O2 CO2 + 2H2O as problems to solve instead of a process of breaking and forming bonds among atoms (Krajcik, 1991). Prior research demonstrates that curricula designed with a knowledge integration approach benefit chemistry learners (Linn et al., 2006). This study used the chemical reactions curriculum unit within the Web-based Inquiry Science Environment (WISE), a computerbased environment that scaffolds inquiry science projects (Slotta & Linn, 2009). WISE projects provide various tools to support knowledge integration and scientific inquiry, such as visualizations, concept maps, idea managers, embedded assessments and online discussions. In addition to scaffolding inquiry, WISE provides detailed log reports of how students progress through projects, including where and when they click on particular steps or interact with certain tools, and if and how they revise answers to embedded assessment steps. The WISE chemical reactions unit guides students through a one-week investigation of how chemical reactions relates to climate change using dynamic molecular visualizations. Prior research with this unit demonstrated that students often overestimate their understanding when using dynamic molecular visualizations, and prompting
ISLS
18
ICLS2012
Volume 2: Symposia
explanations helped students identify gaps in their knowledge (Chiu & Linn, 2008). Building upon this research, this study explored: (1) if students could accurately assess their explanations using explicit knowledge integration criteria; (2) if evaluating explanations encouraged students to revisit and/or revise their explanations, and (3) if students ability to assess their explanation had any impact on overall learning of the unit.
Method
High school chemistry students (n = 93) from the same teacher completed chemical reactions as part of normal class instruction. After students first interactions with visualizations, they explained what they learned. The next step provided a rubric for students to evaluate their explanation based on the scientific ideas and connections within the explanation. Students rated their explanation and provided a justification for their evaluation. Students KI ratings were compared to researcher ratings of the same explanations. Log data were analyzed to determine if students revisited or revised their explanations. Pretest and posttest assessments that contained both conceptual and self-assessment items were used to determine overall impact on self-assessment ability and learning.
Results
Compared to researcher ratings, almost half (48%) of the students accurately assessed their explanations. Students also overestimated (32%) and underestimated (20%) their scores. When asked to justify their selfevaluations, around 30% of students explicitly used KI terms of scientific ideas and/or connections (i.e. We should get a three because we only made one scientific connection in our explanation and our explanation was not very complex.) Most students made general statements about their understanding, such as, i [sic] chose two because in some of the questions i only gave half an answer because some parts i didnt get that much, but i explained what i did know. Log data results reveal that only 42% of students went back to either their explanation or the associated visualization step immediately after the self-assessment step. If students went back to their explanation, they were likely to revise their explanation for a higher KI score (2(1,93) = 14.4, p < 0.01). Student scores significantly increased from pretest to posttest, replicating earlier results that the Chemical Reactions unit helps students make connections among representations in chemistry (t(92) = 15.08, p < 0.01, ES g = 1.08). Regression analysis with pretest score, accuracy of explanation self-evaluation, and revisits to explanations as explanatory variables and posttest score as the dependent variable indicated that neither accuracy of the selfassessment nor revisiting the explanation significantly impacted posttest score. Controlling for pretest ability, explanation evaluation accuracy, and revisiting, students average self-ratings residuals tended to decrease from pretest to posttest, indicating less inflated self-ratings on the posttest (R2 = .2, F(4,88) = 5.50, p < .001; = .44, t = 4.55, p < .001).
Conclusion
Results show that many students could use explicit knowledge integration criteria to assess their open-ended embedded explanations and encouraging students to go back can help them refine their explanations. Results suggest that encouraging students to assess their understanding helped students become more critical of their understanding from pretest to posttest. Since many students did not give KI-based justifications of their scores or go back to revise their explanation after giving themselves a less-than perfect score, students selfassessments may need to be accompanied by external feedback or support to encourage students to act upon their judgments.
Paper 2: Iterative Design Enhances Inquiry-Based Teaching

Cheryl Madeira, James D. Slotta Teacher professional development can be realistically said to translate directly into more effective student learning (Davis & Varma, 2008; Gerard et al., 2011; Krajcik & Blumenfeld, 2006). Teachers need to understand their topic domain, make effective choices of instructional strategy, and assess student learning in ways that inform their pedagogical approach. Substantial literature has addressed teacher knowledge and professional development (e.g., Borko et al., 1997), including the different kinds of knowledge held by teachers, such as general instructional strategies for a specific content domain called Pedagogical Content Knowledge (PCK). This type of teacher knowledge depends on experience and builds on prior knowledge. Teacher learning can be very difficult to measure, as it is often implicit and embedded within the teachers practice (Davis & Varma, 2008). How can a professional development model help capture and promote teacher learning? This paper presents data from a three-year design-based study that examined teacher learning in relationship to design and enactment of an inquiry-based science lesson. We investigated the impact of two interventions reflection and peer-exchange across three stages: (1) lesson planning, (2) enactment, and (3) revision of lesson, with the three stages repeated two or three times, iteratively, by teachers. We describe how
ISLS
19
ICLS2012
Volume 2: Symposia
teachers developed understandings within the context of these activities, contributing to our understanding of situated collaborative learning and teacher professional development.
Method
This study used a design-based methodology to investigate the development of pedagogical content knowledge of nine science teachers (N = 9) in relation to their instructional practices (e.g., lesson design, preparations, classroom interactions, assessment and feedback), and student understanding. These teachers, who volunteered to participate in this study, entered with a range of experience (between 3 and 30 years) and disciplinary expertise (i.e., physics, biology, chemistry, or general science). The teachers came from 5 different schools located in a large urban city in North America and had a wide variety of technology experience and access. For confidentiality, all participants were given pseudonyms. Data sources include teacher surveys, interview questions, lesson plans, reflections (captured in a wiki), videotaped classroom enactments, field notes, student artifacts and responses, peer exchanges (on wiki, and in group meetings). This paper reports on 3 iterations of the study, which gradually introduced the conditions of the intervention. Iteration 1 of the study included four teachers (n = 4) who worked individually with the researchermentor to co-design, enact and reflect the inquiry-based science lesson. Iteration 2 added five more science teachers and improved the reflective prompts by connecting them directly to lesson planning and enactment, while adding community elements (face-to-face and online). In iteration 3, we continued to refine the reflections and community exchange, connecting teacher activities of lesson planning and enactment to topics of student prior knowledge, project-based learning and technology implementation. In order to analyze the various data from Wiki contributions, interviews and field notes, two coding schemas were developed one for lesson planning and one for enactment that included elements to measure teacher knowledge, following Grossmans (1990) taxonomy: pedagogical content knowledge (PCK), content knowledge (CK), pedagogical knowledge (PK), technology pedagogical content knowledge (TPCK), contextual knowledge (CXK). Each code was ranked for quality with a value from 1 to 3, where 1 represented fragmented evidence of that knowledge, and 3 represented a highly coherent instance of the knowledge, and 2 was anything in between. In addition to these codes, we employed the characteristics of project-based learning as measures of inquiry-based lessons (e.g., student-driven questions, collaborative activities). Thus, if the lesson incorporated more inquiry-based approaches, the lesson plan itself scored higher. These coding schemes allowed us to capture improvements in teachers plans and enactments between iterations of our study, reflecting the added benefits of our improved intervention. They also allowed us to correlate those improvements to specific features of teacher knowledge and inquiry-based teaching. Only the lesson planning data will be presented in this paper, in relation to teachers prior knowledge. The elements in the coding schema reflected teachers understanding of student learning (i.e., PCK), including how teachers would respond to student ideas in their lesson revisions and subsequent enactments. The coding schema for Lesson Planning had high intercoder reliability (Kappa = .80).
Results
We hypothesized that when teachers revise lessons based on their reflections about the enactment, they would improve the quality of those lesson plans. This proved true for five of the six participants who completed two iterations, with the exception of Bill (teacher), whose lesson plan included less student collaboration and interactions in Iteration 2 than it did in Iteration 1 (based on results from analysis of lesson planning coding schema). The correlations provide evidence of a link between the quality of teachers reflections and the quality of their designed lessons. Teachers who were able to reflect in detail and link their lesson objectives to student learning were able to improve their lessons in the following iteration. Teachers who used the tools and followed the rules of reflection consistently showed improvements in their overall lesson planning score from the first to the second iteration.
Conclusion
Presumably, these improvements were due to teachers reflections about the strengths and liabilities of their lesson in the previous enactment. The teachers were able to see what worked and didnt work with the lesson plan, reflect on this in a concrete way, and then improve on their design and approach for the next iteration. Thus, scaffolded reflection throughout the course of these two planning and enactment cycles appears to play a productive role in helping teachers inquiry into their own practice, resulting in improved lesson designs and improved teacher knowledge of lesson planning.
ISLS
20
ICLS2012
Volume 2: Symposia
Paper 3: Inquiry Learning with Elementary School Children: Prior or Subsequent Presentation of Theoretical Background Information?
Alexander Rachel, Christof Wecker, Eva Heran-Drr, Hartmut Wiesner, Frank Fischer The idea behind inquiry learning is that learners discover knowledge about scientific phenomena independently (de Jong, 2006). As most learners struggle with the activities required for this purpose, it has been suggested to provide scaffolding for them (de Jong & van Joolingen, 1998). A more far-reaching goal of science education, however, is to acquire knowledge on the theory level that goes beyond observable phenomena. For example, in phenomena related to magnetism, theoretically assumed molecular magnets cannot be observed directly. A problem during inquiry learning might be that theories that assume such unobservable entities cannot easily be discovered because the learners hypothesis spaces (Klahr & Dunbar, 1988; van Joolingen & de Jong, 1997) will hardly contain assumptions about these entities. In a prior study we found that secondary school students can acquire more knowledge on the theory level if they receive a presentation of theoretical background information prior to investigating the phenomena themselves, whereas subsequent presentation of theoretical background information after an inquiry phase had no lasting effect (Rachel et al., 2010). The current study focused on the question whether primary school children can likewise acquire knowledge on the theory level and knowledge on the level of phenomena during inquiry activities if supported accordingly. In particular, we investigated the short- and longer-term effects of prior presentation of theoretical background information, subsequent presentation of theoretical background information and specific scaffolding during inquiry learning activities on knowledge on the level of phenomena and knowledge on the theory level.
Method
Three to four intact 4th grade classes from German primary schools were randomly assigned to each condition of a 2x2x2-factorial design with scaffolding (unspecific/specific), prior presentation of theoretical background information (without/with) and subsequent presentation of theoretical background information (without/with) as independent variables. The 612 participants from 31 classes were on average M = 9.25 (SD = 0.52) years old; 305 of them were girls, 307 were boys. The students first completed a 20 minute pretest. Then they worked on an inquiry unit about magnetism for 120 minutes in which they conducted hands-on experiments in dyads at up to eleven learning stations. Finally they completed a 25-minute posttest. Two months later they completed a 25 minute delayed posttest in their classrooms. Dyads in the conditions with unspecific scaffolding received general prompts to engage in the three inquiry activities of predicting, describing and explaining. Dyads in the condition with specific scaffolding received content- and task-specific prompts for these inquiry activities. In the conditions with prior presentation of theoretical background information, initially a 30-minute introduction to the theoretical background of magnetism was provided by a teacher. No such introduction was given in the conditions without prior presentation of theoretical background information. In the conditions with subsequent presentation of theoretical background information a 30-minute teacher-led wrap-up phase about the same topics as in prior presentation of theoretical background information took place at the end of the learning phase, while there was no such phase in the conditions without subsequent presentation of theoretical background information. Identical knowledge tests were used in the immediate and delayed posttests. They consisted of ten groups of tasks with several subtasks with multiple-choice, true-false and an open answering format that required the learners to insert labels into line drawings. The subtasks were coded as correct or incorrect on separate coding variables. The scale for knowledge on the level of phenomena comprised 15 coding variables and had sufficient reliability (immediate posttest Cronbachs = .60; delayed posttest Cronbachs = .61). The scale for knowledge on the theory level comprised 15 coding variables too and had satisfactory reliability (immediate posttest Cronbachs = .81; delayed posttest Cronbachs = .82). A subset of tasks from the parts of the test that captured knowledge on the level of phenomena was used for the pretest (9 coding variables, Cronbachs = .61).
Results
The main results were the following: With respect to knowledge on the level of phenomena, in the immediate posttest no significant main or interaction effects of the independent variables were found. In the delayed posttest a small main effect in favour of the conditions with prior presentation of theoretical background information was detected, F(1; 22.13) = 5.51; p = .03; partial 2 = .02. With respect to knowledge on the theory level, in the immediate posttest a significant main effect in favour of the conditions with unspecific scaffolding, F(1; 22.12) = 8.29; p = .01; partial 2 = .04, and an
ISLS
21
ICLS2012
Volume 2: Symposia
interaction effect between prior presentation of theoretical background information and subsequent presentation of theoretical background information, F(1; 22.12) = 4.46; p < .05; partial 2 = .02, were found. In particular, in the presence of prior presentation of theoretical background information or subsequent presentation of theoretical background information, learners acquired more knowledge on the theory level than without both of them. In the delayed posttest, however, this interaction effect disappeared. Instead, there was a significant main effect in favour of the conditions with prior presentation of theoretical background information, F(1; 22.10) = 7.55; p = .01; partial 2 = .03.
Discussion
The results indicate a superiority of unspecific scaffolding during inquiry activities with respect to knowledge on the theory level. With general prompts for inquiry, the learners have to set sub-goals for inquiry themselves. As a consequence, the function of each current activity for the overall goal of investigating the theoretical assumptions might be more transparent to the learners than with highly specific questions. Furthermore, prior presentation of theoretical background information appeared beneficial for knowledge on the level of phenomena and knowledge on the theory level. While immediately after the learning phase a summary in the context of subsequent presentation of theoretical background information might be at least a functional equivalent to prior presentation of theoretical background information (as evidenced by the interaction of prior and subsequent presentation of theoretical background information), in the long run prior presentation of theoretical background information seems to be superior. This effect might be explained by the opportunity provided by prior presentation of theoretical background information to apply the theory to be learned during inquiry activities, thereby yielding higher levels of knowledge on the theory level. In sum, this study demonstrates that primary school children can learn about challenging topics involving scientific theories that cannot readily be discovered, provided that they are supported appropriately.
Paper 4: Using Worked Examples to Scaffold Students Understanding of the Inquiry Learning Process
Yvonne G. Mulder, Ard W. Lazonder, Ton de Jong Technology-enhanced inquiry learning environments enable students to develop a deep understanding of science content and processes. Computer simulations have long been incorporated in these environments, and are increasingly being supplemented with opportunities for students to build computer models of the phenomena they are investigating via the simulation. Even though inquiry and modeling are potentially powerful ways of learning, students often lack the skills to take full advantage of these activities (e.g., de Jong & van Joolingen, 1998). A recent study showed that domain novices are quite capable of identifying relevant variables, but experience considerable difficulties in specifying the relations between these variables. Instead of gradually working toward a full-fledged scientific equation to specify a relationship, novices tried to induce and model these equations from scratch (Mulder, Lazonder, & de Jong, 2010). Novice learners tendency to jump the gun points to a lack of knowledge about inquiry strategies that can in principle be controlled by organizing the inquiry learning process in successive phases of increasing complexity (i.e., model progression; White & Frederiksen, 1990). Model progression indeed enhances inquiry and modeling performance (e.g., Mulder, Lazonder, & de Jong, 2011; Swaak, van Joolingen, & de Jong, 1998)but not to a sufficient degree. It thus seems that students need additional support in order to better understand what the activities in each model progression phase entail, and how they should be performed. The present study examined the effectiveness of heuristic worked examples to deliver this support. Heuristic worked examples were proposed by Hilbert and colleagues (Hilbert & Renkl, 2009; Hilbert, Renkl, Kessler, & Reiss, 2008) as means to extend the application of worked examples from well-structured problemsolving tasks to more ill-structured, and hence more complex tasks. Unlike traditional worked examples that provide students with a single algorithm to solve one particular type of problem, heuristic worked examples outline a series of problem solving strategies and demonstrate their usage in or across a range of related tasks. Heuristic worked examples have been successfully applied in, for instance, concept mapping and second language learning tasks (Renkl, Hilbert, & Schworm, 2009), and are expected to be beneficial to inquiry and modeling tasks as well. The present study sought to validate this expectation by comparing the learning and performance of high-school students who worked in an inquiry learning environment with modeling facilities. All students were supported by model progression, but only students in the worked example condition received additional heuristic worked examples that exemplified the activities students should perform within each of three model progression phases. Students in this condition were accordingly expected to exhibit more proficient inquiry and modeling behavior and, as a result, perform better and learn more than students from the control condition who were not supported by heuristic worked examples.
ISLS
22
ICLS2012
Volume 2: Symposia
Method
Eighty-two high-school students (aged 15-17) were randomly assigned to either the worked examples condition (n = 46) or the control condition (n = 36). Students in both conditions had to investigate a charging capacitor and create a computer model of its behavior. This task was divided in three model progression phases that asked students to first identify and sketch the model with its variables and relations, then specify all relations in qualitative terms (e.g., if resistance increases, then current decreases), and finally transform these qualitative specifications into physics equations (e.g., I = V / R). Students in the worked examples condition could consult two heuristic worked examples for each phase. These examples came in the form of annotated videos that showed the inquiry respectively modeling activities of an anonymous person on a comparable task in a different domain. Students in the control condition did not receive these worked examples. The study took place during four 50-min science lessons. In lesson 1, students were introduced to the learning environment and completed the knowledge pretest which addressed the meaning of key domain concepts (i.e., voltage source, resistance, capacitor, and capacitance) and the physics equations that govern the behavior of a charging capacitor. Lesson 2 and 3 were devoted to the inquiry and modeling task, and lesson 4 was used to administer the knowledge posttest. The posttest contained the same items as the pretest, although phrased in modeling terms to maximize resemblance with the learning task, plus six additional items about all qualitative relations in the simulation's underlying model.
Results
Main findings indicate that students in both conditions had comparable and low pretest scores, F(1, 80) = 0.03, p = .866, needed quite the same amount of time on task, F(1, 80) = 0.70, p = .404, but spent this time differently. As instructed by the worked examples, students in this condition did more experiments with the simulation, F(1, 80) = 12.57, p = .001, and took more time to analyze and interpret the outcomes, F(1, 80) = 9.37, p = .003. Control students, by contrast, largely ignored the simulation and spent most of their time creating and testing their model, F(1, 80) = 57.00, p < .001. This proved rather ineffective, as students in the worked example condition performed significantly better, F(2, 79) = 7.65, p = .001. That is, their models contained both correct variables, F(1, 80) = 15.38, p < .001, and relations, F(1, 80) = 9.45, p = .003. Despite this performance difference, posttest scores were comparable across conditions, F(1, 75) = 0.10, p = .759, suggesting that worked example students performed better, but did not learn more.
Conclusion
These results confirm two of the three expectations. As predicted, heuristic worked examples caused students to perform the inquiry and modeling activities as intended, and enhanced the quality of their models. The expected difference in posttest scores failed to show, which suggests that heuristic worked examples have an immediate effect on students learning activities and performance, but little impact on the knowledge students (should) attain from these activities. This suggests that scaffolding inquiry learning strategies is insufficient to acquire domain knowledge; additional content explanations might be needed for students to develop (a deep) understanding of the topic of inquiry.
References
Borko, H., Mayfield, V., Marion, S., Flexer, R., & Cumbo, K. (1997). Teachers developing ideas and practices about mathematics performance assessment: Successes, stumbling blocks, and implications for professional development. Teaching & Teacher Education, 13, 259278. Chiu, J., & Linn, M. C. (2008). Self-Assessment and Self-Explanation for Learning Chemistry Using Dynamic Molecular Visualizations. In International Perspectives in the Learning Sciences: Cre8ting a Learning World. Proceedings of the 8th International Conference of the Learning Sciences (Vol. 3, pp. 16-17). Utrecht, The Netherlands: International Society of the Learning Sciences, Inc. Davis, E. A., & Varma, K. (2008). Supporting teachers in productive adaptation. In Y. Kali, M. C. Linn, & J. E. Roseman (Eds.), Designing coherent science education: Implications for curriculum, instruction and policy (pp. 94122). New York, NY: Teachers College. de Jong, T. (2006). Technological advances in inquiry learning. Science, 312, 532 f. de Jong, T. & van Joolingen, W. R. (1998). Scientific discovery learning with computer simulations of conceptual domains. Review of Educational Research, 68(2), 179-201. Dochy, F., Segers, M. & Buehl, M. M. (1999), The Relation between Assessment Practices and Outcomes of Studies: The Case of Research on Prior Knowledge. Review of Educational Research, 69(2), 145-186. Dunning, D., Heath, C., & Suls, J. M. (2004). Flawed self-assessment. Psychological Science in the Public Interest, 5(3), 69.
ISLS
23
ICLS2012
Volume 2: Symposia
Gerard, L.F., Varma, K., Corliss, S.B. & Linn, M.C. (2011). Professional Development for Technology Enhanced Inquiry Science. Review of Educational Research, 81(3), 408-448. Gijlers, H. & de Jong, T. (2005). The Relation between Prior Knowledge and Students Collaborative Discovery Learning Processes. Journal of Research on Science Teaching, 42(3), 264-282. Grossman, P. L. (1990). The making of a teacher: Teacher knowledge and teacher education. New York, NY: Teachers College Press. Hilbert, T. S., & Renkl, A. (2009). Learning how to use a computer-based concept-mapping tool: Selfexplaining examples helps. Computers in Human Behavior, 25, 267-274. Hilbert, T. S., Renkl, A., Kessler, S., & Reiss, K. (2008). Learning to prove in geometry: Learning from heuristic examples and how it can be supported. Learning and Instruction, 18, 54-65. Klahr, D. & Dunbar, K. (1988). Dual space search during scientific reasoning. Cognitive Science, 12(1), 1-48. Krajcik, J. (1991). Developing students' understandings of chemical concepts. In S. Glynn, R. Yeany, & B. Britton (Eds.), The psychology of learning science (pp. 117-147). Hillsdale, NJ: Erlbaum. Krajcik, J. S., & Blumenfeld, P. C. (2006). Project-based learning. In R. K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (pp. 317333). New York, NY: Cambridge University Press. Linn, M. C., & Eylon, B.-S. (2006). Science education. In P. A. Alexander & P. H. Winne (Eds.) Handbook of Educational Psychology, 2nd edition. Mahwah, NJ: Erlbaum. Linn, M. C., & Eylon, B.-S. (2011). Science Learning and Instruction: Taking Advantage of Technology to Promote Knowledge Integration. New York: Routledge. Linn, M. C., Lee, H.-S., Tinker, R., Husic, F., & Chiu, J. L. (2006). Teaching and assessing knowledge integration in science. Science, 313, 1049-1050. Mulder, Y. G., Lazonder, A. W., & de Jong, T. (2010). Finding out how they find it out: An empirical analysis of inquiry learners need for support. International Journal of Science Education, 32, 2033-2053. Mulder, Y. G., Lazonder, A. W., & de Jong, T. (2011). Comparing two types of model progression in an inquiry learning environment with modelling facilities. Learning and Instruction, 21, 614-624. Quintana, C., Zhang, M., & Krajcik, J. (2005). A framework for supporting metacognitive aspects of online inquiry through software-based scaffolding. Educational Psychologist, 40(4), 235-2244. Rachel, A., Wecker, C., Heran-Drr, E., Waltner, C., Wiesner, H. & Fischer, F. (2010, April/May). A place and a time for expository instruction during inquiry learning? Its role for the understanding of scientific theories that are hard to discover. Paper presented at the Annual Meeting of the American Educational Research Association (AERA) "Understanding Complex Ecologies in a Changing World", Denver, 30 April 4 May, 2010. Renkl, A., Hilbert, T., & Schworm, S. (2009). Example-based learning in heuristic domains: A cognitive load theory account. Educational Psychology Review, 21, 67-78. Scardamalia, M., & Bereiter, C. (1991). Higher levels of agency for children in knowledge building: A challenge for the design of new knowledge media. The Journal of Learning Sciences, 1, 37-68. Swaak, J., van Joolingen, W. R., & de Jong, T. (1998). Supporting simulation-based learning; the effects of model progression and assignments on definitional and intuitive knowledge. Learning and Instruction, 8, 235-252. van Joolingen, W. R. & de Jong, T. (1997). An extended dual search space model of scientific discovery learning. Instructional Science, 25, 307-346. White, B. Y., & Frederiksen, J. R. (1990). Causal model progressions as a foundation for intelligent learning environments. Artificial Intelligence, 42(1), 99-157. White, B., & Frederiksen, J. (1998). Inquiry, modeling and metacognition: Making science accessible to all students. Cognition and Instruction, 16(1), 3-118. Zoller, U., Fastow, M., Lubezky, A., & Tsaparlis, G. (1999). Students self-assessment in chemistry examinations requiring higher- and lower-order cognitive skills. Journal of Chemical Education, 76(1). 112-113.
ISLS
24
ICLS2012
Volume 2: Symposia
Assessing Interests in the Service of Supporting Personalized Learning Through Networked Resources
Brigid Barron, Caitlin K. Martin, Stanford University School of Education, 485 Lasuen Mall, Stanford, CA 94305-3096 Email: barronbj@stanford.edu, ckm@stanford.edu Robert B. W. Ely, Mary Ainley, Jon Pearce, Disna Wijayawickrama, Josie Chan, Psychological Sciences, University of Melbourne, Victoria 3010 Australia Email: r.ely@student.unimelb.edu.au, maryda@unimelb.edu.au, jonmp@unimelb.edu.au, d.wijayawickrama@student.unimelb.edu.au, josiecchan@gmail.com K. Ann Renninger (Organizer), Mark Chin, Dennis Fan, Department of Educational Studies, Swarthmore College, 500 College Ave., Swarthmore, PA 19081 Email: krennin1@swarthmore.edu, mchin1@swarthmore.edu, dennisfan9@gmail.com Abstract: Papers by four researchers working to assess interest online in order to support learning provide the basis for discussion in this symposium. Session participants will use current work to address four questions: (a) What are the research questions being addressed and what is the context of the research? (b) How are interest and learning conceptualized and measured in this study, especially earlier phases of interest and learning? (c) What adjustments were made to the measures due to context? (d) What are the implications of these data for practice and the design of learning resources? Following these presentations, discussion with the audience will focus on how interests should be assessed for the advancement of theory, pedagogy, and the design of networked resources.
Overview of Symposium Panel

Interest has been repeatedly demonstrated to affect the attention, goals, and learning strategies of learn. Even if they initially have low levels of self-efficacy, lack learning goals, and/or have difficulty taking initiative or selfregulating, their interest can be supported to develop (see Renninger & Hidi, 2011, for a review). How interest can be supported to develop is linked both to triggers for interest and to the forms of interaction that are available, recognized, and engaged (see Ainley, 2012; Barron, Martin, Takeuchi, & Fithian, 2009). The present symposium brings together a group of researchers working to assess interest in a variety of online educational contexts in order to allow discussion of their methods and decision making, and consideration of the implications of their findings for supporting personalized learning. Session participants will use current work to address four session questions: 1. 2. 3. 4. What are the research questions being addressed and what is the context of the research? How are interest and learning conceptualized and measured in this study, especially earlier phases of interest and learning? What adjustments were made to the measures due to context? What are the implications of these data for practice and the design of learning resources?
Following the presentations, the panelists will engage the audience in thinking together about how and why interests should be assessed in online contexts in order to support learning.
Presentation Summaries
Developing Interest in Science: The Role of Prior Experience and Engaging Classroom Experiences Brigid Barron and Caitlin K. Martin Problem
A recent longitudinal study documented a statistically significant association between eighth graders projected careers and their completion of a related college degree (Tai, Liu, Maltese, & Fan, 2006). Students who indicated that their ideal jobs were science related were three times more likely to complete a degree in science than students who identified non-science related jobs as ideal. In fact, interest was a stronger predictor of the completion of a science major than achievement test scores in math, a subject that can be a barrier to those interested in natural science. This research suggests that if we want to better understand the origins of interest in science as a career, we focus attention on the resources and experiences that learners encounter that trigger and sustain interest during the preschool, elementary, and middle school years, and that we track experiences both in and out of school (Barron, 2006). In particular, we need to understand how networked affinity groups, learning
ISLS
25
ICLS2012
Volume 2: Symposia
resources, and online communities can become spark and sustain interest (Ito, et al, 2009). The current paper reports on a comparative study that had two primary goals. The first was to contribute theoretically to our basic understanding of the emergence of interest in science. The second was to provide design recommendations about how networked resources might nurture the development of interest in science as a career, as well as science more generally as a topic relevant to all citizens.
Methods
We focus on a middle school curriculum unit that has strong potential to generate interest, and we carry out quantitative and qualitative research to understand how prior experiences and different aspects of the curriculum interact to generate and sustain patterns The specific research questions we address include: (1) How do students with more and less expressed interest in science experience different aspects of an engaging citizen science curriculum?; (2) How do pre-course science experiences, hobbies, and social learning networks, and use of networked learning resources nurture interest in scientific content? Vital Signs is a citizen science networked system located in the state of Maine, linked statewide to schools and accessible not only to the focal participants (teachers and students in seventh and eighth grade classrooms), but to anyone who wants to learn and contribute. Vital Signs has high potential to generate excitement, interest, and a desire to learn about the natural world by engaging learners directly in observing, documenting, and sharing information about real world phenomena. Through Vital Signs, learners participate in learning activities designed around scientific issues in their local communities using authentic tools and collaborating with scientists. Interest in science was measured through a set of survey items and data was collected from 217 middle school students taught by one of four teachers. We examined this data in relation to the more situated judgments of interest in the unit. In the qualitative analysis, eight case study learners were selected based on their initial differential interest in a multi-week Citizen Science project. Mid-way through the unit, teachers made judgments of learner interest. Half of the cases represent low Vital Signs interest and half high Vital Signs interest. Because we were focused on the development of interest in relation to an opportunity to become interested, we followed learners out in the field, observed classroom sessions, and collected examples of work. Because we also wanted to understand their broader learning ecology, we interviewed students to assess both personal social learning networks related to science activities, and interest in science.
Results
In our presentation we focus on a subset of the survey items. For example, we asked students to indicate how much they agreed with the sentence, I find working with science very interesting on a likert-scale from 1 (disagree strongly) to 5 (agree strongly). Of these students, 21% disagreed with the statement, 22% were neutral, and 52% agreed. Open-ended questions that asked students to indicate whether they could become a scientist and explain why revealed the majority of students believed they could not become a scientist and also gave explanations that included low levels of domain interest (e.g., it is boring), judgments about the nature of the work (e.g., it would be stressful), and low ratings of competence or other personal characteristics (e.g., Im not smart enough; I have no patience). This disparity in beliefs about science and science learning was also found between individual student ratings of their science interest and teacher ratings of a childs interest in the Vital Signs project. Our case study data helps to provide an account of these patterns and we will showcase a subset of our cases in our talk. For instance, Laura, an outgoing 13-year-old whose science teacher saw her as highly engaged with Vital Signs but who indicated that she did was not interested in working with science. More detailed observations and interviews shared that Laura had always been extremely involved with sports but also had an affinity towards science. She enjoyed experimenting and told of mixing household products together to see what would happen when she was in elementary school. Though she used Facebook and played some online games, she was not active online with science-related content until Vital Signs. Her team worked efficiently to get things done and Laura and a teammate even set up a mini-experiment within the observation. They captured a bug they knew ate the plant they were looking for and put both it and a leaf sample from their plant in a container to see what happened. She recounted seeing the natural world around her differently due to her species exploration and documentation and was very proud of the positive ID she made and the science expert confirmation she received online. Josh on the other hand, a shy and quiet seventh grader, was rated as not so engaged in Vital Signs by his teacher, but rated himself very engaged in working with science. Interviews with Josh shared stories of many activities done with his entire family, including dirt-biking in the woods around a trailer park, playing first-person-shooter video games, and using Internet resources to pursue science-related topics. Joshs parents became interested in a dwarf star after Josh shared the story he had seen on a you tube video and the entire family talked about new developments, checked for more information online, and rented related videos from Netflix. The family was also using Google Earth together to locate relatives houses around the United States. Despite these exciting family-wide science and technology activities, Josh did not connect with the Vital Signs project. Joshs team did not manage to successfully submit their submission to Vital Signs and he felt that his team members did not listen to him, though he expressed interest in learning more about the
ISLS
26
ICLS2012
Volume 2: Symposia
things that frustrated him, including developing a research question and uploading and organizing digital photos between a camera and the computer.
Discussion
Our detailed case accounts have implications for the design of both classroom experiences and networked resources. We conceptualize topical or activity-based interest as a dynamic psychological state that in its early phases is subject to rapid shifts in relation to a learners affective experience of the activity and in relation to identity fit (Hidi & Renninger, 2006). In addition to these metrics of individual interest, we assess the childs environment for its potential to stimulate interest. Our research is intended to inform the design of networked resources. Our approach builds on methods developed within learning sciences research programs designed to provide rich case study portraits that can guide theory development and design. We plan to develop partnerships with the case study teachers that will allow us to engage them in imagining and prototyping new features and resources that will amplify the potential of cyberlearning for citizen scientist work. We are particularly focused on identifying opportunities to design resources that will allow for personalized pathways for teachers and students based on their specific domain or activity interests.
Establishing the Interests of Young People, a New Exploratory Approach: The My Interest Now for Engagement (MINE) Project Robert B. W. Ely, Mary Ainley, Jon Pearce Problem
A central concern for educators involves identifying what young people are really interested in, and at what level are they interested? Is it possible to trigger and/or establish both the content and level of students interests using a process of online exploration? In contrast to traditional approaches to identifying interest content and levelusing ethnographies, observations, questionnaires and/or interviews, or even a good chat with a studentwe have developed a new methodological approach to measure and trigger interest: the My Interest Now for Engagement (MINE) tool, using iFISH software (Pearce, 2008). The measurement of triggered interest in the moment is possible as the MINE tool possesses the ability to trigger and measure interest as part of the one process, and requires the student to respond with minimum reflection. Identifying the content and phase of interest in students can be critical in the teaching and learning processes of all students, but more particularly for those students who are disengaged from the learning process in school. The Four-Phase Model of Interest Development (Hidi & Renninger, 2006) provides an experiential, cognitive, and affective framework for investigating the development of interest as a dynamic psychological state. Interest is triggered, grows, stabilizes, or recedes depending on an individuals experience, cognition, and affect.
Method
MINE uses an interactive and playful environment to facilitate students reporting and commenting on their interests. Students indicate the experiential, cognitive, and affective dimensions of their involvement in a wide range of interest content generated by, and specific to, the context of their student community. A wide range of factors including age appropriate content, language, culture, sub-culture and gender has a relationship with the potential for interest in content. Students identifying interest content appropriate to their own context was perceived as integral to the MINE method. The interactive nature of the tool allows students to explore and select from a pool of 60 potential interests aggregated from an original 160 potential interests generated by six participants aged 14 to 16. Interactive sliders that represent five dimensions of experience, indoor-outdoor, creative-practical, technological-natural, solitary-social, and serious- fun, facilitate exploration. Manipulation of these sliders animates and re-orders graphical representations of the 60 potential interests in an online and real-time environment. Each student is able to select from three to eleven different interests. In contrast to previous approaches, this process allows for triggering of new interests as well as reporting existing interests. Analysis of the reliability and validity of the process of the MINE tool is derived from data generated by 136 first-year university students and 60 secondary students. Analysis of content is derived from 260 secondary students, aged 12 to 15, enrolled in two low-socioeconomic-status (low SES) secondary schools.
Results
The findings of the project were obtained in a three-stage process. Firstly, 136 first-year university students established values for the settings of five interactive sliders for 60 potential interests. Secondly, evidence for reliability and validity of the MINE process was established using 60 secondary students in a test/re-test pilot study. There was correlation between experiential, cognitive, and affective dimensions of interests selected by the same students on two occasions, two months apart. Finally, once development of the MINE tool was completed, data gathered using MINE established patterns of interest content aggregated into broader categories as described in Figure 1. Results describe a complex pattern of the 1599 interests for the 260 students in these two schools.
ISLS
27
ICLS2012
Volume 2: Symposia
Figure 1. Interest content for 260 students. The students as a cohort most frequently reported being the most interested in electronics (10%) and money (8%), and were very interested in sport (5%), clothes (5%), game consoles (5%) and shoes (3%). Students had no reported interest in being an individual or global warming. Quantitative data relating to the experiential and cognitive and affective dimensions of interest were supplemented by qualitative data gathered from students comments. Four profiles of interest dynamic were identified by cluster analysis of the experiential and cognitive variables: duration (how long?), frequency (how often?), effort (how much?) and flow (sense of time passing?). The first profile is characterized by relatively low duration and frequency and moderate effort and flow, represents a beginning interest. The second profile is defined by high duration and frequency, low effort, and very low flow and represents maintained interest. The third interest profile is characterized by very high duration, frequency, effort, and flow and represents active interest. The fourth interest profile showed very high duration, frequency, and flow but negligible effort, indicating effortless interest. The affective dynamics associated with these four interest profiles are also examined. The exploratory nature of the MINE tool allows student to select objects, activities, or ideas that interest them with duration of just now, and there were 53 instances of such selections. These 53 triggered interests possess a lower frequency but greater effort and flow than beginning interests. Further analysis of qualitative data suggests that students whose interest was triggered by the MINE process itself may have had previous experience with the selected interest content, emphasizing the dynamic nature of interest development.
Discussion
Almost by definition, educators wish to engage students in the process of learning. The MINE findings provide data that will be useful for the development of content for learning resources intended to provide engaging curriculum. Establishing interest content, and identifying four interest profilesbeginning interest, maintained interest, active interest, and effortless interestdescribes both what these 260 students are interested in, individually and collectively, and at what level they are interested. This understanding can inform what curriculum is presented to them, and how these students may be engaged with that curriculum. Subsequent research may use the unique exploratory nature of the MINE process to examine content students might be interested in, described by Hidi and Renninger (2006) as triggered situational interest.
The Tension Between Autonomy, Choice, and Structure when Task Interest is Low Mary Ainley, Disna Wijayawickrama, Josie Chan Problem
It is clear from a range of research findings that interest and self-regulated learning skills function together to produce high-quality learning and performance. Students who are interested in the task, or who have welldeveloped individual interest in the task domain, readily accept new learning challenges and revel in being able to choose their own topic. However, this is not the case for students who are not interested in the task domain or who do not have their interest triggered when the new task is presented. One interpretation of these general findings is that the functional relation between interest, self-regulation, and task characteristics may take different forms according to students level of initial task interest. Online tasks designed to encourage problem solving, reasoning, and argumentation skills by providing opportunities for choice and decision making may be
ISLS
28
ICLS2012
Volume 2: Symposia
advantageous for students who are highly motivated because their interest has been activated, and with strong motivation, their learning skills are energized. At the other end of the spectrum where students respond to the task with low interest, those same task characteristics providing autonomy and choice may mean that students never quite get into the task. The skills they have are not engaged, with resulting low levels of achievement. Just as students with triggered situational interest may require support to sustain that interest, providing support in the form of prompts and scaffolds may serve to increase interest in students who initially have low task interest, thereby assisting them to engage with the task. Hence, it is our expectation that providing more structure for students who have low initial task interest will function to increase interest as the task progresses. In addition, task engagement will increase as manifested in the quality of note taking, and higher task engagement will be associated with higher quality answers.
Methods
We compare how students with high task interest and students with low task interest respond to two versions of the same online, open-ended problem task, solving a murder mystery. One version of the task gave students autonomy to direct their own path through the task. Students chose when and how to access a set of information resources to generate a reasoned answer to the problem. In the second version, access to the information resources followed a fixed sequence and each information resource came with prompts or scaffold questions to guide how students approached the information. Two groups of 8th-grade students from the same secondary school participated in different years. The first version (autonomy version) was completed by 80 students, 55 male and 25 female, and the second version (scaffolding version) by 65 students, 34 males and 31 females. Interest was conceptualized as an immediate psychological state and measured as a rating on a 5-point Likert scale. However, while for some students this state can be akin to a triggered situational interest, for other students who bring to the task some level of interest in the task domain, this immediate psychological state may represent deeper knowledge and valuing of the domain. To record the trajectory of on-task interest, the online interest rating scale appeared three times; after the task had been explained but just before students started work on the problem, mid-way through the problem, and immediately after students submitted their answer. Note-taking quality was assessed as (i) the number of items of important information recorded, and, (ii) the level of organization of the items of information recorded using the SOLO taxonomy (Biggs & Collis, 1982). Answer quality was assessed using an index derived from the theme development scoring in the Wechsler Individual Achievement Test (WAIT-II, 2002) and the Solo Taxonomy.
Results
When high- and low-initial-task-interest groups were compared across the autonomy and scaffolding versions of the task, there was no significant difference in the interest trajectory for the high interest groups. Initial interest was high and this was maintained across the task. However, there were significant differences in the trajectories for the low interest groups. For the autonomy version low interest remained low across the task, while for the scaffolding version interest increased across the task. Note-taking quality also varied between the groups. Of particular significance is the finding that quality of note taking for the low interest group with the scaffolding version showed significantly higher-quality notes both on number of important items of information and integration of information. In addition, this higher-quality note taking was significantly related to increased interest. Further analyses will model relations between interest, note taking, and answer quality.
Discussion
In this investigation, interest has been conceptualized as an immediate psychological state that may represent an immediate response to the new taska triggered situational interest. It may also draw on more developed interests that have been activated by the new task. However, in previous research we have shown that within the immediate situation of an online open-ended problem task, the initial level of interest triggered is predictive of the trajectory across the whole task. The scaffolding version of the task was an attempt to provide support for students who initially have low task interest and who generally show little task engagement if left to make their own way. The implication is that learning conditions supporting self-regulation and task engagement may vary according to students interest. These findings are consistent with the relation between interest development and teacher support as described in the Four-Phase Model (Hidi & Renninger, 2006). In addition, they indicate that the provision of support for student learning cannot be reduced to one size fits all.
ISLS
29
ICLS2012
Volume 2: Symposia
Interest and Productive Disposition: Pre-service Teachers and Virtual Fieldwork in Mathematics K. Ann Renninger, Mark Chin, and Dennis Fan Problem
Session questions will be addressed using findings from a multi-method, use-informed study of pre-service teachers (PTs) work with the Math Forums Virtual Fieldwork Sequence (VFS)online modules designed to enable PTs to develop their capacities to work with challenging mathematics and develop their abilities to mentor elementary-aged students. Evidence from research on interest (e.g. Hidi & Renninger, 2006), self-concept of ability (e.g., Denissen, Zarrett, & Eccles, 2007), anxiety (e.g, Ma & Xu, 2004), and level of prior mathematics (e.g., Carlson, Oehrtman, & Engelke, 2010) suggests that each of these variables points to distinct ways in which PTs are likely to engage with mathematics. There are also indications that these variables are potentially coordinated. In the present study, we first determined that for the PTs, these variables formed two clusters which we termed higher and lower productive disposition, following on the National Research Councils (2001) discussion of productive disposition as a critical but little understood strand of mathematical proficiency. We then examined the impact of the level of PTs productive disposition on (a) the PTs developing abilities to engage in mathematical thinking and mathematical communication, and (b) their perceptions of mathematics and the learning environment. Lastly, we considered the implications of these findings for design.
Methods
Target participants included a total of 88 (13 M, 75 F) PTs who participated in one of four different class implementations (add-on, supplement, integrated supplement, and class focus). A matched control group for each implementation included a total of 49 (4 M, 45 F) PTs. Three sources of data inform the present study: (a) pre-and post-module online surveys that included tasks parallel to the virtual fieldwork, (b) artifacts from PTs work with module tasks, and (c) instructor interviews. The pre- and post-module online descriptive surveys were used to gather data regarding motivation and learner characteristics expected to describe their productive disposition. Artifacts from the PTs work with the modules were used to assess their mathematical thinking and mathematical communication. Finally, interviews of the PTs instructors were used to inform understanding of data from PTs responses to the survey and their work with the modules.
Findings
While PTs varied individually, two clusters of PTs were clearly identified: those with lower productive disposition had low interest in mathematics, low self-concept of ability in mathematics, high anxiety about mathematics, and no calculus background, whereas those with higher productive disposition had high interest in mathematics, high self-concept of ability in mathematics, low anxiety about mathematics, and calculus background. In terms of engagement and learning, PTs with lower productive disposition engaged the content of the modules and learned from them, while those with higher productive disposition did not, or at least not in the same way. The PTs with lower productive disposition made positive gains with respect to their ability to communicate about mathematics as well as their ability to address mathematical content during mentoring. Their matched controls did not. The work of the PTs with higher productive disposition, on the other hand, declined on the same indicators on which the PTs with lower productive disposition improved. Their matched controls did not evidence the same declines. It appears that the present design of the VFS may primarily benefit the PTs with lower productive disposition and may not meet the needs of the cluster of PTs with higher productive disposition. Given that the performance of the PTs with higher productive disposition declines, it seems that they could benefit from different supports to engage with mathematics. For example, while PTs with lower productive disposition appear to be learning to be fluent and to communicate about mathematics, this is different from learning to provide elementary students with either effective strategies for working with mathematics or supportive mathematical feedback. They are not yet able to accomplish goals such as these. However, developing strategies for working with mathematics and learning to provide supportive mathematical feedback could be set as expectations for PTs with higher productive disposition. PTs with higher productive disposition were fluent with mathematics when they started work with the VFS and appear to need support to continue to develop their abilities to work with mathematics in their mentoring.
ISLS
30
ICLS2012
Volume 2: Symposia
Discussion
Interest is here defined developmentally as both a psychological state and a predisposition to return to engagement with particular disciplinary content (e.g., mathematics)(Hidi & Renninger, 2006). Phases in the development of interest have been identified as ranging from an initial triggered situational interest that may last only a few moments to a well-developed individual interest that is relatively long lasting. In the present study, interest was assessed using survey items addressing the PTs feelings, knowledge, and value for mathematics; however, inconsistency in response patterns on Likert ratings and PTs descriptions of their preferences and patterns of engaging mathematics revealed that what mathematics meant to PTs varied, requiring that their responses be cross-validated. As such, PTs who did not like to work with challenging problems or talk about mathematics were not considered to have a developed interest in mathematics despite their own rating of themselves as high on liking. We think that the PTs in this study, regardless of whether they have lower or higher interest, are likely to be in earlier phases of interest development. As a result, we consider the distinction we have made between levels of productive disposition as lower and higher to be relative to the population, potentially requiring qualification of how the findings from this study are understood. In terms of implications, findings from the present study do suggest that learners in both phases of interest that were identified, and with differing levels of productive disposition, need to be supported to learn, and that their strengths and needs as learners differ. In the case of the VFS, it appears that at least two different tracks could be built into the VFS modules allowing learners to opt for the type of problems and/or the types of challenges they take on in providing mentoring to others. Providing instructors with information about differences in the ways in which their students are positioned by their productive disposition to work with online modules is also warranted.
References
Ainley, M. (2012). Students interest and engagement in classroom acitivities. In S.L. Christenson, A.L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 283-302). New York: Springer. Azevedo, F. W. (2006). Personal excursions: Investigating the dynamics of student engagement. International Journal of Computers for Mathematical Learning, 11, 5798. Barron, B. (2006). Interest and self-sustained learning as catalysts of development: A learning ecologies perspective. Human Development, 49, 193224. Barron, B., Kennedy-Martin, C., Takeuchi, L., & Fithian, R. (2009). Parents as learning partners in the development of technological fluency. International Journal of Learning and Media, 1(2), 5577. Biggs, J. B., & Collis, K. (1982). Evaluating the quality of learning: The SOLO taxonomy. New York: Academic Press. Carlson, M., Oehrtman, M., & Engelke, N. (2010). The precalculus concept assessment: A tool for assessing students reasoning abilities and understanding. Cognition and Instruction, 28(2), 113145. Denissen, J. H., Zarrett, N. R., & Eccles, J. S. (2007). I like to do it, Im able, and I know I am: Longitudinal couplings between domain-specific achievement, self-concept, and interest. Child Development, 78, 430447. Hidi, S., & Renninger, K. A. (2006). The four-phase model of interest development. Educational Psychologist, 41(2), 111127. Ito, M., Sonja B., Matteo B., boyd, D., Cody, R., Herr-Stephenson, B., Horst, H.A., et al. 2010. Hanging out, messing around, and geeking out: Kids living and learning with new media. Cambridge, MA: MIT Press. Ma, X. & Xu, J. (2004). The causal ordering of mathematics anxiety and mathematics achievement: A longitudinal panel analysis. Journal of Adolescence, 27(2), 165179. Maltese, A. V., & Tai, R. H. (2011). Pipeline persistence: Examining the association of educational experiences with earned degrees in STEM among US students. Science Education, 95(5), 877907. National Research Council (2001). Adding it up: Helping children learn mathematics. J. Kilpatrick, J. Swafford, and B. Findell (Eds.), Mathematics Learning Study Committee, Center for Education, Division of Behavioral and Social Sciences and Education. Washington, DC: National Academy Press. Pearce, J. M. (2008). A system to encourage playful exploration in a reflective environment. Paper presented at the EdMedia 2008 Conference. Renninger, K. A. & Hidi, S. (2011). Revisiting the conceptualization, measurement, and generation of interest. Educational Psychologist, 46(3), 168-184. The Psychological Corporation. (2002). Wechsler Individual Achievement Test (2nd ed.). San Antonio, TX. Tai, R. H., Liu, C. Q., Maltese, A. V., & Fan, X. (2006). Planning early for careers in science. Science, 312(5777), 11431144.
ISLS
31
ICLS2012
Volume 2: Symposia
Acknowledgments
The research entitled Developing Interest in Science: The Role of Prior Experience and Engaging Classroom Experiences (Brigid Barron and Caitlin K. Martin), was supported by a grant from the National Science Foundation (REC-1124568). The research entitled Interest and Productive Disposition: Pre-service Teachers and Virtual Fieldwork in Mathematics (K. Ann Renninger, Mark Chin, and Dennis Fan) was supported by a grant from the National Science Foundation (DUE-0717732)
ISLS
32
ICLS2012
Volume 2: Symposia
Supporting teachers in capturing and analyzing learning data in the technology-rich classroom
Peter Reimann, MTO, Tbingen, Germany; p.reimann@mto.de (Organizer); Friedrich Hesse, Knowledge Media Research Centre, Tbingen, Germany F.hesse@iwm-kmrc.de (Organizer); Peter Freebody, University of Sydney, Australia; peter.freebody@sydney.edu.au (Discussant) Gabriele Cierniak, Birgit Imhof, {g.cierniak, b.imhof}@iwm-kmrc.de; Barbara Wasson, Cecilie Hansen. University of Bergen & InterMedia, Uni Helse, Uni Research, Bergen, Norway; Cecilie Hansen {Barbara.Wasson, Cecilie.Hansen}@uni.no; Wilfrid Utz, BOC, Vienna, Austria, wilfrid.utz@boc-eu.com; Wolfgang Halb, Joanneum Research, Graz, Austria, wolfgang.halb@joanneum.at; Ravi Vatrapu, Computational Social Science Laboratory (CSSL), Dept. of IT Management; Copenhagen Business School, vatrapu@cbs.dk; Susan Bull, Matthew Johnson, Electronic, Electrical & Computer Engineering, University of Birmingham, s.bull@bham.ac.uk, m.d.johnson.1@bham.ac.uk; Rosemary Luckin, Brock Craft, Katerina Avramides, The London Knowledge Lab, Institute of Education, London. r.luckin@ioe.ac.uk.; Michal Kossowski, BOC Information Technologies Consulting, Warsaw, michal.kossowski@boc-pl.com Abstract: While the technology-rich classroom makes it comparatively easy to gather, store and access data on students activities, turning those into information on learning that can inform pedagogical decision-making is still hard to achieve. In this symposium, it is argued that teachers are an important if not the most important source of knowledge about the necessary diagnostic assessment methods, and that therefore teachers should be supported in describing, sharing and deploying these methods. We also argue that teachers should play a more central role in analyzing learning data for the purpose of creating knowledge about how specific pedagogical and technical innovations play out in the context of specific classroom situations, and describe a new approach to teacher-led inquiry into students learning data. Introduction Schools are slowly yet inevitably entering the information age. But while the level of technology infusion is increasing, and with it the capacity to distribute information for learning and gather information about learning quickly and efficiently, we are still far away from the vision of the school as a high performance, personalized learning community (Hamilton & Jago, 2010). The barrier to that is increasingly not the absence of information, but the absence of the right information, at the right time, in the right format. The classroom may be increasingly data-rich, but is still comparatively information-poor. One reason is that a good part of the data made available to teachers and students have limitations to inform pedagogical decision making: Large-scale assessment data are usually not linked to classroom practices and outcomes, and are not available close enough in time to learning and teaching activities (Crawford, Schlager, Penuel, & Toyama, 2008). Another reason is that classroom technologies that are closer to the performance level focus on activity tracking rather than knowledge tracking. For instance, classroom response systems, also known as "clickers", are an effective, easyto-use way for teachers to generate instant data about students thinking. However, classroom response systems do not in any way guide the teacher in formulating the 'right' questions, such as questions that can help to identify residual misconceptions in students. A third group of reasons for why learning data are often not brought to bear on classroom-level decision-making has to do with teacher capacity. Some studies report that teachers as harboring views of the nature and the role of evidence that are not conducive to data-oriented decision-making (Parr & Timperley, 2008). And even when teachers are appreciative of learning data, they often do not feel sufficiently qualified and/or not provided with sufficient time to be working with detailed learning data. This symposium brings together a number of researchers who are addressing the challenges of the datarich classroom in an international research cooperation funded by the European Commission, the NEXT-TELL project (www.next-tell.eu). NEXT-TELL seeks to address the emergent needs of teachers and schools through the participatory development of a set of methods and tools which support teachers engagement with advanced learning technologies, with a particular focus on the use of student data to promote innovation and change as a form of teacher professional development aligned to schools strategic planning goals. The projects philosophy is that teachers need not only be seen as the users of classroom technologies, and the recipients of information, but also as the innovators of technology-supported teaching and assessment practices, and as the creators of knowledge about students learning. The projects methodological approach is design-based research (Barab & Squire, 2004; Wang & Hannafin, 2005). In accordance with design-based research, we think that multiply
ISLS
33
ICLS2012
Volume 2: Symposia
cycles are needed to successfully develop methods and tools together with teachers and within the classroom. Cierniak et al. present findings from a baseline study that probed into participating teachers conceptions of formative assessment in the classroom, and their conceptions of teacher-led research. As participating teachers come from a number of European countries, the findings reflect not only individual differences, but also differences that can be attributed to variations in educational policies and teacher training across nations. Using the base-line study and findings from the wider research literature as basis, the next three contributions describe a first set of interventions, which will be further refined by means of intervention studies with teachers in schools. Reimann, Utz and Halb describe an intervention that aims at enabling teachers to describe ICTintegrated formative classroom assessment methods, to share these with other teachers, and to deploy these rapidly in their classrooms. The main idea of this intervention is to make the process of formative assessment an open one, open the sense that it can be inspected by stakeholders, as well as open in the sense that all stakeholdersteachers, students, parentscan actively contribute to how it is conducted. Vatrapu, Bull and Johnson extend this idea to the manner learning data get analysed by suggesting a new form for distributing analytical work on such data. This contribution provides also a link from tactical, classroom-level decision making (What do I do next with my students?) to strategic, school-level decision making: What kind of innovations and reforms work for our students? The Teacher Inquiry into Student Learning (TISL) strand of work in the NEXT-TELL project is concerned with fostering in-service teachers professional development by providing new methods and tools for designing and sharing inquiry projects into students learning. It also aims to increase a schools capacity for data-driven decision-making by means of leadership development, including ICT support for the strategic planning of teachers professional development. Luckin et al. describe the first steps into the development of the TISL methodology and the TISL software model.
Teachers Views of the Role of Formative Assessment and Teacher-led Research. Gabriele Cierniak, Birgit Imhof, Friedrich Hesse, KRMC, Tuebingen, Schleichstrae 6, Germany, {g.cierniak, b.imhof, f.hesse}@iwm-kmrc.de; Barbara Wasson, Cecilie Hansen. University of Bergen & InterMedia, Uni Helse, Uni Research, Bergen, Norway; Cecilie Hansen {Barbara.Wasson, Cecilie.Hansen}@uni.no. The NEXT-TELL project aims on the one hand to enhance adaptive teaching practices in the technology-rich classroom by co-designing and co-developing ICT that support evidence-based formative assessment. On the other hand the project aims to support teachers professional development with regard to educational ICT-use and formative E-assessment by co-designing and co-developing a method of teachers inquiry into student learning (TISL). Hence, the project wants to support teachers as key stakeholders of educational innovation with regard to formative E-assessment (Reimann, Utz, Halb, this symposium) and teacher research (Luckin, Craft, Avramides, & Kossowski, this symposium). In order to work with teachers interested in the project according to their current teaching practices, we investigated their current views on and practices in formative E-assessment as well as teacher research in a baseline study. We first developed two semi-structured interviews. Interview 1 was designed to find out more about teachers current practices concerning (1) lesson planning, (2) classroom teaching, (3) monitoring, (4) assessment, (5) providing feedback, (6) homework, (7) communication with students, and (8) communication with parents. At the beginning of each topic teachers were asked to tell how they currently practice the topic (e.g., how they assess their students). If not already included in teachers reports, we asked whether they use any kind of ICT in practice and whether they wished to be supported in any activities by ICT. Interview 2 was designed to find out more about the (country-) specific school systems the teachers are in concerning (1) assessment, (2) teacher research, and (3) professional development. Here, teachers were asked about their understanding of the three topics and their interrelatedness and to describe how they see the topics are handled within their educational systems. We conducted the interviews with 34 teachers teaching in secondary classes in five EU-countries (Austria (8 teachers), Denmark (3 teachers), England (7 teachers, 4 pre-service teachers: in England the interview questions were embedded in general discussions in workshops), Germany (6 teachers), Norway (6 teachers). The teachers teaching experiences varied from 1 to 38 years. The analysis of the interview data analyses showed that despite many similarities across the teachers, there are crucial variations with regard to several dimensions in current practices of lesson planning, assessment, and providing feedback. Concerning lesson planning, teachers differed with regard to the dimensions of how collaboratively they plan, how much they consider student assessment, and how as well as how much ICT they use for planning. Concerning assessment, teachers differed with regard to how formative their assessment practices are and how much ICT they use in assessing their students. Concerning the provision of feedback, teachers also differed with regard to whether they use ICT or not and which type of feedback they provide. Here, the teachers descriptions of their feedback practices also reflected their knowledge concerning formative
ISLS
34
ICLS2012
Volume 2: Symposia
assessment (or assessment for learning: Black, Harrison, Lee, Marshal, & Wiliam, 2003). Whereas some teachers were interested in enhancing their possibilities to enhance feed-forward, others did not mention any emphasize on feedback. Moreover, teachers differed with regard to their explicit knowledge of formative assessment (e.g., Black & Williams, 1998) and teacher research in general (Bannan-Ritland, 2008). Whereas some of the teachers are from systems that promote formative assessment initiatives and teacher research approaches (e.g., teachers in England), others have not yet heard about such initiatives (e.g., teachers in Germany). Thus, a few teachers were irritated and even showed misconceptions when they were confronted with the term teacher research. In general, the more elaborated teachers conceptions on formative assessment and teacher research were, the more interrelated they saw the different levels of teaching practices concerning formative assessment with professional development and with teacher research, but also with school development as whole. Although our results are not representative in a statistical sense, they are nevertheless in line with what is known about national and school-level differences (e.g., Law, Pelgrum, & Plomp, 2006) and hence, suggest different teacher profiles (see Figure 1) which result from the level of understanding of the components (classroom practices/pedagogy, teacher research, and school strategy) as well as their interrelatedness: 1. Particularised-incomplete: Teachers with this profile are not yet familiar with all components to be supported in NEXT-TELL. They also see connections between the different components of formative (E)-assessment as classroom practices, teacher research as professional development, and school development as rather weak or loose. 2. Particularised: Teachers recognize the importance of all components (even if not practiced so far) but do not have a fully integrated view of their interrelations 3. Integrated: Teachers have a differentiated understanding of the components and are aware of their interrelationships.
Figure 1: Visualized examples of teacher profiles. In order to integrate teachers with different profiles concerning formative assessment and teacher research into the project appropriately, we will chose different interventions as next steps in our process of co-development (Penuel, Roschelle, & Shechtman, 2007).
Open Models of ICT-embedded Formative Classroom Assessment. Peter Reimann, MTO, Tbingen, Germany (p.reimann@mto.de); Wilfrid Utz, BOC, Vienna, Austria (wilfrid.utz@boc-eu.com); Wolfgang Halb, Joanneum Research, Graz, Austria (wolfgang.halb@joanneum.at). As mentioned in the introduction, an important step for making classroom data useful for teachers and students tactical decision making (What to do next?) is to express the information on the level of knowledge and skill development, processes of learning, motivation and engagement. Teachers usually get this information from direct observations, and from formative assessments, such as quizzes or problem solving exercises. In the technology-rich classroom, a third source of information are the recordings of learning activities as they unfold in digital media, such as software applications (e.g, MS Excel) , learning management systems (e.g, Moodle), and increasingly on Cloud tools and services (such as Google Docs). While these digitally enacted learning activities are easily recorded (e.g., as log files), they usually need further processing in order to be interpretable as information on learning and knowledge. So far, methods to do this automatically have been confined to socalled Intelligent Tutoring Systems, e.g., Cognitive Tutors (Koedinger & Corbett, 2006) and personalized learning systems (Heller, Steiner, Hockemeyer, & Albert, 2006). This approach, however, needs a very detailed analysis of the knowledge/skill structures to be learned, and a very fine-grained learning application in order to trace the students actions on a level that is appropriate for the diagnostic algorithms. As a consequence, such
ISLS
35
ICLS2012
Volume 2: Symposia
systems have only been developed so far for a few curriculum areas, and the diagnostic machinery has been coupled with specifically designed learning software. Teachers that dont teach in domains covered, and/or dont want to use the specific application with their students are out of luck. In addition, even if more teachers would have access to these applications (some of which are web-based by now), it is still an open question how many would actually make use of them for formative assessment purposes, given that the assessment method is opaque. Teachers would face the dilemma that for providing learning-relevant information to their students they would need to do so based on an assessment method they cannot explain, and which is hidden in programming code. We therefore decided to build on a glass-box approach to technology-enhanced formative assessment; both the diagnostic data transformation procedures as well as the resulting learner model should in principle be open. And beyond that: not only should the assessment be open, it should also be the teacher who, in principle, can develop and modify technology enhanced classroom assessment methods. This takes also into account that diagnostic information on students learning can come from many sources: we consider in particular teachers, students, (in the role of self and peer assessors), parents, and software applications. They all can produce diagnostically relevant information that are candidates for visualisation in the Open Learner Model (OLM). An important design consideration in NEXT-TELL is that we believe that all assessment methods, independent of who employs them (teacher, student, parent, software) should adhere to certain quality criteria, in particular concerning their validity and reliability. For establishing validity, we build on the Evidencecentered assessment Design methodology (Mislevy & Riscontente, 2006), and for establishing reliability of assessments in NEXT-TELL the users need to document their assessment process, thus creating provenance data (Groth et al., 2006). In short, we require that humans as well as computational assessment services describe how they, starting from observations on what learners do in the course of their learning activities (performance) and from the artifacts produced in the course of learning activities, come to conclusions about learners knowledge and skills.
Figure 2: One of the views of the NEXT-TELL authoring tool, integrating learning activities with formative assessments. On the left, a Notebook is shown that renders the attributes that are available with each modeling object, and makes them available for editing. To achieve this, the creators of assessment methods, including teachers, need to be provided with a language to express their assessment ideas. If, as in our case, the assessment is to be of the formative kind, it needs to be integrated with the teaching/learning process. Therefore, we equip teachers with an authoring tool for designing learning activity sequences and for relating these to expected knowledge changes (learning progressions) as well assessment methods (see Figure 2). We treat any assessment process as an instantiation of an assessment model, and any learning activity sequence as an instance of a learning sequence model. Technically, we use a meta-modeling shell and the Open Models approach (Karagiannis, Grossmann, & Hoefferer, 2008) for modeling formative assessment processes as well as learning activity sequences). The models, then, are not only descriptions, but can also serve as the basis for rapid deployment in an ICT environment. Currently, we provide adaptors to Moodle and to Google Apps.
Towards Visual Analytics for Teachers: An Open Learner Model Approach
ISLS
36
ICLS2012
Volume 2: Symposia
Ravi Vatrapu, Computational Social Science Laboratory (CSSL), Dept. of IT Management, Copenhagen Business School, vatrapu@cbs.dk; Susan Bull, Matthew Johnson, Electronic, Electrical and Computer Engineering, University of Birmingham, s.bull@bham.ac.uk, m.d.johnson.1@bham.ac.uk The NEXT-TELL project aims to provide computational and methodological support for teachers in real-time and in-situ classroom settings. Towards this end, we integrate emerging developments in visual analytics and the established methodological approach of design-based research (DBR) in the learning science into Teaching Analytics and propose a model of teaching analytics, termed triadic model of teaching analytics (TMTA) (Vatrapu, et al., 2011), discussed next. We adapt and extend the dyadic model of pair analytics in visual analytics to a Triadic Model of Teaching Analytics (TMTA) as shown in Figure 3:
Figure 3: Triadic Model of Teaching Analytics (TMTA) At its core, our model sees collaborative knowledge building between teachers, analysts and researchers. We think of the relationships between the TE, VAE and DBRE as a dynamic socio-technical system. The design considerations are about creating feedback loops between the three individuals, such that each one drives the other two to higher levels of performance on the positive side (with the cost of anxiety in the negative case). That is, feedback from the teacher inspires the VAE to create new, better visualizations and for the researcher to better understand the ongoing teaching and learning processes while feedback from the VAE perhaps in the form of visualization artifacts allows the teachers to better understand what is going on in the classroom from a learning activity design perspective and the research to hypothesize, test and predict student learning trajectories and performance outcomes. All in all, these feedback loops should culminate in the teacher providing timely, meaningful actionable, customized and personalized feedback to students. The key point here is that each member of the triumvirate of TE, VAE, and DBRE can gain from the other two, not that each partner's role is to highlight deficiencies of the other two. Therefore, TMTA involves a close collaboration between the TE, VAE, and the DBRE. It includes teaching practitioners in the design process and invites them to contribute significantly to the innovation of the visual analytics tools. This allows these learning analytics tools to address pedagogical issues as they arise and evolve in real classrooms. In the next section, we outline an approach to TMTA based on open learner models (OLM). An obvious starting point for developing the TMTA approach is to base it around the existing work in Artificial Intelligence in Education, on open learner models. A learner model holds information (usually) about an individual learner, and the model is automatically and dynamically updated during the user's interaction with a computer-based/online educational environment. The learner model typically includes data about the learner's knowledge state, which may include specific difficulties and misconceptions; and it can also have data on other aspects of the learning process (e.g. representation, content, teaching style preferences; motivational, social, affective attributes). The learner model is then used by the educational environment to adapt its teaching to the specific needs of the individual learner (the environment 'understands' the user's understanding). An "open learner model" is a learner model that can also be externalised to the user (Bull & Kay, 2007). This externalised (open) learner model may be simple or complex in format using, for example: text, skill meters, concept maps, hierarchical structures, animations (Bull et al., 2010). Normally the user who accesses the learner model is the learner. Common purposes of externalising the learner model to learners are to promote metacognitive activity such as awareness-raising, reflection, selfassessment and planning (Bull & Kay, 2008). Some learner models have, however, also been made available to teachers (Bull & McKay, 2004; Eyssautier-Bavay, Jean-Daubias, & Pernin, 2009; Zapata-Rivera, Hansen, Shute, Underwood, & Bauer, 2007). Teacher access to the learner models of their students can help them to better understand learners' needs as individuals and as a group, and can therefore enable teachers to adapt their teaching. Of particular interest in NEXT-TELL is the possibility of open learner models to support the routine but dynamic decision-making that teachers need to perform in the classroom. While the above describes the typical situation of open learner models, it is easy to envisage this being extended for use in TMTA. A range of visualisations or externalisations of the learner model have been
ISLS 37
ICLS2012
Volume 2: Symposia
explored (e.g. Bull et al., 2010), and these are currently being further extended to support the synthesis of work between teaching experts, visual analytics experts and design-based research experts, as required for the proposed TMTA approach within the NEXT-TELL project.
Teachers Inquiry into Student Learning: Supporting Design-based Research towards Formative Assessment in the Technology-rich Classroom. Rosemary Luckin, Brock Craft, Katerina Avramides, The London Knowledge Lab, Institute of Education, London. r.luckin@ioe.ac.uk.; Michal Kossowski, BOC Information Technologies Consulting, Warsaw, michal.kossowski@boc-pl.com A recent report on the impact and use of ICT in schools across 27 EU countries confirms that schools effective engagement with ICTs is patchy and that there exist considerable differences in levels of e-maturity within and between countries (Empirica, 2006). At the same time, the existence of these ICT resources is raising expectations for increased and improved communication about students learning (Becta, 2008, 2009). Teachers and schools face a continual struggle in understanding how to effectively embed these new tools into their everyday pedagogic practices, particularly in areas where ICTs are not perceived to be core elements (Clark et al., 2009; Freebody et al., 2008). In response to these dissonances around technology use in education, some design researchers suggest that a more holistic and systemic approach to understanding the learning context is needed (Fishman, 2004; Luckin, 2010; McKennay et al., 2008; Ravenscroft, 2009). The development of the TISL methodology and software tool has therefore adopted a participatory approach through which teachers directly inform the design process and outputs. Our approach is supported by two theoretical frameworks (1) Teacher Design Research (TDR) (Bannan-Ritland, 2008); and (2) The Ecology of Resources model and design framework (Luckin, 2010). The former is used to generate a dialogue around a conceptualisation of teachers as innovators, whilst the latter is used to expand the TDR model to take into account the complexity of technology-rich learning contexts and the emergent properties of interactions between teachers, learners and available resources. In practice, the literature on teacher inquiry, teacher design research, and teacher-led research has shown that, in order to pursue this kind of research, teachers often require guidance and support. For TISL, such guidance and support must be developed in collaboration with teachers. To facilitate this need, TISL, as with related components within NEXT-TELL, is being developed in two phases: (1) researcher-led; and (2) teacher-led. The TISL approach outlined in this paper is being developed as part of the researcher-led phase of the project. It provides a preliminary framework for identifying relevant tools and approaches on the use of available data for evaluating students learning (Table 1) in alignment with teachers professional development. TISL aims to enable teachers to conduct sustainable and relevant inquiry into students learning and related school-based practices (e.g. teaching, assessment, etc.). Testing and evaluating the initial TISL method. Table 1: 10 steps to systematising teacher inquiry with TISL TISL Method establishing a trigger choosing a lens (researcher or teacherled) planning for and collecting evidence analysing practices enacting and adapting an action/innovation TISL Tools and Data Handling identifying tools (ALTs) and potential data sources planning for data capture and data sharing collaborative data analysis and interpretation evaluating data and reflecting on inquiry process data-driven decision-making for innovating practice
1. 2. 3. 4. 5.
1. 2. 3. 4. 5.
The modelling procedure for the TISL inquiry planner is drawn from the initial 5-Step TISL method (see Table 1 above). It identifies the systematic sequence of the 5 steps to be taken by the user (e.g. teacher participants) when planning an inquiry process (into students learning). The TISL method is designed to produce results that can be used by other teachers to inform teaching, learning and formative assessment. . The structure and dependencies for the TISL inquiry process planner are shown in Figure 4 below.
ISLS
38
ICLS2012
Volume 2: Symposia
Figure 4: Time line view of the TISL Inquiry Process Planner. An example of these steps in action drawn from empirical work with teachers would be: Step 1 Establishing a trigger: how can video data be used to support teacher inquiry into students learning? Step 2 Choosing a lens: constrain to 1 minute video segments at the beginning-middle-end of a learning activity to focus inquiry. Step 3 Plan and Collect Evidence: seek appropriate segments of video from system repository, or plan collection of fresh data. Step 4 Analyse Practices: explore of learning revealed in the video clips for formative assessment and identify framing patterns for future practice and analysis. Step 5 Adapt and Enact: implement and test framing patterns in practice. References Bannan-Ritland (2008). Teacher Design Research. An emerging paradigm for teachers' professional development. In A. E. Kelly, R. A.Lesh & J. Y. Baek (Eds.), Handbook of design research methods in education (pp. 246-262). New York: Routledge. Becta (2008), Harnessing Technology Review 2008, 2009: The role of technology and its impact on education. Coventry: Becta. Black, P., Harrison, C., Lee, C., Marshal, B., & Wiliam, D. (2003). Assessment for learning. New York: Open University Press. Black, P. & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education, 5 (1), 7-74. Bull, S., & Kay, J. (2007). Student Models that Invite the Learner In: The SMILI Open Learner Modelling Framework. International Journal of Artificial Intelligence in Education 17(2), 89-120. Bull, S., & Kay, J. (2008). Metacognition and Open Learner Models. In I. Roll & V. Aleven (Eds.), Proceedings of Workshop on Metacognition and Self-Regulated Learning in Educational Technologies, International Conference on Intelligent Tutoring Systems (pp. 7-20). Bull, S., & McKay, M. (2004). An Open Learner Model for Children and Teachers: Inspecting Knowledge Level of Individuals and Peers. In J. C. Lester, R. M. Vicari & F. Paraguacu (Eds.), Intelligent Tutoring Systems (pp. 646-655). Berlin Heidelberg: Springer. Bull, S., Gakhal, I., Grundy, D., Johnson, M., Mabbott, A., & Xu, J. (2010). Preferences in Multiple View Open Learner Models. In M. Wolpers, P. A. Kirschner, M. Scheffel, S. Lindstaedt & V. Dimitrova (Eds.), EC-TEL 2010 (pp. 476-481). Berlin Heidelberg: Springer. Clark, W., Logan, K., Luckin, R., Mee, A. and Oliver, M. (2009) 'Beyond Web 2.0: mapping the technology landscapes of young learners', Journal of Computer Assisted Learning 25(1), 56-69. Crawford, V. M., Schlager, M. S., Penuel, W. R., & Toyama, Y. (2008). Supporting the art of teaching in a datarich, high-performance learning environment. In E. B. Mandinach & M. Honey (Eds.), Data-driven school improvement (pp. 109-129). New York: Teachers College Press. Empirica (2006), Benchmarking Access and Use of ICT in European Schools 2006. Bonn: Empirica. Eyssautier-Bavay, C., Jean-Daubias, S., & Pernin, J.-P. (2009). A Model of Learners Profiles Management Process. In V. Dimitrova, R. Mizoguchi, B. D. Boulay & A. Graesser (Eds.), AIED09 (pp. 265-272). Amsterdam: IOS Press. Fishman, B. et al. (2004) Creating a Framework for Research on Systemic Technology Innovations, Journal of the Learning Sciences, 13(1), 43-76 Freebody, P., Reimann, P. and Tiu, A. (2008) Alignment of perceptions about the uses of ICT in Australian and New Zealand Schools. University of Sydney.
ISLS
39
ICLS2012
Volume 2: Symposia
Groth, P., Jiang, S., Miles, S., Munroe, S., Tan, V., Tsasakou, S., & Moreau, L. (2006). An architecture for provenance systems Retrieved from http://eprints.ecs.soton.ac.uk/12023/ Hamilton, E., & Jago, M. (2010). Toward a theory of personalized learning communities. In M. J. Jacobson & P. Reimann (Eds.), Designs for learning environments of the future (pp. 263-282). New York: Springer. Heller, J., Steiner, C., Hockemeyer, C., & Albert, D. (2006). Competence-based knowledge structures for personalised learning. International Journal on E-Learning, 5(1), 75-88. Karagiannis, D., Grossmann, W., & Hoefferer, P. (2008). Open Model Initiative. A feasibility study. Vienna: University of Vienna. Koedinger, K. R., & Corbett, A. (2006). Cognitive tutors. In R. K. Sawyer (Ed.), The Cambridge Handbook of the Learning Sciences (pp. 61-77). New York: Cambride University Press. Law, N., Pelgrum, W. J., & Plomp, T. (2008). Pedagogy and ICT Use in Schools around the World: Findings from the IEA SITES 2006 Study. Berlin: Springer. Luckin, R. (2010) Re-designing Learning Contexts: Technology-Rich (Learner-Centred) Ecologies, London: Routledge. McKenney, S. et al. (2006) Design Research from a Curriculum Perspective in Educational Design Research (Ed. van den Akker, J. et al.), London: Routledge Mislevy, R. J., & Riscontente, M. M. (2006). Evidence-centered assessment design. In S. M. Downing & T. M. Haladyna (Eds.), Handbook of test design (pp. 61-90). Mahwah, NJ: Lawrence Erlbaum. Parr, J. M., & Timperley, H. S. (2008). Teachers, schools and using evidence: Considerations of preparedness. Assessment in Education: Principles, Policy & Practice, 15(1), 57-71. Penuel, W. R., Roschelle, J., Shechtman, N. (2007). Designing formative assessment software with teachers: An analysis of the co -design process. Research and Practice in Technology Enhanced Learning, 2(1), 5174. Ravenscroft, A. (2009) Social software, Web 2.0 and learning: status and implications of an evolving paradigm, Journal of Computer Assisted Learning, 25(1), 1-5 Vatrapu, R., Teplovs, C., Fujita, N., & Bull, S. (2011). Towards Visual Analytics for Teachers' Dynamic Diagnostic Pedagogical Decision-Making. Paper presented at the 1st International Conference on Learning Analytics & Knowledge (LAK 2011), Banff, Canada. Wang, F., & Hannafin, M. J. (2005). Design-based research and technology-enhanced learning environments. Educational Technology Research and Development, 53(4), 5-23. Zapata-Rivera, D., Hansen, E., Shute, V. J., Underwood, J. S., & Bauer, M. (2007). Evidence-Based Approach to Interacting with Open Student Models. International Journal of Artificial Intelligence in Education, 17(3), 273-303.
ISLS
40
ICLS2012
Volume 2: Symposia
Gathering Evidence of Scientific Argumentation Practices: From Pre-Kindergarten to High School

Tiffany R. Lee, Hiroki Oura, Giovanna Scalone, Kari Shutt, John Bransford, Andrew Shouse, Katie Van Horne, Nancy Vye, University of Washington, LIFE Plaza, Box 354941, Seattle, WA, USA 98195-4941 Email: tlee13@uw.edu, oura@uw.edu, gscalone@uw.edu, shuttk@uw.edu, bransj@uw.edu, awshouse@uw.edu, katievh@uw.edu, nancyvye@uw.edu Maureen Munn, University of Washington, Genome Sciences, Box 355065, Seattle, WA, USA, 98195, mmunn@uw.edu Randy Knuth, Knuth Research, Inc., 717 East Blackhawk Drive, Spokane, WA, USA 99208, randy@knuthresearch.com Philip Bell (discussant), University of Washington, LIFE Plaza, Box 354941, Seattle, WA, USA 98195-4941, pbell@uw.edu Abstract: The purpose of this symposium is to explore students scientific argumentation practices in relation to the recently released Framework for K-12 Science Education (NRC, 2011) in the United States. We will present data from four research efforts on prekindergarten, upper elementary school, and high school students science learning in classroom settings, followed by a reflective presentation by a discussant. Given that our work spans multiple grade levels of formal school instruction, we will discuss our observations across these groups of students while mapping to the goals and suggested progression of argumentation practices described in the NRC framework.
Symposium Overview
The purpose of this symposium is to explore students scientific argumentation practices in relation to the recently released Framework for K-12 Science Education (NRC, 2011) in the United States. We will present data from four research efforts on pre-kindergarten, upper elementary school, and high school students science learning in classroom settings, followed by a reflective presentation by a discussant. Given that our work spans multiple grade levels of formal school instruction, we will discuss our observations across these groups of students while mapping to the goals and suggested progression of argumentation practices described in the NRC framework. Argumentation is recognized as a necessary and central component of science, and an understanding of scientific argumentation is essential for all people for scientists to conduct their work and for citizens to productively engage with everyday topics such as the environment or personal health. The NRC framework (2011) specifically highlights the importance of scientific practices in addition to crosscutting concepts and disciplinary core ideas, in quality science education. Indeed, this aligns with trends in science education and the emphasis on moving away from teaching science as purely factual knowledge and instead incorporating scientific discourse, explanations, and argumentation (NRC, 2007, 2009; AAAS, 1989, 1993). Not only does the overall emphasis on scientific discourse better align with actual disciplinary practices of professional scientists (e.g., Bell, 2004; Edelson & Reiser, 2006; Newton, Driver, & Osborne, 1999), but opportunities to engage in argumentation in the science classroom have been shown to increase students conceptual understanding and reasoning skills (e.g., Mercer, Dawes, Wegerif & Sams, 2004). Despite the general consensus on the importance of understanding how to argue scientifically, it is still often missing or ineffectively utilized in typical science instruction across grade levels. The research presentations in this symposium highlight four curricular design efforts to support students capacities to engage in scientific reasoning and argumentation while also integrating their personal experiences, interests, and relevant topics in their community to increase their knowledge of and participation in science. Our goals for the symposium are to examine how students engage in scientific argumentation within these units and add to the discussion of the progression of scientific argumentation (Berland & McNeill, 2010; NRC, 2011) practices across pre-kindergarten through high school science education. The first paper examines the developing abilities of pre-kindergarten children, ages 4-5 years, to engage in scientific argumentation in classroom science discussions. Although these young children do not systematically use evidence and construct complete arguments every time, these data show that they are capable of using data to support their claims. Often the students draw upon their everyday experiences and knowledge of the natural world, but they also reference shared classroom experiences or artifacts as they make their claims. The next two papers focus on fifth grade students participation in re-designed science curricular units. One paper looks at how 5th Grade students in a science classroom engage in collaborative debate on a controversial issue on Triclosan. The other paper compares argumentation across three curricular contexts. One
ISLS
41
ICLS2012
Volume 2: Symposia
unit is a traditional, teacher-directed unit. The other two units, of our own design, feature student agency and student-directed investigations. Argumentation varies widely across contexts with differing amounts of agency. The last paper examines high school students argumentative reasoning during hypothesis development and testing with a real dataset in a refined science curriculum designed for engaging students in an authentic practice of genomic and epidemiological research. The authors analyzed student conversations during their investigations. They found that argumentative reasoning emerges during both hypothesis development and statistical analysis. Students did frequently articulate logical relationships among claim, evidence, and reasoning, and they worked to develop alternative explanations when they encountered results contradictory to their original hypothesis. However, they also found such reasoning is limited during causal inference. The authors developed a hypothesis that both students and teachers may have developed a singular view of scientific method for causal reasoning as a reflection of the existing science curricula, and it might interfere when they learn other scientific methods. We intentionally present data from across pre-kindergarten through high school curricular enactments to advance conversations about the development of and expectations for school aged youths argumentation practices. Our discussant, Philip Bell, will provide reflective commentary on the presentations.
Paper 1: Supporting Pre-Kindergarten Students Emergent Scientific Argumentation Practices (Tiffany R. Lee)
Views on science education have undergone significant changes in the twentieth century. Science has traditionally been taught as a well-defined process using the scientific method (hypothesis, experimentation, control of variables, etc) with theories, concepts and vocabulary to memorize. In the last 50 years, science education reforms have worked to impart more accurate accounts of scientific ideas, practices, and reasoning skills to schoolchildren. In particular, the recently released NRC Framework for K-12 Science Education (2011) has explicitly called for increased attention to students understanding of scientific practices. In this paper, we identify instances of pre-kindergarten students engaging in scientific argumentation during classroom science instruction and provide evidence of the progression of argumentation practices. Osborne (2010) suggests that the nature student participation in reasoning and argumentation is complex, dependent upon their existing domain-specific knowledge and their reasoning capabilities at a given age. For very young children, science instruction is often limited to direct observations and hands-on activities, with the assumption that they are not yet capable of abstract thinking and reasoning. Much of the existing early science curricula and teaching approaches are based upon an overextension of Piagets conjecture that young children are concrete, simplistic thinkers, resulting in unnecessary and oftentimes inappropriate constraints on young childrens learning (Metz, 1995). The research effort described in this paper presupposes that young children are capable science learners and should be provided with opportunities to support their developing science knowledge and skills (cf. Brown, Campione, Metz, & Ash, 1997; Eshach & Fried, 2005) and builds upon recent research that demonstrates young childrens developing knowledge and experiences with science (Lee, 2010) and their abilities to engage in extended discourse and argumentation (e.g., Lee, 2011; Corsaro, 2003; Gallas, 1995). The overarching goals of the curricular design are to leverage young childrens existing knowledge and interests about the natural world and engage them in the processes of science. We draw upon theory about how people learn, particularly in relation to engaging learners interests and preconceptions (NRC, 2000); Vygotskys (1978) notion that knowledge is co-constructed by people and that childrens learning is optimally supported by identifying and leveraging zones of proximal development; and applications of Brown and Campiones (1994) work on creating classroom communities of learners. This paper focuses specifically on pre-kindergarten students engagement in scientific argumentation and their use of everyday experiences and knowledge to support their claims. We will show the emergent argumentation practices of these young students and discuss how the classroom teachers facilitated these opportunities for argumentation.
Methods
The data presented in this analysis are part of a year-long curricular intervention in two half-day prekindergarten classrooms in the Northwest region of the United States. Each of the pre-kindergarten classrooms comprised of 12-15 students (ages 4-5 years) one lead teacher, and one assistant teacher. Researchers worked with the teaching team to develop weekly science lessons that emphasized student-led discourse, argumentation, and connections to their existing knowledge of the natural world. Lessons included exploration of artifacts brought into the classroom (e.g., live animals, plants, fossils), observations of the outdoors, and questions to spark student discussion, and typically lasted between 30 to 60 minutes each. To preserve the authenticity of student-generated topics of study, the lessons were designed throughout the school year and built upon students interests and questions. Data sources included 1) video recordings of the weekly science lessons, 2) fieldnotes from classroom observations and teacher interviews, 3) digital photographs of classroom activities, 4) artifacts
ISLS
42
ICLS2012
Volume 2: Symposia
such as student-made drawings and writing samples, materials used for instruction, etc, 5) teacher interviews to reflect on science instruction throughout the year, and 6) parent surveys about students out-of-school activities.
Analysis and Findings

Video data of class discussions were analyzed for examples of scientific argumentation between two or more students. As hypothesized, questions that tapped into childrens common experiences but did not have obvious factual answers allowed for extended discourse and argumentation. One unit at the beginning of the school year focused on living vs. non-living things. We began by asking the students what it means to be alive and having them identify characteristics of living things. Students offered ideas from their existing knowledge of the living world: it moves, it talks, it eats, etc. Teachers then asked questions to challenge students ideas. One question, Are plants alive? prompted an extended discussion between the students do plants make noise and do they eat? In this discussion, we observed students making references to their existing knowledge of plants (e.g., knowing that plants cannot make noises or move on their own), and then trying to reconcile these ideas to support the argument that plants are living things. In another lesson, students are presented with two hamsters one real hamster and one toy hamster that runs on a wheel. The students are asked to decide whether each hamster is alive based on the qualities they have identified for living things. Students easily agree that the real hamster is alive it moves, breathes, eats, and maybe it can make a noise. They also know that living things can reproduce (often talked about as making more or having babies), and they guess that the real hamster can do this. The toy hamster is puzzling at first: several students claim that it could be alive because it moves on its own. Upon closer investigation, one student notices that it has wheels instead of feet and guesses that it uses a battery to move. After more discussion, they agree that the toy hamster is not alive. Throughout the data, we see that students are often quick to give an opinion in response to the teachers questions. When the teacher follows up by asking for a reason behind that opinion or encourages other students to voice a differing opinion, the students are given opportunities to explain their thinking. In these instances, we see that the pre-kindergarten students often support their claims with evidence from their out-of-school experiences or shared classroom activities.
Discussion
Findings from this work show that even very young children are able to engage in scientific argumentation and the development of these practices needs to be supported in science instruction. In these data, pre-kindergarten students readily engage in extended discourse and argumentation when presented with topics that relate to their early experiences with the natural world and encouraged by the teacher to give opinions and agree or disagree with their peers. Although these students do not systematically use evidence to support their claims or necessarily construct thorough arguments each time, they easily leverage their experiences and knowledge to support their claims. Increased opportunities for young children to engage in argumentation practices will likely improve their abilities to successfully use data to support their claims and allow them to become more proficient in their argumentation practices over time. Further research is being conducted to document the range of young childrens abilities, particularly as they relate to how children engage in scientific discourse and argumentation and how they make connections to their everyday experiences. In addition, we seek to better understand the role of the teacher in creating an environment that supports this kind of discourse and science exploration. As a result of these and other related findings about young childrens reasoning abilities, we argue that the goals and approaches for science education need to be revised to support the development of early scientific skills and prepare children for future success in science.
Paper 2: Achieving Coordination in Collaborative Debate (Giovanna Scalone)

Teaching students to argue and collaboratively debate are central features of inquiry in the natural sciences (Bell, 2004). We draw on Bells framing of argumentation and debate in science as the exploration of a theoretical controversy involving the coordination of evidence with theoretical ideas (p. 138). When students engage in collaborative argumentation, they are arguing to learn. This implies that students are collaboratively exploring solutions dialogically (Andriessen, 2006). From a Vygotskian perspective, it is through language that we come to know things, and transform knowledge into actions, objects, and other symbol systems. Argument is a form of discourse, which according to Kuhn (1991) requires one to not only think in a metacognitive and metalinguistic framework, but to also place ones ideas in the planes of the possible, the probable, and the disputable (Pierut-Le Bonniec & Valette, 1991). Elementary school students are rarely asked to participate in collaborative debate and discussion of controversial topics in science education (Levinson & Turner, 2001). As a result, in classroom debate students find difficulties in anticipating objections in their own arguments as well as in the arguments of others (Reznitskaya et al., 2001). This issue also arises because students have difficulties in understanding the
ISLS
43
ICLS2012
Volume 2: Symposia
evidentiary nature of scientific arguments and the social dimensions of science (Bell, 2004; Kuhn, 1993; Driver et al., 1996). In this paper, we discuss the findings of two elementary school science debates on Should hand sanitizers be banned? Perspective taking is essential in science where students come to understand the standpoints of various stakeholders associated with the controversy around triclosan. In addition, we wanted students to move beyond their epistemological naivet and consider a theory as well as evidence and its bearing on a theory (Kuhn, 1991). Helping students to think about their own thinking will help them learn how to evaluate evidence. Finally, it is possible that helping students weigh different forms of evidence enables them to formulate and reformulate ideas, which, in turn, fosters conceptual change (Koslowski, 1996). In this paper we ask: How do elementary school students in science achieve coordination in debate?
Methods & Data Sources

This study took place in two fifth grade science classrooms at Granite Elementary School in an urban neighborhood. There were five iterations of the Micros & Me curriculum, in which the content was framed around microbiology and health; and students were given choice and agency to document community-relevant health topics; design fair tests; and choose a personally-relevant, related topic and conduct research. We report on the findings from the fifth iteration, specifically on a debate, Hand-sanitizers should be banned. We helped students structure their debates by drawing on Bells (2004) framing of argumentation and debate. We think of argument as a product, which includes claims, evidence, and reasoning; and we think of argumentation as the process of assembling these components in order to make ones case. Students debated on whether hand sanitizers that contain Triclosan (an antibacterial, antifungal agent used in consumer products, such as hand sanitizer, toothpaste, etc.) should be used since it affects the microorganisms in the water. In Fall of 2010, participants included 29 consented fifth graders (all between 10 and 11 years old) and two teachers, Ms. Jones and Mr. A. Ethnographic techniques were incorporated into the design to leverage students everyday expertise. The data sources came from participant observation using field notes and digital photography of instruction in the science classrooms, video recording, student illustrations, and students notebooks.
Findings
By following Bells (2004) framing of argumentation and debate within student discourse as well as verbal and nonverbal (cf. Lemke, 1998) argument, in this paper we endeavor to understand how students coordinate their evidence in classroom debate. Preliminary findings reveal that as students engage in different modes of evidence to support their position in the triclosan debate, some were faced with difficulties separating theory and evidence, producing pseudoevidence (Kuhn, 1991). Other students made good evidence-based arguments, however their arguments where disassociated with the counterarguments.
Discussion
The findings have implications for how we can help students not only think about perspective taking in science, but also scaffold their thinking in ways that can help them critically evaluate arguments for the opposing position and make every argument associated with a counterargument. Overall, this contributes to helping students understand how scientists work by having them engage in critique and evaluation. Consequently, students come to understand that science is a body of knowledge that is rooted in evidence-based argumentation (NRC, 2011).
Paper 3: Student Agency and Argumentation Across Curricular Contexts (Kari Shutt, Nancy Vye, & John Bransford)
An important goal for science education is to provide students with the opportunity to engage in authentic science practices (National Research Council, 1996, 2011). Argumentation is a key component of scientific discourse and plays a central role in the construction of knowledge. It is the means by which the scientific community reaches consensus, and is therefore a crucial scientific practice (NRC, 2007, 2011). In this study, we compare argumentation across three different curricular contexts that differ in terms of their level of student agency. Each unit is taught to 5th grade students over the span of 12 weeks. One unit, the FOSS Environments unit, is a kit-based unit that features discrete, teacher-directed investigations. The other two units, the Isopod Habitat Challenge and My Skokomish River Challenge, were designed by our research team. The cornerstones of the latter units are student agency and student-directed collaborative investigations related to an overarching problem (Vye et al., 1998).
Methods
The study took place in a mid-sized suburban school district. We selected one teacher and class from each curricular context as cases for consideration. Within each class, we selected one small group of students to
ISLS
44
ICLS2012
Volume 2: Symposia
follow closely. Each of the three units involves a series of investigation cycles. We chose to examine an investigation cycle that took place midway through the units. We focused our attention on the selection of the research question and the experimental design process. The primary data source was naturalistic video. Videos were transcribed for analysis, and episodes were coded using an adaptation of Chin and Osbornes (2010) argumentation coding scheme.
Findings Case 1: FOSS Environments (Comparison)

In the FOSS-Environments unit (Delta Education (Firm), Lawrence Hall of Science, & University of California Berkeley, 2000), students undertake a series of life science investigations. In most instances, students investigate questions that are determined by the teacher, and students typically follow prescribed procedures to investigate those questions. In this investigation cycle, Ms. Stark gave her students the investigative question at the start of the class session, and asked students to brainstorm ways that they might conduct the investigation. George: Well if its like too cold or something it [the isopod] could roll up into a ball or just stay very still. Eileen: I think it rolls up into a ball when it's scared because um George: Or when it just doesn't like the climate. Drew: You can pick yours [isopod] up now [Off Task] Student interaction was limited for the Comparison students, and there was minimal evidence of argumentation, with only one weak rebuttal offered during the conversation.
Case 2: Isopod Habitat Challenge (IHC)

The Isopod Habitat Challenge addressed the same learning objectives as the FOSS Environments unit, but in the redesigned unit, the IHC, small groups of students conducted sustained, original inquiry in which they determine what and how to design a classroom habitat for isopods. Ms. Atwell began this investigation cycle with a 90 minute, student-led discussion about researchable questions. She interjected open-ended questions periodically to move the discussion forward. After this discussion, Ms. Atwell released the students to generate a question in their small groups. The students considered seven possible questions before settling on their choice. Design of the procedure flowed naturally from the conversation about investigative questions, with students designing their experiment as they debated the merits of the possible questions. In total, IHC students spent 59 minutes in small group conversation around design issues. The IHC students offered more new ideas about investigative questions and experimental design than students in the FOSS, and they demonstrated more evidence of argumentation around those ideas. This included an emerging ability to indentify weaknesses in claims. Lisa: Do isopods drink water? John: You can't find that out. We can't find that out. Jessi: All organisms need water. Lisa: Nu-uh. They could eat like- they could eat the- they eat the carrots that has moisture in them as their water. John: Really, you can't tell if they're drinking really. Lisa: You could. Really, you could. You put a certain amount of water in a little cup or something and measure it and then you see if later on there's a little less amount. John: Well it would be so tiny you could barely find it
Case 3: My Skokomish River Challenge (MSRC)

MSRC is an earth science unit based on similar design principles to the IHC. It is challenge-based and features opportunities for sustained inquiry, revision of ideas, and student agency. It also features a number of studentled investigations. To set up the investigative question selection, Ms. Donovan told students what experiments they might do and how they might go about conducting them. When given the opportunity to select their question in their small group, one student simply stated a question from the list. For the procedure, Ms. Donovan gave students a fill-in-the-blank procedure sheet. Students inserted their variables on the blank spaces; the remainder of the procedure was provided for them. There was no discussion of the procedure in the small group. Despite the fact that the unit was designed for student agency--where students would generate questions to investigate and design ways to investigate these questions--that agency was not offered to them in this investigation cycle. These activities were highly-constrained and structured by the teacher. Only 14 minutes were provided for small group conversation. This particular group did not choose to interact during that time, and thus, there was no evidence of argumentation.
ISLS
45
ICLS2012
Volume 2: Symposia
Discussion
Findings from the IHC unit suggest that authentic, agentive science inquiry can provide a rich context for engaging students in productive argumentation. However, we saw widely different outcomes in the IHC and the MSRC, despite their common design principles. Curriculum can create an environment that fosters productive argumentation, but particular pedagogical moves are needed to support this practice. Positioning students as knowledgeable and capable of independent inquiry, for example, by asking open-ended questions and encouraging student-to-student interaction and decision-making, appear to foster agency and productive argumentation.
Paper 4: High School Students Argumentative Reasoning During Hypothesis Development and Testing In Epidemiological Analysis (Hiroki Oura, Katie Van Horne, Andrew Shouse, and Maureen Munn, Randy Knuth)
Introduction
Developing a scientific argument can be seen as marshaling relevant evidence and justifying an explanation relative to other alternative explanations for particular observations or phenomena (Osborne & Patterson, 2011). As a basic form for instruction, students should develop an argument with a set of claim, evidence, and reasoning, and they also need to hedge their argument by articulating the limitations and possible rebuttals (Toulmin, 1958; McNeill & Krajcik, 2012). The key for meaningful classroom practice is to provide the appropriate contexts in which students may reach multiple reasonable explanations, for instance, by analyzing a real dataset (instructional context), by supporting explanations with relevant evidence (argumentative product), and by defending their explanations and questioning explanations from others (argumentative process) (Berland & McNeill, 2010). The authors have refined an existing curriculum aimed at engaging students in an authentic practice of genomic and epidemiological research (Munn et al., 2010) and developed a set of scaffolds in collaboration with genome scientists in a three-year project. The goal of this design-based research is to engage students in authentic data analysis with a real dataset in which they develop arguments for factors that increase a persons risk of becoming a regular smoker. Smoking is a multifactorial trait, and various environmental and physiological (or genetic) factors contribute to its initiation and the difficulty in quitting (cessation). The dataset is from a real case control study of nearly 300 adult smokers and non-smokers and includes items from the research survey completed by subjects and limited genotyping on their DNA. Students develop explanatory hypotheses by defining the exposures and non-exposures for selected survey items as candidate factors. Students test their hypotheses by calculating a significance test for association and determining causal inference by applying a set of criteria used by epidemiologists. For each test (limited to only a few), students estimate the odds ratio as the magnitude of the difference in likelihood for the exposure between regular smokers (cases) and those who tried smoking but did not continue (controls), and evaluate the significance with the 95% confidence interval. Once students identify the association by the statistical evidence and their reasoning, they may make causal inference by scrutinizing the degree to which their results meet a set of criteria for causality such as the strength of association, temporal sequence, consistency with other studies, and lack of confounding factor (Bradford Hill, 1965).
Method
In this study, we examined high school students argumentative reasoning as they conduct hypothesis development and testing during our first-year implementation. Our data include video- and audio- recordings during data analysis and associated field notes from 13 high school student groups (2 to 4 students per group) in urban or suburban public schools or a local summer program. All teachers in the classrooms had participated in a training workshop in which they learned basic concepts in the curriculum and practiced data analysis with their own investigations. In the classrooms, students conducted their investigations after learning basic concepts related to the model of smoking, case control study, and evaluation with the odds ratio and confidence interval, mostly in a lecture format. We analyzed student conversations during their data analysis and evaluated their argumentative reasoning using the instructional model articulated by McNeill & Krajcik (2012). Due to the variability in the lengths and situations of the recordings among the participating classrooms, we focus on and report noteworthy themes on students typical patterns and performance followed by our discussion along with the learning progression and instructional concerns.
Findings
Our analyses indicate that argumentative reasoning can emerge during hypothesis development based on informal evidence such as students beliefs, experience, and observations. Students often developed their hypothesis as they identified a survey item related to their overarching hypothesis and justified their hypothesis to the item by drawing on such informal evidence. As an ideal case, for instance, when one student in a group
ISLS 46
ICLS2012
Volume 2: Symposia
was developing a test to see if having a close relationship with parents who smoke can increase a risk of smoking, another student proposed an alternative hypothesis that people might not smoke because their parents smoke by drawing on her fathers case (they called it rebelling). They decided to test this hypothesis and found that the odds ratio was insignificant; they concluded the hypothesis was not supported, though they each appeared to maintain their belief in their own hypothesis after testing. During statistical analysis, students frequently consider and articulate logical relationships among claim, evidence, and reasoning when they evaluate the association from the odds ratio and confidence interval. Most groups appropriately evaluated the significance of the association from the confidence interval by reasoning in a way such as one is outside the interval, so there is association (the null is one). Their reasoning seemed to become more active particularly when they encountered results contradictory to their original hypothesis, and students worked to develop an alternative explanation fitting the results. Lastly, we found that argumentative reasoning rarely emerged during discussions of causal inference, and when students did talk about the criteria for causality, they focused on the strength of association. This is likely due to the curriculum structure, as its main focus is placed on how to appropriately interpret the odds ratio and confidence interval rather than asking students to evaluate and construct causal inferences. Yet, these criteria were explicitly shown in a table in the curriculum and teachers had practiced the causal reasoning in the prior training workshop. We identified some cases that both students and teachers seemed to reject the idea that they could make causal inference because the case control study is not controlled (according to their explanation of characteristics of a study needed to infer cause). In reality, however, epidemiologists often make causal inferences based on evidence for association from observational studies due to several ethical and practical reasons (Koepsell & Weiss, 2003). We could hypothesize that some teachers and students may hold a singular view of scientific method for causal reasoning (Lederman et al., 2002), and it might interfere when they learn other methods, since the randomized and manipulated control experiments are the predominant form of hypothesis testing in todays K-12 science curricula and standards (Windschitl et al., 2007). Although it is hard to generalize the issue within the present study, this epistemological aspect of argumentative reasoning is worth further investigation to the existing literature in our future works.
References
American Association for the Advancement of Science/Project 2061 (1989). Science for all Americans. New York: Oxford University Press. American Association for the Advancement of Science/Project 2061 (1993). Benchmarks for science literacy. New York: Oxford University Press. Andriessen, J. (2006). Arguing to learn. In R.K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (pp. 443-460). Cambridge, UK: Cambridge University Press. Bell, P. (2004). Promoting students argument construction and collaborative debate in the science classroom. In M. C. Linn, E. A. Davis & P. Bell (Eds.), Internet environments for science education (pp. 115-143). Mahwah, NJ: Erlbaum. Berland, L. K., & McNeill, K. L. (2010). A learning progression for scientific argumentation: Understanding student work and designing supportive instructional contexts. Science Education, 94(5), 765-793. Bradford-Hill, A. (1965). The environment and disease: Association or causation? Proceedings of the Royal Society of Medicine, 58: 295300. Brown, A. L., & Campione, J. C. (1994). Guided discovery in a community of learners. In K. McGilly (Ed.), Classroom lessons: Integrating cognitive theory and classroom practice (pp. 229270). Cambridge, MA: MIT Press/Bradford Books. Brown, A., Campione, J., Metz, K. E., & Ash, D. (1997). The development of science learning abilities in children. In A. Burgen & K. Harnquist (Eds.), Growing up with science: Developing early understanding of science. Goteborg, Sweden: Academia Europaea. pp. 7-40. Corsaro, W. A. (2003). Were friends right? Inside kids culture. Washington, DC: Joseph Henry Press. Delta Education (Firm), Lawrence Hall of Science, & University of California Berkeley. (2000). FOSS Environments: Grades 5-6. Nashua, NH: Delta Education. Driver, R., Leach, J., Millar, R. & Scott, P. (1996). Young peoples images of science. Buckinghamshire, England, Open University Press. Edelson, D.C., & Reiser, B.J. (2006). Making authentic practices accessible to learners. In R.K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (pp. 335-354). Cambridge, UK: Cambridge University Press. Eshach, H., & Fried, M. N. (2005). Should science be taught in early childhood? Journal of Science Education and Technology, 14(3), 315-336. Gallas, K. (1995). Talking their way into science: Hearing childrens questions and theories, responding with curricula. New York: Teachers College Press.
ISLS
47
ICLS2012
Volume 2: Symposia
Koepsell, T. D., & Weiss, N. S. (2003). Epidemiologic methods: Studying the occurrence of illness. Oxford: Oxford University Press. Koslowski, B. (1996). Theory and evidence: The development of scientific reasoning. Cambridge, MA: MIT Press. Kuhn, D. (1991). The skills of argument. Cambridge: Cambridge University Press. Kuhn, D. (1993). Connecting scientific and informal reasoning. Merrill-Palmer Quarterly, 39(1), 74-103. Lederman, N. G., Abd-El-Khalick, F., Bell, R. L., & Schwartz, R. S. (2002). Views of Nature of Science questionnaire: Toward valid and meaningful assessment of learners conceptions of nature of science. Journal of Research in Science Teaching, 39(6), 497-521. Lee, T. R., Vye, N., & Bransford, J. D. (2010, March). Social influences on the development of young childrens understandings of science. Paper presented at the annual meeting of NARST, Philadelphia, PA. Lee, T. R., Salgado, N. L., Vye, N., & Bransford, J. D. (2011, April). Argumentation in the pre-kindergarten classroom: Young childrens use of everyday experiences and knowledge in scientific discourse. Paper presented at the annual meeting of NARST, Orlando, FL. Lemke, J. L. (1998). Multiplying meaning: Visual and verbal semiotics in scientific text. In J. R. Martin & R. Veel (Eds.), Reading science: Critical and functional perspectives on discourses of science (pp. 87 113). New York: Routledge. Levinson, R. & Turner, S. (2001). Valuable lessons: Engaging with the social context of science in schools. London: The Wellcome Trust. McNeill, K. L. & Krajcik, J. (2012). Supporting grade 5-8 students in constructing explanations in science: The claim, evidence and reasoning framework for talk and writing. New York, NY: Pearson Allyn & Bacon. Mercer, N., Dawes, R., Wegerif, R., & Sams, C. (2004) Reasoning as a scientist: Ways of helping children to use language to learn science. British Educational Research Journal, 30(3), 367-385. Metz, K. E. (1995). Reassessment of developmental constraints on childrens science instruction. Review of Educational Research, 65(2), 93-127. Munn, M. Brown, M., Martinez, K., Alan, D., Booth, G., Kelly, C., Mouat-Rich, N., Santucci, S., Thomson, L., and Welch, MM 2010. Investigating the Effects of genes and Environment on Smoking Behavior. Available at: https://gsoutreach.gs.washington.edu/files/investigating_smoking_behavior_jan2011.pdf National Research Council. (1996). The National Science Education Standards. Washington, D.C.: National Academy Press. National Research Council. (2000). How people learn: Brain, mind, experience, and school. Washington, D.C.: National Academy Press. National Research Council. (2007). Taking science to school: Learning and teaching science in grades K-8. Washington, D.C.: National Academies Press. National Research Council. (2009). Learning science in informal environments: People, places, and pursuits. Washington, D.C.: National Academies Press. National Research Council. (2011). A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas. Washington, D.C.: The National Academies Press. Newton, P., Driver, R., & Osborne, J. (1999). The place of argumentation in the pedagogy of school science. International Journal of Science Education, 21(5), 553 576. Osborne, J. (2010). Arguing to learn in science: The role of collaborative, critical discourse. Science, 328, 463466. Osborne, J., & Patterson, A. (2011). Scientific argument and explanation: A necessary distinction? Science Education, 95(4), 627-638. Pierut-Le Bonniec, G., & Valette, M. (1991). The development of argumentative discourse. In G. Pierut-Le Bonniec & M. Dolitsky (Eds.), Language bases to discourse basics. Amsterdam: Benjamins. Reznitskaya, A., Anderson, R. C., McNurlen, B., Nguyen-Jahiel, K., Archodidou, A., & Kim, S. (2001). Influence of oral discussion on written argument. Discourse Processes, 32(2-3), 155-175. Toulmin, S. E. (1958). The uses of argument. Cambridge: Cambridge University Press. Vye, N. J., Schwartz, D. L., Bransford, J. D., Barron, B. J., Zech, L. K., & Cognition and Technology Group at Vanderbilt. (1998). SMART environments that support monitoring, reflection, and revision. In D. Hacker, J. Dunlosky & A. Graessar (Eds.), Metacognition in Educational Theory and Practice (pp. 305-346). Mahwah, NJ: Erlbaum. Vygotsky, L. S. (1978) Mind in society: The development of higher psychological processes. (M. Cole, V. JohnSteiner, S. Scribner, & E. Souberman, Eds. and Trans.). Cambridge, MA: Harvard University Press. Windschitl, M., Dvornich, K., Ryken, A. E., Tudor, M., & Koehler, G. (2007). A Comparative Model of Field Investigations: Aligning School Science Inquiry with the Practices of Contemporary Science. School Science and Mathematics, 107, 1, 382-390.
ISLS
48
ICLS2012
Volume 2: Symposia
Engaging Middle School-Aged Students in Classroom Science and Mathematics: Implications for Design and Research
Kimberley Pressick-Kilborn University of Technology, Sydney, PO Box 222, LINDFIELD NSW, AUSTRALIA 2070 Email: kimberley.pressick-kilborn@uts.edu.au Melissa Gresalfi Indiana University, 1900 E. 10th Street, Eigenmann 532, Bloomington, IN 47406 Email: mgresalf@indiana.edu K. Ann Renninger & Jessica E. Bachrach Department of Educational Studies, Swarthmore College, Swarthmore, PA 19081 Email: krennin1@swarthmore.edu, jbachra1@gmail.com Nicole Shechtman (Organizer / Chair), Britte Cheng, Patrik Lundh, & Gucci Trinidad Center for Technology in Learning, SRI International, 333 Ravenswood Ave., Menlo Park, CA 94025 Email: nicole.shechtman@sri.com, britte.cheng@sri.com, patrik.lundh@sri.com, gucci.trinidad@sri.com Richard Walker (Discussant) University of Sydney, Education Building A35, NSW 2006 Email: richard.walker@sydney.edu.au Abstract: A critical educational challenge and core issue for the ICLS 2012 theme of the Future of Learning, is how to keep students, particularly those who attend the most underresourced schools, engaged in science and math as they progress through secondary school. The four papers in this symposium discuss key issues, challenges, and progress around interest and engagement in science and math for students in grades 5 to 7. From a range of theoretical and methodological perspectives, the panelists discuss what interest and engagement are as research constructs; some of the design principles, key features of instructional materials, and contextual factors that trigger and shape interest and engagement; how learner characteristics influence the ways students select or respond to opportunities to engage; the roles that interest and engagement play in learning; and issues of measurement. This 90-minute session will include both formal presentations and moderated discussion among the panelists and audience.
Symposium Overview
A critical educational challenge is how to keep students, particularly those who attend the most under-resourced schools, engaged in science and math as they progress through secondary school. Motivation and engagement are essential to learning at all ages (National Research Council, 2000), yet it is well-documented that students interest and enjoyment in both science and math generally decline as they move from primary to secondary school (e.g., Hidi & Harackiewicz, 2000; Krapp, 2006; Speering & Rennie, 1996; Eccles & Wigfield, 1992; National Center for Education Statistics, 2006). The late primary / middle school years are a particularly important and vulnerable transition point in the school trajectory, as concepts become increasingly difficult and abstract (Nathan & Koellner, 2007; Leinhardt, Zaslavsky, & Stein, 1990). Many students begin to lose interest in science and math, fall behind in achievement (Oakes, 1990), and consolidate motivational attitudes (Eccles, Wigfield, & Schiefele, 1998). Declining interest is of special concern for urban inner-city students who face challenges of poverty, cultural differences, and language. In the US, the National Research Council-commissioned book, Engaging Schools: Fostering High School Students Motivation to Learn (National Research Council & Institute of Medicine, 2004), reports on a federal charge to review and synthesize the broad spectrum of educational research to establish a set of recommendations for the educational community for how to promote and maintain student engagement and genuine improvements in achievement throughout the high school years. This symposium focuses on the Committees first recommendations, that courses and instructional methods be redesigned in ways that will increase adolescent engagement and learning (p. 214). This charge is highly relevant to the international ICLS 2012 theme of the Future of Learning, in particular to the key prototypic research questions:
ISLS
49
ICLS2012
Volume 2: Symposia
How can the energies and motivations that accompany a learners interests be matched with learning resources to enable productive learning pathways? How can teachers productively create teaching and learning environments that support the needs of learners of diverse linguistic, cultural and economic backgrounds?
These questions indicate the need for deep understanding of motivation as a research construct and its implications for design: What are key aspects of instructional materials and context that trigger and shape interest and engagement? How do learner characteristics influence the ways students select or respond to opportunities to engage? The four papers in this symposium provide complementary approaches to addressing these issues in the teaching of science and mathematics, especially for students in the late primary / middle grade years. The two papers by Pressick-Kilborn and Gresalfi describe design principles and design research undertaken with the aim of developing and understanding specific instructional affordances for motivation in science and math classrooms. Renninger & Bachrach report on findings addressing the relation among triggers for interest and learner characteristics in an out-of-school science program for at-risk youth, and Shechtman et al. consider issues of operationalization and measurement in a study of engagement as a multidimensional construct in the urban mathematics classroom. The papers represent a range of theoretical perspectives, rooted in both sociocultural and psychological approaches; a range of methodological approaches, including design research, qualitative case studies, and psychometrics; both formal and informal instructional contexts; technology-based curriculum and curriculum less reliant on technology; and settings in both the US and Australia. Furthermore, within these issues, the papers speak to a number of themes important to the ICLS community: collaborative learning, metacognition and self-regulation, developing flexible understandings that can be used beyond formal schooling and throughout life, and the capabilities of interactive technology and computational models for supporting STEM learning. The 90-minute session format will be as follows. The first hour will be formal presentation; each paper will be presented for 15 minutes. The last half hour will be dedicated to a discussion among the panel and a moderated discussion with the audience. The discussant is Richard Walker, whose research interests center on ways of enhancing the learning, motivation, and academic achievement of students at all levels of education. His work investigates from a sociocultural perspective learning in electronic learning environments designed to support collaborative and cooperative interactions among students, the use of textbooks and other learning resources, after-school homework support, and identity formation. He will pose questions to the panelists about the design implications from their findings regarding the prototypic questions, for example, (a) the match between middle school students interests and learning resources; and (b) how teaching and learning environments might productively address the needs of learners of diverse linguistic, cultural and economic backgrounds.
Paper 1: Instructional Design Principles for Engaging Students in Learning Science and Fostering Interest Development
Kimberley Pressick-Kilborn
Problem and Theoretical Rationale

A number of studies have reported that while students have a great deal of enthusiasm for science in the primary years, interest and enjoyment generally decline during the early years of learning science in secondary schools (Hidi & Harackiewicz, 2000; Krapp, 2006; Speering & Rennie, 1996). This finding leads to the question of what it is that makes primary science engaging and interesting. More specifically, what features of primary contexts for science learning contribute to creating, maintaining and developing student engagement and interest? The focus of this paper is the articulation and illustration of guiding instructional principles for supporting the development of students interest in learning science. The project was framed from a Vygotskian sociocultural perspective, which emphasises the social origins of learning and motivation (Walker, PressickKilborn, Sainsbury & Arnold, 2004; Walker, 2010). Strengths of adopting a sociocultural perspective lie in the emphasis on studying interest as it develops in authentic learning contexts over time, enabling the relationship between situational and individual aspects of interest to be considered as inclusively separate (Valsiner, 1998) so that a study of the total activity is maintained.
Method
The principles were developed as part of a qualitative, classroom-based project that drew on a design-based research methodology (Brown, 1992; Guthrie & Alao, 1997), aimed at informing and extending both theory and practice. Seven guiding instructional principles were established at the outset of the research, informed by the literature relating firstly to student motivation and interest (for example, Mitchell, 1993; Paris & Turner, 1994;
ISLS
50
ICLS2012
Volume 2: Symposia
Guthrie & Alao, 1997; Bergin, 1999; Ainley, 2001), and secondly to classrooms as learning communities (Bereiter & Scardamalia, 1993; Brown, 1997; Collins, 1998). Two science learning units were subsequently designed in collaboration with a classroom teacher, guided by these principles, and team-taught in a grade 5 classroom in Sydney, Australia. Detailed researcher field notes were recorded for observations of class activities, student engagement and interest, as students participated in weekly science lessons over 6 months. Students shared self-reports of their interest through charting retrospective interest trajectory graphs and semistructured interviews were conducted with six focus students at different time points during the study.
Results
The key features of the learning units that contributed to creating a community of learners, designed to create and support interest and engagement in science and technology, were identified and considered in relation to (1) physical features of the classroom context (classroom layout, access to materials), (2) features to support social interaction within and beyond the classroom (grouping structures and strategies, access to experts in the wider community) and (3) pedagogical features (opportunities for hands-on involvement, varying degrees of choice, reflection on learning, interactive noticeboard). Individual student interest trajectory ratings were collated to create a class mean trajectory of interest development, which revealed patterns of interest in the classroom learning community. Interest peaked during engagement in hands-on investigations and design and make activities with multiple pathways for task completion and possibilities for negotiation. Interest peaks also were evident for guest speaker presentations and an excursion. Troughs were observed for class discussions, when there was relatively more passive engagement of students who were listening to peers, more limited opportunities for active participation and greater teacher control.
Discussion and Implications

The seven instructional principles articulated at the outset of the research related to the organisation of science learning units in three distinct phases, and emphasised the importance of active student engagement in hands-on tasks with possibilities for choice. The principles also supported the inclusion of both individual and collaborative activities, as well as opportunities for engagement with topics and issues of concern to the broader scientific community, and contact with domain experts and communities of practice beyond the classroom. In addition, the principles guided the teacher to share emotions during the learning process and to lead the learning as a more expert learner, whilst also fostering student reflection on the learning process. My researcher perspective as a participant observer in that classroom led to the refinement, elaboration and extension of the initial guiding design principles. In particular, three additional principles were developed, focusing on play, imagination and the physical layout of the classroom. The focus of this paper on guiding instructional principles has clear implications for classroom practice and more specifically, science curriculum design. The principles are illustrated by drawing on data gathered in the classroom-based research project, with the focus on elaborating the relationships amongst interest, engagement and learning.
Paper 2: Designing to Support Critical Engagement with Content

Melissa Gresalfi

In this paper I describe a set of design experiments that were undertaken with the goal of supporting a particular form of engagement that we call critical engagement, which involves making intentional choices about which procedures to leverage in order to support particular claims. The past twenty years have brought technological innovations that have changed the world as we know it. Continuous access to online information has supported a transformation of the relationship between individuals and knowledge; with information so readily accessible, people have been repositioned to move beyond acquisition of facts to consider when to access those facts, interrogate them, and integrate them into activity. For these reasons, knowledgeable participation in mathematics must involve more than proficiency using key procedures. Instead, knowledgeable participation must involve engaging in acts of decision-making, determining which procedures enable the resolution of defensible solutions, how, and why. It is clear that supporting students to engage with mathematics in this way is not simply a matter of learning more mathematics content. Instead, it is crucial that students have opportunities to learn new content in ways that are consistent with how we actually want them to use that information (Boaler, 2000; Bransford & Schwartz, 1999; Cobb, Stephan, McClain, & Gravemeijer, 2001; Engle, 2006; Greeno, 1991; Lave, 1997). When creating new curricula, it is therefore crucial that designers attend not just to the content goals of the unit (the mathematical ideas you want students to learn and understand), but also participatory goals of the unit (how you hope students will engage). This does not mean that becoming a successful engineer, for example, requires
ISLS
51
ICLS2012
Volume 2: Symposia
learning all mathematics at the elbow of a practicing engineer. Instead, this suggests that the kinds of practices that a practitioner is expected to leverage (such as experimenting, reviewing, collaborating, justifying, testing, and inventing) are the practices engaged in during the learning experiences.
Data Sources and Methods

This paper presents two rounds of a design cycle of a statistics unit, created in the context of an online, interactive videogame called Quest Atlantis. The design experiment was conducted with the same teacher over two years, focusing on two different 5th grade math classes. For both studies, the entirety of the two-week unit was captured with videotapes of whole class interaction, small group discussion, and individual interviews. In addition, students submitted work was collected and coded for accuracy and for evidence of critical engagement.
Results and Significance

Examining the design history of the unit, this paper demonstrates how iterative refinements of the unit supported increasing critical engagement with the content, and specifically, the importance of fostering both intentionality and experiences of consequentiality in the designs. Finally, the paper considers the implications of the lessons learned for this targeted intervention for our understanding of how elements of design support students mathematical reasoning more generally.
Paper 3: Design Implications from Study of Potential Triggers in an Out-ofSchool Biology Workshop
K. Ann Renninger & Jessica E. Bachrach

For educators, an essential question is how to pique, or trigger, the interest of learners. However, little is understood about how triggering really works outside of the experimental setting. A working assumption among motivation researchers is that potential triggers for interest result in engagement even though research findings sometimes suggest otherwise (e.g., Durik & Harackiewicz, 2007). Moreover, the assumption that potential triggers result in engagement presumes that all learners perceive and respond to potential triggers in the same waya point that has also been challenged (e.g., Bartlett, 1932; E. J. Gibson, 1982; J. J. Gibson, 1950, 1977, 1979; Herrenkohl, Tasker, & B. White, 2011; Norman, 1999). Better understanding of the triggering process is needed to begin to explain when and why potential triggers for interest are effective. In order to consider the relation between potential triggers for interest and learner engagement, as well as the relation between engagement and learner characteristics, findings from two studies of the activity of eight inner-city middle school participants in an out-of-school biology workshop will be described. Study 1 addresses the identification and generalizability of potential triggers for interest across activities. Study 2 is a post hoc analysis that explores the relationship between triggers for interest and learner characteristics.
Participants
Eight youths (3 males, 5 females) participated in the biology workshop. They were African-American or racially mixed and ranged in age from 9-12 years; mean age was 10.5 years. The youth were participants in a rigorous choral training program for inner-city youth and were enrolled in the workshop as a summer enrichment activity. They had had no previous experience with formal instruction in science and they were identified as being in the earliest phase of interest development.
Data Sources and Methods

Informed by grounded theory (Charmaz, 2000; Strauss & Corbin, 1998), potential triggers for interest were tracked using participant observation notes collected as part of a larger project focused on participant learning and motivation in an out-of-school, inquiry-oriented biology workshop for inner-city youth. Multiple additional sources of data (workshop artifacts, participant interviews, caretaker interviews, and educator reports) were available to the researchers as reference material for each of the two studies. Analyses for both studies were undertaken at the level of the activity and/or the group. Each of the studies to be reported targeted participants behaviors during ongoing workshop activity (Stake, 2005); other studies of the workshop have focused on individual participants. Categorical directed content analysis (Hickey & Kipping, 1996; Hsieh & Shannon, 2005; Potter & Levine-Donnerstein, 1999) was used for the first part of both Study 1 and Study 2. This approach involves using existing research and theory to inform data reduction, in addition to allowing identification of emergent categories for coding. The categories identified in Part I of each study were then used to code data in Part II of each study.
ISLS
52
ICLS2012
Volume 2: Symposia
Potential Triggers for interest identified based on a review of the literature and tracked for purposes of analysis included affect, autonomy, challenge, character identification, computers/technology, group work, hands-on activity, instructional conversation, novelty, ownership, and personal relevance.
Findings and Significance

Taken together, findings from both studies confirm that learners do not perceive and respond identically to potential triggers for interest and provide insight into the circumstances affecting when potential triggers will and will not work for learners who are new to science. They indicate that the triggering process is nuanced and is informed both by the features of the activity and the readiness of the learner to pick up on potential triggers. Findings from Study 1 indicate that triggers examined in experimental studies could be identified in the naturally occurring workshop setting. Furthermore, the same triggers could be identified in multiple activities, although they may or may not work for all learners at all times. Moreover, some activities included multiple triggers. As in studies from work on situational interest, the present findings indicate that when triggers work, attention is captured. In this sense, triggering enabled engagement (Fredricks, et al., 2004) and was often dependent on other people in the environment, the design of activities, or, as reported in Study 1, serendipity such as when a participant walked into a spiders web. In contrast to discussions suggesting that situational interest might be expected to work the same way for all learners (e.g., Hidi & Berndorff, 2001), the present findings suggest that the process of triggering is complex and idiosyncratic, and that knowledge of the learning environment and the learner may be essential for predicting whether potential triggers will work. Findings from Study 2, furthermore, underscore that learner characteristics are likely to affect when and how potential triggers work. While this notion is not new, Study 2 represents an initial attempt to systematically investigate this relation and to do so in a naturally occurring context with all of its complexity. A number of previously unreported relations among potential triggers and learner characteristics surfaced. In particular: (a) Some learner characteristics were found to be more relevant to some triggers than others (e.g., sociability appears most likely to influence whether group work works as a trigger for interest); (b) The success of some potential triggers was found to always be tied to particular learner characteristics (e.g., personal relevance was always affected by awareness); (c) Furthermore, potential triggers were found to be affected by different numbers of learner characteristics (e.g., affect, challenge, and group work were affected by all of the identified learner characteristics).
Paper 4: Unpacking the Black Box of Engagement: Cognitive, Behavioral, and Affective Engagement in Learning Mathematics
Nicole Shechtman, Britte Cheng, Patrik Lundh, & Gucci Trinidad

Keeping students engaged in mathematics as they progress through secondary school is a critical national educational challenge for practitioners and researchers. It is also called out in many recent national policy documents (e.g., Engaging Schools [National Research Council& Institutes of Medicine, 2004], National Mathematics Advisory Panel [NMAP, 2008], Adding It Up [Kilpatrick et al., 2001]). However, engagement in mathematics learning is still somewhat of a black boxthe field is grappling with a gap in our understanding of what engagement in mathematics truly is, how to measure it, what its relationships to learning are, and how we could leverage what we understand to increase engagement for all students. The goal of the Math Engagement Project (MEP) is to examine ways to unpack this black box. MEP is an exploratory research study of middle school mathematics learning and engagement to build theory and instrumentation around engagement in mathematics learningwhich we treat as a complex, multidimensional, context-specific construct. We will present findings with respect to a theoretical framework we are developing to capture the dimensions of engagement in math learning as it is embedded in the classroom as an activity system. In a review of the broader literature on school engagement, Fredricks et al. (2004) describe several dimensions that are implicit within engagement. We apply this specifically to the domain of mathematics learning, teasing apart three components: (a) behavioral engagement (e.g., being on-task), (b) affective engagement (e.g., enjoying or being interested), and (c) cognitive engagement (e.g., doing the cognitive work). Within the cognitive space, there are also a variety of ways of engaging generally (e.g., memorizing facts vs. solving novel problems) and specifically with mathematical content (e.g., critical [see Paper 2] vs. procedural vs. conceptual). We also distinguish engagement at two different time horizons: (a) dispositional engagement over a long period of time (e.g., engagement in secondary math), and (b) proximal engagement in the moments of instruction (e.g., engagement in todays math lesson). We use this framework to unpack the elements of engagement and learning of middle school students (ages 12-13) in the classrooms of two teachers teaching a 4-week unit on rate and proportionality. In prior large-
ISLS
53
ICLS2012
Volume 2: Symposia
scale RCTs, this unit, which integrates curriculum and the interactive, dynamic software SimCalc MathWorlds, was shown to promote gains in conceptual understanding for a wide variety of students (see Roschelle et al., 2010). However, while learning gains overall were robust, there was also substantial variation between studentsindicating that not all students engage with the unit to the same extent or in the same ways. In the current study, we use multiple methods to examine individual students behavioral, affective, and cognitive engagement during several lessons in the unit. We seek to characterize the range of engagement and examine how it may be shape the mathematics that students learn. Furthermore, we examine how the ways individual students engage may be shaped by (1) the dispositions students bring to the unit, and (2) the affordances for engagement provided by the curriculum and the teachers.
Methods
For the 302 seventh-grade students across the two teachers 11 classrooms, sources of data include: (1) pre-unit survey of attitudes and dispositions toward learning and mathematics (e.g., achievement orientation, interest in mathematics, mathematical confidence and anxiety, school engagement); (2) unit pretest and posttest to measure baseline knowledge and learning gains for the units mathematical content; (3) survey administered multiple times throughout the unit to capture students self-reported behavioral, affective, and cognitive engagement in the lessons; (4) teachers report of students behavioral, affective, and cognitive engagement for a subset of 98 students; (5) analyses of written mathematical work for evidence of cognitive/mathematical engagement; and (6) classroom observations focused on the mathematical discourse. Additionally, 11 case study students (7 girls, 4 boys) were interviewed before, during, and after the unit to investigate their engagement and learning in greater depth. This paper will focus on the behavioral, affective, and cognitive engagement of the students over the course of the unit, using the 11 case study students as illustrative examples. We draw from these multiple sources of data to characterize the different ways students engage, how their engagement relates to their learning, and the dispositional and contextual factors that shape their engagement.
Preliminary Results and Discussion

The context of the study was a suburban area in Northern California. The school has less than 10% of students qualifying for the free or reduced lunch program, and the majority of students are Caucasian or Asian/Pacific Islander (Note that we are in the process of recruiting for the next study which will focus on students in highpoverty, high-minority schools). The 11 classrooms across the two teachers represented the remedial and gradelevel pre-algebra tracks (more advanced students had a different teacher). Not surprising in this context, student attendance was high and behavioral disruptions were relatively low. Both teachers taught with high fidelity to the workbook. On average, students reported a relatively positive classroom climate, in terms of both teacher and student support. The mean student pretest and gains were slightly above the average of the RCT treatment group, which represented a wide variety of classrooms across the state of Texas. Within this context, the 302 students represent a range of prior achievement levels (as measured on the pretest) and prior attitudes and dispositions toward learning and mathematics. While most of the students tended to be behaviorally engaged (i.e., on task) much of the time, preliminary findings suggest marked differences in the ways students engage affectively and cognitively. Affectively, while some students engagement is marked by interest and enthusiasm, other students is marked by boredom, anxiety, or indifference. Cognitively, while some students engage in building their own knowledge as the curriculum was intended through inquiry with the software and asking questions, other students appear quite satisfied to passively absorb the lessons, while others are strongly disaffected and simply trying to get by. Preliminary findings also suggest that these ways of engaging may be mediated by dispositions students had reported at baseline (e.g., how they see themselves as learners in the classroom, achievement orientation as mastery vs. performance, mathematical self-efficacy), the ways in which the teachers scaffold the lesson, and opportunities to interact productively or unproductively with peers. For example, the more passive cognitive engagement was observed in a highly math-anxious student in the classroom of the teacher who scaffolded the lesson in a way that emphasized correct answers. Findings also suggest that these ways of engaging have consequences for the depth of conceptual understanding that students attain. These and other themes will be discussed with respect to the broader literatures on engagement and motivation. We will explore implications for the measurement of engagement and how to find leverage points for developing ways to support engagement in learning mathematics.
References
Ainley, M. (2001). Interest in learning and classroom interactions. In D. Clarke (Ed.), Perspectives on practices and meaning in mathematics and science classrooms (pp. 105-130). Dordrecht: Kluwer Academic Press. Bartlett, F. C. (1932). Remembering: A study in experimental and social psychology. London: Cambridge University Press.
ISLS
54
ICLS2012
Volume 2: Symposia
Bereiter, C., & Scardamalia, M. (1993). Surpassing ourselves: An inquiry into the nature and implications of expertise. Chicago: Open Court. Bergin, D.A. (1999). Influences on classroom interest. Educational Psychologist, 34(2), 87-98. Boaler, J. (2000). Exploring situated insights into research and learning. Journal for Research in Mathematics Education, 31, 113-119. Bransford, J. D., & Schwartz, D. L. (1999). Rethinking transfer: a simple proposal with multiple implications. Review of Research in Education, 24, 61-100. Brown, A.L. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. Journal of the Learning Sciences, 2, 141-178. Brown, A.L. (1997).Transforming schools into communities of thinking and learning about serious matters. American Psychologist, 52(4), 399-413. Charmaz, K. (2000) Grounded theory: Objectivist and constructivist methods. In N. Denzin & Y. Lincoln (Eds.), Handbook of qualitative research (2nd ed., pp. 509-536). London: Sage Publications. Cobb, P., Stephan, M., McClain, K., & Gravemeijer, K. (2001). Participating in classroom mathematical practices. Journal of the Learning Sciences, 10, 113-164. Collins, A. (1998). Learning communities: A commentary on papers by Brown, Ellery and Campione, and by Riel. In J.G. Greeno & S.V. Goldman (Eds.), Thinking practices in mathematics and science learning. Mahwah, NJ: Lawrence Erlbaum Associates. Durik, A., & Harackiewicz, J. M. (2007). Different strokes for different folks: How individual interest moderates the effects of situational factors on task interest. Journal of Educational Psychology, 99(3), 597-610. Eccles, J. S., & Wigfield, A. (1992). The development of achievement-task values: A theoretical analysis. Developmental Review, 12, 265-310. Eccles, J. S., Wigfield, A., & Schiefele, U. (1998). Motivation to succeed. In N. Eisenberg (Ed.), Social, emotional, and personality development handbook of child psychology (Vol. 3, pp. 1017-1096). New York: Wiley. Engle, R. A. (2006). Framing interactions to foster generative learning: A situative explanation of transfer in a community of learners classroom. Journal of the Learning Sciences, 14(4), 451-498. Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74(1), 59-109. Gibson, E. J. (1982). The concept of affordances in development: The renascence of functionalism. In W. A. Collins (Ed.), The concept of development: The Minnesota symposia on child psychology (Vol. 15, pp. 55-82). Lawrence Erlbaum Associates, Inc. Gibson, J. J. (1950). The perception of the visual world. Boston: Houghton Mifflin. Gibson, J. J. (1977). The theory of affordances. In R. Shaw & J. Bransford (Eds.), Perceiving, acting, and knowing: Toward an ecological perspective (pp. 67-82). Hillsdale, NJ: Lawrence Erlbaum Associates, Publishers. Gibson, J. J. (1979). The ecological approach to visual perception. Boston: Houghton Mifflin Company. Greeno, J. G. (1991). Number sense as situated knowing in a conceptual domain. Journal for Research in Mathematics Education, 22, 170-218. Guthrie, J.T., & Alao, S. (1997). Designing contexts to increase motivations for reading. Educational Psychologist, 32(2), 95-105. Herrenkohl, L. R., Tasker, T., & White, B. (2011). Pedagogical practices to support classroom cultures of scientific inquiry. Cognition and Instruction, 29(1), 1-44. Hickey, G., & Kipping, C. (1996). Issues in research: A multi-stage approach to the coding of data from openended questions. Nurse Researcher, 4, 81-91. Hidi, S. & Harackiewicz, J.M. (2000). Motivating the academically unmotivated: A critical issue for the 21 st century. Review of Educational Research, 70(2), 151-179. Hidi, S., & Berndorff, D. (2001). Interest, reading, and learning: Theoretical and practical considerations. Educational Psychology Review, 13(3), 191209. Hsieh, H.-F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15(9), 1277-1288. Kilpatrick, J., Swafford, J., & Findell, B. (2001). Adding it up: Helping children learn mathematics. Washington, DC: National Academy Press. Krapp, A. (2006). Structural and dynamic aspects of interest development: Theoretical perspectives from an ontogenetic perspective. Learning and Instruction, 12(4), 383-409. Lave, J. (1997). The culture of acquisition and the practice of understanding. In D. Kirshner & J. A. Whitson (Eds.), Situated cognition: Social, semiotic, and psychological perspectives (pp. 17-35). Mahwah, NJ: Erlbaum. Leinhardt, G., Zaslavsky, O., & Stein, M. (1990). Functions, graphs, and graphing: Tasks, learning, and
ISLS
55
ICLS2012
Volume 2: Symposia
teaching. Review of Educational Research, 60, 1-64. Mitchell, M. (1993). Situational interest: Its multifaceted structure in the secondary school mathematics classroom. Journal of Educational Psychology, 85(3), 424-436. Nathan, M. J., & Koellner, K. (2007). A framework for understanding and cultivating the transition from arithmetic to algebraic reasoning. Mathematical Thinking and Learning, 9(3), 179-192. National Center for Education Statistics. (2006). The nation's report card: Mathematics 2005 (No. NCES 2001 571). Washington, DC: U.S. Department of Education. National Mathematics Advisory Panel. (2008). Foundations for success: The final report of the National Mathematics Advisory Panel. Washington DC: United States Department of Education. National Research Council. (2000). How people learn: Brain, mind, experience, and school, Expanded edition. In J. D. Bransford, A. L. Brown, & R. R. Cocking (Ed.), Committee on Developments in the Science of Learning, Committee on Learning Research and Educational Practice, Commission on Behavioral and Social Sciences and Education. Washington D.C.: National Academy Press. National Research Council & Institute of Medicine. (2004). Engaging schools:Fostering high school students motivation to learn. Washington, DC: The National Academies Press. Norman, D. A. (1999). Affordance, conventions, and design. Interactions, 6(3), 38-42. Oakes, J. (1990). Opportunities, achievement, and choice: Women and minority students in science and mathematics. Review of Educational Research, 16, 153-222. Paris, S.G., & Turner, J.C. (1994). Situated motivation. In P.R. Pintrich, D.R. Brown & C.E. Weinstein (Eds.), Student motivation, cognition and learning (pp. 213-237). Hillsdale, NJ: Lawrence Erlbaum. Potter, J., & Levine-Donnerstein, D. (1999). Rethinking Validity and Reliability in Content Analysis. Journal of Applied Communication Research, 27, 258-284. Roschelle, J., Shechtman, N., Tatar, D., Hegedus, S., Hopkins, B., Empson, S., Knudsen, J. & Gallagher, L. (2010). Integration of technology, curriculum, and professional development for advancing middle school mathematics: Three large-scale studies. American Educational Research Journal, 47(4), 833878. Speering, W., & Rennie, L. (1996). Students perceptions about science: The impact of transition from primary to secondary school, Research in Science Education, 26(3), 283-298. Stake, R. E. (2005). Qualitative case studies. In N. K. Denzin & Y. S. Lincoln (Eds.), The Sage handbook of qualitative research (3rd ed., pp. 443-466). Thousand Oaks, CA: Sage Publications. Strauss, A., & Corbin, J. (1998). Basics of qualitative research techniques and procedures for developing grounded theory (2nd ed.). London: Sage Publications. Valsiner, J. (1998). Dualisms displaced: From crusades to analytic distinctions. Human Development, 41, 350354. Walker, R.A. (2010). Sociocultural issues in motivation. In P. Peterson, E. Baker & B. McGaw (Eds.), International Encyclopedia of Education (3rd ed., Vol. 6, pp. 712-717). Oxford: Elsevier. Walker, R.A., Pressick-Kilborn, K., Arnold, L. & Sainsbury, E.J. (2004). Investigating motivation in context: Multiple dimensions, domains and assessments. European Psychologist, 9, 245-256.
ISLS
56
ICLS2012
Volume 2: Symposia
Robot Facilitation as Dynamic Support for Collaborative Learning

Naomi Miyake, The University of Tokyo, Hongo 7-3-1, Bunkyo-ku, Tokyo 113-0033, Japan, nmiyake@p.u-tokyo.ac.jp Sandra Y. Okita, Teachers College, Columbia University, 525 West 120th Street, New York, NY 10027, okita@tc.columbia.edu Abstract: Automated as well as remotely controlled robots have high potential to expand our research and practice in collaborative learning environments. The area of computer supported collaborative learning (CSCL) is one in which advanced technologies have been used effectively to automatically monitor and dynamically structure group discussions for learning. Traditional desktop environments for collaborative learning have included interactions between groups and computer agents with conversational abilities. Robots can take the collaboration out of the computer and into the three dimensional, real world environments of students. This symposium raises the important question of how advanced technologies, such as robotics, may enable us to take the same insights from the area of scripted collaboration and use them to support groups of learners in these environments. Other topic areas include design choices in robotic features that may influence scripted interactions, social behavior, meaningful engagement, and conditions that dynamically support situational factors and generate collaborative learning among students.
Introduction
As collaborative learning has proven to be effective in helping students learn (Chi et. al, 2001; Graesser, Person, & Magliano, 1995; Palincsar & Brown, 1984), it is only natural to ask how sophisticated artifacts can be used as facilitating tools to dynamically support collaborative learning. Conversational agents, avatars, and humanoid robots are highly directable, allowing researchers to capitalize on a dynamic nature that enables humans to share knowledge and ideas. Similar to the use of conversational agent technologies in desktop CSCL environments, having a robot participate in collaborative learning allows researchers to control some part of the collaborative activities, which enables us to better understand the basic mechanisms of collaborative learning. This work has built on theories and findings from a wide range of areas of the Learning Sciences as well as the technical fields of Machine Learning, Language Technologies, and Human-Computer Interaction. Some of the most cutting edge work in this area has been the use of robots to support collaborative learning. The symposium takes a close look at three areas that may contribute to the unique value of robots in assisting collaborative learning and assessment. One area that is critical in the design of meaningful engagement is scripted collaboration. In the past decade, the area of scripted collaboration has produced tremendous gains in terms of insight into how to elevate the occurrence of valuable discussion-based learning behaviors from students in online and offline settings (Dillenbourg, 2002; Kollar, Fischer, & Slotta, 2005). This insight has also led to advanced technologies, such as robotics, that may enable us to take these findings and use them to support groups of learners in the three-dimensional world. More recently, advances in the area of automatic collaborative learning process analysis have enabled the first attempts at dynamic support for collaborative learning, which builds on the insights learned from static forms of scripted collaboration (Ros, Wang, Cui, Arguello, Stegmann, Weinberger, & Fischer, 2008). The second area takes from the theories and findings of Human-Computer Interaction (HCI) and Human-Robot Interaction (HRI). This area examines how specific features in robots may influence; how interactive scenarios are delivered to humans (e.g., with an angry voice, with gesture), how behavioral scripts based on familiar narratives and routines can be generated from humans (e.g., common children play routines like tea party, play doctor), and how robotic features may influence the quality of interaction (e.g., testing the robot or meaningful engagement). In examining the situational factors that promote collaborative learning, much attention has been given in making machines more responsive and sophisticated. The advancement in sensors and audio-visual tools has helped robots detect human behavior (e.g., facial expressions), while the sophisticated automation and expressive tools have helped robots provide dynamic and expressive responses toward humans (e.g., gesture, tone of voice, attention). Identifying the delicate balance between detection and response has improved the overall quality of human-robot interaction. However, little research has been done to examine how these features when combined with specific scripts and scenarios can assist collaborative learning and meaningful engagement. The final area of interest in this symposium is taking the insights from the first two areas (i.e., scripted collaboration and human-robot interaction) and exploring conditions that dynamically support and generate the optimal level of collaboration among groups of students. Robots and other agent technologies can participate in
ISLS 57
ICLS2012
Volume 2: Symposia
and support reflection in active ways, for example by providing learners with precise replay of past learning experiences upon request, which has not been as feasible with human support. In addition to these capabilities, however, the use of robots goes beyond that, allowing us to implement basic design principles of group activities in more tangible ways. They offer the unique opportunity to support groups of students in real world activities. Automated as well as remotely controlled robots have high potential to expand our research and practice in collaborative learning. As a learning collaborator, robots can present their prepared explanations to their human partners, comment on others ideas, and exchange questions and answers, at least to the extent those exchanges could be either prepared prior to the classes or provided by human operators. As a research partner, robots can run controlled experiments during a collaborative learning process in classroom-like settings. One example of such control is to have a robot in each discussion group and to have it deliver the same information to different groups, so that we can explore the effects of a particular discourse in collaboration. Reflectively it is illuminating that the various effects of this kind of dynamic and interactive support have not been carefully researched in the past. Collaborative learning has been practiced widely in the community of learning scientists, with relative to profound change in the quality of learning achievements (Bransford et al., 1999; Sawyer, 2006). In order to better design such practices within a wider variety of cultural settings, we need stronger scientific evidence for such success, as well as a more precise explanation of the mechanisms of how and why it works (cf., Miyake, 2008). One of our motivational factors for proceeding in this direction is the need for us to better equip ourselves with stronger sets of evidence of the power of collaborative, learner-centered orientation in education, to work effectively with students in the collaborative setting.
Target Audience
Our audience would include a wide range of learning science researchers who seek to evolve research on collaboration. We particularly aim this symposium at practice-oriented, classroom-based research, from which we could learn how to build sustainable, effective communities, to foster impacts on real world learning. We are also keen to invite engineers and robotics researchers seeking focused research fields to foresee what is needed for the creation of symbiotic robots. Collaborating with them would open up a stronger promise for us to reach out to real world classrooms.
Symposium Presentations
The symposium presentations will focus on three topic areas: 1) dynamic collaborative learning support techniques with scripted collaboration, 2) influence of robotic features on interactive scenarios and facilitating engagement, and 3) exploring conditions that dynamically support situational factors that generate the optimal level of collaboration among groups of students. The research in the first presentation covers interactions that are text-based in a two-dimensional chat environment with group interactions, but the architecture and dynamic collaborative learning support techniques are highly applicable to physical robots environment. The research in the second presentation involves a close examination of physical humanoid robots and humans in a one-to-one (robot-to-human) interaction (not in groups), but the findings has helped identify important perceptual cues and responses that need to be recognized and handled respectively that are applicable to collaborative groups settings. The research in the final presentation involves multiple physical humanoid robots spread across multiple student groups and addresses many of the common insights and issues from the first two presentations. To help highlight the three interest areas, we discuss them separately, but all three are at play in each presentation. The symposium consists of two parts. The first part involves presentations of recent work that shows promising results, strengths of, and challenges in robot facilitation. The second part of the session leads into a discussion organized by discussants that engage presenters and audiences to explore practical use of robots and agents/avatars in collaborative learning (e.g., how different situational factors require different dynamic support, and how robots may facilitate in each setting).
Presentation 1 Basilica 3D: Towards Architecture for Operating Robotic Accountable Talk Facilitation Agents
Carolyn Penstein Ros Language Technologies Institute and Human-Computer Interaction Institute School of Computer Science, Carnegie Mellon University cprose@cs.cmu.edu
ISLS
58
ICLS2012
Volume 2: Symposia
While theories of discussion-based learning both within the computer supported collaborative learning (CSCL) community as well as in the classroom discourse community are many, there is consensus about what types of group interactions are desirable. For example, the value placed on taking ownership of reasoning, articulation of reasoning, evaluation of reasoning, and building on the reasoning of self and others are almost ubiquitous. Yet it is widely acknowledged that groups do not operate at an ideal level without support. Thus, researchers in the area of scripted collaboration have worked to develop design principles to guide development of support that elicits the kind of group behaviors that are valued within the CSCL community. However, while much of that work was developed and evaluated with static forms of support, there has been a growing interest in making that support dynamic, both in the sense of triggering based on real time analysis of the ongoing collaboration (Ros et al., 2008; Mu et al., 2011), as well as in the sense of support that is itself interactive (Kumar et al., 2006). Conversational agents have a long history of successful support for individual learning with technology (Ros et al., 2001; Ros & Van Lehn, 2005; Kumar et al., 2006). A series of results offer hope that they can be used productively to offer support for collaborative learning, especially in chat environments (Kumar & Ros, 2011; Kumar et al., 2010; Kumar et al., 2011; Chaudhuri et al., 2009; Ai et al., 2010; Howely, Mayfield, & Ros, 2011). The Basilica architecture has been developed for the purpose of enabling efficient development of intelligent agents with rich conversational behaviors to participate with groups of students in online chat environments and to play a facilitative role (Kumar & Ros, 2011). Early work in this area simply imported the same conversational agents used in individual learning with technology into group learning settings. However, more recent work has directly employed techniques from the classroom discourse community to develop conversational agents that can employ to some limited extent the practices of classroom facilitation techniques. In early work on dynamic support for collaborative learning, agent behaviors were triggered through analysis of text-based interactions in the chat environment. Recent work on monitoring group interactions through speech (Gweon et al., 2011) gives hope that this dynamic collaborative learning support technique can be adapted from the two-dimensional text-based chat environment to the three-dimensional, face to face collaboration setting where the facilitation may be conducted by robots controlled through this same architecture. This talk takes a visionary look at how this Basilica work may be extended for this purpose.
Presentation 2: Multimodal Approach to Facilitating Affective Human-Robot Interactions

Sandra Y. Okita Communication, Computing, and Technology in Education Teachers College, Columbia University okita@tc.columbia.edu The advancement in sensors and audio-visual tools has helped robots detect human behavior (e.g., facial expression), while sophisticated automation and expressive tools have helped robots provide dynamic and expressive responses toward humans (e.g., gesture, tone of voice, attention). Identifying the delicate balance between detection and response has improved the overall quality of human-robot interaction. However, little research has been done to examine how these features when combined with specific interactive scenarios can facilitate meaningful and affective engagement. The research shares some preliminary findings on how specific features in robots may influence how interactive scenarios are delivered to humans (e.g., with an angry voice, exaggerated gesture), how familiar scenarios and behavioral scripts can be generated from humans (e.g., hide-and-seek), and how robotic features may influence the quality of interaction (e.g., child testing the robot but not communicating with the robot), affective engagement (e.g., robotic voice scares child) and learning (e.g., peer-like robot vs. teacher-like robot). Having physical humanoid robots in the collaborative learning setting provides researchers with a range of design choices (e.g., tone of voice, attention, gesture, detection and response timing) on how interactive scenarios can be delivered. For example, is it the content of the interaction or the timing of feedback that is more important? If the interactive scenario is accompanied with robotic gestures (Ng-Thow-Hing, Luo, & Okita, 2010; Okita, Ng-Thow-Hing, & Sarvadevabhatla, 2009), will the interpretation be different? Will interaction be more affective if the robot maintains longer eye contact, pay more attention, mimic human gestures, or stand more closely to students? As some features may be common across different platforms (e.g. computer desktop, virtual reality environments, robot) some features are unique to the physical environment (e.g., proxemics). Examining these characteristics can be potentially useful when incorporating the findings into designing interactive and dynamic scenarios for collaborative learning. In the first line of work, we applied familiar play routines when interacting with robots (e.g., turntaking scenario, setting up the dinner table). Often times, the childs ability to pretend or engage in
ISLS
59
ICLS2012
Volume 2: Symposia
collaborative play was constrained by what the robot could do in response (e.g., timing and limitation of response). In another line of work we found that when the robot activated a familiar schema (e.g., going to the zoo, story-telling time, The Three Little Pigs), the child sustained a social relationship with the robot (Okita, Ng-Thow-Hing, & Sarvadevabhatla, 2011). Children seemed to start out by drawing on prior knowledge through which they decide the can and cannot do interactions with robots (Okita & Schwartz, 2006). The studies helped identify important perceptual cues and responses that needed to be recognized and handled respectively to improve the quality of interaction. Until robots have the intelligence to flexibly respond to a wide range of interactive bids, following a well-known script or scenario seemed quite successful in guiding the interaction. In the third line of work, we examined whether familiar game scenarios (e.g., children's game "Captain May I?") could influence physical distance (e.g., proxemics) between human and robots (Okita, NgThow-Hing, & Sarvadevabhatla, 2012). Current limitations in sensor technology made physical distance a crucial factor to detect and respond to situations (e.g., avoid collision during collaborative tasks), user state (e.g., facial expression, speech recognition), and better choice of expressive communication (e.g., verbal or gesture).
Presentation 3: Robots facilitate constructive listening for strengthening individualized learning in collaborative learning situations
Naomi Miyake Consortium for Renovating Education of the Future, School of Education The University of Tokyo nmiyake@p.u-tokyo.ac.jp Remotely operable robots in collaborative classrooms can serve at least two roles. One is to work with children as a learning friend who helps members in the class enjoy and stay engaged in dynamic learning, where each individual member promotes her/his own understanding. The second role is to work with researchers as a datacollector and a reflector of our own facilitating behavior so that we can learn both the basic mechanisms and the design principles of productive activities in collaborative classrooms. In this presentation I will focus on the former, to illustrate what kind of new research can be developed by using robots, and then lead the way toward the second role. In order to establish the research context for this study, we have devised a strongly scripted yet dynamically collaborative learning situation based on the Jigsaw method. We call this framework the Knowledge Constructive Jigsaw, where it is emphasized that each individual student is responsible for integrating perspectives given by the learning materials and from other students with their own understanding. The class design involves a shared question to be answered and some relevant learning materials from different perspectives distributed among the different groups first in expert groups, to be later exchanged and integrated to answer the question in the jigsaw groups (Miyake, 2011). In accordance to some basic mechanisms of constructive interaction (e.g., Miyake, 1986), the design naturally requires each student to become a task-doer in the jigsaw group. It also provides each student with a chance, or chances, to be a monitor who infers what the other students say and why they say that, to integrate others ideas with their own. Yet the proportion of being the doer could differ from individual to individual. We now have data from nearly thirty classes of different grade levels on different topics. The analyses of these classes of middle schools on different science topics have revealed that there is little correlation between the achievement levels and the proportion of the role exchange. Rather, the monitors, who could spend almost the entire class without speaking up, learned a lot from just attending to others talks and inwardly working to integrate such inputs into their own understandings. This result suggests the importance of constructive monitoring, similar to the notion of constructive listening proposed by Greeno and van de Sande (2007), for better analyses and design of the interactive learning processes (Damsa et al., 2010). In a series of follow-up studies of this construct, we used remotely operable robots so that we could control the levels of soliciting constructive monitoring. We used the established learning plans and their associated materials with new groups of children and a robot, to see whether we could recreate similarly successful classes with robots as facilitators, and also to see what kind of conversational cues the monitoring children would use to start taking responsibility of their own learning. The basic analyses of these classes have revealed that children around 10 year olds have a keen sense of distinguishing whether the robot knows the answer but does not say it or the robot is just like the other kid who does not know the answer, but sincerely working to know the answer. When the robots are accepted as the latter, the children had a better chance to get involved in the constructive conversation, resulted in better learning (Miyake, 2011; Miyake, et al., 2011; Oshima, et al., 2011).
ISLS
60
ICLS2012
Volume 2: Symposia
Discussion Topics (Tentative) 1. Comparison with alternative technologies. How robots can enhance spontaneous, creative collaborative discourse among learners without interrupting or distracting from the inter-personal interaction (Gerry Stahl)
How can robots create values beyond existing devices for learning and collaboration? Can we use existing measures for collaborative learning to evaluate this type of learning partnership? 2. Pragmatic considerations of deployment to formal and informal learning environments. What needs to be considered when taking robots out of the lab and into real world (e.g., user support).
Discussant 1: Gerry Stahl

College of information Science and Technology Drexel University Gerry.Stahl@drexel.edu
Area of Expertise:
Dr. Stahl is trained in computer science, artificial intelligence, social philosophy, cognitive science, and learning science. Since 2002, he has taught human-computer interaction (HCI), computer-supported collaborative learning (CSCL), computer-supported cooperative work (CSCW) and social informatics (SI) at Drexel University. Dr. Stahls research approach includes theory building, system development and empirical studies of software usage. He has developed software systems and prototypes to explore support for collaborative learning, design rationale, perspectives and negotiation. His theory combines various sources from philosophy, education, sociology, communication and anthropology. He has developed a methodology of fine-grained empirical investigation into how groups of people learn to use artifacts like groupware systems in real-world settings such as school classrooms and virtual math teams. Dr. Stahl is a world-class researcher in CSCL, having organized international conferences, founded an international journal, published a volume on Group Cognition in MIT Press and one on Studying Virtual Math Teams in Springer Press and written over 150 professional papers. His website and blog are major resources for the CSCL research community. Retrieved excerpts from Drexel University faculty www site (November 7, 2011).
Discussant 2: Frank Fischer

Munich Center of the Learning Sciences Ludwig-Maximilians-Universitt Mnchen Frank.fischer@psy.lmu.de
Area of Expertise:
Frank Fischer earned his Doctorate in Psychology in 1997, and his Habilitation (professorial dissertation) in Psychology and Educational Science in 2002, both from the University of Munich. He is a Full Professor of Educational Science and Educational Psychology at the University of Munich. Since 2008 he is serving as the Director of the Department of Psychology at this university. Since 2009, he is the speaker of the Munich Center of the Learning Sciences, an interdisciplinary collaboration of more than 30 research groups focusing on advancing research on learning From Cortex to Community. In this context, he is also directing the Doctoral Training Program Learning Sciences. He is currently a member of the Board of Directors of the International Society of the Learning Sciences. His own research revolves around learning and instruction, with projects on collaborative learning, problem based learning as well as inquiry and simulation based learning. An overarching question is how technology-enhanced learning environments can advance knowledge and skills of learners in school, higher and further education. With respect to methodology, he has been contributing to the development of use inspired basic research approach in the field of learning and instruction. He has published more than 100 journal articles and book chapters, and co edited 6 books and special issues of scientific journals.
ISLS
61
ICLS2012
Volume 2: Symposia
Retrieved excerpts from Ludwig-Maximilians-Universitt Mnchen faculty www site (November 7, 2011).
Implications
In order to better design such practices within a wider variety of cultural settings, we need stronger scientific evidence for such success, and more precise explanation of the mechanisms of how and why things works. The studies in the presentations provide some portrayal on how students interact, engage in conversation, respond to, and work with robots and agents in a collaborative learning setting. This has some practical importance, as we hope to examine whether or not advanced technologies have promising roles for students in collaborative learning. The recent findings in the presentations, and the insight and expertise from the discussants, is sure to generate a strong discussion on the pros and cons, challenges and promises in using such artifacts in formal and informal learning environments. What is most crucial in such an endeavor is have an opportunity to gather, exchange ideas, rack brains, and plan future steps that will lead to successful research and implementation.
References
Ai, H., Kumar, R., Nguyen, D., Nagasunder, A., Ros, C. P. (2010). Exploring the Effectiveness of Social Capabilities and Goal Alignment in Computer Supported Collaborative Learning, in Proceedings of Intelligent Tutoring Systems. Bransford. J., Brown, A.L. & Cocking R.R. (Eds.). (1999). How People Learn: Brain, Mind, Experience, and School. Washington D.C.: National Academy Press for National Research Council. Chaudhuri, S., Kumar, R., Howley, I., Ros, C. P. (2009). Engaging Collaborative Learners with Helping Agents. In Proceedings of AI in Education Chi, M.T.H., Silver, S.A., Jeong, H., Yamauchi, T., Hausmann, R.G. (2001). Learning from human tutoring. Cognitive Science, 25, 471-533. Graesser, A.C., Person, N., Magliano, J. (1995). Collaborative dialog patterns in naturalistic one-on-one tutoring. Applied Cognitive Psychologist, 9, 359-387. Damsa, C. I., Kirschner, P. A., Andriessen, J. E. B., Gijsbert, E. & Sins, P. H. M. (2010). Shared Epistemic Agency: An Empirical Study of an Emergent Construct. The Journal of the Learning Sciences, 19, 143186. Dillenbourg, P. (2002). Over-scripting CSCL: The risks of blending collaborative learning with instructional design. In P. A. Kirschner (Ed.), Three worlds of CSCL Can we support CSCL (pp. 61-91). Heerlen: Open Universiteit Nederland. Drexel University (2011, November 7). Faculty Details Gerry Stahl, Ph.D. Retrieved November 7, 2011, retrieved excerpts from http://www.ischool.drexel.edu/Home/people/faculty/facultydetails/?facultyid=23 Greeno, J. G., & van de Sande, C., (2007). Perspectival understanding of conceptions and conceptual growth in interaction, Educational Psychologist, 42, 9-23. Gweon, G., Agarwal, P., Udani, M., Raj., B., Ros, C. P.(2011). The Automatic Assessmnet of Knowledge Integration Processes in Project Teams, in Proceedings of Computer Supported Collaborative Learning Howley, I., Mayfield, E., Ros, C. P. (2011). Missing Something? Authority in Collaborative Learning, in Proceedings of Computer Supported Collaborative Learning Kollar, I., Fischer, F., & Slotta, J. D. (2005). Internal and external collaboration scripts in web-based science learning at schools. In T. Koschman, D. Suthers, & T.W. Chan (Eds.), Computer Supported Collaborative Learning 2005: The Next 10 Years! (pp. 331-340). Mahwah, NJ: Lawrence Erlbaum Associates. Kumar, R., Ros, C. P., Aleven, V., Iglesias, A., Robinson, A. (2006). Evaluating the Effectiveness of Tutorial Dialogue Instruction in an Exploratory Learning Context, Proceedings of the Intelligent Tutoring Systems Conference. Kumar, R., Ai, H., Beuth, J. and Ros, C. P. (2010) Socially-capable Conversational Tutors can be Effective in Collaborative-Learning situations, Intelligent Tutoring Systems, Pittsburgh, PA Kumar, R. and Ros, C. P. (2011), Architecture for Building Conversational Agents that Support Collaborative Learning, IEEE Transactions on Learning Technologies Kumar, R., Beuth, J., Ros, C. P. (2011). Conversational Strategies that Support Idea Generation Productivity in Groups, in Proceedings of Computer Supported Collaborative Learning Ludwig-Maximilians-Universitt Mnchen (2011, November 7). Curriculum Vitae, Frank Fischer, Retrieved November 7, 2011, retrieved excerpts from http://www.psy.uni-muenchen.de/ffp_en/download/lebenslaeuufe/cv_frank_fischer.pdf
ISLS
62
ICLS2012
Volume 2: Symposia
Miyake, N. (1986) Constructive interaction and the iterative processes of understanding, Cognitive Science, 10, 151-177. Miyake, N., (2011) Fostering conceptual change through collaboration: Its cognitive mechanism, socio-cultural factors, and the promises of technological support, Proceedings of the 9th International Conference on Computer-Supported Collaborative Learning, (CSCL2011), Hong Kong Miyake, N., Oshima, J., & Shirouzu, H. (2011). "Robots as a research partner for promoting young children's collaborative learning." Proceedings of the 6th ACM/IEEE International Conference on Human-Robot Interaction. Lausanne, Switzerland. Mu, J., Stegmann, K., Mayfield, E., Ros, C. P., Fischer, F. (2011). ACODEA: A Framework for the Development of Classification Schemes for Automatic Classification of Online Discussions, in Proceedings of Computer Supported Collaborative Learning. Ng-Thow-Hing, V., Luo, P., & Okita, S. Y. (2010). Synchronized gesture and speech production for humanoid robots. Proceedings of the 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). (pp. 4617-4624). October 18-22, Taipei, Taiwan. Okita, S. Y., Ng-Thow-Hing, V., & Sarvadevabhatla, R. K. (2012). Captain May I? Proxemics Study Examining Factors that Influence Distance between Humanoid Robots, Children, and Adults during Human-Robot Interaction. Proceedings of the 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI), (pp.203-204). March 6-8, Boston, MA. Okita, S. Y., Ng-Thow-Hing, V, & Sarvadevabhatla, R. K. (2011). Multimodal Approach to Affective HumanRobot Interaction Design with Children. ACM Transactions on Interactive Intelligent Systems (TiiS). ACM, 1,1, Article 5, 1-29. Okita, S. Y., Ng-Thow-Hing, V., & Sarvadevabhatla, R. K. (2009). Learning together: ASIMO developing an interactive learning partnership with children. Proceedings of the 18th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN) (pp.1125-1130). September 27-October 2, Toyama, Japan. Okita, S. Y., & Schwartz, D. L. (2006). Young childrens understanding of animacy and entertainment robots. International Journal of Humanoid Robotics (IJHR), World Scientific, 3, 393-412. Oshima, J., & Oshima, R., (2011) Collaborative Reading Comprehension with a Robot as a Learning Partner: Implementation of Robots in the Jigsaw Method, Proceedings of the 9th International Conference on Computer-Supported Collaborative Learning, (CSCL2011), Hong Kong Palincsar, A.S., Brown, A.L. (1984). Reciprocal teaching of comprehension-fostering and comprehension monitoring activities. Cognition and Instruction, 1, 117-175. Ros, C. P., Moore, J. D., VanLehn, K., Allbritton, D. (2001). A Comparative Evaluation of Socratic versus Didactic Tutoring, In Proceedings of the Cognitive Sciences Society Ros C. P., & VanLehn, K. (2005). An Evaluation of a Hybrid Language Understanding Approach for Robust Selection of Tutoring Goals, International Journal of AI in Education 15. Ros, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (2008). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, The International Journal of Computer Supported Collaborative Learning, 3, pp237-271. Sawyer, R. K. (Ed.). (2006). The Cambridge handbook of the learning sciences. New York: Cambridge University Press.
ISLS
63
ICLS2012
Volume 2: Symposia
Embedded Phenomena for Knowledge Communities: Supporting complex practices and interactions within a community of inquiry in the elementary science classroom
Rebecca Cober1, Cresencia Fong1, Alessandro Gnoli2, Brenda Lpez Silva2, Michelle Lui1, Cheryl Madeira1, Colin McCann1, Tom Moher2, Jim Slotta1, and Mike Tissenbaum1 1 Ontario Institute for Studies in Education, University of Toronto 2 Department of Computer Science and Learning Science Research Institute, University of Illinois at Chicago rebecca.cober@utoronto.ca, cresencia.fong@utoronto.ca, agnoli2@uic.edu, brenda.ls@gmail.com, michelle.lui@gmail.com, madeirc@gmail.com, colindmccann@gmail.com, moher@uic.edu, jslotta@gmail.com, miketissenbaum@gmail.com Abstract: The work presented here is a product of a collaborative effort to develop a knowledge community and inquiry curriculum for elementary science, where students engage in extended investigations of simulated scientific phenomena presumed to occupy the physical space of their classrooms. By their immersive nature, these embedded phenomena lend themselves to a collective epistemology, and hence to new forms of learning and instruction that depart from the conventional didactic approach. The symposium centers on the design and enactment of a seven-week elementary school ecosystems unit, WallCology, developed in close collaboration with partner teachers and school administrators during summer and fall of 2011. Six posters highlight different facets of our effort, including descriptions of the immersive environment, the instructional narrative, the inquiry support technologies, the role of aggregate representations, discourse processes, and the classroom experiences of the 37 students and two teachers who participated in the unit.
Introduction
The goal of establishing a science classroom where students and teachers work together as a community of learners has been championed by Brown and Campione (1996), Scardmalia and Bereiter (1996; 2006), Bielaczyk & Collins (2006) and many others. In this vision, students work together with peers to investigate issues or phenomenon, develop their own theories, build upon one anothers ideas, and make progress toward some commonly held learning goal. While this knowledge community approach (Slotta & Najafi, 2010) has received many accolades for its vision of a collective epistemology, it has been quite difficult for researchers to enact. The FCL approach (Brown & Campione, 1996) has never been fully replicated, and the knowledge building model has been recognized by Scardamalia (2006) and others (van Aalst & Chan, 2008) as being quite challenging to enact. What are some of the major obstacles to making a knowledge community approach more tractable for researchers and practitioners alike? First, there must be some object of inquiry. The design or selection of this object of inquiry is of crucial importance, as it must be sufficiently intriguing for students, accessible to their investigations, and broad enough in scope to ensure that all students must be involved to make progress in the inquiry. Second, there must be pedagogical and technological scaffolds to support student inquiry, allowing students to engage in authentic science practices and helping make ideas visible for students, peers and teachers. Such scaffolds, when well designed, help students focus on important scientific aspects of the inquiry objective. Third, a community knowledge base serves to captures the aggregated observations and insights of all participants, and serves as a resource to their subsequent inquiry. Finally, there is an important epistemological element, where students must come to understand the purpose of their inquiry as being quite different from individual learning, and instead more concerned with the progress of the whole class in terms of inquiry activities and theoretical ideas. These are extremely important theoretical constructs in the learning sciences that are fundamentally distinct from the dominant body of work that explores individual learning, or even the more conventional models of collaborative learning. These ideas represent a revolutionary departure from conventional instruction, and a step toward inclusive, equitable, engaging designsif only we can get past the substantial barriers to entry. We must assume that our own understanding about how to conduct such research will grow in parallel with our scientific output, as we observe and participate in the evolution of learning and instruction within our partner classrooms. Perhaps this is why such research remains limited to just a few exemplars, despite its compelling theoretical position. The work presented here is a product of a new collaborative effort to engage in a knowledge community approach in an elementary classroom. One group, led by Tom Moher at the University of Illinois at Chicago, has spent the past six years developing a design framework called Embedded Phenomena (Moher, 2006), which engages learners in extended investigations of simulated scientific phenomena presumed to occupy the space of their classrooms. By their immersive nature, these embedded phenomena lend themselves to a collective epistemology, and hence to new forms of learning and instruction that depart from the conventional
ISLS 64
ICLS2012
Volume 2: Symposia
didactic approach. The other group, led by Jim Slotta from the University of Toronto, has developed a new pedagogical model called Knowledge Community and Inquiry (KCI) that provides a scaffolding framework for collective inquiry (Slotta, 2007). The project seeks to enrich the representational space of the embedded phenomena framework with the addition of comprehensive inquiry support layer for students and teachers enactment of complex investigations. The symposium centers on the design and enactment of a seven-week elementary school ecosystems unit, WallCology, developed in close collaboration with partner teachers and school administrators during summer and fall of 2011. Six posters highlight different facets of that process, including descriptions of the immersive environment, the instructional narrative, the inquiry support technologies, and the classroom experiences of the 37 students and two teachers who participated in the unit. The symposium will begin with a fifteen-minute overview to orient participants to the poster suite, followed by an hour of individual discussion with poster presenters. Chris Quintana from the University of Michigan will serve as chair and discussant, offering comments during the final fifteen minutes of the symposium.
WallCology: Representational space and instructional narrative

Brenda Lpez Silva, Alessandro Gnoli, and Tom Moher In WallCology, learners are challenged to maintain the biodiversity of a simulated ecosystem contained within a controlled environment imagined to occupy the walls of their classroom. The ecosystem contains several species of flora and fauna which students observe through persistent (continuously animated) portals (WallScopes) attached to the walls of the classroom, allowing them partial access to the controlled habitats. In addition, students are given control over several environmental variables that differentially impact vegetation growth: temperature, humidity, and lighting (Fig. 1). While initially stable, over the course of seven weeks, the ecosystem is perturbed in ways that threaten the viability of one or more species. Students are asked to respond to threat through the manipulation of the environmental controls, based on their understanding of the system food web and population dynamics. Students engage in a progressive series of activities designed to foster learning across a range of ecosystem concepts and science practices. Initially, students document the biodiversity and abundance of the Figure 1. WallScope view of a WallCology system through systematic observation of habitats and h bit t species that they see in the WallScopes. In the second phase, students focus on identifying the life cycles of the animal species by observing egg hatchings, transitions from larvae to pupa, etc. These understandings are needed in order for students to estimate species populations, an activity that is repeated periodically throughout the unit in order to monitor population trends that are not readily apparent to the naked eye. (Population estimates are obtained by counting species visible within the WallScopes and computing estimates based on the ratio of the area of the classroom wall to that of the display screens). Students observe the feeding patterns of species in order to construct the food web defining the indirect interactions within the ecosystem. An initial perturbation (the failure of the lighting system in several of the habitats), and its concomitant impact on population levels, alerts the children to the sensitivity of the system. While this failure is quickly resolved, it creates an opportunity to engage the learners in consideration of what-if scenarios involving other potential threats to the system. Rather than experimenting on the target ecosystem directly and risking possible catastrophic results, learners are provided with an ecosystem-modeling tool that allows them to rapidly model outcomes associated with variations in population mixtures and environmental conditions. The community knowledge growing out of this phase becomes critical when a second perturbation is introduced: the arrival of an invasive species in the form of a new predator. In the final phase of the unit, students directly intervene in the WallCology ecosystem by modifying environmental conditions to mitigate the impact of the predator and ensure the survival of all of the original species. Student activity within the unit involves a combination of structured observation, question and hypothesis formulation, experimental manipulation and hypothesis testing, and whole class and small group discussion, all within the context of a technology tool suite that allows students and teachers to contribute to, and draw from, an emerging community knowledge base. Students record observations, utilize the modeling tool, and synthesize con-tributions to the knowledge base using tablet computers; teachers orchestrate discussions using a SmartBoard interface that gives them access the knowledge base and scaffolds themed discussions aligned with unit milestones.
ISLS
65
ICLS2012
Volume 2: Symposia
Smart Classrooms for Knowledge Communities: EPIC Technology Environment

Jim Slotta, Mike Tissenbaum, Michelle Lui, and Matt Zukowski New forms of knowledge media offer a wealth of opportunity for researchers who can take advantage of the varying contexts (i.e., within the classroom, at home, or in field activities) and devices (e.g., laptops, smartphones, interactive tabletops, and large format displays) to design exciting new kinds of instruction where students dynamically generate knowledge, build on peers ideas, and investigate questions as a knowledge community. We have advanced the notion of a smart classroom, which integrates such technologies for purposes of supporting a spectrum of collaborative inquiry and knowledge construction activities. The work centers around the development of a flexible open source platform called SAIL Smart Space (S3), which offers a framework for integrating devices, materials, and user interactions, as well as a set of core technologies: (1) a portal for student registration and software application management; (2) an intelligent agent framework for data mining and tracking of student interactions in real time; (3) a central database that houses the designed curriculums and the products of student interactions; and (4) a visualization layer that controls how materials are presented to students (Slotta, 2010). We have implemented S3 in several distinct scenarios, including (1) PLACE. Web: a content community for high school physics (Tissenbaum & Slotta, 2011); (2) EvoRoom: an immersive, room-sized simulation of a Sumatran rainforest where students investigate evolution (Lui, Tissenbaum & Slotta, 2011), and (presently) (3) Embedded Phenomena for Inquiry Communities, where tablet computers and shared interactive displays support student investigations of simulations running within their classroom walls, floors and ceilings. Through these developments, we investigate how smart classrooms can scaffold the collection, aggregation, and representation of information across contexts and environments. We also strengthen the underlying S3 core technologies so they may be extended to an increasing range of applications. Our poster will detail the S3 implementation for the EPIC designs. We focus on several dimensions, including the role of the core Figure 2. The S3 Software Architecture. technology (e.g., to register students, support groups, store and retrieve data, etc), our use of intelligent agents (i.e., in support of the various interaction designs discussed in other posters within this session), and the specific forms of visualization used for various purposes. Our Wallcology enactment relied on two loosely coupled systems: (1) the Embedded Phenomena server, which managed the bug colonies (and all perturbations, environmental dependencies, etc.) within the classroom walls, and (2) an S3 server that supported student and teacher interactions, including log-ins, data collection and management, tablet and Smartboard applications. S3 supported our development of a peer-to-peer network of tablet-based input devices, where students add their observations of the phenomenon (e.g., counts of species appearing on the wallscopes), as well as more abstract representations (e.g., the relationships between two species of insects). Observational data are aggregated in real time, using Extensible Messaging and Presence Protocol (XMPP), which updates student tablets continuously. Intelligent agents are also implemented using XMPP; agents are implemented as software routines that simply listen in to the relevant message stream, and take action whenever certain conditions occur. WallCology employs an Archivist agent, which translates incoming messages from the tablets into database entries, and a Notetaker agent, which passes student notes from tablet to database and to the Smartboard. Several distinct forms of representation were designed, including aggregate tables of student observations, graphical representations (e.g., of species counts, dependencies across locations in the room, etc.), and the phenomenon itself both in a concrete form (i.e., 1 bug in the wall is 1 bug on the screen) and a more symbolic form (graphs of bug populations). The representations were populated by active calls to the database that occurred via XMPP in real time, so that all tablets and the Smartboard were updated instantaneously. Our poster will discuss the role of smart classroom technologies in supporting research of complex pedagogical designs, including the scaffolding of interactions, as well as distinct affordances for real time aggregation and execution of pedagogical logic (e.g., dynamic grouping, real time assignment of resources, and content sensitive feedback), using intelligent agents. S3 offered an accessible means of developing the sophisticated array of interdependent applications needed to coordinate our Wallcology designs.
Materials that scaffold collective inquiry: The role of aggregate representations

Rebecca Cober, Colin McCann, Jim Slotta, and Tom Moher
ISLS
66
ICLS2012
Volume 2: Symposia
In previous enactments of WallCology, students used paper lab books to record observations about the phenomena (Moher et al., 2008). In the current version, our interest was to help students work collectively as a "knowledge community, with the contributions of individuals or small groups readily accessible to others. Moreover, the results of individual inquiry actions, such as observations or reflections, should ideally combine into aggregate representations (ARs) that make important patterns visible to both teachers and students. To achieve this goal, we developed a new technology layer to support student investigations and to share knowledge with each other. A custom tablet application was developed to support student investigations of Wallcology, effectively replacing the lab books of previous iterations. Most importantly, students had access to ARs, displayed on the tablets and interactive whiteboards, comprised of their own and their peers' observations and insights, compiled in real-time using XMPP (a Twitter-like messaging technology). Our presentation will describe the design and evaluation of these representations and the role they played within the WallCology narrative. Our guiding research question: How do aggregate representations of collective knowledge advance students' and teachers' understanding of the object of scientific inquiry and provide a resource for subsequent inquiry activities? Method. Our designs were informed by the Knowledge Community and Inquiry (KCI) framework, which seeks to integrate elements of community knowledge construction with scaffolded inquiry (Slotta, 2007). We designed a tablet application to facilitate four learning activities within the WallCology unit: recording observations of (1) the organisms and (2) habitats of the ecosystem, and constructing representations of (3) life cycles and (4) food webs (Fig. 3).
Figure 3. Aggregate representations of community knowledge. Text-based data tables for learning activities 1 and 2, and pair-wise matrices depicting tallies of relational statements, for learning activity 4. We begin by describing the targeted pedagogical goal for each AR, in relation to its corresponding learning activity. We then evaluate each AR in the context of the wider instructional design: does it serve to achieve the broader desired pedagogical goals of the unit? Next, we evaluate the enactment of the curriculum, recognizing that it may diverge considerably from the intended design, due to variations in teaching styles and classroom dynamics. Finally, for each AR, we capture and represent the pedagogical patterns that actually occurred in this iteration of WallCology, and compare them against our original design goals. Outcomes. The AR designs supported collective inquiry in three important ways. First, teachers used the ARs to guide whole-class discussions (WCDs). One pattern, called "aggregate observations and discuss" occurred across all four ARs, where teachers displayed relevant strands of student-contributed observations to facilitate discussion, moving the classroom inquiry toward a desired pedagogical goal. The outcome of these WCDs frequently set the stage for further work, revealing important next steps in the investigation of the EP. Second, ARs were used to form a basis for consensus particularly the relationship tallies in ARs 3 and 4. High tallies were strong indicators of the veracity of relational statements (e.g., that green bugs hatch from dark blue eggs), enabling the community to resolve the life cycle and food web relationships (led by the teacher). Third, the ARs revealed areas of disagreement, moving the knowledge community forward in their understanding of the embedded phenomena. Closely contested observations (e.g., two organisms tallied almost equally for the same life cycle stage) were readily apparent in the ARs, providing specific areas for students to focus on in their continued investigations, and for teachers to focus discussions.
ISLS
67
ICLS2012
Volume 2: Symposia
The Role of Discussion in Orchestrating Inquiry

Cresencia Fong, Rafael Pascual-Leone, and Jim Slotta WallCology engages students in cognitive, social, and technological interactions with their environment orchestrated by the teacher, typically through discussions that served to advance the understanding of the community. How do such idea-centred (Scardamalia & Bereiter, 1996) discussions proceed? How should the teacher orchestrate (Fischer & Dillenbourg, 2006) and script (Kollar, Fischer & Slotta, 2007) such discussions to foster students sense of contribution and personal progress? What role does technology play in representing student ideas, and supporting the growth of collective understanding? We investigate those questions by examining discussions that occur within WallCology classrooms, describing: (1) the pedagogical characteristics of discussions that occur throughout the unit, (2) how the various WallCology technologies were employed within those discussions, and (3) how a new discourse tool impacted those pedagogical characteristics. We begin by analyzing the discussions that occurred in each classroom. We then introduce Common Knowledge (CK)a technology environment designed to promote idea growth and exchange while enabling teachers to guide discussions. Method. In WallCology, the teachers role as guide on the side is pivotal in facilitating meaningful discussions. Such discussions may be short and spontaneous to orient students toward productive paths, or lengthier and strategically placed in the curriculum to promote conceptual learning goals. Using video and observation notes, we analyzed discussions occurring during the first month of the enactment, in terms of their goal or purpose, conversational dynamics, the growth of ideas, and pedagogical outcomes or consequences. We also analyzed the use of technology and paper-based resources. In CK, a handheld computer tablet application enables students to contribute their questions, evidencebased hypotheses, theories, and ideas; and tag their contributions from a list of science content and practice keywords (e.g., Theory, Observation, Question). These notes dynamically appear on all tablets and on the classrooms SMART Board, allowing teachers to manipulate notes during oral discussions, swiftly filtering topic-related notes as the discussion progresses. We integrated CK discussions into the WallCology curriculum, specifying discussion goals and pre-programming relevant keyword tags. Teachers also launched spontaneous CK discussions, as they felt warranted. Outcomes. Four Figure 4. Tablet (left) and SMART Board (right) mediated discussions. WallCology sessions in both grades 5/6 classrooms were analyzed as described above. Data sources included classroom observations, video and audio recordings, teacher-researcher debriefings, paper artifacts of student work, teacher and student interviews, and data logs of CK discussions. The goal was to produce a schema for productive inquiry discourse, and a model by which to evaluate CK discussions. Discussion analysis revealed several distinct discourse patterns, including: Encouraging new ideas. Teacher displays observation data or CK notes on SMART board; helps students recognize that there might be additional avenues for reflection or observation; Resolving divergence. Teacher displays observation data or CK notes on SMART board; helps students notice divergence, discuss each position, and consider ways to resolve the divergence; Introduce topics or processes. Teacher builds upon existing ideas from observation data or CK notes, grouping ideas together to promote insights about a new topic (e.g., organism life cycles); Motivate approaches. Teacher encourages students to reflect on data or hypotheses, brainstorm limitations to current approaches, and possible new activities (e.g., comparing numbers of species in different habitats). We analyzed the CK discussions that occurred within the first 4 weeks of the Wallcology curriculum, to see how teachers achieved these patterns using the technology environment. Students were engaged, with 4 times the number of in-class CK contributions per minute than oral discussions, and substantial reading and response to peer notes. Idea growth and symmetric knowledge advance were heavily reliant on teacher orchestration of lesson activities and spontaneous teacher scaffolding. Design improvements are concerned with smarter filtering and commenting.
Teacher Orchestration of Complex Inquiry Patterns

Cheryl Madeira, Cresencia Fong, and Richard Messina
ISLS
68
ICLS2012
Volume 2: Symposia
How do teachers orchestrate complex inquiry-based activities, and how can technology scaffolds be developed that support teachers as they acquire such skills? This paper examines teacher practice within the Wallcology project, following two master teachers as they worked closely with researchers and technologists to co-design, enact, and revise (often in real time) the inquiry-based activity patterns for their grade 5-6 classes. Indeed, from the outset, these teachers (knowing better) asserted that it would not be possible to choreograph every twist and turn of the curriculum, but that they would need substantial leeway to interpret the situation and respond accordingly. Such adaptive practice, illustrated below, can be viewed as orchestration of an inquiry design: After reviewing the days lesson plan, the teacher, Brendon (pseudonym), notices that students have not entered details in their Observation tab. He says, Only a couple of notes on life cycle were enteredmaybe today we can observe more. Then he instructs: under the Organism tab, you can talk about organism again; and then go into life cycle. Even though the plan had called for whole class session of Observation of organisms, Brendon realized that by turning kids attention forward to Lifecycle observations, he could transition into the important step of Relationships. (FieldNotes 2011_1017). This except illustrates adaptive teacher practice, where the teacher gages students' prior knowledge (Penuel & Gallagher, 2009), develops collaborative opportunities for students' inquiry processes (Slotta & Linn, 2009), and facilitates students' emerging knowledge (Zhang et al., 2010). In recent years, learning scientists have embraced the term 'orchestration,' defined as the process of productively coordinating supportive interventions across multiple learning activities occurring at multiple social levels (Fischer & Dillenbourg, 2006). In orchestrating the flow of activities, materials, and interactions within the classroom, teachers develop pedagogical knowledge. Method. To capture the role of the teacher in our EPIC classrooms, we examined teachers orchestration, including their adaptive responses to student ideas, and how they make use of technology in ways both intended (i.e., by designers) and unintended. While the teachers in our study were both masters of inquirybased pedagogy, having used it exclusively for 3 and 5 years respectively, neither had any experience with embedded phenomena nor the specific forms of student inquiry it was designed to support. Thus, while our teachers entered with substantive pedagogical knowledge, there would be many challenging and novel aspects of teaching with the Wallcology materials and activities. Our research seeks to understand how these teachers develop new practices (and implicitly, new pedagogical knowledge), and to inform our understanding of how such orchestration leads to in-service professional development. We observed the teachers throughout the co-design and enactment of the 8-week Wallcology unit that addressed science topics related the diversity of life. Data included classroom observation notes, video, photos and audio recordings, time logs of teachers activities, teacher and student interviews, and other relevant school documentation. Ten co-design sessions were held (all audiotaped) where teachers, administrators, researchers and technologists designed the curriculum activities, reviewed drafts of materials and technology tools, and reflected actively about the nature of learning and instruction with Wallcology. We observed all classroom sessions, and debriefed regularly with teachers, including larger scale research meetings where the teachers met with the entire research team to discuss pedagogical issues and inform ongoing design. Working first from observation notes, then from video, we segmented each lesson into an orchestrational move. For each segment, we described the purpose, or goal, the role of technologies or representations, and any consequential outcomes. We paid particular attention for adaptive moves where teachers responded to characteristics of the classroom, evidence of student knowledge, or intuitions about where more attention might be necessary. Outcomes. It became apparent early in the implementation that the teachers were not at ease unless they were able to adapt and improvise at any moment during the enactment. They felt strongly about their decisions to veer away from certain design requirements. We saw evidence of teachers ignoring the technology when face-to-face communications were more comfortable and effective. In one segment after the next, there were indications that teachers assessed student engagement and understanding, then responding in ways that departed from the lesson plan. Each night, a revised lesson plan was made for the following day. While we had envisioned that the plan would be a reference guide (expecting some variance), Brendon commented that he had the score in his head summoning images of a jazz performance rather than orchestral conducting. At least for these early trials, our curriculum was specified more as a series of chord progressions than a strict melody. Our summary of teacher enactment patterns, including technology use, can inform the design of EPIC curriculum to reinforce those patterns and encourage teachers with less expertise to engage in them.
ISLS
69
ICLS2012
Volume 2: Symposia
Fidelity of science practice in complex instructional designs

Alessandro Gnoli, Cheryl Madeira, Tom Moher Contemporary research in science pedagogy emphasizes the importance of engaging learners in authentic science practices, not only for the development of specific inquiry skills, but also for the development of agency and understandings about the nature of scientific work (Duschl, Schweingruber, & Shouse, 2007; Zimmerman, 2007). The embedded phenomena (EP) framework affords classroom learners opportunities to engage in a broad range of science practices, particularly the kind of patient scienceextended inquiry over long time periods that elementary school students rarely encounter (Fulp, 2002). The underlying simulations are designed to present an extended narrative, gradually revealed through the accumulation of data over time. The clarity of that narrative, as a result, is highly sensitive to students consistent and accurate applications of observational and manipulative procedures, making the timely detection and remediation of practice errors critical capabilities in successful enactment. When errorsof observation, of computation, of transcription, of interpretationgo undetected, and erroneous data are added to the historical representation of the investigation, they can obscure the underlying narrative of the unit, undermining days or weeks of classroom effort. In WallCology, students gather and transcribe substantial amounts of objective data involving habitats, life cycle patterns, producer/consumer relationships, and environmental conditions. They count individual animals within WallScopes (samples) and computer population estimates based on physical measurements of the classroom walls. Errors in these practices can have significant impacts on the resultant representations used by students to reason about the state of the ecosystem. For example, an incorrect characterization of the life cycle of one species can result in incorrect classification of individuals within sampling activities, leading to population estimation errors. Inconsistent procedures for counting individuals (even if properly classified), or computational errors in determining count averages can have similar effects. In WallCology, the cumulative impact of these errors can easily mask Figure 5. Errors in counting mask important population shifts or introduce representational population trends in prior WallCology enactment. artifacts unrepresentative of the actual state of the ecosystem. Method. Moving embedded phenomena from paper to technology-based media creates the opportunity to identify observational, transcription, and computational errors of practice at the point of data entry. While straightforward enough to implement, our goal is not to use discrepancies as the basis for directly notifying students of their errors (e.g., using pop-up dialogs to inform students that the data they just recorded are incorrect). Rather, our longer-term strategy is provide notifications to teachers, who can use the information to more subtly intervene within undermining students agency as investigators (e.g., by walking over to a group thats making significant errors in practice and suggesting that their data seem unusual, perhaps warranting a second look by the group). Research surrounding this strategy will focus on the capacity of teachers to manage this information flow, the choice of media to be used for delivery (e.g., tablet notifications, smartphone text messages, Bluetooth earphones), message representations (individual, small group, whole class), and information origination (e.g., push vs. query strategies). Outcomes. We have developed software agents that continuously compare student data entries with the current state of the simulation, and we used those agents to collect baseline data on the frequency and magnitude of errors during the present WallCology enactment. Preliminary analysis suggests that moving to electronic rather than paper-based media has not impacted the rate of practice errors. As in prior enactments, cumulative errors obscured the representation of population trends, requiring us to dramatize the impact of ecosystem perturbations in order to motivate population modeling and interventions. The rate of observed errors suggests that the design of notification mechanisms could be a significant challenge, requiring a careful balance between real-time and off-line approaches, and the adoption of strategies that are sensitive to the severity of errors as well as historical error frequency among individuals and within groups. We will be designing and developing our initial tools and procedures to support practice error notification and remediation for a second iteration of the new WallCology in spring 2012.
ISLS
70
ICLS2012
Volume 2: Symposia
References
Bielaczyc, K., & Collins, A. (2006). Fostering knowledge-creating communities. Collaborative Learning, Reasoning, and Technology, 37-60. Brown, A. L., & Campione, J. C. (1996). Psychological theory and the design of innovative learning environments. Environments, 289-325. Duschl, R., Schweingruber, H. & Shouse, A. (2007). Taking Science to School: Learning and Teaching Science in Grades K-8. National Research Council report. Fischer, F., & Dillenbourg, P. (2006). Challenges of orchestrating computer-supported collaborative learning. Paper presented at the 87th Annual Meeting of the American Educational Research Association (AERA), San Francisco, California, USA. Fishman, B., Marx, R. W., Best, S., & Tal, R. (2003). Linking teacher and student learning to improve professional development in systemic reform. Teaching and Teacher Education, 19, 643658. Fulp, S. (2002). The Status of Elementary Science Teaching. Horizon Research (One of a series of reports from the NSF-sponsored 2000 National Survey of Science and Mathematics). Retrieved from: http://2000survey.horizon-research.com/reports/elem_science.php Kollar, I., Fischer, F., & Slotta, J. D. (2007). Internal and external scripts in computer-supported collaborative inquiry learning. Learning & Instruction, 17(6), 708-721. Moher, T. (2006). Embedded phenomena: Supporting science learning with classroom-sized distributed simulations. Proceedings ACM Conference on Human Factors in Computing Systems (CHI 06), Montreal, Quebec, Canada. 691-700. Moher, T., Uphoff, B., Bhatt, D., Lpez Silva, B., and Malcolm, P. (2008). WallCology: designing interaction affordances for learner engagement in authentic science inquiry. Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems. CHI '08. ACM, New York, NY, 163-172. Penuel, W. R., & Gallagher, L. P. (2009). Preparing teachers to design instruction for deep understanding in middle-school earth science. Journal of the Learning Sciences, 18, 461508. Prieto, L. P. (2010). The many faces of orchestration: Towards a (more) operative definition, Unpublished Technical Report, University of Valladolid, Spain. Retrieved from http://www.mendeley.com/research/faces-orchestration-towards-more-operative-definition/. Scardamalia, M., & Bereiter, C. (1996). Engaging students in a knowledge society. Educational Leadership, 54(3), 6-10. Scardamalia, M., & Bereiter, C. (2006). Knowledge building: Theory, pedagogy, and technology. The Cambridge Handbook of the Learning Sciences.Cambridge University Press, Cambridge, 97-115. Slotta, J. D. (2007). Supporting collaborative inquiry: New architectures, new opportunities. In J. Gobert (Chair), Fostering peer collaboration with technology. Symposium conducted at the biennial Computer Supported Collaborative Learning (CSCL) Conference, New Brunswick, NJ. Slotta, J.D., Tissenbaum, M., & Lui, M. (2011). SAIL smart space: Orchestrating collective inquiry for knowledge communities. 9th International Computer-Supported Collaborative Learning Conference (CSCL), Hong Kong, China. 3, 1081-1088. Slotta, J. D., & Linn, M. C. (2009). WISE science: Web-based inquiry in the classroom Teachers College Press. Slotta J D and Najafi H (2010), Knowledge Communities in the Classroom. In: Penelope Peterson, Eva Baker, Barry McGaw, (Editors), International Encyclopedia of Education, Volume 8, pp. 189-196. Oxford: Elsevier. Tissenbaum, M. (2011). Co-designing collaborative smart classroom curriculum for secondary school science. Proceedings of the Red-Conference: Rethinking Education in the Knowledge Society, Switzerland. van Aalst, J., & Chan, C. K. (2007). Student-directed assessment of knowledge building using electronic portfolios. Journal of the Learning Sciences, 16(2), 175-220. Zhang, J., Scardamalia, M., Lamon, M., Messina, R. & Reeve, R. (2007). Socio-cognitive dynamics of knowledge building in the work of 9 and 10 year olds. Education Tech Research Dev, 55, 117-145. Zimmerman, C. (2007) The development of scientific thinking skills in elementary and middle school. Developmental Review 27(2), 172-223.
Acknowledgments
The authors would like to thank Joel Brown, Julia Murray, Armin Krauss, Francesco Novellis, and Ben Peebles, Gabriel Resch for their invaluable contributions to the design and enactment of the WallCology unit. This material presented here is based upon work supported by the U.S. National Science Foundation under grant IIS1065275 and Canadian Social Sciences and Humanities Research Council under grant 410-2011-0474.
ISLS
71
ICLS2012
Volume 2: Symposia
Critical Aspects in Learning with Technologies

Ching Sing Chai, Nanyang Technological University, chingsing.chai@nie.edu.sg Choon Lang Quek, Nanyang Technological University, choonlang.quek@nie.edu.sg Chwee Beng Lee, University of Western Sydney, chwee.lee@uws.edu.au Huang-Yao Hong, National Chengchi University, Taiwan, hyhong@nccu.edu.tw Joyce Koh Hwee Ling, Nanyang Technological University, joyce.koh@nie.edu.sg Naomi Miyake, University of Tokyo, nmiyake@p.u-tokyo.ac.jp Peter Reimann, University of Sydney, peter.reimann@sydney.edu.au Abstract: In this symposium, we present a comprehensive overview on the unique position of technologies in the context of learning sciences. According to Jonassen, Howland, Marra and Crismond (2008), learning with technology necessitates the designs of learning and the learning environments in order to engage learners. The overarching purpose of this symposium is to bring forth a comprehensive discussion on the critical aspects of learning with technologies which are fast becoming an integrated part of learning. To accomplish our purpose, the presenters will (a) provide an overview of the context of learning by discussing the importance of technological pedagogical content knowledge and socio-cultural issues, (b) examine issues related to the designs of learning, this includes scaffolding learning and intentional knowing, (c) discuss one of the most important outcomes of learning, which is conceptual change. In our symposium, our presenter will also discuss the roles of technologies in fostering conceptual change.
Introduction
Technologies are pervasively used in education. But are we using it for productive thinking and meaning making? What are the critical aspects to consider when we intend to integrate technologies into meaningful learning? Have we made progress in designing for learning with technology rather than merely from technology? With these questions in mind and relating to the emphasis of the powerful role that technologies play in transforming learning (Sawyer, 2006), we seek to assess the position of technologies in the context of learning sciences. According to Jonassen, Howland, Marra and Crismond (2008), when learning with technologies, it necessarily consists of the designs and the environments that engage learners. Most importantly, technologies should function as intellectual partners where the cognitive responsibility is distributed to the partner that performs it better. The overarching purpose of this symposium is to bring forth a comprehensive discussion on the critical aspects in learning with technologies. To accomplish our purpose, the presenters will (a) provide an overview of the context of learning by discussing the importance of technological pedagogical content knowledge and sociocultural issues. Learning scientists argue that deep learning is most likely to occur in complex social and technological environments (Sawyer, 2006) and this is supported by Koehler and Mishra (2009) while previously Mishra and Koehler (2006) who also argue that good teaching requires knowledge of content, pedagogy, and technology. Understanding the situative perspective (Greeno, 2006) of learning is critical as this helps researchers and educators to understand the complexity nature of learning for better design of learning environments, (b) examine issues related to the designs of learning, this includes intentional knowing. One of the main concerns of learning sciences is to provide powerful learning environments that foster deep learning (Sawyer, 2006). Our symposium will also examine the importance of intentional knowing in relation to the use of technologies for learning, (c) discuss one of the most important outcomes of learning, which is conceptual change. According to Vosniadou (2008), perhaps one of the biggest breakthroughs in the learning sciences was the examination of conceptual change. In our symposium, our presenter will also discuss the multiple roles of technologies in fostering conceptual change. Our discussion begins with Ching Sing Chai, Huang-Yao Hong, Joyce Koh Hwee Leng giving the general perspectives of technological pedagogical content knowledge and design thinking. Next, Naomi Miyake will address socio-cultural issues such as collaborative learning and the contexts of learning. She will discuss urgent needs for researchers pay more attention to these harder to observe collaborative processes. Chwee Beng Lee and Choon Lang Quek will discuss the components of intentional knowing and its role. Lastly, Peter Reimann will focus on the pivotal roles of technologies in the context of conceptual change research.
Technological pedagogical content knowledge and design thinking

Ching Sing Chai, Nanyang Technological University
ISLS
72
ICLS2012
Volume 2: Symposia
Email: chingsing.chai@nie.edu.sg Huang-Yao Hong, National Chengchi University, Taiwan Email: hyhong@nccu.edu.tw Joyce Koh Hwee Ling, Nanyang Technological University Email: joyce.koh@nie.edu.sg The advancement of networked technology and the development of myriad e-learning platforms and social networks have provided ample opportunities where ideas, insights, experiences, and knowledge can be articulated, shared, co-constructed and distributed (Chai & Lim, 2011). These ideas, insights, experiences, and knowledge can be referred to as conceptual artifacts or world 3 objects (Bereiter, 2002; Popper, 1978). Once publicized through networked technologies or traditional media, these cognitive objects become improvable ideas. They are subjected to examination, critique, refinement, elaboration, re-contextualization etc. Bereiter and Scardamalia (2006) argue that ability of an individual to participate and contribute to a community effort in improving the ideas is the key competency that citizens of the Knowledge Age must possess. Many educators have also advocated shifting the focus of education from reproductive transmission to that of creative construction (Collins & Halverson, 2010; Paavola, Lipponen, & Hakkarainen, 2004). In this paper, building on Bereiter and Scardmalias (2006) proposal of building students ability to work collaboratively with conceptual artefacts in design mode" (p. 702), we propose that teachers need to develop and create the corresponding technological, pedagogical and content (TPACK) knowledge (Mishra & Koehler, 2006) through continuous design thinking. We also argue that nurturing design epistemology is the key task that educators should focus on, beginning from the teachers and cascading down towards the students, especially when supported by the affordances of collaborative learning platforms. In other words, we argue in this paper that teachers need to become knowledge creators who generate the know-how in using collaborative technologies to help students to co-construct knowledge. This involves teachers capacity in providing the epistemic framing (Elby & Hammer, 2010) for classroom learning, scaffolding students collaborative sense making processes and building students epistemic repertoire before, during and after their learning episodes.
The learning of a monitor: How much and why

Naomi Miyake, University of Tokyo Email: nmiyake@p.u-tokyo.ac.jp One intriguing outcome of socially well-organized collaborative learning is the students who sat silent during the group work still come out with high performance at the end. In this presentation, I will showcase three examples of this learning of a monitor, give some theoretical framework to understand how this could happen, and then propose promising use of technology to promote this kind of learning. I will take examples of the learning from the Hypothesis-Experiment Instruction, (Itakura, 1997, henceforth HEI), the Knowledge Constructive Jigsaw, (Miyake, 2010, henceforth KCJ), and Manabi-noKyodotai (Comminity for Learning, Japan, led by Manabu Sato, henceforth MK). They differ in their basic curriculum designs, but all support strongly the learners discussions either in whole class (HEI), in small groups with member change (KCJ), or four-member groupings (MK). Careful analyses of the students learning processes and the associate learning outcomes often reveal that the students who mainly monitored during the class did show substantial learning toward the end of class. Theoretically this could be the case, as suggested by studies on two-person, understanding and problem solving process such as trying to find an answer to how a sewing machine makes its stitches (Miyake, 1986), because in her analyses, the monitor, who took the role of listening and integrate what the other member, the task-doer, tried to explain to her/his own understanding, did progress her/his part of understanding almost independently from the task-doers. It was not rare in her other analyses that the monitors played the critical role to bring the pair to find a hidden course to solve the problem, mainly because they were pursuing some different course from the task-doers. If this theoretical framework should work at classrooms where more than two people engage in the groups simultaneously, we should be able to identify cases where those students who mainly stay silent should also learn, as well as those who actively participate in the group work. For the HEI, I will take a case where 21 third graders discussed to predict forthcoming results of experiments, to build a first stage understanding about the air as substance, (i.e., to understand and become able to explain why the air does not share the same space with water), based on Saitos analyses (Saito & Miyake, 2010). She analyzed the frequency and the order of class utterances of all of the students, and found that the frequency of talk differed greatly, but all of them could predict the result of the last experiment correctly. In addition, all of them could write after the class the reason for their choice in scientifically acceptable forms. The KCJ is a jigsaw class, with a carefully designed class question to be answered only by integrating different pieces of information contributed by different expert groups into the
ISLS
73
ICLS2012
Volume 2: Symposia
jigsaw. For the KCJ, I will report three cases which all showed that those relatively silent students could write what they learned during the class as well as those who had been active during the class. The MK uses group work more flexibly yet the recommended form of grouping is four, so that when a teacher asks the class to solve a problem with expected answer rate of 50%, there is some assurance that often two of the members could explain how and why of the solution. In MK also, there has been observed that the students who had been identified as not fit for the subject of the class sometimes gives a critical utterance to lead the whole class to clear understanding. After showcasing these examples, I will develop a model for this type of learning of monitor based on my constructive interaction framework. To conclude my presentation, I would like to discuss urgent needs for us to collect and study the learning processes in more detail during both those experimental and regular classes, so that we could pay more attention to these harder to observe processes. I will illustrate how technology could help us do this more extensively, with more ease, so that not only researchers like us but teachers could participate in this endeavour to understand how people learn, and how we could support the learning.
The role of intentional knowing in learning with technology

Chwee Beng Lee, University of Western Sydney, School of Education Email: chwee.lee@uws.edu.au Choon Lang Quek, Nanyang Technological University Email: choonlang.quek@nie.edu.sg Berieter and Scadamalia (1989) highlighted that learners are not only active in their construction of meaning, but they can be intentional. They are intentional because they are fulfilling their goals that in turn determine their human behaviour. This would also mean that they are cognitively engaged in the learning process, monitoring and regulating their learning (Sinatra & Pintrinch, 2003). To learn intentionally, students must consciously understand and be able to define their strengths and weaknesses, their learning processes, how they examine the way they execute learning tasks, monitor learning, evaluate learning and whether they innovate in order to learn intentionally. Berieter and Scadamalia (1989) used the term intentional learning to refer to cognitive processes that have learning as a goal rather than an incidental outcome. It is an achievement, not an automatic consequence of human intelligence. (pp. 366). As put forth by Scadamalia and Berieter (2002), intentional cognition is more than self-regulated learning; it is the active pursuit of a mental life whereas selfregulated learning is usually a set of study skills and learning-to-learn strategies. They mentioned that there is a need for metaknowledge to foster intentional learning. Such an understanding is very much aligned with Efklidess (2008) metatheoretical model in which at the personal-awareness level, there are conscientious efforts in monitoring and controlling processes. Bereiter and Scadamalia (1989) defined it as knowledge about knowledge, access to ones own knowledge, and skills, which take the knowledge as an object to operate on. From a constructivist standpoint, learners are aware of the central role of ideas in the development of knowledge and how ideas are revised through a process of conjecture, argument, and test (Smith, Maclin, Houghton, & Hennessey, 2000). This type of processing is not a mere response to the external circumstances or environment. In a sense, intentional level processing is not only initiated by the learners, it is under the learners conscious control (Sinatra & Pintrich, 2003, pp. 4). On the other hand, if a learner is not intentional, the learning process is determined largely by the external factors such as prior knowledge, types of tasks, facilitating conditions, etc. In this section of the symposium, we will introduce the term intentional knowing through the lens of metacognition and we argue that it is a critical element to propel intentional learning. We have conducted several empirical studies to re-conceptualize metacogntion and a six-factor model was found to explain intentional knowing. Next, we argue that intentional knowing plays a pivotal role in learning with technology. This stance aligns with what most learning scientists would advocate, and that is the emphasis of technology in fostering deep learning, for instance, helping learners to collaborate meaningfully or to reflect on their understanding (Sawyer, 2006). Using technology to represent knowledge, articulate thinking, manipulate and revise conceptual understanding and to reflect upon knowledge and learning process are effortful and goaloriented activities for the learners. Such activities necessarily require intentional knowing which is the conscious knowing of ones capacity and learning process, the knowing of how one examines his or her own monitoring, evaluation and execution processes, in addition, how he or she innovates to cope with the demand of tasks. Intentional learners not only cognitively engage in the learning process, but also regulate their learning. For instance, Vosniadou and Kollias (2003) reported that when using Knowledge Forum for science learning, students were able to collaborate effectively and question their own conceptual framework. From the pedagogical perspective, in order to develop learners who have the intentionality to learn, we suggest that educators not only create learning opportunities to cognitively challenge the students, but also embed
ISLS
74
ICLS2012
Volume 2: Symposia
meaningful tasks and activities in technology-rich environments to them to apply thinking skills and take stock of their own strengths and weakness in the learning process. In this section of our symposium, we discuss the components and importance of intentional knowing in using technology for learning through our recent empirical study (N=2404). In this study that was conducted with high school students, we developed a statistical model to explain the intricate relationships between intentional learning and intention to use technology.
The multiple roles of technology for learning: Lessons learned from conceptual change research
Peter Reimann, University of Sydney, Faculty of Education and Social Work Email: peter.reimann@sydney.edu.au Research on conceptual change is an interesting case for studying various conceptualizations of the role educational technology can play for learning. While the notion of technology as a tool to learn with has probably found its strongest expression in math education, science educators have seen the role of technology more conservative: more to learn from than to learn with. At the same time, research on science education particularly concerned with conceptual change has been strongly influenced by Vygotskian notions of symbolic and social mediation. Scientific concepts and theories are cultural resources provided through (school, university, adult) education. Vygotsky made a crucial distinction between experiences produced by the immediate contact of the individual with the environment, and experiences mediated by symbolic tools, with writing as the major class of symbolic mediators (Kozulin, 2003). With respect to science education, unmediated contact gives rise to empirical learning, leading to the acquisition of spontaneous conceptions (of which some will be misconceptions), whereas the second, mediated form of contact affords theoretical learning, resulting more often in scientific conceptions (Karpov, 2003). In effect, for the case of theoretical learning, this does away with the dualism between process and content: How humans learn is largely dependent on the kind of concepts they master in the course of learning. In a profound sense, human learning is all about learning to learn. Process and content are thus two sides of the same coin. While a socio-cultural perspective allows us to overcome some of the problems inherent in the processcontent dualism, it is committed to the mind-environment dualism (with environment comprising the social as well as the natural). Due to the commitment to internalization, problem solving and thinking are achieved in the mind, by operations on a represented environment. In socio-cultural theory, the focus on the symbolic nature of tools goes along with a focus on internalization: Those tools are out there, and provided to us through education, but in order to use them we have to gain command over them by making them part of our cognitive repertoire. Cognition is seen as rooted in the mind/brain, 'between our ears'. However, for a number of reasons, not the least the rapid growth of mobile devices, the question of what needs to be learned from tools versus what we can accomplish with tools (Salomon, 1990) needs to be revisited. This requires us to make a less strong distinction between mind and environment. Jonassens (1996) for instance, identified the role that widely available computational tools, such as spreadsheet software, can play as mindtools. Despite its success, and the publication of a second edition (Jonassen, 2000), it is probably fair to say that the central idea has not been fully taken up. What we find today in (science) education is the acceptance of computers as productivity tools, which are good for creating artifacts such as essays and graphs and models. Learning, however, is still seen as resulting from these activities only in the form of cognitive residues, as changes in long-term memory. What is true for education in general also holds for the perceived value of ICT for conceptual change: the predominant question asked is: How does technology x contribute to fostering change? And the place we look for the answer is in the learners head, typically in a setting where the use of the technology is not allowed. To fully unravel the potential of computers, this view needs to be widened; the vision articulated in Jonassen (1996) needs to be revitalized. This should not be so hard, given that the socio-cultural perspective widely accepted by educators provides us with an appropriate conceptual framework: Learning scientific concepts means learning to make use of cultural tools, of the concepts and methods that science has developed and continuous to develop. If we are willing to entertain the idea that the distinction between psychological and technical instruments (Vygotsky, 1930/1981) is getting increasingly blurred, then the notion of learning as extending to changes in the use of external toolsnot confined to changes to long term memoryshould be a rather natural one. For instance, the notion of learning by modeling has evolved from a notion of learning from models (Reimann & Thompson, 2009), but to play out the full potential we need to move on to the notion of knowledge creation/problem solving/decision making with modeling. When the formal learning ends, tool use does not necessarily end. For example, students today can continue to carry their Netlogo software (Tisue & Wilenski, 2004) with them on their laptop, hopefully soon also on their smartphone or tablet device. Indeed, further learning (including communal and societal learning) requires the continuous use and further
ISLS
75
ICLS2012
Volume 2: Symposia
development of such tools. In order for such a (realistic) view of technology to take hold in schools, what is required are changes in the manner we assess learning. A still largely underconceptualized and underused potential of technology lies in its role for enhancing perception (Young, 2004). The main value of ICT may be that it provides ways to see things differently by providing conceptual tools, and to make those tools available persistently and ubiquitously. The best use of ICT maybe augmentation--scaffolding without the fading. Computer tools for augmenting reality can enhance perception in addition to thinking and problem solving.
References
Bereiter, C., & Scardamalia, M. (1989). Intentional learning as a goal of instruction, In L. B. Resnick (Ed.), Knowing, learning and instruction: Essays in honor of Robert Glaser (pp.361-392). Hillsdale, NJ: Lawrence Erlbaum Associates. Bereiter, C. (2002). Education and mind in the knowledge age. Mahwah, NJ: Lawrence Erlbaum. Bereiter, C., & Scardamalia, M. (2006) Education for the Knowledge Age. In P. A. Alexander, and P. H. Winne (Eds.), Handbook of Educational Psychology (2nd ed.). (pp. 695-713). Mahwah, NJ : Lawrence Erlbaum. Chai, C. S. & Lim, C. P. (2011). The Internet and Teacher Education: Traversing between the Digitized World and Schools. The Internet and Higher Education, 14, 3-9. Collins, A., & Halverson, R. (2010). The second educational revolution: rethinking education in the age of technology. Journal of Computer Assisted Learning, 26, 18-27. Elby, A., & Hammer, D. (2010). Teachers' personal epistemology and its impact on classroom teaching. In L. D. Bendixen & F. C. Feucht (Eds.), Personal epistemology in the classroom: Theory, research, and implications for practice (pp. 409-434). Cambridge: Cambridge University Press. Efklides, A. (2008). Metacognition: Defining its facets and levels of functioning in relation to self-regulation and co-regulation. European Psychologist, 13(4), 277-287. Greeno, J. (2006). Learning in activity, In K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (pp. 79-96). NY: Cambridge University Press Itakura, K. (1997). Kasetsu-Jikken-Jugyo no ABC, Dai 4 han. (The ABC of the Hypothesis-ExperimentInstruction: Invitation to enjoyable classes, Ver.4.) Tokyo: Kasetsu-Sha. [in Japanese] Jonassen, D. (1996). Computers in the classroom: mind tools for critical thinking. New Jersey: Prentice Hall. Jonassen, D. (2000). Computers as mindtools for schools. engaging critical thinking. (2nd ed.): Merrill. Jonassen, D., Howland, J., Marra, R., & Crismond, D. (2008). Meaningful learning with technology. Pearson: Upper Saddle River, NJ. Nussbaum Karpov, Y. (2003). Vygotsky's doctrine of scientific concepts: Its role for contemporary education. In A. Kozulin, B. Gindis, V. S. Ageyev & S. M. Miller (Eds.), Vygotsky's Educational Theory in Cultural Context (pp. 65-82). Cambridge: Cambridge University Press. Koehler, M. J., & Mishra, P. (2009). What is technological pedagogical content knowledge? Contemporary Issues in Technology and Teacher Education, 9(1). Retrieved from www.citejournal.org/vol9/iss1/general/article1.cfm Kozulin, A. (2003). Psychological tools and mediated learning. In A. Kozulin, B. Gindis, V. S. Ageyev & S. M. Miller (Eds.), Vygotsky's Educational Theory in Cultural Context (pp. 15-38). Cambridge: Cambridge University Press. Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A framework for integrating technology in teacher knowledge. Teachers College Record, 108(6), 1017-1054. Miyake, N. (1986) Constructive interaction and the iterative processes of understanding, Cognitive Science, 10(2), 151-177. Miyake, N., Oshima, J., & Shirouzu, H. (2011). "Robots as a research partner for promoting young children's collaborative learning." Proceedings of the 6th ACM/IEEE International Conference on Human-Robot Interaction. Lausanne, Switzerland. Paavola, S., Lipponen, L., & Hakkarainen, K. (2004). Models of Innovative Knowledge Communities and Three Metaphors of Learning. Review of Educational Research, 74(4), 557-577. Popper, K. (1978, April 7). Three worlds. Retrieved 16th March 2010 from http://www.tannerlectures.utah.edu/lectures/documents/popper80.pdfReimann, P., & Thompson, K. (2009). Ecosystem modeling for environmental education: From stocks and flows to behavior and interactions. In P. Blumenschein, D. Hung & D. Jonassen (Eds.), Model-based approaches to learning: Using systems models and simulations to improve understanding and problem solving in complex domains (pp. 111-148). Rotterdam, The Netherlands: Sense Publishers. Salomon, G. (1990). Cognitive effects with and of computer technology. Communication Research, 17(1), 2644. Saito, M., & Miyake, N. (2011) Socially constructive interaction for fostering conceptual change, Proceedings
ISLS
76
ICLS2012
Volume 2: Symposia
of the 9th International Conference on Computer-Supported Collaborative Learning, (CSCL2011), Hong Kong, Sawyer, K. (2006). The Cambridge handbook of the learning sciences. NY: Cambridge University Press Scardamalia, M., & Bereiter, C. (2003). Knowledge Building. In Encyclopaedia of Education. (2nd ed., pp.1370-1373). New York: Macmillan Reference, USA. Sinatra, G. M., & Pintrich, P. R. (2003). The role of intentions in conceptual learning. In G. M. Sinatra & P. R. Pintrich (Eds.), Intentional conceptual change (pp. 1-18). Mahwah, NJ: Lawrence Erlbaum. Smith, C. L., Maclin, D., Houghton, C., & Henessey, M. G. (2000). Sixth-grade students epistemologies of science: The impact of school science experiences on epistemological development. Cognition and Instruction, 18(3), 349-422. Tisue, S., & Wilenski, U. (2004, May). NetLogo: A simple environment for modeling complexity. Paper presented at the International Conference on Complex Systems, Boston. Vygotsky, L. S. (1930/1981). The instrumental method in psychology. In J. V. Wertsch (Ed.), The concept of activity in Soviet psychology (pp. 134-143). Armonk, NY: Sharpe. Vosniadou, S., & Kollias, V. (2003). Using collaborative, computer-supported, model building to promote conceptual change in Science. In De Corte, E., Verschaffel, L., Entwistle, N., & Van Merrienboer, J. (Eds.). Powerful learning environments: Unravelling basic components and dimensions (pp.181-196). Amsterdam: Earli. Vosniadou, S. (2008). International handbook of research on conceptual change NY: Routledge. Young, M. (2004). An ecological psychology of instructional design: learning and thinking by perceiving. In D. H. Jonassen (Ed.), Handbook of Research on Educational Communication and Technology (2 ed., pp. 169-177). Mahwah, NJ: Lawrence Erlbaum.
ISLS
77
ICLS2012
Volume 2: Symposia
Interactive Surfaces and Spaces: A Learning Sciences Agenda

Michael A. Evans, Virginia Tech, 306 War Memorial Hall, Blacksburg VA 24060, mae@vt.edu Jochen Rick, EduTech, Saarland University, Campus C5 4, Saarbrcken, D-66123, j.rick@mx.uni-saarland.de Michael Horn, Northwestern University, 2120 Campus Drive, Evanston IL 60208, michael-horn@northwestern.edu Chia Shen, Harvard University, 33 Oxford St, Cambridge MA 02138, cshen@seas.harvard.edu
Emma Mercier, James McNaughton, Steve Higgins, Elizabeth Burd Durham University School of Education, Leazes Road, Durham, DH1 1TA, UK emma.mercier, j.a.macnaughton, s.e.higgins, liz.burd @durham.ac.uk Mike Tissenbaum, Michelle Lui, and James D. Slotta, University of Toronto, 252 Bloor St. W., Toronto, Canada miketissenbaum@gmail.com, michelle.lui@utonronto.ca, jslotta@gmail.com Roberto Martinez Maldonado and Andrew Clayphan, University of Sydney, roberto@it.usyd.edu.au, andrew.clayphan@sydney.edu.au Abstract: Interactive surfaces and spaces are entering classrooms and other learning settings. This symposium brings together leaders in the field to establish a coherent research agenda for interactive surfaces inside the learning sciences. We demonstrate the broad applicability of these technologies, outline advantages and disadvantages, present relevant analytical frameworks, and suggest themes to guide future research and application.
Supporting Learning with Interactive Surfaces and Spaces

Though the mouse-and-keyboard controlled interface is still the standard in classrooms, museums, and homes, interactive surfaces and spaces are emerging as a prominent alternative in the form of handhelds, tablets, tabletops, and whiteboards, used alone or in provocative configurations. As these devices become more capable and affordable, the case for applying these technologies to support learning becomes increasingly compelling. Now is the time for the learning sciences to get involvedto research the benefits and deficits of these technologies and to develop learning solutions that take advantage of the former while avoiding or compensating for the latter. The goal of this interactive poster session is to exhibit current work in this field.
Gesture, Metapragmatics & Multimodal Analysis Techniques for Surfaces & Spaces
Michael A. Evans Our work examines PreK students (ages 4-5), a group of three boys compared to a pair of girls attempting to solve geometric puzzles on a tabletop (Evans, Feenstra, Ryon, & McNeill, 2011). Results showed significant differences in the way that boys and girls negotiated the task at hand. These differences can be located in the relative frequency of object co-references and task co-references, and also in the function of metapragmatic speech, either to assert ones presence and correctness or to co-create a coherent structure for the interaction. Accordingly, the boys and girls also differed in the way that they involved the teacher in their interactions. Where our work extends the current literature is by investigating multimodal interaction among peers and triads around a multi-touch tabletop surface. For the reported research we focus on the SMART Table, a tabletop computer that projects bottom-up images on a horizontal surface that can be directly manipulated by users simultaneously. The incorporation of the SMART Table in our research allows us to experimentally manipulate learning conditions with changes to the software, requiring less reliance on verbal cues and policy. Thus, our research examined the interrelationships among social constraints (free, divided, and single ownership modes) and instructional technologies (physical and virtual manipualtives) using a multi-touch surface. The questions that follow from this arrangement of social constraints and instructional technologies are these: 1) How does discourse (quality and type) differ among the three levels of social constraint? 2) How does discourse (quality and type) differ between physical and virtual manipulatives? 3) Are there distinct patterns of interaction between social constraint and instructional technologies (i.e., physical and virtual manipulatives? Multi-modal techniques that examine talk, gesture, gaze, and activity are not unknown to the LS research community. Works by Strijbos and Stahl (2007) have demonstrated the benefits of using multimodal techniques to examine collaborative learning. Though Cakir et al. (2010) were investigating the use of a digital whiteboard in a virtual mathematics chat room setting, results from this work corroborate our emphasis on focusing on coreferential or joint problem solving moments in the discourse to find traces or evidence of group cognition. Moreover, the techniques adopted by Cakir et al. (2010) justify our adoption of microgenetic ethnographic methods to identify discrete moments of group cognition. Where our work is distinguished from theirs is the emphasis of co-located interaction and collaboration. The virtual chat room space decreases, or entirely removes, indications rendered by gesture. Though our work aligns in emphasizing gaze and how it might establish a dual space, we extend these efforts by including the gestural component that has been found critical in conveying mathematical ideas. Collaboration emerges when participants explicitly describe themselves as collaborating, or implicitly fit themselves into the structures we associate with collaboration as a way of interacting, such as turn-taking
ISLS
78
ICLS2012
Volume 2: Symposia
structures marked by both time and repeating poetic structures from turn to turn. Within the scheme of references and co-references, metapragmatic speech that functions to make an interaction coherently collaborative is necessarily co-referential (McNeill et. al., 2010). Collaboration operates on a scale larger than a moment; an emergent collaborative character of what we are talking doing now depends on chain linkages of collaborative metapragmatics throughout an interaction.
Proportion: A Tablet Application for Supporting Collaborative Learning

Jochen Rick I designed the Proportion iPad application to research the potential of one table to support collaborative learning. For this application, the tablet is positioned vertically on a table in front of two learners, aged 910. Learners work together to solve a series of ratio / proportion problems. The interface has two columnsone on the left and one on the right. For each problem, the children must size the left and right columns in proportion to their respective numerical labels. Through using Proportion, they gain competence in proportional reasoning.
Figure 1. Four interfaces to scaffold learners in solving problems. Proportional reasoning is a challenging mathematical domain. One problem is that it is usually taught and tested with mathematical notation through word problems; these provide neither feedback on task progress nor tools to scaffold users. Proportion provides several interfaces to scaffold users (Figure 1). Without any support (a), learners must estimate the ratios. Embodied proportional reasoning, relying on rules-of-thumb (e.g., larger denominator means smaller amount) and estimation (e.g., 9 is about twice as much as 4), are particularly important for learners to relate their everyday experiences to mathematical concepts (Abrahamson & Trimic, 2011). Proportion provides two levels of feedback on task progress. If the ratio of the two columns is close to the correct answer, a small star with a close label is shown. If the ratio is within a very small zone, then it is pronounced as correct, a large star with a correct label is shown, and the application moves on to the next problem. With a fixed 10-position grid (b), learners have precise places that they can target, thereby using their mathematical understanding of the task to quickly solve problems. One strategy is for users to select the grid line that corresponds to their respective numbers. This works well for simple ratios (e.g., 4 : 9). For the common-factor problem shown in Figure 1b, that strategy does not work. As illustrated, the children tried a novel strategy of positioning the columns based on the last digit of the number. Of course, this did not work and they were able to realize that this was not a viable strategy. With relative lines (c) that expand based on the position of the columns, learners can use counting to help them solve the problem. They can also learn more embodied strategies, such as maximizing the size of the larger column to make it easier to correctly position the smaller column. When the lines are labeled (d), other strategies can be supported. For instance, in the fractionbased problem shown, a useful strategy is to arrange columns so that whole numbers (e.g., 1) are at the same level. Proportion has been through two rounds of user testing to improve the interface and fine-tune the sequence of problems. The research with Proportion aims to shed light on two broad research topics. First, it will investigate how children communicate to collaborate. Previous work on interactive tabletops has demonstrated that children readily use their interactions with the interactive surface to communicate with their partners (Rick, Marshall, & Yuill, 2011). This work aims to tease apart the role of verbal and gestural communication. Second, it will investigate issues of equity of collaboration for tablet-based collaboration. On tabletops, it becomes difficult for users to access all parts of the surface; therefore, users tend to concentrate their interactions in areas closer to their position at the tabletop (Rick et al., 2009). Such separation is not possible for a tablet: Every user has good access to all parts of the interactive surface. Proportion was designed to have an interface split across the users. Children quickly grasp that they should control the column on their side. Do children stay with this convention? What happens when the convention breaks down? How does this affect the equity and effectiveness of the collaboration?
ISLS 79
ICLS2012
Volume 2: Symposia
Multi-Touch Tabletops in Natural History Museums

Michael S. Horn and Chia Shen In this presentation, we discuss one exhibit that we have designed for this project called Build-a-Tree (or BAT for short). Build-a-Tree is a multi-level puzzle game that asks visitors to construct trees showing the evolutionary relationships of organisms (Figure 1). We represent organisms using black silhouettes superimposed on colored circles that visitors drag around the table. Touching any two circles together joins them into a tree. The reverse action, dragging circles apart, removes an organism from an existing tree. We designed the game levels to be increasingly difficult and to build on one another conceptually. For example, level four asks visitors to construct a tree showing the relationships of spiders, scorpions, and insects. These same relationships appear again as sub-problems in larger puzzles on levels six and seven. We conducted a study of Build-a-Tree at a well-known university natural history museum. In the study, we recruited and video recorded 30 family groups (consisting of at least on parent and one child) using the exhibit. We transcribed visitor conversation up to and including the first six levels of game play.
Figure 1. Screenshot of the Build-a-Tree game (left) and visitors playing the game in the museum (right). Based on these data, we iteratively developed a coding scheme that included three high-level categories: game talk, content talk, and off-topic talk. Among our findings, we saw that visitors spent long periods of time interacting with the exhibit (14 minutes on average; SD = 6) and engaged in minimal off-topic conversation (off-topic utterances occurred 0.16 times per minute on average; SD = 0.3). However, while engagement was high, evidence of visitor reasoning about evolutionary relationships was less satisfactory. Content related talk made up 26.6% of the overall conversation, but statements about relationships were rarely backed up with explanation. We see this study as one point in an ongoing design-based research effort, and we are in the midst of updating our design based on this analysis to increase evolutionary reasoning in visitor conversation. In attempting to explain these results, it was clear that game talk played a significant role in helping visitors coordinate their activity around the tabletop. For example, negotiating turn-taking was a common mechanism for groups consisting of multiple children. This is perhaps not surprising. In ethnographic research of children playing video games in homes, Stevens, Satwicz, and McCarthy (2007) observed a variety of learning arrangements that young people spontaneously adopted to support game play. In effect, when visitors encountered our game in the museum, they already had a repertoire of social practices on hand to support collaborative interaction. Identifying similar ways to cue effective social practices will be critical to support the use of tabletop surfaces in informal learning environments.
Orchestrating Learning in the Multi-touch Classroom: Developing Appropriate Tools

Emma Mercier, James McNaughton, Steve Higgins, Elizabeth Burd Interactive surfaces provide new ways to support collaborative learning with technology, however, new technology in classrooms has rarely lived up to its promise, in part because less attention has been paid to the pedagogical and classroom orchestration issues of introducing new technology (Dillenbourg & Jermann, 2010; Slotta, 2010). The SynergyNet multi-touch classroom (Higgins, et al., 2011) was designed with four networked, multi-touch student tables, teacher tools and an interactive whiteboard (IWB) that allow for the movement of content between the tables and to the IWB, and teachers tools to manage table functions, deliver content and monitor activities. Over two years we have developed tools for the teacher to use, iteratively designing the teacher controls as we learn more about the orchestration needs of a multi-touch classroom.
Stage 1: The Multi-touch Classroom and Orchestration Desk

Initially teacher orchestration tools were on an angled multi-touch table (Fig. 1). These controls were used to select activities for the class, send content to and monitor the student tables, and project content to the IWB.
ISLS
80
ICLS2012
Volume 2: Symposia
This classroom was used to study a series of collaborative learning activities; two researchers with prior teaching experience taught the classes. For each of the activities, the teacher started the activity by sending the content to the student tables, then walked around the room, intervening to support groups where necessary, and locking the tables for whole-class discussions. For this the teacher projected content from one of the student tables to the IWB; the group whose table was displayed described their process and ideas. The teachers noted that not having access to controls as they moved around the room constrained their ability to take an interesting idea from one group to the whole class to support uptake of student ideas.
Stage 2: Mobile Orchestration Devices

To allow control while the teacher moved around the room, the teacher controls were transferred from the orchestration desk to an iPad (Fig. 2), which the teacher carried and used to make changes to the activities and view information about students contributions to tasks. The control features allow the teacher to start, pause or end an activity, project the content of a student table to the IWB, and move content from one table to the next. In addition, the teacher can view the work that the students are doing to all quick intervention if a student appears to be struggling. In one study, a member of the research team taught three classes using the iPad, reporting that the ability to control the activities from anywhere in the room was useful, but that having to hold and interact with the tablet took away from some of the ability to support the students by working directly on the student tables.
Stage 3: Gesture-based Teacher Controls

Currently, we are moving away from using direct-touch devices for orchestration, building on tracking capabilities of depth cameras such as the Microsoft Kinect. The teacher is identified by the Kinect, which is mounted on the teacher orchestration desk. Using a series of pre-defined gestures the teacher can manage the student tables from anywhere in the classroom (Fig 3), including locking and unlocking the students tables, taking screen-shots of a particular table and projecting the content from one table onto the IWB. These should provide the teacher with enough control to manage the technology, while future development will explore the use of teacher-specific tools that the teacher can activate on any of the interactive surfaces in the room, building on the recognition capabilities of the Kinect, which would allow the teacher to conduct more complex functions without the need for an additional device. Issues of orchestrating collaborative learning in a technology enhanced classroom are concerned with both the management of the technology and ways in which technology can provide additional information for optimal teacher intervention when groups get stuck or go astray. While our orchestration tools have been designed to support both needs, some of the management issues and technical interaction and control challenges must be solved before a teacher can give all their attention to the learning activities of their students.
Gesture, Scaffolding Collaborative Knowledge Construction in High School Physics with Tablet Computers and interactive White Boards
Mike Tissenbaum and James D. Slotta To effectively integrate collaborative technologies and practices into Science, Technology, Engineering, and Math (STEM) curriculum many learning theorists have advocated a knowledge community approach, where students work together to investigate salient issues, collaboratively develop theories, build ideas, and develop conclusions with technology as a central scaffold to the learning process (Brown & Campione,1996). Handheld tablets are increasingly popular within these technology mediated environments, as they allow for a 1:1 device to student ratio which can improve the organization and distribution of materials, and the coordination, communication, and negotiation of students within real-time activities (Zurita & Nussbaum, 2004). However, handheld screen sizes can inhibit group collaboration, and as such, should be augmented with large-format displays in order to better facilitate small group and whole class interactions (Tissenbaum, Lui, Slotta in press). By using large, interactive surfaces, we can more easily mimic the ecological dexterity of paper, allowing students to rearrange, reassemble, and annotate aggregated products of prior interactions (Everitt et al., 2005). The use of such technologies can allow students to collaboratively interact within an information space, build shared representations, and allow students to slide information between their own device and these representations. Another aspect of this combination of technologies (i.e., within a single physical setting) is the ability to dynamically adjust the representations and scripts sent to devices within the room. This capacity to
ISLS
81
ICLS2012
Volume 2: Symposia
orchestrate the pedagogical flow of students, materials and activities can facilitate specific instructional designs, and serve to embody the broader epistemological goals of the research (Tissenbaum, Lui, Slotta, in press). The room could conceivably respond to individual students, moving them between groups or activities, or sending relevant materials. To achieve these complex pedagogical moves we require a robust technology infrastructure, including a role for intelligent software agents that can perform data mining and help coordinate pedagogical flow (Slotta, 2010). To this end, our group has been developing a flexible, open source smart classroom framework called SAIL Smart Space (S3) that supports the integration of technologies, including the aggregation, interpretation and response to student activities. The goal is to develop a technology environment capable to supporting a wide range of collaborative inquiry and knowledge construction scripts including interactive media and multi-touch surfaces. Our current design involves two grade 12 high school physics classes each involved in a two-day smart classroom activity, where students developed their understandings of real-world physics phenomena, in part through classifying physics problems, mapping those problems to the phenomena, and setting up problems concerning the phenomena, in ways similar to physics experts (Chi, Feltovich & Glaser, 1981).
Figure 1: In-Class collaborative equation negotiation
Figure 2: Smart Classroom configuration (Step 1)
Figure 3: SMART Board aggregated principle screen
Students first worked at home to sort various physics problems according to a core set of underlying principles (Newtons laws, conservation of energy, and equations of motion), before working in small groups in a unique form of equation sorting and negotiation towards solving assigned problems. Throughout the activity intelligent agents performed student grouping based on past work, and tracked in-class activity, providing student groups feedback about their progress on consensus on their final equation sorts (Figure 1). The teacher used his own touchscreen tablet to flip through the aggregated group pages (like pages in a book), gaining insight into groups progress. During the smartroom activity students entered the room (Figure 2) in batches of 12, and divided into 4 groups of 3. Each student was given a set of sub-principles from one of three larger themes, and moved from station to station (by a traffic flow agent) and: (1) watched a video clip of a popular Hollywood movie that illustrated or violated one or more physics principles, and (2) flung (swiped from tablet to wall) any of their assigned principles at the video wall, before moving on to the next station as directed by the agent. As students moved throughout the room, flinging principles, the aggregated collections appeared on the SMART boards at the front (Figure 3). Once completed, agents regrouped students to one of the four stations based on their tags from the previous step and provided a scaffolding question to help them approach the video as a real physicist would. Using their tablets students submitted assumptions about the scenario within the video (e.g., the weight or speed of a car, if relevant to solving the implicit problem) and collectively debated these assumptions, towards forming consensus. In the next pedagogical step, students moved to a new station, where they saw the assumption of the previous group and received a small set of problems, assigned by an agent based on the principles attached to the video (from the homework stage). Their task was to pick the problem that most directly helped in understanding the video, and its related equations (attached during the preactivities). The groups were reconfigured for a final step, where they used the work of previous groups to order the sequence of equations towards solving the implicit physics problem by dragging equations to a field on their tablets and negotiating any conflicts (similar to the Day 1 activity). The activity concluded with the teacher engaging the class in a debrief around the final results and using the SMART boards to show the evolution of the class constructed knowledge. This intervention highlights how a smart classroom setting coupled with tangibly interactive technologies can achieve a variety of class configurations, interactions with student generated content, and epistemological goals that could not be achieved by traditional paper and pen approaches and physical layouts. Our demo will create a simplified activity that can be performed by session participants within the context of
ISLS
82
ICLS2012
Volume 2: Symposia
this symposium (i.e., in 15 minutes) and also provide a video of our actual smart room session and accompanying poster.
EvoRoom: An Immersive Simulation for Smart Classrooms

Michelle Lui and James D. Slotta EvoRoom is a room-sized simulation of a rainforest environment designed for a smart classroom in an urban high school. Collaborating closely with a biology teacher, we designed immersive learning activities for teaching topics on biodiversity and evolution. These immersive simulation activities engages students in collective inquiry, where students take on the role of field researchers and use aggregated student inputs and collective knowledge to solve a problem or to understand a phenomena. Students work in various group configurations (e.g., individually, small groups, and as a collective) to complete tasks delivered to them on their personal tablet computers. The tablets, which contain custom designed application, provide scaffolds, place students in small groups, and give real-time updates and dynamic resources. Student contributions (e.g. written reflections) are aggregated and displayed on interactive whiteboards (IWBs) as they come in, and stored within a collective knowledge base. There are a number of surfaces involved in an immersive simulation, and each affords a different interaction (e.g., student-to-student interactions, student-to-teacher, and student-to-digitalartifacts). This poster examines how students engaged with the various components of EvoRoom. Below, we describe the environment, outline one of the immersive activities and detail the outcomes with respect to the various surfaces involved in the sessions, relating student engagement to the kinds of interactions that these components encourage. Our technology framework, Sail Smart Space (S3) supported the activity by providing a portal that where students login to the experience, an intelligent agent architecture that tracked real-time communications, and a central database that housed curriculum materials and the products of student interactions (Slotta, 2010). The room is set up with two sets of large projected displays (6 meters wide, achieved by stitching together 3 projector displays) that students would examine together (Figures 1, 2). The simulation files are networked and controlled with a custom tablet application, allowing the teacher to manage the time spent in each portion of the activity, controlling the pedagogical flow within the room. Two IWBs are located at the front of the room to display aggregated student inputs, which both teacher and students manipulate as a locus of discussions. In this study, 45 students from two sections of our co-design teachers grade 11 Biology course spent 75 minutes (i.e., one class period) working on the activity. Students were video recorded and student artifacts (e.g. written notes) were collected for analysis. A post activity questionnaire that queried student perceptions of environment was answered by 39 of 45 students.
Figure 1. Smart classroom setup
Figure 2. Three projectors make up a sidewall of EvoRoom
Students who visited EvoRoom were each given a tablet to work with; on the walls were different versions of the rainforest ecosystem, each showing an outcome from a specific environmental variable (e.g., high rainfall, tsunamietc.). We designed a tablet-based interface that challenged students to explore the differences between these rainforests and to match them to the different environmental variables. As part of the activity, a sorting agent assigned students different organisms to look for. When students scanned QR codes at the different stations, the agents recognized their location and sent further, contextualized instruction to the tablets. For each station, students were prompted to record whether their assigned organisms were present. All responses were aggregated on the IWB at the front of the room, presenting a tally for reference by teachers and students alike. Students worked in groups of two or three to begin eliminating rainforest stations that likely did not result from their assigned variable. During group activities, different tasks were distributed to different students in each group. For example, in a group of three, one person was designated to be the scribe, while the second group member was instructed to look up information from field guide, with the third assigned to look up prediction from their class website (accessed as a link on the tablet). All of the group's decisions and notes were
ISLS
83
ICLS2012
Volume 2: Symposia
collected on the IWB at the front of the room. The aggregated results were all shown on the IWB, which the teacher used to lead a discussion of how the variables affect the flora and fauna of the rainforest ecosystem. Our analysis of student performance and perceptions in EvoRoom uncovered several findings about student learning in immersive environments. Results of the classroom trial indicate that students are able to effectively allocate their attention between the immersive simulation and the various technologies supporting their tasks (e.g., tablets, laptops). Of the 16 groups that participated in the activity, 8 groups (50%) correctly identified the related rainforest as their first choice, while 4 (25%) groups chose the correct rainforest as their second choice. Video analysis will allow researchers to better understand which patterns of interactions were more conducive to supporting collective inquiry. On a scale of 1 to 10 (with 1 being unsuccessful and 10 being very successful), students rated the use of personal tablets in the smart classroom with an average of 7 (SD = 2.76). The immersivity of EvoRoom was rated highly, at an average of 9 out of 10 (SD = 1.48). One student indicated that the QR scanning feature offered a more hands-on experience compared to, for example, selecting an option from a list of item. On the whole, we found that students were excited about the immersivity of EvoRoom and the use of tablet computers for supporting their learning.
Analysing and Supporting Collaborative Learning at the Interactive Tabletop

Roberto Martinez Maldonado and Andrew Clayphan One of the major challenges faced when developing computer-supported collaborative learning tools is the provision of adequate support to the activities according to the particular needs of the group members. In group work, teachers often divide their attention among groups and they typically focus on the groups which they think need more advice and guidance. As a result, they typically only see the final product. The process to build such a solution and individual contributions may be hard to determine. Emerging pervasive shared devices, such as multi-touch interactive tabletops, are very promising, providing an enriched learning environment and a medium to capture groups social dynamics to adapt the affordances of the tool to students and their teachers needs. We present Collaid (Martinez et al., 2011), a digital framework to support collaboration at the classroom by providing both support to learners and awareness tools for their teachers in a multi-tabletop classroom. Collaid augments the capabilities of the hardware with a set of key features: identification of proximity of learners around a tabletop and authorship of each touch (using a depth sensor situated on top of the surface device), recognition of people speaking and capture of multimodal data about collaboration (video, audio and application logs in a common repository). Any touch or oral participation around the tabletop is automatically traced, with its author, and logged.
Figure 1. Examples of Collaid in use. We show how personal devices can be used to establish a link so to identify the source, provide personalisation and tracking of learners during activities and across activities. We demonstrate how our system uses technology for ways to capture the digital footprints of users without impacting the main task. This provides the opportunity to model groups participation and mirror back indicators of collaboration to their teachers so they can best orchestrate the collocated collaborative process. Collaid is presentedin two learning domains: collaborative concept mapping and brainstorming. Concept mapping is a technique that permits learners to visually externalise a representation of their knowledge about a topic. It permits group members to confront different perspectives to solve misunderstandings and build new knowledge. The interface permits learners to merge their individual concept maps in face-to-face sessions at the tabletop. Secondly, we demonstrate brainstorming (Clayphan et al., 2011) on an interactive digital tabletop as a collaborative activity to help groups generate original ideas encouraging egalitarian and cooperative participation. Digital tabletops combine natural face-to-face discussion found in conventional settings with increased flexibility gained from computerised support. Tabletops allow collaborative process to be captured, helping measure learner
ISLS
84
ICLS2012
Volume 2: Symposia
engagement. We present work on tabletops and brainstorming with scripted collaboration around user control. We identify stages in the process and enable mechanisms to guide learners with the aim of improving support. We demonstrate the system with a recent integration of user tracking, allowing deeper metric collection and understanding of active contribution.
Conclusion
As demonstrated in this symposium, interactive surfaces and spaces offer a new computing paradigm to supplement, complement, or even supplant the desktop user interface. Learning scientists and educational technologists adopting these technologies have begun to carve niches for themselves in education. Over time, more powerful hardware will become available at a more affordable price, software development environments will improve, more applications will be available, and our understanding of how to use them to support learning will increase. Consequently, these interactive environments will play an increasing role in research and in the general support of learning.
References
Abrahamson, D., & Trimic, D. (2011). Toward an embodied-interaction design framework for mathematical concepts. In Proceedings of IDC 11 (pp. 110). New York: ACM Press. Brown, A. & Campione, J. C. (1996). Psychological theory and the design of innovative learning environments. 289325. Cakir, M.P., Zemel, A., & Stahl, G. (2010). The joint organization of interaction within a multimodal CSCL medium. International Journal of Computer-Supported Collaborative Learning,4(2), 115-149. Chi, M. T. H., Feltovich, P. J., & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5(2), 121-152. Clayphan, A., Collins, A., Ackad, C., Kummerfeld, B. and Kay, J. Firestorm: A brainstorming application for collaborative group work at tabletops, Proc. of Interactive Tabletops and Surfaces ITS'11, ACM (2011), pages 162-171. Dillenbourg, P., & Evans, M. (2011). Interactive tabletops in education. International Journal of ComputerSupported Collaborative Learning, 6(4), to appear. Evans, M.A., Feenstra, E., Ryon, E., & McNeill, D. (2011). A multimodal approach to coding discourse: Collaboration, distributed cognition, and geometric reasoning. International Journal of Computer Supported Collaborative Learning, 6(2), 253-278. Higgins, S. E., Mercier, E. M., Burd, E., & Hatch, A. (2011). Multi-touch tables and the relationship with collaborative classroom pedagogies: a synthetic review. International Journal of Computer-Supported Collaborative Learning, 6(4), 515538. Martnez R., Collins A., Kay J., Yacef K., Who did what? Who said that?: Collaid: an environment for capturing traces of collaborative learning at the tabletop, Proc. of Interactive Tabletops and Surfaces ITS'11, ACM (2011), pp. 172-181. McNeill, D., Duncan, S., Franklin, A., Goss, J., Kimbara, I., Parrill, F., Welji, H., Chen, L., Harper, M., Quek, F., Rose, T. & Tuttle, R. (2010). Mind-Merging. In a Festschrift for Robert Krauss, Ezequiel Morsella (ed.), Expressing oneself / expressing one's self: Communication, language, cognition, and identity. London: Taylor and Francis. Rick, J., Harris, A., Marshall, P., Fleck, R., Yuill, N., & Rogers, Y. (2009). Children designing together on a multi-touch tabletop: An analysis of spatial orientation and user interactions. In Proceedings of IDC 09 (pp. 106114). New York: ACM Press. Rick, J., Marshall, P., & Yuill, N. (2011). Beyond one-size-fits-all: How interactive tabletops support collaborative learning. In Proceedings of IDC 11 (pp. 109117). New York: ACM Press. Slotta, J. D. (2010). Evolving the classrooms of the future: The interplay of pedagogy, technology and community. In K.M\{a}kitalo-Siegl, F. Kaplan, J. Zottmann & F. Fischer (Eds.), The classroom of the future orchestrating collaborative learning spaces (pp. 215-242). Rotterdam: SensePublisher. Stevens, R., Satwicz, T., & McCarthy, L. (2007). In-game, in-room, and in-world: Reconnecting video game play to the rest of kids lives. In K. Salen (Ed.), The Ecology of Games: Connecting Youth, Games, and Learning, pp. 4166. MIT Press Strijbos, J. W., & Stahl, G. (2007). Methodological issues in developing a multi-dimensional coding procedure for small group chat communication. Learning & Instruction. Special issue on measurement challenges in collaborative learning research, 17(4), 394-404. Tissenbaum, M., Slotta, J. D., & Lui, M. Co-designing collaborative smart classroom curriculum for secondary school science. Jorunal of Universal Computer Science. Zurita, G., & Nussbaum, M. (2004). Computer supported collaborative learning using wirelessly interconnected handheld computers. Computers \& Education, 42(3), 289-314.
ISLS
85
ICLS2012
Volume 2: Symposia
Instructional Approaches to Promote Conceptual Change

Stellan Ohlsson (stellan@uic.edu) Stella Vosniadou, Department of Philosophy and History of Science, National and Kapodistrian University of Athens, Greece, (svosniad@phs.uoa.gr) Naomi Miyake, School of Computer and Cognitive Sciences, Chukyo University, Japan, (nmiyake@sist.chukyo-u.ac.jp) David D. Cosejo, Department of Psychology, University of Illinois at Chicago, USA, (dcosej1@uic.edu) Marcia Linn (mclinn@berkeley.edu) Doug Clark, Graduate School of Education, University of California at Berkeley & Vanderbilt University, USA, (doug.clark@Vanderbilt.Edu) Michael J. Jacobson , Education and Social Work, University of Sydney, Australia, (michael.jacobson@sydney.edu.au) Keywords: learning, conceptual change, re-categorization, instructional analysis, integration, classroom discussion
Introduction
Learning usually involves the enrichment of prior knowledge and constructive approaches to learning and instruction emphasize the activation of prior knowledge so that the new information can be productively integrated into what is already known. What happens however when the new, to-be-learned, information is highly counter-intuitive and contradicts what is already known? What are the instructional interventions that can be used to help students restructure what they already know and achieve conceptual change? The problem of belief revision in the process of conceptual change is one of the most important problems in learning and instruction research and a problem that unfortunately has not received adequate attention so far. The purpose of the present symposium is to present current theoretical analyses of this problem and to discuss constructivist instructional approaches that attempt to deal with it. Researchers coming from different perspectives will present new empirical evidence as well as new theoretical accounts of the processes involved in conceptual change and the instructional interventions that can facilitate it. A number of different proposals will be presented and their implications for the design of curricula and instruction will be discussed. Attention will be given to the role of implicit learning, to explicit hypothesis testing, to the importance of integrating students ideas, to the deliberate use of instructional analogies, and finally to productive use of collaboration and classroom discussion. Even though the problem of conceptual change is fundamental for a theory of learning it has had little presence so far in the conferences of the International Learning Science Society. The aim of this symposium is to bring some of the current discussions and different perspectives on instructional interventions to promote conceptual change to the attention of the learning science audience. All the speakers are key developers of the different approaches and have led their development.
Explicit and Implicit Processing in a Laboratory Model of Conceptual Change

Stellan Ohlsson and David D. Cosejo. Conceptual change is a temporally extended process with infrequent and course-grained expression in observable behavior. Developmental psychologists, educational researchers and historians of science work with temporally scarce data that provide scant information about the processes involved. To study these processes under controlled conditions, we have developed a laboratory model of conceptual change that we call recategorization. In a re-categorization paradigm, participants learn an unfamiliar category, to criterion, at which point the target category is re-defined. The participants are not informed of this but continue the standard categorization cycles of stimulus presentation, categorizations, and feedback until the revised category has been learned to criterion. The question is how, by what processes, the initially acquired category the misconception is unlearned, and the revised, target category is acquired. We present some preliminary data obtained with this paradigm that supports the view that conceptual change (in this paradigm) is an interaction between explicit and implicit processing. Explicit processing such as deliberate hypothesis testing is sequential and hence strongly impacted by the complexity of the category to be learned, while implicit processing (e.g., automatic updating of association strengths) occurs in parallel and hence is less affected by category
ISLS
86
ICLS2012
Volume 2: Symposia
complexity. The relation of the laboratory model to naturalistic conceptual change is raised, and the question of how the model can be improved to capture more of naturalistic processes is discussed.
Using instructional analogies to promote the comprehension of counterintuitive text

Stella Vosniadou Learning science often requires children to radically restructure their nave explanations of physical phenomena. Are children successful in engaging in such restructuring processes and can instructional analogies help in this process? For sometime now we have been investigating primary and secondary school students comprehension of counter-intuitive text in science and mathematics with and without the use of verbal analogies and/or dynamic visual models. Our results showed that students fail to understand even the most fundamental ideas in counter-intuitive science and math texts. In addition, the students often generate erroneous inferences which reveal intrusions from prior knowledge and which are similar in many respects to the misconceptions, or synthetic conceptions, revealed in cross-sectional and longitudinal developmental studies. We hypothesized that analogies would have the potential to help in the restructuring process because they draw upon students existing knowledge from a different but familiar domain to facilitate the understanding of a new and unfamiliar explanation. We also hypothesized that the presence of a dynamic visual model would facilitate the comprehension of the scientific explanation not only because it draws on familiar past knowledge but also because it can make more explicit the structure and function of the explanatory mechanism. Results confirmed both hypotheses showing significant gains for the analogies and dynamic models groups compared to the controls. They also showed that the interventions were not equally successful for all students but had the greatest effect on the children who were in the transition to understanding the scientific explanation and had rejected some of the presuppositions of their nave theories that constrained their understanding of the scientific explanations. In general, the students who were able to understand the analogies were also able to use them to restructure their physical explanations.
Synergies in conceptual change perspectives

Marcia Linn & Doug Clark Seeking similarities between varied perspectives on conceptual change helps to clarify the field and offers a foundation for future research (Clark & Linn, in press). We identify synergies between framework (Vosniadou, 2008) and elemental (Clark, 2006; diSessa, 2008; Linn & Eylon, 2011) perspectives. Researchers offer differing perspectives about whether students understandings in science are better characterized as a coherent unified scheme of theory-like character or an ecology of quasi-independent elements or ideas. But the disagreement is a matter of degree. To identify synergies we consider a graduated spectrum for the magnitudes of influence that elements may exert on one another. We draw on research exploring knowledge integration to clarify the synergies. An examination of the diverse, creative, and unique ideas that students formulate has led researchers to argue for a constructive process of knowledge generation and change. When students are asked to explain scientific phenomena in an abstract de-contextualized frame, they often respond quite differently from when asked to explain an observed phenomenon. Analyzing the conceptual change processes involves explaining how students make sense of ideas when they occur in a new situation. Students methods for grappling with new observations strengthen the view that students engage in a creative process of trying to make sense of their world (Hatano & Inagaki, 2003). Many researchers have focused on the diversity and character of student ideas. These ideas have been referred to as misconceptions, alternative conceptions, beliefs, intuitive ideas, and constructed ideas. Any theory of conceptual change needs to explain the emergence of these views, as well as their role in conceptual change. A large body of evidence has led researchers to argue that students construct multiple, contradictory, and fragmented ideas that stem from their interactions with the material and social world. Many researchers have shown that the ideas students generate arise from observations, analogies with related events, cultural practices, or colloquial uses of language.
ISLS
87
ICLS2012
Volume 2: Symposia
The knowledge integration perspective on conceptual change has emerged through a series of empirical studies. It was spurred by evidence of the impact of context on student reasoning (Linn, 1983). It celebrates the ideas students generate and views these ideas as intellectual accomplishments rather than intellectual constraints (Linn & Hsi, 2000). Important evidence for this view comes from a longitudinal study carried out over five years that gives insight into student lifelong learning (Clark, 2006; Clark & Linn, 2003; Linn & Hsi, 2000). This longitudinal study illustrated how students maintain conceptual ecologies involving multiple conceptual elements and ideas at various levels of sophistication, connection, and conflict. These conceptual elements and ideas include cultural and observational information and beliefs spanning both epistemological and ontological aspects of knowing and learning. As with Vosniadou et al.s (2008) framework theory perspective and diSessas knowledge in pieces perspective (1993), the knowledge integration perspective acknowledges that ideas can be introduced (through schooling and other experiences) that result in conflicts, fragmentation, or integration. Learning occurs through a process of restructuring and reorganizing these ideas. These studies reveal underlying (although often unarticulated) ideas that shape students thinking, explanations, and predictions. Students use these multiple ideas to interpret the phenomena they encounter in their everyday lives. The particular ideas students consider and connect depend on contextual cues. Some connections arise from experience (e.g., metals feel cold), some connections are situationally specific and less broadly useful (e.g., cooling on the stove is different from heating), some are imported from another domain such as electricity and may or may not be useful or accurate in the new domain (e.g., glass is not a conductor of electricity so it will be a poor thermal conductor), and some have their roots in classroom instruction (e.g., metals have heavier molecules). Some connections that students make are spontaneous and ephemeral, whereas some connections are much more durable and persistent. As students learn, they reorganize, reconnect, and sort through their ideas. Some ideas become much more central and pivotal as a student uses them as focal points around which to integrate other ideas, while other ideas are demoted in priority and centrality. The knowledge integration framework emphasizes creating opportunities so students can productively distinguish among their ideas to achieve conceptual change and coherent understanding. Researchers have shown that the ideas students articulate to make sense of school and everyday situations illustrate students capabilities to sort out confusing observations rather than illustrating developmental constraints (e.g., Gilbert & Boulter, 2000; Redish, 2003). For example, students often argue that metal must be a naturally colder material because metal feels cold at room temperature. Some researchers see these efforts as evidence for powerful reasoning ability that can be guided by instruction. More specifically, when students make an effort to sort out ideas, even if the view they formulate is not supported by all the empirical data, they are engaging in the sort of reasoning that can lead to understanding. In summary, the knowledge integration framework calls for capitalizing on students ability to make sense of scientific phenomena by empowering them to consider new ideas, distinguish among existing and new ideas, and promote the most promising ones. Students generate a broad range of ideas about any scientific phenomenon. These ideas represent multiple types of explanations, vary across contexts, and may not be recognized as applying to the same topic. The knowledge integration framework takes these ideas as building blocks and mobilizes the same processes that generated them to focus the learning trajectory on coherent understanding. Synthesizing framework theory perspectives with the knowledge integration perspective and other elemental perspectives reveals overlaps and consistencies. In particular, the magnitude of influence of certain ideas from framework theory perspectives can mesh well with the focus on the rich interactions within conceptual ecologies highlighted by elemental perspectives. These synergies clarify and strengthen the accounts of conceptual change collected across studies. This integrated perspective also supports the value of instructional sequences, such as the knowledge integration pattern, that scaffold students in refining and consolidating their conceptual ecologies around productive focal ideas. Essentially, highlighting the magnitude of influence of certain ideas in students conceptual ecologies clarifies systematicities in student explanations while respecting the rich range of ideas and interactions between ideas. This approach clarifies the nature and process involved in the fragmentation of ideas and emergence of synthetic models. It also sheds light on the difficulties students have when grappling with the abstract formal ideas introduced in science classes.
Designs and analyses of multi-person constructive interaction in real classrooms for adaptive conceptual change
ISLS
88
ICLS2012
Volume 2: Symposia
Naomi Miyake The University of Tokyo This presentation aims at clarifying structures of successful classroom discussion that support learners to change their folk knowledge into scientific understandings. It has been suggested that sequential, cumulative discussion across classes has strong positive effects to help learners expand their understandings adaptively (e.g., Hatano & Inagaki, 1991). Yet, the details of such classroom discussion have rarely been analyzed fully to explore whether there is some specific structure leading to adaptive conceptual change. In this talk I report the results of our analyses of two types of classroom discussions, the Hypothesis-Experiment Instruction (henceforth HEI; Itakura, 1997) and the Knowledge Constructive Jigsaw classes (henceforth KCJ; Miyake, 2011). To analyze these two types of classroom activities, I adopt the frameworks from previous research on collaborative conceptual change (Roschelle, 1992) and two-person, constructive interaction for the abstraction of levels of understanding (Miyake, 1986). The combined framework requires that there occur interaction, or role exchange, between the task-doer engaged in explaining or externalizing the on-going problem solving at hand and the monitor who watches over such development and tries to integrate her/his own understanding. It also suggests that the progression of the aimed conceptual change could be analyzed in terms of levels of understanding, from more directly related to every-day, mundane understanding to more abstract, scientifically acceptable understanding. Choosing classrooms where the students succeeded in attaining primitive yet scientific understandings, I analyzed their discussion patterns to see whether there occurred the role-exchange and if so in what forms. The relationship between the shifts of the identified patterns and the levels of understandings of the students, exhibited in forms of their conversations, class presentations, and the end-of-theclass written reports. For the HEI, four classes of third graders were analyzed (Saito & Miyake, 2011). The topic was to understand physical identity of objects, or to understand that two objects like air and water cannot share the identical physical space to be specific. Overall, our analyses show that the kids discussion represents the socially expanded version of two-person, constructive interaction, where not only an individual student but also small groups of students could serve the role of task-doing while the rest of the class monitors. The proportion of the two roles differs from class to class, but more group-based interaction tended to lead to changes of understandings. The class design framework of the KCJ involves a shared question to be answered and some relevant learning materials from different perspectives that are distributed among the different groups first in expert groups, to be later exchanged and integrated to answer the question in the jigsaw groups (Miyake, 2011). The design naturally requires each student become a task-doer in the jigsaw group, yet the proportion of the two roles could differ from individual to individual. We have analyzed data from several classes of different grade levels from elementary to middle school on different topics of science. The analyses of three classes of middle schools on different topics have revealed that there is little correlation between the achievement levels and the proportion of the role exchange. Rather, the monitors, who could spend almost the entire class without speaking up, tended to learn a lot from just attending to others talks and inwardly working to integrate such inputs to their own understandings. I will report some evidence for this silent yet active learning identifiable from the memos taken during the class by such monitors.
References
Clark, D. B. (2006). Longitudinal conceptual change in students' understanding of thermal equilibrium: An examination of the process of conceptual restructuring. Cognition and Instruction, 24(4), 467-563. Clark, D. B., & Linn, M. C. (In press). The Knowledge Integration Perspective: Connections Across Research and Education. In S. Vosniadou (Ed.), Handbook of Research on Conceptual Change (2nd ed.). New York: Routledge. Clark, D. B., & Linn, M. C. (2003). Scaffolding knowledge integration through curricular depth. Journal of Learning Sciences, 12(4), 451-494. DiSessa, A. (2008). A Birds-Eye View of the Pieces vs. Coherence Controversy (From the Pieces Side of the Fence). In S. Vosniadou (Ed.), Handbook of Research on Conceptual Change. New York: Routledge. Gilbert, J. K., & Boulter, C. J. (2000). Developing models in science education. Dordrecht: Kluwer. Hatano, G. & Inagaki, K. (1991). Sharing cognition through collective comprehension activity. In B. Resnick, J. M. Levine, & S. D. Teasley (Eds.). Perspectives on socially shared cognition. 331-348. APA Hatano, G., & Inagaki, K. (2003). When is conceptual change intended?: A cognitive-sociocultural view. In G. M. Sinatra & P. R. Pintrich (Eds.), Intentional conceptual change (pp. 407-427). Mahwah, NJ: Lawrence Erlbaum Associates.
ISLS
89
ICLS2012
Volume 2: Symposia
Itakura, K. (1997). Kasetsu-Jikken-Jugyo no ABC, Dai 4 han. (The ABC of the Hypothesis-ExperimentInstruction: Invitation to enjoyable classes, Ver.4.) Tokyo: Kasetsu-Sha. [in Japanese] Linn, M. C. (1983). Content, context, and process in adolescent reasoning. Journal of Early Adolescence, 3, 6382. Linn, M. C., & Eylon, B.-S. (2011). Science Learning and Instruction: Taking Advantage of Technology to Promote Knowledge Integration. New York: Routledge. Linn, M. C., & Hsi, S. (2000). Computers, Teachers, Peers: Science Learning Partners. Mahwah, NJ: Lawrence Erlbaum Associates. Miyake, N. (1986) Constructive interaction and the iterative processes of understanding, Cognitive Science, 10(2), 151-177. Miyake, N., (2011) Fostering conceptual change through collaboration: Its cognitive mechanism, socio-cultural factors, and the promises of technological support, Proceedings of the 9th International Conference on Computer-Supported Collaborative Learning, (CSCL2011), Hong Kong Redish, E. F. (2003). Teaching Physics with the Physics Suite. New York: John Wiley and Sons, Inc. Roschelle, J. (1992). Learning by collaboration: convergent conceptual change, The Journal of the Learning Sciences, 2, 235-276. Saito, M., & Miyake, N. (2011) Socially constructive interaction for fostering conceptual change, Proceedings of the 9th International Conference on Computer-Supported Collaborative Learning, (CSCL2011), Hong Kong Vosniadou, S. (Ed.). (2008). International Handbook of Research on Conceptual Change. New York: Routledge.
Acknowledgment
The work entitled Using instructional analogies to promote the comprehension of counter-intuitive text (Stella Vosniadou) is supported by the project ANALOGY: Human-The Analogy Making Species, financed by the FP6 NEST Program of the European Commission. (STREP Contr. 029088).
ISLS
90
ICLS2012
Volume 2: Symposia
Everyday Interactions and Activities: Field Studies of Early Learning Across Settings
Reed Stevens, Lauren Penney, Danielle Keifert, Pryce Davis Northwestern University, School of Education and Social Policy 2120 Campus Drive, Evanston IL, 60208 USA Email: reed-stevens@northwestern.edu, lauren.penney@u.northwestern.edu, keifert@u.northwestern.edu, pryce@u.northwestern.edu Siri Mehus, University of Washington, College of Education 1100 NE 45th St., Suite 200, Box 354941, Seattle, WA 98195-3600 USA Email: smehus@u.washington.edu Discussant: Rich Lehrer, Vanderbilt University Abstract: This symposium presents research that uses video-based ethnographic methods to sample the range of ecological events and contexts in young childrens lives and thereby document the what and how of naturally occurring learning and development. All four papers presented focus on the interactional arrangements of social, cultural, and material supports for learning within contexts. The specific focus is on the social functions and social occasioning of practices across settings. The first two papers examine two common interactional arrangementsquestioning and imitationand the qualities of these arrangements across settings. The final two papers examine two common childhood activities in contextbuilding with blocks and watching televisionin order to understand how children engage in these activities and how that engagement is mediated by different interactional arrangements across settings. The combined effect of these papers is to begin to map the socio-material arrangements and interactional routines that contribute to young childrens everyday learning, a neglected focus in the learning sciences.
Symposium Overview
The learning sciences have largely neglected the study of young children and have generally ceded learning at this age to more traditional developmentalists and their methods. The dominant methods of developmental research over recent decades have been experimental. Laboratory studies offer assurances of statistical reliability, internal validity, and generalizability. What this dominant tradition lacks as a focus is a commitment to adequately sample the range of ecological events and contexts in young childrens lives and thereby to directly study the what and how of naturally occurring learning and development. For this we need to capture and analyze the daily learning activities and settings of early childhood. These studies involve video-based microethnographic analyses of childrens routine activities at home and/or preschool. The data are drawn from two large-scale field projects investigating childrens interactions and learning in and across the socio-material contexts of their everyday lives. The first three papers are based on video data recorded in young childrens (ages 2-5) preschool classrooms and homes. Activities in seven classrooms at three preschools were recorded weekly for three-five months in each school. In addition, ten focal children (five boys and five girls) were video-recorded in out-of-school contexts. Approximately 500 total hours of interaction were video-recorded. The video records were viewed, logged, and selectively transcribed for further microanalysis. The analysis in the final paper draws from data collected for a study of childrens learning from screen media, particularly television. We simultaneously recorded the video stream from the television and the embodied activities of children and others in the viewing space, synchronizing the two videos to allow analysis of how childrens interactions in the room were affected by and coordinated with the television shows they viewed. Participants included 13 children (ages 1-6 years old) from nine families, as well as their parents, friends and siblings. Approximately 60 hours of video were collected for this study. Two overarching themes unify these studies. First, each study documents the interactional, organizational, and material supports for learning in the naturally occurring activities of young children. Secondly, these studies specify qualities of how children think and learn with others. Two further analytic foci connect these studiesfoci distinctively available to field methods; these studies all seek to explore the social functions and social occasioning of childrens practices within and across settings. By social functions, we refer to the work that specific practices (e.g. questioning) do among people (e.g. how questions serve attentiongetting functions for children with adults as well as information-getting functions). By social occasioning, we refer to the precipitating contextual conditions that give rise to a specific practice (e.g. a family-local practice of inquiry about scientific or natural things). Both of these questions are important for the learning sciences in that they address questions of how and under what conditions practices emerge in the flow of activity and where they lead in downstream activity.
ISLS
91
ICLS2012
Volume 2: Symposia
These studies could be seen to lack unity if viewed from the perspective of the four focal practices: questioning, imitating, block building, and TV viewing. Conventional wisdom would suggest two of these practices are to be understood as much more general (i.e. imitating and questioning) and two are more specific (e.g. block building and TV viewing). By bringing studies on these four practices together, we mean to problematize this conventional wisdom. We mean for a comparison of these practices to open up new questions about the movement and transformation of practices across contexts and about the movement of people across contexts, into and out of practices. There are arguments of course that imitation is likely to be a general learning practice, because imitating is built into the human genetic structure (Rizzolatti, Fadiga, Fogassi, & Gallese, 2002) and that questioning is a general practice, because questions are encoded into the syntactic and/or prosodic structure of languages. TV viewing and block building would not seem to have these sorts prestructured affordances, but perhaps that only seems the case if we bias our view of prior structure to personinternal structure (e.g. grammar, prosodic production, or mirror neurons) and not extend considerations of prior structuring to the external environment as well. TV viewing makes a strong case for thinking in these terms as well, because screen-based viewing opportunities (on televisions, computers, tablets, and smart phones) are nearing ubiquity, at least in Western culture contexts. The comparative questions we want to ask with the different practices are fundamentally transfer questions, understood differently from traditional mentalist transfer: how do practices circulate and stabilize across contexts and how do people circulate among practices across contexts? Together, these four papers examine learning over several levels of interactional arrangementfrom the moment-to-moment analysis of questioning and imitation to the more extended interactional arrangements that surround play and media usage. These papers also explore learning across multiple contexts and among multiple interactional partners, which include parents, siblings, and school peers. The combined goal of these papers is to partially map the socio-material arrangements and interactional routines that contribute to young childrens everyday learning and to open up new questions about learning and participation in practices, within and across contexts.
Questioning among Preschool Children

Lauren Penney & Reed Stevens It is generally accepted that children use questions as tools to learn about the world, and for the past century much research has been done on the topic of childrens questions. Researchers have catalogued the types of questions children ask (e.g. what, where, why, how) and at what age they begin to ask each type of question (Davis, 1932; Meyer & Shane, 1973; Kearsley, 1976; Tyack & Ingram, 1976; Hood & Bloom, 1979). Brown (1968) studied the grammar and structure of the question forms that children produce. Other researchers have analyzed the different ways adults respond to childrens questions (Tizard, Hughes, Carmichael, & Pinkerton, 1983; Van Hekken & Roelofsen, 1981). More recently, there has been investigation into how childrens questions contribute to their causal reasoning (Frazier, Gelman, Wellman, 2009; Chouinard, 2007; Callanan & Oakes, 1992; Hood & Bloom, 1979). This prior research analyzes the spontaneous questions of children, yet focuses specifically on questions that indicate information gathering and knowledge building (i.e. Whquestions: what, where, when, why, how), and analyzes questions out of the context of the larger conversation. Our research extends prior work by more closely examining the role that questions play in the lives of young children. We use conversation analysis (e.g. Goodwin & Heritage, 1990) as well as quantitative analysis to explore all of the spontaneous questions asked by two-year-old to five-year-old children in their everyday environments. In this paper we expand upon existing taxonomies that have been created to categorize the various functions that questions play in the conversations of children. Our expanded taxonomy includes a missing category of questions we call social which serve various social functions. We then argue that unlike existing research, childrens questions should be analyzed at a unit level larger than just the question or question-answer pair. Using turn-by-turn analysis we take a detailed look at several sequences we call lines of question-based inquiry (LOQBI) and explore how different the analysis would look if these sequences were analyzed as individual question-answer pairs. We conclude this paper with a discussion about how questions function as a tool that children employ to interact socially with each other. Ultimately this paper makes two new arguments: 1) childrens questions often need to be analyzed in extended sequences (as opposed to broken into question-answer pairs), and 2) these questions are multifunctional and frequently not only seeking information. The question sequences we are calling LOQBI have been noted, but not fully explored by prior research (Chouinard, 2007; Tizard et al., 1983). In this paper we document and analyze inquiry sequences from video-recordings of preschool children across multiple settings. Our findings extend recent work by Chouinard (2007) in that we find childrens questions are not one-offs but are sustained for sense making or information gathering purposes. In Chouinards study, questions continued until satisfactory adult answers were provided, while in our study children continue questioning, even after initial answers are provided, to explore a line of inquiry. For example, when Charlie commented that he was supposed to see Santa Claus but didnt, his teacher
ISLS
92
ICLS2012
Volume 2: Symposia
replied, Maybe hes still working in the North Pole. Charlie then asked several questions about this topic, starting with, What is the North Pole? This LOQBI continued for two minutes and was question-heavy, with Charlie asking 11 questions all of which had something to do with understanding the Santa Claus story. This conversation covered where Santa Claus lives, what he does, with whom he associates, and how he does his work. Charlie received an adequate response to his initial What is the North Pole question, yet he continued asking questions about the Santa Claus story. This three-year old child was asking a stream of questions about a topic he was interested in, and asked different questions as he learned more about the topic. These questions were all related and the topic was the reason for the continued questioning not inadequate answers. Next we argue that childrens questions perform various functions in a conversation beyond seeking information. Freeds (1994) taxonomy of question functions in adult didactic conversations, for example, is a continuum from information sought (the speaker seeks information from the hearer) to information conveyed (the speaker conveys information to the hearer). Freed says questions that fall into the information sought category ask for factual information (e.g. What time are you going home?) while information conveyed questions are used to express importance or emotion (e.g. And you know whats upsetting?) (p. 626-629). Research on childrens questions has focused mainly on questions that are about information gathering, yet the research on adult questions indicates this is not the only way questions are used in conversation. Many taxonomies of childrens questions have been created, many of which stem from Piagets (1932) categorization of why questions (Piaget, 1923; Davis, 1932; Meyer & Shane, 1973), others of which are developed independently (Tyack & Ingram, 1977; Callanan & Oakes, 1992). However, these taxonomies focus primarily on the information gathering and knowledge building questions asked by children. This field study captures and begins to explore the full range of questions asked by young children in their natural environments. One of our findings is that many of the questions children direct towards other children perform a social function, that is, children use questions to invite others to play (Whos the conductor?), as well as to negotiate and arrange their play together (Wanna watch some dog movies? or How much player is this?). These questions are not asked to simply seek information, but the children also use them as an interactional social tool. Our first argument in this paper is that analyzing these LOQBI as sequences allow us to see the connected nature of this type of questioning something that is lost when questions are analyzed individually or as question-answer pairs. Our second argument is that childrens questions perform various functions in a conversation beyond seeking information; children use them as they socially interact with each other during pretend play. We hope this paper begins a discussion about the range of questions asked by young children.
Childrens Imitation in its Natural Environments

Siri Mehus, Reed Stevens, & Lauren Penney Imitation is one of the fundamental means by which humans learn from other humans. Our particular propensity for doing as we see and hear others do is quite possibly unique to humans and forms the basis of human culture (e.g. Gergely & Csibra, 2005; Meltzoff et al., 2009; Meltzoff, 1988; Tomasello, 1999). While the importance of imitation for human learning has been well demonstrated in experimental laboratory research, few studies have investigated the ways in which children imitate other children and adults in their natural habitate.g. in the homes and childcare classrooms in which they spend their days. Even fewer have employed the ethnographic approach of investigating such practices from the participants perspectives. Laboratory studies help us understand how children can learn through imitationi.e. what the human brain is capable ofbut they do not tell us much about how children do learn through imitationi.e. the role of imitative learning in the course of childrens development in their social worlds. In this study we set out to investigate imitation in terms of how it is occasioned and how it functions in the socio-material contexts of childrens everyday lives. We ask, in other words, when does imitation happen, and when it does, what happens next? Our data consist of video-recorded interactions between young children and their families, teachers and peers in home and preschool environments. Our analysis reveals that imitation in childrens daily lives often occurs in the context of some social routine, game, or shared activity. Within these interactional environments, the purpose of making an appropriate and successful move in the activity motivates the imitative act. Examples of social moves that often involve some form of imitation include reciprocating, one-upping, joining in, aligning, mocking, arguing for equity, and participating in improvisational games. Many of childrens imitative acts in everyday life are partial and transformative. By this we mean that the imitator repeats some part or aspect of another person's initial action, but also adds or changes something in the repetition. We submit that the ways in which children imitatee.g. whether and how they transform or adapt the repeated actare shaped by the activities in which these imitative acts are embedded. For instance, we analyze one sequence that takes place in a preschool classroom in which a group of children (3-5 years old) are seated around a table. They engage in a spontaneous improvisational game that involves repeating and upgrading the previous utterances of other children (Transcript 1). We compare this to a sequence that occurs in the home of a three-year old girl (Marie), in which she and her five-year old brother prepare to engage in a
ISLS
93
ICLS2012
Volume 2: Symposia
battle against their father, employing Star Wars ships they have built out of Legos (Transcript 2). We find that in the context of this activity, direct imitation with minimal transformation is a resource for the child in forming an alliance with her brother, in opposition to their father. Transcript 1: Im a tiger Transcript 2: Forming an Alliance
5 6 7 8 9 Hey, what about this? What if I made a ship and you guys, I let you guys destroy it? Evan: Yeah:::! Marie: Yeah:::! Dad: Would that be fun? All right. ((approx. 1.5 minutes while Dad builds his ship and repeatedly predicts that he will win the battle, which Marie and Evan contest.)) 49 Dad: I'm having a little trouble keeping my ship together. 50 Do you have any glue? Am I allowed to use glue? 51 Evan: No! 52 Marie: No! 53 Dad: Okay:: The door is opening. ((sound effect)) That'll be my driver. 54 Marie: Well [(I have your driver) ((chanting)) 55 Dad: [Oh. I know what I need. (.) This. The ultimate56 You will be fighting the ultimate iron giant. Ha ha ha. ((approx. 40 seconds of talk about the ships and the battle)) 71 Evan: Dad, you only have three guns? 72 Dad: I don't know. You wait and see. There's a lot of surprises coming 73 for you mister. 74 Evan: Phuh huh huh 75 Marie: Ha ha ha. 76 Evan: Ha ha ha ha ha. 77 Dad: Ha ha ha? 78 Marie: Ha. 79 Evan: Blah blah blah. Your ship is really dumb. 80 Marie: Blah blah blah. [((makes "talking" gesture)) 81 Dad: [It's very very powerful. 82 Evan: Very very ugly. 83 Marie: [Very very ugly. 84 Dad: [You know why it's so powerful? Cuz you think it's weak but it's 85 actually ((engine sound, stands up and zooms ship through air)) The battle begins. Dad:
These naturally occurring sequences of talk and action do not provide evidence that learning has occurred, at least not in the form we are accustomed to finding it in reports of laboratory experiments or results of standardized tests. Rather, they provide a glimpse of the everyday activities through which children, over time, learn about their worlds and develop the interactional competence to participate in them (e.g. Schieffelin & Ochs, 1986). Furthermore, by examining these sequences closely, we can gain insight into how children might learn from these activities by identifying the specific opportunities for learning they provide. By analyzing the two instances described above, we find that the different forms of imitation occasioned by participation in these activities have different implications for learning. As Marie (Transcript 2) echoes her brothers words and actions, she has the opportunity to try on new phrases and mannerisms, using her whole body to inhabit a role and enact its relationships with the roles taken on by her brother and father (ally and enemy). Though Marie mimics her brother quite closely, she seems to be taking up his actions and integrating them into her understanding of her own role and purpose, rather than simply parroting him this can be seen in lines 79 and 80 where Marie directly imitates part of her brothers utterance (blah blah blah) but makes it her own by adding an appropriate gesture (the fingers-to-thumb talking hand gesture). The learning opportunities for participants in the improvisational game in Transcript 1 are perhaps even more substantial. Designing a next move in the game requires parsing the syntax of previous moves and working out the overall structure and rules of the one-upping game. Children comment on one anothers moves, providing feedback on previous moves, as well as guidance and motivation for improvement of future moves. This extended play sequence provides a rich environment for building linguistic and interactional competence at lexical, syntactic, and discourse levels. Finally we note that in our data, a primary interactional environment for many types of imitation is the activity frame of play. Imitation is not just a move within these games, it can be the fundamental building block out of which these games are created. It is troubling to consider that the dominant social organizational conventions of formal schooling, with its emphasis on assessment of individual performance, may tend to shut down this powerful, child-driven, engine for social learning.
ISLS
94
ICLS2012
Volume 2: Symposia
Block Building at Home and in Preschool

Danielle Keifert & Reed Stevens Prior research has shown that parents and teachers support childrens reasoning in different settings, including home, school, and museum (Callanan & Jipson, 2001; Callanan & Oakes, 1992; Crowley, et al., 2001; Hood & Bloom, 1979). While providing important insight into joint parent-child interactions, this work typically limited to one type of learning (parent-supported childrens learning) in single settings (designed science activities). Yet, the majority of day-to-day encounters with science phenomena likely occur independent of predetermined social arrangements and outside of designed settings (Crowley & Jacobs, 2002). What we need to better understand is how spontaneous interactional arrangements in multiple settings shape everyday practices that form the foundation of later STEM learning and how preschool children support each others learning during these practices. Through a comparative case study of two episodes of a preschooler building with blocks at home and preschool, this paper examines the interactional arrangements for a common activity in and across these settings. We adopt the lens of Newman, Griffin, and Cole (1989), who argue that the individual is not the correct unit of analysis to compare across settings, favoring instead the larger functional system within which tasksor activities like building with blocksare defined by the ways in which people orient to and accept different goals during interaction. Thus, while the comparison in this paper is of the same child building with blocks in two settings (home and preschool), the analysis of the differences in those episodes considers the interactional arrangements that constitute the system within which the task occurs (playing with Jenga blocks at home with mom nearby versus building a Bad-Guy block trap with peers at school). Following interaction analysis traditions, a turn-by-turn analysis provides the grounding for understanding both how the interactions and block-building activity unfold, as well as how the interactional arrangements shape that unfolding activity (Goodwin & Heritage, 1990; Heritage, 2008; McDermott, et al., 1978; Schegloff, 1972). The focus is on two episodes of four year-old Jamie building with blocks. At home, the interactional arrangement allows Jamie to self-direct his block-building activity with his mother nearby. The construction of the block structure co-emerges with an evolving narrative about the buildings function as verbalized by Jamie. He begins by narrating that he is building a stacka freight stack, but quickly states that he is making a building. Jamie then encounters trouble with his building saying, my building thats kind of not staying up, Mommy and his mother chimes in with Well you know what I think the problem is sweetheart, is that your side pieces are a little bit taller than your stack in the middle. Initially Jamie rejects this advice saying that if he switches the sidepieces to another orientation (so they are oriented like the blocks in the middle) That, it would be too low! When Mom suggests using more pieces he says he only wants to use three. By rejecting Moms advice Jamie remains the sole constructor of the block structure, so even when he later decides to rebuild following Moms advice, he still can claim ownership of the building. Throughout the 24 minutes of block play observed, Jamies story telling and building of his block-structure co-emerge, changing only when he, the sole constructor of both, chooses to make changes and based on Jamies own development as a problem-solver. In the school episode, Jamie participates in block building in an interactional arrangement consisting of several participating peers and building blocks. The video segment begins with Jamie at work with schoolmates building a Bad-Guy Trap (so named by the children). At school the block-building activity is also shaped by an overarching story, which the co-builders draw from cultural forms of storytelling (good guy versus bad guy, setting a trap for the bad guy) as well as a form of division of labor where each builder is responsible for a component of the building. This can be seen when a violation of the ownership of different parts of the structure occurs as when one child reaches over and takes blocks from Jamies area of work. Jamie quickly claims to the group Noams taking, was taking those o:::ff! in a whiny tone. This form of policing of each others behavior is a form of maintaining the norms and rules established within the working group (McDermott, Gospondinoff, & Aron, 1978). The construction of both the story and the building structure also include opportunities for negotiation of roles and activity (Lets make half like that and half like that!). Each of the components of the interactional arrangement (peers, story, available blocks, division of labor) contributes to the overall of the structure, which in turn shapes and is shaped by the co-authored story, all of which is must be negotiated. Across these two settings we see Jamie working on constructing both a story and a building structure made of blocks. However, the experience of block building for Jamie is very different when the larger functional system, the interactional arrangement, is considered. The interactional arrangements in each setting, particularly the sole ownership of the activity at home and the shared ownership of the activity with peers while at school, shapes the way in which story and building structure co-emerge. When building at home, Jamies talk focuses around his co-evolving story and building structure, and he is the sole decision maker regarding structural and narrative features. In this arrangements Jamie is able to focus on physical-spatial problem solving, choosing when, if at all, he takes his mothers advice. However, when building at school, much of Jamie and his peers talk is about negotiating the story, the building structure, and access to materials. Jamies peers contribute significantly to the process making Jamie one of many contributors to specifying and constraining the activity. In fact, much of Jamies activity at school surrounds social negotiation with his co-builders. So although we
ISLS
95
ICLS2012
Volume 2: Symposia
may say that Jamie is performing the same task of building with blocks in both settings, the nature of the task is strikingly different as a result of the differing interactional arrangements. The activity is effectively transformed by the nature of the interactional arrangement and through this transformation Jamie is exposed to very different forms of story and building construction, various problems solving techniques (physical and social), and how to effectively use the affordances of different arrangements to achieve his own goals. Recognizing the differences in opportunities for learning, problem solving, and negotiating available resources in differing interactional arrangements provides important considerations for researchers. As we look to better understand learning across multiple contexts (whether designed or non-designed), and how learning connects across these contexts to build foundations for STEM skills, we must consider the ways in which interactional arrangements shape these foundational experiences. Although in the comparison provided the experiences seem to provide complementary opportunities for Jamie to develop competence that may support later STEM learning, this may not be the case for all comparisons of similar activities. In continuing research, we will broaden our analysis of Jamies opportunities for learning in both home and school contexts, and consider how experiences may mutual enhance or inhibit the development of STEM reasoning across settings.
TV Viewing Practices and Childrens Reasoning

Pryce Davis & Reed Stevens Television is a controversial mediumcriticized for wasting valuable time, promoting violence in children, and harming family relationships (Wartella & Robb, 2008). Despite possible negative effects, TV viewing dominates young peoples media consumption. From age two to age eleven, TV viewing among children increases from an average of one-and-a-half to five hours a day (Wartella, Richert, & Robb, 2010; Kaiser Family Foundation, 2010). If we accept the premise that television viewing is central aspect of preschool-age childrens lives, then uncovering the types of thinking and learning that accompany TV viewing should be a major subject of study. While many studies have focused on childrens learning from TV (e.g. Fisch, 2004; Uchikoshi, 2006; etc), these studies are interested in the aftereffects of TV viewing. Instead, we should try to understand what kinds of thinking happen[ during TV viewing. Rather than focusing on questions about whether kids watch too much TV (e.g. Condry, 1993), if TV viewing is positive or negative in and of itself (e.g. Zimmerman & Christakis, 2005), or how we can design interventions to reduce TV viewing (e.g. Dennison, et al., 2004), researchers should be asking: How do children make sense of what they see on television? How can we make visible and understand childrens television-mediated reasoning? In what contexts does reasoning occur? Only after we address these questions can we as educators encourage fruitful television watching or take advantage of the skills children develop while watching TV at home. In this study, we draw data from a corpus of videos and ethnographic field notes of children watching television and interacting with their families in their homes (see Dugan, Stevens, & Mehus, 2010). We analyze side-by-side synchronized recordings of kids watching TV and what they are viewing. This data, together with parental journal data about their kids everyday activities and viewing habits, forms our in-room, in-show, inworld framework. This framework allows us to accurately and completely as possible record childrens actual experience of watching and reasoning about television. For this paper we present a case study of two siblingsLeAnne (6) and Harrison (4). These siblings were selected as being generally representative of the data as a whole. They typically watch 4-5 hours of television per week. Their mother claims they are interactive and talkative television watchers, who often repeatedly watch the same episode. In this particular case study, we present these siblings viewing two different episodes of their favorite childrens science program (Henrys Amazing Animals). The two vignettes represent the two relevant trends in the data. First, children often seem to passively view TV. Secondly, children do display active reasoning about TV under two particularly important practices: Co-viewing with adults and repeated viewing of single episodes. In our first vignette, the children watch an episode for the first time. While watching the show the children sit quietly by themselves and hardly take their eyes from the screen. There is scant activity or talk taking place in the room and little visible reasoning. Of course, the show is presenting a near constant stream of information about animal camouflage. Overall, this vignette reflects the stereotyped couch potato view of children watching television, and demonstrates none of the activity and reasoning claimed by the childrens mother. This isnt to say that the children are not reasoning, but that their reasoning is invisible to researchers. In the second vignette, the children watch an episode that their mother says is one they have viewed many times as it is Harrisons favorite. In this episode the kids talk almost constantly, both to each other and to the researcher in the room. Their talk reveals that the children are engaged in many types of reasoning. The children not only repeat information from the show, but also give information minutes before the show gives the same information. Furthermore, the show prompts them to share novel information as when LeAnne sees something on the screen that causes her to share how starfish eat (which is not presented in the show). In other instances, the children connect information from the show to their personal lives. For example, an image of the
ISLS
96
ICLS2012
Volume 2: Symposia
ocean prompts LeAnne to share a personal story about a trip to the beach that gives her an emotional connection to the animals in the show and how she has researched the animals on her computer. At other times the kids build coherent explanations using causal inferences. In one case, the show talks about sunken ships making good homes for fish, as long as they are not filled with oil and LeAnne infers that fish cannot breathe oil. In other cases, the kids draw inferences to explain the structure of the show, the function of particular features of animals, and even the relationship between the shows characters. Finally, the children share previous ideas about the show in order to generate new questions. Overall, children display complex reasoning practices while watching TV, but these typically only occur (or only become visible) during certain viewing practices and with certain kinds of social support. These practices are highlighted in the differences between the two vignettes. First of all, the children hadnt seen the first episode before so the information may be novel to them; they stay silent because they are trying to comprehend what is being presented. However, they had seen the second episode multiple times and are familiar with the information. So, they are free to actively grapple with the information in order to construct their understanding. Little work exists on unprompted repeated viewing, although experimental work has found forced repeated viewing to increase comprehension without decreasing interest (Crawley, et al., 1999). Secondly, in the second vignette there is a more immediately present adult with the kids than in the first vignette. Previous research points to adult directed co-viewing as an important practice (Friedrich & Stein, 1975; Valkenburg, Krcmar, & de Roos, 1998). In the first vignette, there are two adults in the room with the children: The mom who is on the computer and the researcher manning the recording equipment, but they are both silent and out of view. So, the children do not interact with them. In the second vignette, the researcher is easily in the kids view. This researcher passively observes, and doesnt ask questions or prompt the kids. However, the kids talk to him and treat him like a co-viewer. In a way, he is given the role of the student and the kids teach him about the animals on the show. So, unlike previous research on co-viewing the adult is not explicitly driving the kids attention, but is merely passive yet attentive to what the kids point out. To conclude, we return to the driving questions of this research. How do children make sense of what they see on television? How can make visible and understand childrens television-mediated reasoning? We contend that, although often passive, kids engage in complex reasoning while watching TV under certain circumstances. In fact, they display quite varied reasoning skills, from sharing novel knowledge to drawing causal inferences. In what contexts does reasoning occur? Certain TV viewing practices are more likely to support reasoning. Repeatedly viewing allows kids to continually engage with the same information, so they can actively construct knowledge about the shows topics, which they take into their everyday lives. Co-viewing helps prompt reasoning, even when the adult is not explicitly driving the childs attention, but is merely passive yet attentive to what the kids thinking. This work extends previous findings by demonstrating instances of child-directed co-viewing and repeated viewing, meaning that the child is requesting the repeated viewing and is the active agent in the co-viewing. Taken together, these findings give a conceptual foundation about the things we should attend to as we attempt to understand the role and effects of television in childrens lives.
References
Brown, R. (1968). The development of Wh- questions in child speech. Journal of Verbal Learning and Verbal Behavior, 7, 279-290. Callanan, M., & Jipson, J. (2001). Explanatory conversations & young children's developing scientific literacy In K. Crowley, C. Schunn & T. Okada (Eds.), Designing for science. Mahwah, NJ: Erlbaum. Callanan, M., & Oakes, L. (1992). Preschoolers' questions & parents' explanations: Causal thinking in everyday activity. Cognitive Development, 7, 213-233. Chouinard, M. (2007). Children's questions: a mechanism for cognitive development. Monographs of the Society for Research in Child Development, 72 (1), 1-129. Comstock, G., & Scharrer, E. (2007). Media and the American child. Burlington, MA: Academic Press. Condry, J. (1993). Thief of time, unfaithful servant: TV & the American child. Daedalus, 122(1), 259-278. Crawley, A., Anderson, D., Wilder, A., Williams, M., & Santomero, A. (1999). Effects of repeated exposures to a single episode of the television program Blue's Clues on the viewing behaviors & comprehension of preschool children. Journal of Educational Psychology, 91(4), 630-637. Crowley, K., Callanan, M., Jipson, J., Galco, J., Topping, K., & Shrager, J. (2001). Shared scientific thinking in everyday parent-child activity. Science Education, 85, 712-732. Crowley, K., & Jacobs, M. (2002). Building islands of expertise in everyday family activity. In G. Leinhardt, K. Crowley & K. Knutson (Eds.), Learning conversations in museums. Mahwah, NJ: Erlbaum. Davis, E. (1932). The Form and Function of Children's Questions. Child Development, 3(1), p57-74. Dennison B., Russo, T., Burdick P., & Jenkins, P. (2004). An intervention to reduce television viewing by preschool children. Archives of Pediatrics and Adolescent Medicine, 158(2), 170-176
ISLS
97
ICLS2012
Volume 2: Symposia
Dugan, T., Stevens, R., & Mehus, S. (2010). From show, to room, to world: A Cross-Context Investigation of How Children Learn from Media Programming. Proceedings of the 9th international conference on Learning sciences (ICLS 2010), 992-999. Duveen, G. (2000). Piaget ethnographer: Qualitative methods in the study of culture & development. Social Science Information, 39(1), 79-97. Fisch, S. (2004). Childrens learning from educational TV: Sesame Street & beyond. Mahwah, NJ: Erlbaum. Frazier, B., Gelman, S., Wellman, H. (2009). Preschoolers search for explanatory information within adultchild conversation. Child Development, 80(6), 1592-1611. Frederich, L., & Stein, A. (1975). Prosocial television & young children: The effects of verbal labeling & role playing on learning and behavior. Child Development, 46(1), 27-38. Freed, A. (1994). The form & function of questions in informal dyadic conversation. Journal of Pragmatics, 21(6), 621-644. Gergely, G & Csibra, G. (2005). The social construction of the cultural mind: Imitative learning as a mechanism of human pedagogy. Interaction Studies, 6(3), 463481. Goodwin, C., & Heritage, J. (1990). Conversation analysis. Annual Review of Anthropology, 19, 283-307. Heritage, J. (2008). Conversation analysis as social theory. In B. Turner (Ed.), The new Blackwell companion to social theory. Oxford: Blackwell. Hood, L., & Bloom, L. (1979). What, when, & how about why: A longitudinal study of early expressions of causality. Monographs of the Society for Research in Child Development, 44(6), 1-47. Kaiser Family Foundation. (2010). Generation M2: Media in the Lives of 8-to-18-Year-Olds. Menlo Park, CA: Kaiser. Kearsley, G. (1976). Questions & question asking in verbal discourse: A cross-disciplinary review. Journal of Psycholinguistic Research, 5(4), 355-375. McDermott, R., Gospondinoff, K., & Aron, J. (1978). Criteria for an ethnographically adequate description of concerted activities & their contexts. Semiotica, 24, 245-275. Meltzoff, A. (1988). Infant imitation and memory: Nine-month-olds in immediate and deferred tests. Child Development, 59, 217-225. Meltzoff, A., Kuhl, P., Movellan, J. & Sejnowski, T. (2009). Foundations for a new science of learning. Science, 325, 284-288). Meyer, W., & Shane. J. (1973). The form & function of children's questions. Journal of Genetic Psychology, 123(2), 285-296. Newman, D., Griffin, P., & Cole, M. (1989). The Construction Zone. NY: Cambridge. Rizzolatti, G., Fadiga, L., Fogassi, L. & Gallese, V. (2002). From mirror neurons to imitation: Facts & Speculations. In A. Meltzoff & W. Prinz (Eds.), The Imitative Mind. (pp. 246-266). Cambridge. Schegloff, E. (1972). Notes on a conversational practice: Formulating place. In D. Sudnow (Ed.), Studies in social interaction (pp. 75-119). NY: Free Press. Schieiffelin, B. & Ochs, E. (1986). Language socialization in two cultures. NY: Cambridge University Press. Stevens, R., Satwicz, T., & McCarthy, L. (2008). In game, in room, in world: Reconnecting video game play to the rest of kids lives. In K. Salen (Ed.), Ecology of games: MacArthur Foundation series on digital media and learning. Cambridge, MA: MIT. Uchikoshi, T. (2006). Early Reading in Bilingual Kindergartners: Can Educational Television Help? Scientific Studies of Reading, 10(1), 89-120. Tizard, B., Hughes, M., Carmichael, H., & Pinkerton, G. (1983). Children's questions and adults' answers. Journal of Child Psychology and Psychiatry, 24(2), 269-281. Tomasello, M. (1999). The cultural origins of human cognition. Cambridge: Cambridge University Press. Tyack, D., & Ingram, D. (1976). Children's production & comprehension of questions. Journal of Child Language, 4(2), 211-224. Valkenburg, M., Krcmar, M., & de Roos, S. (1998). The impact of a cultural children's program & adult mediation on children's knowledge & attitudes towards opera. J. of Broadcast. Elec. Media 42, 315-26. Van Hekken, S., & Roelofsen, W. (1981). More questions than answers: A study of questionanswer sequences in a naturalistic setting. Journal of Child Language, 9(2), 445-460. Wartella, E., Richert, R. & Robb, M. (2010) Babies, Television & Videos: How Did We Get Here? Developmental Review, 30(2), 116-127. Wartella, E., & Robb, M. (2008). Historical & recurring concerns about childrens use of the mass media. In S. Calvert & B. Wilson (Eds.), The Handbook of Children, Media, & Development (pp. 726). Blackwell. Zimmerman, F., & Christakis, D. (2005). Children's television viewing & cognitive outcomes: A longitudinal analysis of national data. Archives of Pediatrics & Adolescent Medicine, 159(7), 619-625.
ISLS
98
ICLS2012
Volume 2: Symposia
Youre It! Body, Action, and Object in STEM Learning

Dor Abrahamson (Chair), UC Berkeley, 4649 Tolman Hall, Berkeley, CA 94720-1670, dor@berkeley.edu Carmen J. Petrick & H. Taylor Martin, UT Austin, 1 University Station D5700, carmenpetrick@gmail.com David J. DeLiema, Noel Enyedy, Francis F. Sten, & Darin Hoyer, UC Los Angeles, ddeliema@gmail.com Mina C. JohnsonGlenberg, David Birchfield, Tatyana Koziupa, Caroline Savio-Ramos, & Julie Cruse, School of Arts, Media + Engineering, Arizona State U., Tempe, AZ, Mina.Johnson@asu.edu Robb Lindgren, Anthony Aakre, & J. Michael Moshell, University of Central Florida, Robb.Lindgren@ucf.edu Cameron L. Fadjo & John B. Black, Teachers College, Columbia, 525 W. 120 th St., NY, clf2110@columbia.edu Mike Eisenberg (Discussant), U. Colorado, Campus Box 430, Michael.Eisenberg@colorado.edu Abstract: In this special double symposium, sixteen established and emerging scholars from seven US universities, who share theoretical perspectives of grounded cognition, empirical contexts of design for STEM content domains, and analytic attention to nuances of multimodal expression, all gather to explore synergy and coherence across their diverging research questions, methodologies, and conclusions in light of the conference theme Future of Learning. Jointly we ask, What are the relations among embodiment, action, artifacts, and discourse in the development of mathematical, scientific, engineering, or computer-sciences concepts? The session offers emerging answers as well as implications for theory and practice. THERE was a child went forth every day; And the first object he lookd upon, that object he became; And that object became part of him for the day, or a certain part of the day, or for many years, or stretching cycles of years. (Walt Whitman)
Introduction: Youre It! Is More Than a Tagline

Youre it!, so mundane a playground exclamation, appears to capture much more of human experience than a game of tag. To each of us16 established and emerging scholars from seven different universities across the USYoure it! bears in profound and empirically substantiated ways on relations among action, embodiment, artifacts, reasoning, and discourse, as these relations pertain to developing competence in some STEM domain (Science, Technology, Engineering, Mathematics). Our distinct yet teaming ideas, we sense, should be gathered and shared among us and the larger learning-sciences community, because apparently these ideas collectively hone and point toward what might be the theoretical core and pedagogical promise of the embodiment approach. In this symposium, we attempt to shed light on the nature and dynamics of enactment, integration, and signification as naturalistic learning phases that can be recruited via pedagogical design for content instruction. Each study interprets cases of mediated interactions designed and facilitated with the objective of fostering knowledge (which we conceptualize and pin down from varying epistemological perspectives as professional perception, insight, models, skills, etc). Moreover, each presentation provides rich qualitative analyses of paradigmatic moments, in which study participants physical actionwhether with, upon, or about objects that are either material, virtual, or imaginarycontributes to individual microgenesis of target subject matter. Yet more specifically, we are all interested in elaborating from a grounded-cognition perspective theoretical models pertaining to manipulationwhether actual, vicarious, or simulatedand how these operations contribute to learning. All the data discussed in this symposium were collected in instructional situations, writ large. Turning to instructional practice, we are also interested in determining and characterizing any unique pedagogical affordances of particular technology and interaction strategies vis--vis student cognition of focal content. Thus, in accord with the ICLS 2012 The Future of Learning theme, this symposium attempts to ground nextgeneration interaction design in established tenets of learning sciences theory. Our discussant, Mike Eisenberg, brings to this symposium his renowned expertise in cognitive science, computer science, mathematics, engineering, and integrated multimedia design for STEM content learning. In writing our individual sections, we chose to use a common format, so as to highlight our similarities and suggest our synergies. The sections present the proposed papers in their intended order of presentation. During the 120 min. session, these six individual papers (6 x 15 min.) will be followed by comments from our discussant (20 min.), and then we will engage in a general discussion with the audience (10 min.). To varying degrees, presenters will bring software, media, and artifacts from their respective research studies that will enhance post-symposium informal follow-up conversations with interested members of the audience.
ISLS
99
ICLS2012
Volume 2: Symposia
You Made It! From Action to Object in Guided Embodied Interaction Design
Dor Abrahamson, Dragan Trninic, and Jos F. Gutirrez, University of California at Berkeley
Object is supposed to mean....something that is either internally or externally present in a certain situation. Thus, not only external things like....wood blocks, or signs and persons, are objects, but it is also possible that a certain form of knowledge or a certain cognitive ability is the object. (Hoffmann, 2007, p. 189, original italics) What role might instructors play in scaffolding students generalizations from embodied interaction? Twentytwo Grade 4-6 students (ages 9-11) participated, either individually or in pairs, in a task-based, semi-structured, tutorial clinical interview. The interventions objective was to gather empirical data for a design-based research study investigating the emergence of conceptual knowledge from physical activity. Under the researchers guidance, participants engaged in an embodied-interaction problem-solving activity. Their task was to make a computer screen green by remote-controlling two virtual objects, one hand each. Unknown to them, the screen would be green only when their hands were at particular heights above the desk, relative to each other (see Figure 1). In Abrahamson, Trninic, Gutirrez, Huth, and Lee (2011) we describe the following typical participation trajectory. Students first developed nave qualitative strategies (e.g., The higher you go, the bigger the distance). Next, when we overlaid a virtual Cartesian grid on the screen, they used this mathematical resource to bootstrap an a-per-b form (e.g., For every one unit I go up on the left, I got up two on the right); and when we supplemented numerals, they determined a multiplicative relation (e.g., The right hand is always double the left hand). Here we report on a study of the tutors function in scaffolding these insights.
a. b. c. d. Figure 1. The Mathematical Imagery Trainer for Proportion (MIT-P) set at a 1:2 ratio, so that the right hand needs to be twice as high along the monitor as the left hand. In an empirically determined schematic interaction sequence, the student: (a) positions hands incorrectly (red feedback); (b) stumbles on a correct position (green); (c) raises hands maintaining constant distance between them (red); and (d) corrects position (green), infers rule. Compare 1b and 1dnote the different distances between the cursors. The following transcription from our empirical data is presented to illustrate the microgenesis of mathematical objects via guided, mediated embodied interaction. In particular, we attend to nuanced language featuressliding uses and substitution of pronounsas indicators of how a dyad zigzags between embodied actions, relations, and rules, ultimately objectifying referents in a shared perceptuomotor field, all so as to repair designed practical and discursive vagueness (Abrahamson et al., 2009; Newman, Griffin, & Cole, 1989; Rowland, 1999). As per Hoffmanns quotation in our motto, indeed our object is an externally present thing an invisible distance between two points. This particular phenomenological object is of critical importance to learning via our design: noticing, controlling, and naming it is the first step to articulate a proto-ratio principle. Dor: Amira: D: A: D: A: D: A: D: A: D: A:
ISLS
So whats the rule? What makes it green? [= scaffolds reflection, requests generalization] Having this one [she indicates her right hand] be higher [...she indicates her left hand]. Hnhn. Ok. So, any higher? [= implies a request for more specificity. Note: the grid is on.] ....about three squares higher. [= complies by offering quantitative relational locator] Three squares higher. Ok. So if you bring your hands down... [= launches generalization] I think its like, if its up here also... Theres like a few spaces where its.... In some spaces you have to be lower down. [= in some areas on the screen the interval is smaller] Aha. In some spaces you have to be lower down. [= echoes; implies positive valorization] Yeah. Right down here. But then when you go up here, you have to be higher. Ah! Ok, so some spaces you have to be down, but then when you go up there, it has to be... [= echoes, but switches you to it; elides higher to invite another, clearer descriptor] Yeah... A bigger distance. [= offers distance as an alternative completion of the assertion, and thus disambiguates that her higher had referred to the hands interval not elevation] A bigger distance. Ok. [= echoes; affirms with explicit positive valorization] Down here you only need about one. [one indicates absolute, not relational interval]
100
ICLS2012
Volume 2: Symposia
D: A:
Ok, so down there you only need about one. [echoes] .... As you keep going up, it has to be more..... Here its 1, ...2...3...4...5 [infers; applies]
We thus witness how goal-oriented interaction situated in discursive interaction begets a mathematical object. During the presentation, I will screen several samples of video footage, in which tutortutee dyads coconstruct mathematical referents using available semiotic means of objectification (Radford, 2003), including speech, gesture, gaze, and materialvirtual resources. I will interpret those unique moments as culminating brief histories of localized discursive interaction around task-based pedagogical activity.
Learning Mathematics: Youre It vs. Its It

Carmen J. Petrick and H. Taylor Martin, University of Texas at Austin
Childrens play is everywhere permeated by mimetic modes of behavior, and its realm is by no means limited to what one person can imitate in another. The child plays at being not only a shopkeeper or teacher but also as windmill and a train. Of what use to him is this schooling of his mimetic faculty? (Benjamin, 1986, p. 333) This presentation reports on part of a larger study comparing students learning geometry through either embodied or observational activities. There is evidence that embodying concepts is beneficial to learners (Abrahamson, 2004; Fadjo, Lu, & Black, 2009; Roth, 2001); yet, little is known about the differences in the learning process between students who physically enact concepts and students who do not. This study compares how high school students remembered an activity in which they explored the concept of ratio by either being it or watching it. By being it, we mean experiencing oneself as the mathematical object. In contrast, we refer to watching it as observing a mathematical object as being remote and separate from oneself. We use a theoretical framework of embodied cognition as we try to understand how being in a body and interacting with objects in the world can influence how students make meaning of mathematics (Anderson, 2003; Barsalou, 2008; Wilson, 2002). Being able to get inside a problem can be a helpful strategy for students learning mathematics (Wright, 2001). Embodiment is one way to help students do that (Noble, Nemirovsky, Wright, & Cornelia, 2001; Petrick, Berland, & Martin, 2011; Wilensky & Reisman, 2006). When a student becomes it, we hypothesize that he or she will have a very different kind of experience than a student who detachedly watches it, because embodiment promotes connections between physical actions and mathematics in a way that observing does not. In this study we examined students written responses to a survey asking them to write what they remembered about a learning activity, and we compared two groups of students: those who learned through embodiment (n = 69) and those who learned by observing (n = 59). We predicted that students in the embodied condition would remember more about the activity as a whole. Also, we predicted that students in the embodied condition would be more likely to write their responses in a first-person narrative, thus showing that they had realized they were it. In the larger study, fourteen classes of high school geometry students followed a two-week curriculum on similarity. Four teachers participated in the study, and half of each teachers classes were randomly assigned to either the embodied or observer condition. Students in the embodied condition participated in eight activities designed to promote direct embodiment of mathematical ideas, such as ratio and similar triangles. In these activities, the students became it by physically enacting the concepts. Students in the observer condition participated in eight activities that were very similar to the embodied condition, except these students observed or drew pictures of abstract symbols enacting the concepts rather than embodying them. Unit pre- and post-tests showed that students in the embodied condition had greater learning gains than students in the observer condition, and that these differences were found on the conceptual items of the test but not the procedural items (see Petrick & Martin, 2011). To look closer at how the students experienced the different types of activities, we administered a survey at the end of the unit asking students to write down everything they remembered about each of the eight activities. We have chosen to focus on the first activity of the unit, called Make the Screen Green, for this presentation. In the embodied condition, students interacted with a version of the Mathematical Imagery Trainer (MIT; Abrahamson & Howison, 2010), in which they used Wii remotes to control the heights of two blocks on a screen (see Figure 1, below). Students took turns moving the remotes to help them find the hidden rule that turned the screen from red to green, and every few minutes the teacher would ask students to use their hands to simulate the positions of the blocks that they thought would make the screen green. As they did this, each students partner would give feedback. Students in the observer condition did not manipulate the blocks or simulate the movements of the blocks with their hands. Instead, they watched a video showing the same blocks and screen that students in the embodied condition saw. Similarly, the screen would turn green whenever the ratio of the heights of the blocks was 2:1. Yet this video was a recording of other students interactions with the
ISLS
101
ICLS2012
Volume 2: Symposia
MIT from an earlier pilot. Here, too, every few minutes the teacher would ask students to describe to their partner what made they thought made the screen turn green, and each students partner would offer feedback.
Figure 1. Classroom setup for Make the Screen Green: embodied condition (left) observer condition (right). After analyzing what all students wrote on their survey at the end of the unit, we found that students in the embodied condition wrote significantly more words overall in their response than students in the observer condition. Upon looking more deeply, we found that the embodied condition wrote significantly more mathematical details (e.g., mentioning a two-to-one relationship) than students in the observer condition, and they also wrote significantly more non-mathematical details (e.g., names of partners). In addition, we found that students in the embodied condition were significantly more likely to adopt first-person narratives, while students in the observer condition were significantly more likely to write from a third-person narrative. These differences appear even though there were no significant differences between conditions in student learning on procedural items on the unit pre- and post-tests. During the presentation, we will discuss further details about the two activities, information about the analyses, and implications of these results for the theory and practice of mathematics education.
Learning Science by Being You, Being It, Being Both

David J. DeLiema, Noel Enyedy, Francis F. Steen, and Darin Hoyer, U. of California at Los Angeles
When I observed phenomena in the laboratory that I did not understand, I would also ask questions as if interrogating myself: Why would I do that if I were a virus or a cancer cell, or the immune system. (Salk, 1983, p. 7) This presentation reports on findings from an ongoing research project investigating the role of physical action, particularly gesture, in learning scientific content. It has been theorized that learning science involves the construction of mental models (cf. Frederiksen, White, & Gutwill, 1999; Gentner & Stevens, 1983; JohnsonLaird, 1983). The objective of my project is to revisit and potentially qualify these theories from the perspective of the rising paradigm of grounded cognition (Barsalou, 2010). The current study, by analyzing students gestures during the incipient moments of mental model construction, is a first step toward that goal. According to classical computationalism (Gentner & Stevens, 1983), modeling transpires entirely inside the head (Greca & Moreira, 2000). Yet this classical theory, which conceptualizes models as formed out of propositional mental symbols, bears the fundamental methodological disadvantage that these cognitive constructs cannot be seen or measured, such as when students study content (Rouse & Morris, 1986, p. 1). Regardless of whether or not these theories obtain, one methodological means of monitoring learning is asking people to represent their emerging understandings in speech and diagrams (Coll & Treagust, 2001; Harrison & Treagust, 1996; Niedderer & Goldberg, 1996). The present study looks to productions in another semiotic modality, hand gestures, as physically instantiated mental models (Schwartz & Black, 1996, p. 464; see also Clement & Steinberg, 2002; Crowder, 1996). A methodological advantage of studying gestures, in this respect, is their relatively unmediated presentation of thought process, as compared to verbally or diagrammatically instantiated mental models. For the current study, I selected the conceptual content of packet switching, the historical technological innovation that gave rise to the Internet. Study participants with no prior knowledge of packet switching were asked to perform interpretive gestures at the same time as they listened to verbal statements about this system. The term interpretive gesture refers to the students attempt to comprehend and then gesturally model the words of the conversation partner, as the discourse unfolds. Next, I had participants discuss packet switching more naturally (i.e. spontaneously using co-speech gestures). As I now elaborate, this non-
ISLS
102
ICLS2012
Volume 2: Symposia
routine procedure offers a unique window onto the microevolution of a mental model. The following builds on intensive analysis of a case study consisting of a single participant (Cynthia [pseudonym], a college student). The analysis builds on the thesis that students reasoning about STEM phenomena naturally enact both Observer ViewPoints (O-VPT, seeing the phenomenon from outside) and first-person, anthropomorphized Character ViewPoints (C-VPT, being the phenomenon; see Nemirovsky & Monk, 2000; Ochs et al., 1996; Wilensky & Reisman, 2006). Indeed, the student in the present study routinely alternated between O-VPT of components and C-VPT as components. In several instances, the student coordinated static O-VPT representations of the component shape (i.e. box-like) with dynamical C-VPT representations of its actions (i.e. grasping or inscribing). (In the following example, the researchers speech is in inscribed in roman type, and Cynthias gestures, which follow each of the researchers speech turns, are in italics.) Each packet [left hand forms horizontal box shape and remains in place] grabs a few small pieces of the email message [right hand reaches out, plucks an imaginary small object from the air (see Figure 1] and stores those pieces inside itself [places the object inside the left handrepeats three times].
Figure 1. Cynthias left hand is a packet; her right hand operates as a packet on itself. Cynthias left hand established narrative context by instantiating an O-VPT packet, even as the right hand, representing a hand of the same packet, acted upon it as a C-VPT element. By this token, Youre it means importing naturalistic interaction schemas into the inquiry process by literally being the phenomenon in question. Gestures, by concretizing your view when you dive in to be it, and concretizing its view when you dive out to observe it, support the dynamic coordination between the You and It viewpoints by blending into a single model traces of their respective allocentric and egocentric experiences. These same viewpoints for depictions of system events tended to reappear in Cynthias co-speech gestures during her later retelling of packet switching. During my presentation, I will show video clips of the learning interaction and provide moment-tomoment analyses of how Cynthias gestures build multiviewpoint models, establish context around action, and produce evolving representations, and I offer implications for the design of virtual STEM environments.
Seeing It versus Doing It: Lessons from Mixed Reality STEM Education
Mina JohnsonGlenberg, David Birchfield, Tatyana Koziupa, Caroline Savio-Ramos, and Julie Cruse School of Arts, Media + Engineering, and the Learning Sciences Institute, Arizona State University
Watching a child makes it obvious that the development of his mind comes about through his movements. Mind and movement are parts of the same entity. (Maria Montessori, 1967) Mixed-reality embodied learning platforms are coming of age. We will present several studies that have demonstrated increased learning when students are randomly assigned to embodied, mixed-reality (MR) environments compared to learning in regular instruction environments, where teacher and content are held constant. The SMALLab (Situated Multimedia Arts Learning Lab) and Serious Games for Embodied Learning groups at ASU create and research content for K-12 education that is embedded in kinesthetic platforms. (See www.smallablearning.com for videos.) We explore the boundaries of environments that use the body as an interface for learning. The two most common platforms are rigid body motion- and skeletal-tracking cameras (e.g., Kinect). We co-design all lessons with classroom teachers to create content that engages the major sense
ISLS
103
ICLS2012
Volume 2: Symposia
modalities (visual, auditory, and kinesthetic). SMALLab uses 12 infrared motion-tracking cameras to send to a computer information about where a student is in a 15 X 15 ft. floor-projected environment. Students step into the active space and hold a wand (a trackable object) that allows the physical body to function as a 3D cursor in the interactive space. Figure 1 shows two students using wands to manipulate elements in a chemistry lesson.
Figure 1. Two students adding molecules into a virtual flask in a chemistry titration scenario.
Figure 2. Comparative mean scores over time for chemistry scenario.
With turn-taking, entire classrooms with 30 students are able to physically experience a learning scenario within a typical class period. We contend that the more modalities (sensorimotor systems) are activated during the encoding of information, the crisper and more stable the knowledge representations will be in schematic storage. These crisper representations, with more modal associative overlap, will be more easily recalled. Better retrieval leads to better performance on assessment measures. If gestures are another modality and they emerge from perceptual and motor simulations that underlie embodied cognition (Hostetter & Alibali, 2008)then creating an embodied learning scenario that reifies the gestures should be a powerful teaching aid. Yet, it is not trivial to create congruent gestures that map to the lesson that is to be learned (we use congruent the way Segal, Black, Tversky, 2010, do). To this end, we will also present some design guidelines for creating meaningful embodied content and how to think about action as a method for deeper encoding. Several studies have been published supporting significantly larger learning gains when students are active in SMALLab versus regular instruction (Birchfield & JohnsonGlenberg, 2010; Johnson-Glenberg et al., 2009, Tolentino et al., 2009). We hypothesize the primary drivers for the learning changes are embodiment, collaboration, and novelty. In addition, the gains also may be mediated by increased peer-to-peer language use and gameplay. Results from several experiments will be presented, including the titration chemistry scenario. Our studies typically use a waitlist design and three invariant tests. Figure 2 shows that significant gains are observed each time the classes are assigned to the SMALLab condition. We have analyzed results from 200 students in a study designed to address the question of watching it versus being it. We label these two conditions as low vs. high embodiment. The two embodied levels are crossed with three learning platforms: SMALLab, an interactive whiteboard (IWB), or a desktop-and-mouse condition. Psychology undergraduates experience a onehour lesson on Centripetal Force. The research has led to several design principles intended to frame the realization of embodied learning experiences in computer-mediated environments (Birchfield, JohnsonGlenberg, MegowanRomanowicz, Savvides, & Uysal, 2010). Specifically: Direct Impact: Learners physical actions should have a direct, causal impact in the simulated environment; Map to Function: A learners gesture should closely align with its function and role in the simulated environment (e.g., physical and simulated throwing gestures should align); Human Scale: Computer interfaces should support movement on a human scale (e.g., degrees of freedom, size, and speed of a gesture); Socio-Cultural Meaning: The communicative aspects of human presence and gesture should be accounted for (e.g., the cultural meaning of a gesture, the information conveyed by a gesture needs to be addressed).
ISLS
104
ICLS2012
Volume 2: Symposia
Youre the Asteroid! Body-Based Metaphors in a Mixed Reality Simulation of Planetary Astronomy
Robb Lindgren, Anthony Aakre, and J. Michael Moshell, University of Central Florida
The brains sensorimotor representations of space gain their coherence not by their subservience to some over-arching, mathematical definition of space but with respect to a repertoire of movement. (Arbib, 1991, p. 379, as quoted by Hagendoorn, 2012) In this paper we describe a specific approach to generating embodied learning, where users are embedded within a simulation and given the opportunity to learn the important relationships from the inside. A number of recent reports have highlighted the benefits of informal, simulation-based learning experience for science education (Bell et al., 2009; Honey & Hilton, 2011), but there has been fairly little specificity about how the interactions one has with these simulations affect learning. Building upon recent work where mixed reality (MR) environments (the merging of physical and virtual elements in interactive spaces) have been shown to have great potential for facilitating learning (e.g., Birchfield & Johnson-Glenberg, 2010; Hughes, Stapleton, Hughes, & Smith, 2005; Kirkley & Kirkley, 2005), we have developed an interaction approach we call body-based metaphors. Unlike the relational metaphors that drive certain kinds of knowledge construction described by Gentner (1988), body-based metaphors are functional metaphors where the source domain (S) functions (or is made to function) like the target domain (T). In the MR environment we have developed, learners enact functional metaphors by using their bodies to act out part of a simulation of planetary astronomy (see Figure 1). We believe that these body-based metaphors are particularly effective for young learners who may struggle with the structure mapping process associated with relational metaphors.
Figure 1. A middle school student uses their body to put an asteroid into orbit. The learning goal for this project is to develop intuitions about physics concepts related to planetary movement (orbits, gravity, etc.). Philosophers have argued that body activity serves as the basis of conceptual understanding (Gallagher, 2005; Johnson, 1987), and this may be especially true of understanding spatial relationships. The use of body activity to teach physics concepts has been met with mixed success previously, likely because Earth does not provide a pure environment for examining elements such as force. With MR, however, it is possible both to isolate these elements and connect bodily movement with the abstract representations (graphs, vector diagrams, etc.) that are typically used to convey knowledge of physics. We will describe data from research we have conducted to investigate whether the body-based metaphor approach of interacting with digital simulations has advantages over the traditional mouse-andkeyboard interface. To conduct this research we created a simulation game we call MEteor that can be run on both a 30-foot-by-10-foot interactive floor space and standard desktop computer. Participants work through a series of game levels that require a basic understanding of Newtons and Keplers laws (e.g., hitting a target on the opposite side of a large planet). A separate display shows the participant their movement within a graph, allowing them to assess their predictions compared to the actual movement of objects as dictated by the laws of physics. We are applying a number of traditional metrics of learning (pre- and post- knowledge questions, questionnaires about science efficacy), but we are also utilizing a number of alternative measures to probe for effects that may be more commensurate with the modality of learning in this simulation. For example, in our preliminary research we found that participants sketches of the simulation included more dynamic elements
ISLS
105
ICLS2012
Volume 2: Symposia
(arrows showing movement, etc.) and less surface features (textures, background objects) when using the MR simulation compared to the desktop simulation. A primary focus of this research is whether or not we can develop effective measures that specifically target embodied learning. We are interested, first, whether or not a learners experience and level of comfort with physical and embodied activities (e.g., sports, dance, 4H, girl scouts, etc.) predispose them to success with the type of interactive learning intervention we have developed. A recent paper on dance and spatial cognition provides a good example of how experiencing different and more types of movements creates a greater repertoire for understanding (Hagendoorn, 2012). To this end we are using a pre-questionnaire that surveys the participants experience in various physical activities and the types of things they do with their bodies to aid their thinking. A second measure we use is recording the degree to which a participants movements are consistent with the normative trajectories of simulation elements, and how these patterns of movement change over time. We hope to see, for example, that participants using the immersive MR simulation quickly adapt their movements to match how things actually move in space (e.g., slowing down when an orbiting asteroid is far away from the planet and speeding up when closer). Finally we observe the quantity and kind of gestures participants use both in the simulation and outside of it when explaining their reasoning about physics. Previous research has shown that gestures have a significant impact on cognition (Goldin-Meadow, 2003; Goldin-Meadow, Nusbaum, Kelly, & Wagner, 2001), and we are interested in whether a greater propensity to gesture is related to the other knowledge and performance measures being applied. We are using motion tracking to quantify the degree that participants are gesturing in the simulation and using video protocols to analyze gesture in pre- and post-interviews. These varied measures are allowing us to produce a more nuanced description and evaluation of embodied learning generally and body-based metaphors specifically.
Youre In the Game: Direct Embodiment and Computational Artifact Construction

Cameron L. Fadjo and John B. Black, Teachers College, Columbia University
Virtual characters have virtual minds and virtual bodies. They become the players surrogate mind and body. (Gee, 2010, p. 258) This presentation reports on findings from a field-based study investigating the learning of abstract computer science concepts and skills through physical action. Recent research in embodied and grounded cognition has examined the roles action, perception, and environment play in the teaching and learning of abstract concepts in mathematics (Abrahamson, 2010), science (Barab et al., 2007; Chan & Black, 2006), dance (Grafton & Cross, 2008), drama (Noice & Noice, 2006), and robotics (Lu, Kang, Huang, & Black, 2011; Petrick et al., 2011). The objective of this study is to examine if physical, or Direct (Fadjo & Black, 2011) and Imagined Embodiment (Fadjo & Black, 2011) during pedagogical activities improves the learning of abstract concepts in computer science. In particular, we are interested in examining if the embodiment of pre-defined dialogue-based scripts during classroom instruction improves the development of certain Computational Thinking skills and concept knowledge (Resnick & Brennan, 2011; Wing, 2006) during the construction of a computational artifact. This work is a first step toward defining a grounded embodied epistemology of pedagogy. Embodied (Glenberg, 2010) and grounded (Barsalou, 2010) perspectives of cognition have emerged over the past thirty years as the traditional Cartesian dualism view of cognition has been challenged (Gibbs, 2006). In particular, the problems of transduction (Barsalou, 1999) and grounding (Harnad, 1990; c.f. Pecher & Zwaan) have led cognitive scientists, cognitive psychologists, and philosophers to question the role the body plays in cognition. Further exploration of these problems has resulted in the emergence of two major developments in support of an embodied cognition (Gibbs, 2006). The first development, dynamical systems theory, emerged from the fields of artificial intelligence and cognitive science as a way to define and explore the complex relationship between mind, body, and environment in cognition (Gibbs, 2006; Robbins & Aydede, 2009). The second development within embodied cognition comes primarily from work in the field of cognitive linguistics that emphasizes the importance of linguistic structures on human conceptual knowledge, bodily experience, and the communicative functions of discourse (Gibbs, 2006). Our study examines the development of Computational Thinking through an embodied approach to learning the language of computing. Recent studies by Glenberg and colleagues (2009, 2004) have demonstrated that an embodied approach to reading comprehension through physical and imagined manipulation is highly effective for learning abstract concepts in novel scenarios. This study extends previous work on embodied learning by immersing the learner in a scenario where she or he becomes the video game character. For the current study, we incorporated physical and imagined embodiment in the instruction of certain Computational Thinking concepts (conditional logic, loops, and parallelism) and skills (pattern recognition, abstraction, and decomposition) during the construction of video game artifacts using Scratch, a block-based, visual programming and design language developed at the MIT Media Lab (Resnick & Brennan, 2011). Sixth ISLS 106
ICLS2012
Volume 2: Symposia
and seventh-grade students from a suburban public middle school with no prior programming experience were asked to use their body to physically enact pre-defined programming scripts during four 10-minute instructional sessions over a six-day period. Instruction began with a pair of instructors modeling the physical embodiment of the pre-defined scripts, while all students observed the teachers being the characters. Then, depending on the group to which they had been randomly assigned, the students either physically embodied and imagined the same pre-defined scripts themselves (see Figure 1) or only imagined these interactions, without physical embodiment. By embodying the characters actions and behaviors, the learners became the characters. So doing, they moved and interacted with one another in a traditional learning environment. At the same time, they created an imaged scenario that was informed by play schemas (Salen & Zimmerman, 2004). This scenario, in turn, was formed by previous game play experience and prior artifact construction (Fadjo & Black, 2011).
Figure 1. Direct Embodiment of a pre-defined Scratch Script. The learner is attempting to read the pre-defined script while physically enacting the sequence of actions and statements. In our presentation we will show split-screen video recordings of students becoming the character and engaging in the Direct Embodiment of pre-defined Scratch scripts with simultaneous tracking of sequential code structures, present findings from our recent study on using this grounded embodied approach to developing Computational Thinking and the effects these actions had on concept implementation, and discuss implications of Direct and Imagined Embodiment on the instruction of advanced computer science concepts and skills.
References
Abrahamson, D. (2004). Embodied spatial articulation: A gesture perspective on student negotiation between kinesthetic schemas and epistemic forms in learning mathematics. In D. E. McDougall & J. A. Ross (Eds.), Proceedings of the 26th Annual Meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education (Vol. 2, pp. 791 797). Toronto, Ontario: Preney. Abrahamson, D. (2010). A tempest in a teapot is but a drop in the ocean: action-objects in analogical mathematical reasoning. In K. Gomez, L. Lyons, & J. Radinsky (Eds.), Learning in the Disciplines: Proceedings of the 9th International Conference of the Learning Sciences (ICLS 2010) (Vol. 1 [Full Papers], pp. 492-499). International Society of the Learning Sciences: Chicago IL. Abrahamson, D., Bryant, M. J., Gutirrez, J. F., Mookerjee, A. V., Souchkova, D., & Thacker, I. E. (2009). Figuring it out: mathematical learning as guided semiotic disambiguation of useful yet initially entangled intuitions. In S. L. Swars, D. W. Stinson & S. Lemons-Smith (Eds.), Proceedings of the Thirty-First Annual Meeting of the North-American Chapter of the International Group for the Psychology of Mathematics Education (Vol. 5, pp. 662-670). Atlanta, GA: Georgia State University. Abrahamson, D., & Howison, M. (2010, April). Embodied artifacts: coordinated action as an object-to-thinkwith. Paper presented at the annual meeting of the American Educational Research Association. Abrahamson, D., Trninic, D., Gutirrez, J. F., Huth, J., & Lee, R. G. (2011). Hooks and shifts: a dialectical study of mediated discovery. Technology, Knowledge, and Learning, 16(1), 55-85. Anderson, M. L. (2003). Embodied cognition: a field guide. Artificial Intelligence, 149, 91-130. Arbib, M. A. (1991). Interaction of multiple representations of space in the brain. In J. Paillard (Ed.), Brain and space (pp. 379-403). Oxford: Oxford University Press. Bakker, A., & Hoffmann, M. H. G. (2005). Diagrammatic reasoning as the basis for developing concepts: A semiotic analysis of students learning about statistical distribution. Educational Studies in Mathematics, 60(3), 333-358. Barab, S., Zuiker, S., Warren, S., Hickey, D., Ingram-Goble, A., Kwon, E.-J., et al. (2007). Situationally embodied curriculum: Relating formalisms and contexts. Science Education, 91, 750-782. Barsalou, L.W. (1999). Perceptual symbol systems. Behavioral and Brain Sciences, 22, 577-660. Barsalou, L. (2008). Grounded cognition. Annual Reviews Psychology, 59, 617-645. Barsalou, L.W. (2010). Grounded cognition: past, present, and future. Topics in Cog. Science, 2(4), 716-724. Bell, P., Lewenstein, B., Shouse, A., & Feder, M. (Eds.). (2009). Learning science in informal environments: people, places, and pursuits. Washington, DC: National Academy Press.
ISLS 107
ICLS2012
Volume 2: Symposia
Benjamin, W. (1986). Reflections: essays, aphorisms, autobiographical writings (E. Jephcott, Trans.). New York: Schocken. Birchfield, D., & Johnson-Glenberg, M. C. (2010). A next gen interface for embodied learning: SMALLab and the geological layer cake. International J. of Gaming and Computer-mediated Simulation, 2(1) 49-58. Chan, M. S., & Black, J. B. (2006). Direct-manipulation animation: incorporating the haptic channel in the learning process to support middle school students in science learning and mental model acquisition. In S. Barab, K. Hay, & D. Hickey (Eds.), Proceedings of the 7th International Conference of the Learning Sciences (pp. 64-70). Mahwah, NJ: LEA. Clement, J. J., & Steinberg, M. S. (2002). Step-wise evolution of mental models of electric circuits: a learningaloud case study. The Journal of the Learning Sciences, 11(4), 389-452. Coll, R. K., & Treagust, D. F. (2001). Learners mental models of chemical bonding. Research in Science Education, 31, 357-382. Crowder, E. M. (1996). Gestures at work in sense-making science talk. The Journal of the Learning Sciences, 5(3), 173-208. Fadjo, C., Lu, M., & Black, J. B. (2009). Instructional embodiment and video game programming in an after school program. Paper presented at the World Conference on Educational Multimedia, Hypermedia and Telecommunications, Chesapeake, VA. Fadjo, C. L., & Black, J. B. (2011). A grounded embodied approach to the instruction of computational thinking. In Proceedings of the 42nd ACM Technical Symposium on Computer Science Education. New York: ACM. Frederiksen, J. R., White, B. Y., & Gutwill, J. (1999). Dynamic mental models in learning science: Journal of research in science teaching, 36(7), 806-836. Gallagher, S. (2005). How the body shapes the mind. Oxford University Press, Oxford, UK. Gee, J. P. (2008). Video games and embodiment. Games and Culture, 3(3-4), 253263. Gentner, D. (1988). Metaphor as structure mapping: The relational shift. Child Development, 59, 47-59. Gentner, D., & Stevens, A. L. (1983) Mental models. Mahwah, NJ: Lawrence Erlbaum. Gibbs, R.W. (2005). Embodiment and cognitive science. New York: Cambridge University Press. Glenberg, A. M. (2010). Embodiment as a unifying perspective for psychology. Wiley Interdisciplinary Reviews: Cognitive Science, 1, 586596. Glenberg, A. M., Goldberg, A. B., & Zhu, X. (2011). Improving early reading comprehension using embodied CAI. Instructional Science, 39(1), 113. Glenberg, A. M., Gutierrez, T., Levin, J. R., Japuntich, S., & Kaschak, M. P. (2004). Activity and imagined activity can enhance young childrens reading comprehension. Journal of Educational Psychology, 96(3), 424436. Goldin-Meadow, S. (2003). Hearing gesture: how our hands help us think. Cambridge, MA: Harvard University Press. Goldin-Meadow, S., Nusbaum, H., Kelly, S., & Wagner, S. (2001). Explaining math: gesturing lightens the load. Psychological Science, 12, 516-522. Grafton, S., & Cross, E. (2008). Dance and the brain. The Dana Foundation. Retrieved October 23, 2009 from http://www.dana.org/printerfriendly.aspx?id=10744 Greca, I. M., & Moreira, A. (2000). Mental models, conceptual models, and modelling. International Journal of Science Education, 22(1), 1-11. Hagendoorn, I. (2012). Inscribing the body, exscribing space. Phenomenology and the Cognitive Sciences, 11(1), 69-78. Harnad, S. (1990). The symbol grounding problem. Physica D, 42, 335346. Harrison, A. G., Treagust, D. F. (1996). Secondary students mental models of atoms and molecules: implications for teaching chemistry. Science Education, 80(5), 509-534. Hoffmann, M. H. G. (2007). Learning from people, things, and signs. Studies in Philosophy and Education, 26, 185-204. Honey, M. A., & Hilton, M. (Eds.). (2011). Learning science through computer games and simulations. Washington DC: National Academies Press. Hostetter, A. B., & Alibali, M. W. (2008). Visible embodiment: gestures as simulated action: Psychonomic Bulletin & Review, 15, 495-514. Hughes, C. E., Stapleton, C. B., Hughes, D. E., & Smith, E. (2005). Mixed reality in education, entertainment and training: An interdisciplinary approach. IEEE Computer Graphics & Applications, 26(6), 24-30. Johnson-Laird, P. N. (1983). Mental models. Cambridge, MA: Harvard University Press. Johnson, M. (1987). The body in the mind: The bodily basis of meaning, imagination, and reason. Chicago: University of Chicago Press.
ISLS
108
ICLS2012
Volume 2: Symposia
Johnson-Glenberg, M. C., Birchfield, D., Megowan-Romanowicz, C., Tolentino, L., & Martinez, C. (2009) Embodied Games, Next Gen Interfaces, and Assessment of High School Physics, International Journal of Learning and Media,1(2). Access http://ijlm.net/knowinganddoing/10.1162/ijlm.2009.0017 Kirkley, S. and Kirkley, J. (2005). Creating next generation blended learning environments using mixed reality, video games and simulations. TechTrends, 49(3), 42-89. Lu, C. M., Kang, S., Huang, S., & Black, J. B. (2011). Building student understanding and interest in science through embodied experiences with LEGO Robotics. Educational Multimedia, Hypermedia, and Telecommunications. Charlottesville, VA: Association for Advancement of Computing in Education. McNeill, D. (1992). Hand and mind: what gestures reveal about thought. Chicago: University of Chicago Press. Nemirovsky, R., & Monk, S. (2000). If you look at it the other way . . .: An exploration into the nature of symbolizing. In P. Cobb, E. Yackel, & K. McClain (Eds.), Symbolizing and communicating in mathematics classrooms: perspectives on discourse, tools, and instructional design (pp. 177221). Mahwah, NJ: Lawrence Erlbaum. Newman, D., Griffin, P., & Cole, M. (1989). The construction zone: working for cognitive change in school. New York: Cambridge University Press. Niedderer, H., & Goldberg, F. (1996). Learning processes in electric circuits. Paper presented at the annual meeting of the National Association for Research in Science Teaching, St. Louis, MO. Noble, T., Nemirovsky, R., Wright, T., & Tierney, C. (2001). Experiencing change: the mathematics of change in multiple environments. Journal for Research in Mathematics Education, 32(1), 85-108. Noice, H., & Noice, T. (2006). What studies of actors and acting can tell us about memory and cognitive functioning. Current Directions in Psychological Science, 15(1), 14-18. Ochs, E., Gonzalez, P., & Jacoby, S. (1996). When I come down, I'm in a domain state: grammar and graphic representation in the interpretive activity of physics. In E. Ochs, E. A. Schegloff, & S. Thompson (Eds.), Interaction and grammar (pp. 328-369). Cambridge, UK: Cambridge University Press. Pecher, D., & Zwaan, R. A. (2010). Grounding cognition: the role of perception and action in memory, language, and thinking. New York: Cambridge University Press. Petrick, C., Berland, M., & Martin, T. (2011). Allocentrism and computational thinking. In G. Stahl, H. Spada, & N. Miyake (Eds.), Connecting computer-supported collaborative learning to policy and practice: CSCL2011 Conference Proceedings (Vol. 2, pp. 666-670). Hong Kong: ISLS. Petrick, C., & Martin, T. (2011). Every body move: learning mathematics through embodied actions. Manuscript in progress (copy on file with author). Radford, L. (2003). Gestures, speech, and the sprouting of signs: a semiotic-cultural approach to students' types of generalization. Mathematical Thinking and Learning, 5(1), 37-70. Resnick, M., & Brennan, K. (2011, January 24). Four questions about Scratch. ScratchEd Webinar Series. Robbins, P., & Aydede, M. (Eds.) (2009). The Cambridge handbook of situated cognition. New York: Cambridge University Press. Roth, W.-M. (2001). Gestures: their role in teaching and learning. Review of Educational Research, 71(3), 365392. Rowland, T. (1999). Pronouns in mathematics talk: power, vagueness and generalisation. For the Learning of Mathematics, 19(2), 19-26. Rouse, W. B., & Morris, N. M. (1986). On looking into the black box: prospects and limits in the search for mental models. Psychological Bulletin, 100, 349363. Salen, K., & Zimmerman, E. (2005). Rules of play: game design fundamentals. Cambridge, MA: MIT Press. Salk, J. (1983). Anatomy of reality: merging of intuition and reason. New York: Columbia University Press. Schwartz, D. L. & Black, J. B. (1996). Shuttling between depictive models and abstract rules: induction and fall-back. Cognitive Science, 20, 457497. Segal, A., Black, J. & Tversky, B. (2010, November). Do gestural interfaces promote learning? Embodied interaction: Congruent gestures promote performance in math. Poster presented at the annual meeting of the Psychonomics Society, St. Louis, MI. Tolentino, L., Birchfield, D., Megowan-Romanowicz, C., Johnson-Glenberg, M. C., Kelliher, A., & Martinez, C. (2009). Teaching and learning in the mixed-reality science classroom. Journal of Science Education and Technology. 18, 6, 501-517. DOI: 10.1007/s10956-009-916. Wilensky, U., & Reisman, K. (2006). Thinking like a wolf, a sheep or a firefly: learning biology through constructing and testing computational theoriesan embodied modeling approach. Cognition & Instruction, 24(2), 171-209. Wilson, M. (2002). Six views of embodied cognition. Psychonomic Bulletin & Review, 9(4), 625-636. Wing, J. (2006). Computational thinking. Communications of the ACM, 49(3), 33-35. Wright, T. (2001). Karen in motion: the role of physical enactment in developing an understanding of distance, time, and speed. Journal of Mathematical Behavior, 20, 145-162.
ISLS
109
ICLS2012
Volume 2: Symposia
The Future of Assessment: Measuring Science Reasoning and Inquiry Skills Using Simulations and Immersive Environments
Jodi L. Davenport, WestEd, Oakland, CA USA, jdavenp@wested.org Edys S. Quellmalz, WestEd, Redwood City, CA USA, equellm@wested.org Jody Clarke-Midura, Harvard University, Cambridge, MA USA, jec294@mail.harvard.edu Chris Dede, Harvard University, Cambridge, MA USA, chris_dede@harvard.edu Janice D. Gobert, Worcester Polytechnic Institute, Worcester, MA USA, jgobert@wpi.edu Kenneth R. Koedinger, Carnegie Mellon University, Pittsburgh, PA USA, koedinger@cs.cmu.edu Marty McCall, Smarter Balanced Assessment Consortium, Portland, OR, mccall.marty@gmail.com Michael J. Timms, Australian Council for Educational Research, Melbourne, Australia, mtimms9@gmail.com Abstract: Simulations and immersive environments provide innovative ways to measure students science reasoning and inquiry skills. These computer-based assessments allow for dynamic displays of science systems that expand how phenomena, information, and data can be represented; they also allow for interactivity that provides new ways for learners to demonstrate their knowledge and skills. A number of groups have been working to create and evaluate next-generation assessments that both evaluate students on 21st Century scientific skills and provide evidence models for making inferences about student proficiency. In this symposium, researchers who are currently developing and testing simulation-based and immersive assessments to meaningfully assess science content and inquiry skills will share findings from classroom-based studies of students using the assessments. The presentations will be followed by a discussion from James Pellegrino, an expert in assessment design.
Symposium Objectives
Simulations and immersive environments provide innovative ways to measure students science reasoning and inquiry skills. However, do assessments using these environments provide more information than traditional tests about student proficiency of science reasoning and inquiry skills? Whereas factual knowledge can be easily assessed with traditional paper-and-pencil tests, more complex scientific reasoning and inquiry skills (e.g., systems thinking, designing investigations, gathering evidence, explaining observations) are more difficult to measure with static assessments. Dynamic and interactive assessment designs expand how phenomena, information, and data can be represented and increase the number of ways learners can show their knowledge and skills. As the field starts to integrate technology-based assessments, the challenges for K-12 educators and assessment developers are to create assessment tasks that allow students to demonstrate 21st century scientific skills and to create evidence models for making inferences about student progress and proficiency. We want to move beyond simply putting multiple-choice questions about declarative knowledge online. In this symposium, we bring together researchers who have developed simulation-based and immersive assessments to meaningfully assess science content and inquiry skills. The researchers will share both the designs of these technology-based science assessments using the conceptual framework of evidence-centered design and the findings from studies using the assessments with students. The goals of the current session are not only to describe next-generation assessments, but also to provide principled frameworks for their design, use, and evaluation.
Symposium Overview
The three papers in this session present research findings on innovative, computer-based assessments that are designed to measure scientific reasoning and inquiry skills. The papers report classroom-based findings that demonstrate novel ways to use student actions in open-ended environments to evaluate student proficiency. In the first paper, Gobert and Koedinger present data from a study of the innovative, simulation-based learning environment, Science Assistments (www.scienceassistments.org). The Science Assistments platform records each move as students engage in inquiry practices. The system then uses model tracing to evaluate the actions that students performed to determine what students know about science inquiry. As students create hypotheses and design experiments, the system updates a model that estimates student proficiency on inquiry and reasoning skills. The Science Assistments system leverages the affordances of model-tracing algorithms to detect different patterns of student behaviors, including genuine discovery and confirmation bias. In the second paper, Clarke-Midura, McCall, and Dede investigate the use of an immersive environment as a platform for assessing student inquiry and reasoning skills. As students move their avatars through a 3-D world, they are able to make observations, gather and analyze data, and draw conclusions. Clarke-Midura, McCall, and Dede found that the performance assessment using the immersive environment was
ISLS
110
ICLS2012
Volume 2: Symposia
a reliable measure of inquiry and reasoning skills, though only a small number of students in the classroom studies demonstrated high-level inquiry skills. Finally, in the third paper, Davenport, Quellmalz, and Timms report the results of an empirical study to determine whether active, animations and interactive, simulation-based assessments are better than static assessments at distinguishing factual knowledge of scientific principles from meaningful inquiry and reasoning skills (e.g., generating predictions from observations, designing experiments, and drawing conclusions). A pool of 1566 students participated in a within-subjects design in their science classrooms. Students took science assessments in each of three different modalities; static, active (using dynamic animations), and interactive (using simulations). The results suggest that the dynamic and interactive assessments were more effective than the static assessment (most similar to traditional, paper-based tests) at distinguishing declarative, factual knowledge from deeper scientific reasoning and inquiry skills. After the presentations, James Pellegrino, an expert on technology and assessment, will provide a brief discussion and lead the question and answer session.
Significance
This symposium will bring together a variety of perspectives on the principled design and evaluation of nextgeneration assessments using simulation-based and immersive technologies. The symposium will create an opportunity for a broader discussion of the theoretical and practical considerations for leveraging emerging technologies to meaningfully assess complex science knowledge and inquiry practices.
Using Model-tracing to Conduct Performance Assessment of Students Science Inquiry Skill at Conducting Experiments Within a Microworld
Janice D. Gobert, Worcester Polytechnic Institute & Kenneth R. Koedinger, Carnegie-Mellon University
Introduction
Many national frameworks for science emphasize inquiry skills (e.g., NRC, 1996). However, in typical classroom practice, science instruction often focuses on rote learning in part because science process skills are difficult to assess (Fadel, Honey, & Pasnick, 2007) and rote knowledge is prioritized on high-stakes tests. Short answer assessments of inquiry have been used (cf., Alonzo & Aschbacher, 2004; Songer, 2006), however, these tend to not align well to current national frameworks (Quellmalz, Kreikemeier, DeBarger, & Haertel, 2006) and it is unclear whether they properly identify inquiry skills (Black, 1999; Pellegrino, 2001). Hands-on performance assessments are more authentic (Baxter and Shavelson 1994; Ruiz-Primo & Shavelson, 1996), however, these are seldom used in schools because of difficulty with reliable administration and the resulting high cost. Science Assistments (www.scienceassistments.org) learning environment assists and assesses (hence, assistments) middle school students on inquiry so teachers can assess their students skills during instruction-in the context in which they are developing (Mislevy et al, 2002).
Framework
As a proof of concept for automated assessment of scientific inquiry skills, we used model-tracing (Corbett & Anderson, 1995; Koedinger & Corbett, 2006) to develop a cognitive model of science inquiry skills, particularly, the control for variables strategy (Chen & Klahr, 1999) and warranting claims with data. This model provides a rich qualitative, process-oriented scoring of students inquiry moves within a guided scientific inquiry simulation for the domain of state change. We address the validity of this automated approach to performance assessment both quantitatively, in terms of reliability and predictive validity, and qualitatively, in terms of providing rich traces of student inquiry steps and mis-steps or haphazard inquiry (Buckley, Gobert et al, 2010). Additionally, we present Cronbachs alphas as reliability measures for each of our variables, and correlations with other inquiry tasks as additional construct validity data.
Methods and Data Sources

Participants. 78 eighth grade students (aged 12-14 years) from a public school in Central MA participated. Students belonged to one of six class sections and had one of two science teachers. Materials. Pre- and post-tests for inquiry skills (n=12) and domain knowledge (n=7) were used. A Phase Change Microworld activity was used with which students engaged in a series of inquiry tasks. Data Collection and Analysis: By applying model-tracing to students log data from their interactions with microworlds, we use production rules to code for: 1) CVS-relevance for each of the trials (using the control for variables strategy and relevant to the students hypothesis), 2) tested-and-true hypotheses for each of the trials
ISLS
111
ICLS2012
Volume 2: Symposia
(whether a claim is supported based on data), and 3) lastly, an average of these scores, referred to as %cvs+truetested for each of the trials. Our model tracer tracked whether: students initial hypotheses were scientifically accurate, whether the experimental trials they ran were relevant to their hypotheses, whether their trials used the control for variables strategy (Chen & Klahr, 1999), whether their final analysis entered was supported or unsupported by their data, and whether they had collected appropriate experimental evidence that supported their final conclusion (relevant controlled trials). Using data from the model-tracer, we calculated Cronbachs alpha for our variables to ascertain the reliability across the 4 trials on each of the measures. The Cronbachs alpha for the 4 CVS-relevant scores was 0.683; the Cronbachs alpha for the 4 true-tested scores was 0.741; and lastly, the Cronbachs alpha for the aggregate of the 2 inquiry scores across the 4 trials, %CVS+true-tested, was 0.774, indicating a high degree of internal consistency for each of the three measures. Correlations were calculated between our auto-scored performance measures of inquiry with specific post-test inquiry items that should, in theory, be related. We obtained moderate correlations between our performance measures of inquiry and our post-test items for identifying an independent variable, identifying a dependent variable, and demonstrating the control of variables strategy (CVS).
Findings
In this paper we have shown that we can use model-tracing as a method of performance assessment for science inquiry skills, an ill defined domain. This builds upon the extensive work that has been done to date for welldefined domains such as math (Corbett & Anderson, 1995; Koedinger & Corbett, 2006). Additionally: 1) the reliability of our machine-scored measures of inquiry are highly consistent across the 4 Assistment activities or trials, suggesting that we can reliably capture students inquiry performance on these rich inquiry tasks, and 2) our measures are moderately correlated with post-test measures of inquiry performance for analogous concepts. Lastly, our data show that model-tracing can detect interesting patterns of student inquiry such as confirmation bias and overcoming confirmation basis. These are important data with respect to demonstrating auto-scoring of rich inquiry behaviors, but are also important, particularly the former, in terms of its implications for adaptive scaffolding of student inquiry, such as that being done by the Science Assistments group (www.scienceassistments.org; Gobert et al, 2007, 2009; Sao Pedro et al, in press).
Significance
This work makes contribution to theoretical understanding of scientific inquiry, to its assessment, and to technical methods to auto-score inquiry. This represents an advance in this area since to date there has been difficulty in separating inquiry from the domain-specific context in which it was learned (Mislevy et al., 2002; Gobert, Pallant, & Daniels, 2010), and difficulty measuring inquiry skills due to their complexity and the amount of data required for reliable measurement (Shavelson et al, 1999).
Assessing Science Inquiry using Immersive Virtual Environments

Jody Clarke-Midura, Marty McCall, & Chris Dede, Harvard Graduate School of Education
Introduction
Scientific inquiry is the method by which scientists interact with and study the world. While detailed definitions of inquiry can be complex, at its core the process is hypothesized to involve theorizing and investigating. For example, Kuhn and colleagues define inquiry learning as investigations where students individually or collectively investigate a set of phenomena (virtual or real) and draw conclusions about it (Kuhn, Black, Keselman, & Kaplan, 2000). Similarly, White, Collins, & Frederiksen (2011) offer a definition of inquiry as a process that oscillates between theory and evidence within the practice of argumentation. Given the multifaceted and open-ended nature of inquiry, it is not surprising that research has found existing methods for assessing science inquiry learning to be limited (Quellmalz, Kreikemeier, DeBarger, & Haertel, 2006; USDE, 2010). Traditional science assessments either fail to align with the active nature of inquiry (Quellmalz et al. 2006) or have difficulty distinguishing what part of the complex reasoning process students do not understand (Gotwals & Songer, 2010). This is partly due to the difficulty of capturing students actions and behaviors as they perform a paper-based or hands-on task. Digital media, such as virtual environments and simulations, allow us to create tasks that are more characteristic of how students engage with inquiry in the real world; these processes and trajectories are unobtrusively captured as log data (Clarke, 2009). In doing so, these techologies allow us to create observations of student learning not possible via hands-on, paper-based, and multiple choice assessments (Clarke-Midura & Dede, 2010). In this paper, we discuss how virtual performance assessments can provide reliable observations of students inquiry knowledge. These virtual performance assessments are delivered via
ISLS
112
ICLS2012
Volume 2: Symposia
an immersive environment and require students to solve an authentic scientific problem in a virtual context (see vpa.gse.harvard.edu).
Framework
We used the Evidence Centered Design framework (ECD; Mislevy, Steinberg, & Almond, 2003) to develop our assessments. ECD provides a framework for developing assessment tasks that elicit evidence (scores) that bears directly on the claims that one wants to make about what a student knows and can do (Shute, et al., 2007, p. 6). Using this framework, we reframed our science inquiry constructs into specific knowledge, skills and abilities (KSAs) aligned with national frameworks. We then converted the KSAs into proficiencies: data gathering, evaluating evidence, experimenting, and reasoning from evidence. Through the process of articulating the exact details of what is being measured and how it is being measured, it is easy to link the KSAs to evidence of student learning. Linking KSAs like this provides a high degree of validity that research has found often lacking in performance assessments (e.g. Linn, Baker, & Dunbar, 1991).

In the virtual performance assessments, students have the ability to walk around the virtual environment, make observations, gather data, and run tests in a laboratory, in order to solve a scientific problem (see figures 1 and 2 below). They then build a claim and support it with evidence they gathered.
Figures 1 and 2: Screenshots of the Virtual Assessments. As mentioned above, we compiled our inquiry skills into proficiencies: data gathering, evaluating evidence, experimenting, and reasoning from evidence (making claims). Observations of students demonstrating these skills as they participated in the assessment were compiled into groupings. Unlike traditional tests that characterize student performance in terms of response correctness, the interactive environment records whether or not students engaged in an activity. Students have varying levels of understanding about the scientific process. They may, for example, know that scientists collect evidence and make observations, but may not understand the role of control data or how to interpret observations in the light of previous research. The pattern of engagement in the interactive environment and the examinees final demonstration of evidence and reasoning from this evidence provide information about understanding of the scientific process. In order to assess the reliability of these observations as evidence of a students proficiency, we asked the following research question: Are observations of students skills (i.e. reasoning from evidence) behaving as if they are governed by a coherent skill? This study involved 643 middle school students in two states in the US (females=341). In order to answer the research question, we used Item Response Theory (IRT). IRT is a type of latent trait analysis that can be viewed as non-linear factor analysis. A trait is any skill or ability that determines the likelihood of a specific response to a test question. Observations generated from the log data were recoded from raw data as degrees of correctness (no, partial, or full credit for demonstrating the skill). Both the ability of the examinee and the difficulty of observation levels are on the same scale. Data were analyzed using WINSTEPS (Linacre, 2003) with a partial credit one-parameter model.
Findings
IRT analysis found the observations were providing evidence of a solid trait. Due to limited space, this paper presents results on students ability to reason from evidence. Results showed moderate fit statistics (Table 1). The item-total correlation coefficients are very highthey should be over .25indicating that student who did well on the overall skill also did well on each observation. This is an indicator that observations are performing as expected by the model, supporting the claim that actions are governed by a coherent trait.
ISLS
113
ICLS2012 Table 1. Item Fit Statistics Overall difficulty Infit Outfit 0.07 0.58 -0.08 -0.2 -0.42 0.23 0.55 -0.73 0.64 0.72 0.67 1.08 1.26 1.12 1.23 1.29 0.61 0.67 0.63 1.11 1.57 1.17 1.3 1.28
Volume 2: Symposia
Question Observation1 Observation2 Observation3 Observation4 Observation5 Observation6 Observation7 Observation8
Item-total correlation 0.76 0.77 0.74 0.42 0.27 0.39 0.49 0.50
While we found that we were able to model latent traits based on observations of students actions in the world, there were surprising findings about how students were responding to the multivariate nature of the problem. Our findings were in line with Kuhn et als research (2000) on students misconceptions around multivariate systems: students had difficulty teasing out causal and non-causal factors on the outcome. students were not able to distinguish the additive effects that individual features contributed to the respective effects on the outcome. students focused on surface level features while solving the problem.
We are in the process of exploring various analytic approaches to modeling the log data that will provide further insight and diagnostic data on what actions lead to lower and higher inquiry abilities.
Significance
Science inquiry is a complex process. Our research shows that we can use virtual environments to simulate the complexity of inquiry, while reliably assessing student learning. We demonstrated the reliability of our measure using traditional methods (IRT). However, we are also exploring how we can model student observational data from the virtual assessment using Bayesian networks and cognitive diagnostic models. Only through deep understanding of how to measure and model inquiry will we better understand the best methods for teaching it and for preparing our students to understand the complexity of multivariate systems.
Affordances of Dynamic and Interactive Assessments for Measuring Science Inquiry and Reasoning
Jodi L. Davenport, Edys S. Quellmalz, WestEd, and Michael Timms, ACER
Introduction
What are the affordances of simulation-based assessments for eliciting science inquiry skills? Computer-based assessments can portray dynamic information and allow for simulations that are interactive and responsive to student input. Technology-based science tests have been piloted in international testing programs including the Programme for International Student Assessment (PISA) (NGB, 2006; Koomen, 2006). These interactive assessments appear to be ideally suited to measuring student proficiency on science inquiry and reasoning skills. However, computer-based assessments are substantially more costly to develop and have many technical requirements. In the current study, we investigate whether dynamic and interactive assessments are more effective than traditional, static assessments at discriminating student proficiency on three types of science practices: identifying scientific principles (e.g., stating or recognizing principles), using science principles (e.g., predicting or explaining), and conducting inquiry (e.g., designing experiments).
Framework
Our project uses the theoretical frameworks of evidence-centered design and model-based learning to identify the interconnected knowledge and skills that form the student models in our assessments. The evidence-centered assessment design framework shapes the development of the assessments in our study. The process begins with domain analysis, then specification of the student models (content and inquiry to be tested), task models (designs of assessment tasks and items) and evidence models (data for content and inquiry learning) (Mislevy et al., 2003). Research on model-based learning suggests that effective science learners form, use, evaluate, and revise their mental models of phenomena in a recursive process that results in more complete, accurate, and
ISLS
114
ICLS2012
Volume 2: Symposia
useful mental models of a science system (Gobert & Buckley, 2000). Further, cognitive research shows that learners who internalize schema of complex system organizationstructure, functions, and emergent behaviorscan transfer this heuristic understanding across systems (e.g., Goldstone & Wilensky, 2008). The assessment tasks were designed to elicit evidence of science practices described in the 2009 National Assessment of Educational Progress (NGB, 2006). The assessments require students to students carry out the science practices in the context of the content domain of ecosystems, and to explicitly consider the components, interactions, and emergent behaviors characteristic of all complex systems. To ensure the construct validity of the items, we carried out expert reviews with 3 independent reviewers and student think-alouds with 10 middle school students. Both expert reviews and student think-alouds revealed that the items elicited the targeted science practices.

A sample of 1566 students from the classrooms of 22 teachers in 12 states in the United States participated in the study. Each student completed 3 versions of computer-based Ecosystems assessments that varied in the level of interactivity (static, active, and interactive). In the static version, students viewed still images. In the active version, students viewed animations, but did not conduct active investigations. In the interactive version, students designed and ran their own experiments. Items were aligned across the assessments to tap into the same science practice skills (identifying scientific principles, using science principles, and conducting inquiry) and subskills (e.g., predicting or explaining, drawing conclusions). See Figure 1 for examples of the different modalities of assessment. The assessments were given in three consecutive sessions and the order of the sessions was counterbalanced (e.g., some students receive the static versions during the first session and others receiving the interactive versions first). To analyze the results, we used a generalizability study (G-Study) analysis which estimates the correlations between items designed to tap into the three science practices, and can determine which sets of items appear to be measuring distinct constructs versus the same construct. Our hypothesis was that the interactive assessment would be the most effective at distinguishing student abilities on conducting inquiry from the other science practices, as only the interactive assessment allows students to actively engage in designing experiments and gathering data based on their designs. We also hypothesized that the active, dynamic assessment would be more effective than the static assessment at distinguishing between using science principle and identifying principles, as the active, dynamic assessment allows students to watch animations and make observations of dynamic systems to generate predictions and explanations.
Figure 1. top left, static; top right, active; bottom, interactive
Findings
ISLS 115
ICLS2012
Volume 2: Symposia
Consistent with our hypotheses, our G-study analysis found that the active, dynamic assessment was the most effective at distinguishing student performance between identifying science practices (e.g., declarative facts) and using science principles (e.g., making predictions and explanations). The interactive, simulation-based assessment was the most effective at distinguishing student performance on conducting inquiry from either of the other science practices. Table 1 shows the correlations between each of the science practices for each of the assessment modalities. Notice that the active, dynamic assessment had the lowest correlation between identifying and using principles, and that the interactive assessment had the lowest correlation between conducting inquiry and either identifying or using principles. Table 1: G-study correlations between science practices across assessment modalities. Correlation between Identifying Principles and Using Principles 0.92 0.80 0.82 Correlation between Identifying Principles and Conducting Inquiry 0.80 0.80 0.72 Correlation between Using Principles and Conducting Inquiry 0.91 1 0.84
Static Active/Dynamic Interactive
Significance
The current study provides the first large-scale empirical evidence of the affordances of dynamic and interactive assessments for discriminating among science reasoning and inquiry skills. Dynamic assessments are more effective than static assessments at differentiating between factual knowledge and the ability to apply that knowledge in meaningful contexts. Simulation-based, interactive assessments, with environments that are responsive to student actions, were more effective than either static or active assessments at uniquely measuring students ability to conduct inquiry.
References
Black, P. (1999). Testing: Friend or Foe? Theory and Practice of Assessment and Testing. New York, NY: Falmer Press. Chen, Z., & Klahr, D. (1999). All Other Things Being Equal: Acquisition and Transfer of the Control of Variables Strategy. Child Development, 70 (5), 1098-1120. Clarke, J. (2009). Exploring the Complexity of Inquiry in an Open-Ended Problem Space. Doctoral dissertation presented to the Fellows of Harvard University. Cambridge: Harvard Graduate School of Education. Clarke-Midura, J., & Dede, C. (2010). Assessment, technology, and change. Journal of Research on Technology in Education, 42(3), 309-328. Corbett, A., & Anderson, J. (1995). Knowledge-Tracing: Modeling the Acquisition of Procedural Knowledge. User Modeling and User-Adapted Interaction, 4, 253-278. Fadel, C., Honey, M., and Pasnick, S. (2007). Assessment in the Age of Innovation, Education Week, Volume 26 (38), 34-40. Gobert, J. D., & Buckley, B. C. (2000). Introduction to model-based teaching and learning in science education. International Journal of Science Education, 22(9), 891894. Gobert, J.; Heffernan, N.; Koedinger, K.; Beck, J. (2009). ASSISTments Meets Science Learning (AMSL). Proposal (R305A090170) funded by the U.S. Dept. of Education. Gobert, J.; Heffernan, N.; Ruiz, C.; Kim, R. (2007). AMI: ASSISTments Meets Inquiry. Proposal NSF-DRL# 0733286 funded by the National Science Foundation. Gobert, J.D, Pallant, A.R., & Daniels, J.T.M. (2010). Unpacking inquiry skills from content knowledge in Geoscience: A research perspective with implications for assessment design. International Journal of Learning Technologies, 5(3), 310-334. Goldstone, R. L., & Wilensky, U. (2008). Promoting transfer through complex systems principles. Journal of the Learning Sciences, 17, 465516. Gotwals, A.W. and Songer, N.B. (2010) Reasoning Up and Down a Food Chain: Using an Assessment Framework to Investigate Students' Middle Knowledge. Science Education (94)2. P. 259-281. Koedinger, K., & Corbett, A. (2006). Cognitive Tutors: Technology Bringing Learning Sciences to the Classroom. In R. Sawyer, The Cambridge Handbook of the Learning Sciences (pp. 61-77). New York, NY: Cambridge University Press. Koedinger, K. R., Suthers, D. D., & Forbus, K. D. (1999). Component-based construction of a science learning space. International Journal of Artificial Intelligence in Education, 10, 292-313.
ISLS
116
ICLS2012
Volume 2: Symposia
Koomen, M. (2006, April). The development and implementation of a computer-based assessment of science literacy in PISA 2006. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA. Kuhn, D., Black, J., Keselman, A., Kaplan, D. (2000). The Development of Cognitive Skills to Support Inquiry Learning. Cognition & Instruction, 18(4), 495-523. Linacre, M. J. (2003). Winsteps Rasch measurementsoftware [Computer software]. Chicago: Winsteps. Mislevy, R., & Haertel, G. (2006). Implications of evidence centered design for educational testing PADI Technical Report 17. Menlo Park, CA: SRI Interantional. Mislevy, R. J., Chudowsky, N., Draney, K., Fried, R., Gaffney, T., and Haertel, G. (2002). Design patterns for assessing science inquiry. Unpublished manuscript, Washington, D.C. Mislevy, R., Steinberg, L. S., & Almond, R. G. (2003). On the structure of educational assessment. Measurement: Interdisciplinary Research & Perspective, 1(1), 3-62. Mislevy, R., Almond, R., Yan, D., & Steinberg, L. (1999). Bayes nets in educational assessment: Where do the numbers come from? In Kathryn B. Laskey and Henri Prade, editors, Proceedings of the Fifteenth Conference on Uncertainty in AI, San Francisco. Morgan Kaufmann Publishers, Inc. National Committee on Science Education Standards and Assessment. (1996). National Science Education Standards, Washington, D.C., National Academy Press. National Governing Board (NGB). (2006). Science framework for the 2009 National Assessment of Educational Progress. Washington, DC: Author. National Research Council. (2000). How People Learn: Brain, Mind, Experience, and School: Expanded Edition. Committee on Learning Research and Educational Practice. M.Suzanne Donovan, John D.Bransford, and James W.Pellegrino, editors. Washington, DC: The National Academies Press. Pellegrino, J. (2001). Rethinking and redesigning educational assessment: Preschool through postsecondary. Denver, CO: Education Commission of the States. Quellmalz, E., Kreikmeier, P., DeBarger, A. H., & Haertel, G. (2006). A study of the alignment of the NAEP, TIMSS, and New Standards Science Assessments with the inquiry abilities in the National Science Education Standards. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, CA. Sao Pedro, M., Baker, R., Gobert, J., Montalvo, O., & Nakama, A. (in press). Using Machine-Learned Detectors of Systematic Inquiry Behavior to Predict Gains in Inquiry Skills. To appear in User Modeling and User-Adapted Interaction. Shute, V. J., Hansen, E. G., & Almond, R. G. (2007). An assessment for learning system called ACED: Designing for learning effectiveness and accessibility. (RR-07-26). Princeton, NJ: Educational Testing Service. Strauss A, Corbin J. (1990). Basics of Qualitative Research: Grounded Theory Procedures and Techniques. Newbury Park, CA: Sage. United States Department of Education (USDE). (2010). National Education Technology Plan 2010. Washington, DC: US Department of Education. White, B., Collins, A., & Frederiksen, J. (2011). The nature of scientific meta-knowledge. In M. S. Khine & I. Saleh (Eds.), Dynamic modeling: Cognitive tool for scientific enquiry. London: Spinger.
Acknowledgments
The research in the paper by Clarke-Midura and Dede was supported by grants from the Institute of Education Sciences (IES#R305A080141) and the Bill and Melinda Gates Foundation to Harvard University. Any findings, interpretations, and conclusions do not constitute an official position of the U. S. Department of Education. The authors gratefully acknowledge the contributions of Drs. Jillianne Code, Geordie Dukas, Michael Mayrath, Nick Zap, as well as the assistance of numerous students at the Harvard Graduate School of Education. The material in the paper by Davenport, Quellmalz, and Timms is based upon work supported by the National Science Foundation under Grant No. DRL-0814776 awarded to WestEd, Edys Quellmalz, Principal Investigator. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
ISLS
117
ICLS2012
Volume 2: Symposia
Scripting Science Inquiry Learning in CSCL Classrooms

Annelies Raes, Tammy Schellens, & Bram De Wever, Dept. of Educational Studies, Ghent University, Belgium Email: annelies.raes@ugent.be, tammy.schellens@ugent.be, bram.dewever@ugent.be Ingo Kollar, Christof Wecker, Sybille Langer, & Frank Fischer, Ludwig-Maximilians-Universitt, Germany ingo.kollar@psy.lmu.de, christof.wecker@psy.lmu.de, sybille.langer@psy.lmu.de, frank.fischer@psy.lmu.de Mike Tissenbaum, James D. Slotta, Ontario Institute for Studies in Education, University of Toronto, Canada Email: miketissenbaum@gmail.com, jslotta@oise.utoronto.ca Vanessa L. Peters & Nancy Butler Songer, University of Michigan, School of Education, USA Email: vlpeters@umich.edu, songer@umich.edu Chair: Bram De Wever, Ghent University, Belgium, bram.dewever@ugent.be Discussant: Pierre Dillenbourg, Ecole Polytechnique Fdrale de Lausanne, Switzerland, pierre.dillenbourg@epfl.ch Abstract: Research on scripting computer-supported collaborative learning (CSCL) has recently received a lot of attention. However, most findings within this research grew out of studies focusing scripting online collaborative learning activities that often had an asynchronous nature and were conducted in artificial settings. This symposium includes an international set of presenters from Belgium, Canada, Germany, and the USA and brings together four studies that focus on scripting face-to-face classroom activities, seeing the classroom as a formal physical learning environment. The common denominator of the contributions is that they are all field studies focusing on computer-supported science inquiry learning, aiming to investigate the optimal conditions for organizing these inquiry learning environments. Each paper will present the research context, method, data, and conclusions on how scripting can be implemented to support science inquiry learning. Broader implications of the findings of these studies will be discussed with the audience.
Aim and Structure of this Symposium

The main aim of this symposium is to highlight the added value of scripting CSCL activities in authentic science classrooms. Within the field of Computer-Supported Collaborative Learning, a lot of attention has been paid to how learning environments need to be organized in order to promote students learning. There is widespread agreement that technology in itself does not support learning (Lai, 2008) and that technology should be well integrated into a learning environment in order to realize its full potential (Weinberger, Reiserer, Ertl, Fischer, & Mandl, 2005). In this respect, the introduction of scripts (see Kobbe et al., 2007) has been proven useful. More specifically, earlier research showed that scripts have proven to be a powerful instructional approach to foster specific collaborative activities and interaction patterns, such as identifying conceptual differences, asking thought-provoking questions, integrating multiple perspectives, and/or constructing arguments and counter-arguments (Weinberger, Stegmann, & Fischer, 2010; cf. a recent discussion at CSCL 2011, see Gijlers, van Dijk, & Weinberger, 2011, p. 1071). While a lot of research towards scripting has been focusing on online usually asynchronous learning environments, the present symposium focuses on studying scripting in CSCL authentic classroom settings. Within these settings, the collaboration script is functioning next to, or in addition with, the teacher in the learning environment or classroom. Through bringing together the work of scholars from four distinct research groups from four different countries, we hope to provide a diversity of perspectives and empirical data to widen the discussion on the issue of implementing scripting in authentic CSCL situations. This symposium comprises four papers, which all focus on (1) science inquiry learning, (2) in authentic situations, (3) supported by technology, (4) in secondary education, and (5) the role of scripting in these learning environments. The first two studies focus more on collaboration scripts within the classrooms, in order to find ways how to introduce small group collaboration scripts and potentially combine them with the teacher as another potential source of support for the learners. The third study investigates an emergent script introducing user-generated content in a physics learning environment. The fourth study centers on implementing a mobile technology application to script students activities while performing field observations. The discussant, Pierre Dillenbourg, who is not affiliated with any of these research programs, but has studied scripting in high detail, will synthesize the ideas and advances in this work and lead a discussion amongst the participants and the audience. He will invite (a) the participants to connect and compare their findings and put forward specific implications for inquiry based science learning in TEL classrooms and (b) the audience to relate to and critically comment on these findings and scientific and practical implications.
ISLS
118
ICLS2012
Volume 2: Symposia
Paper 1: Scripting Computer-Supported Collaborative Information Problem Solving on the Web

Annelies Raes, Tammy Schellens, & Bram De Wever
Introduction
Previous research has indicated that learning science content by means of a web-based inquiry project is effective in enhancing learners knowledge acquisition (Slotta & Linn, 2009) and their metacognitive awareness while solving information problems on the web (Raes, Schellens, De Wever, 2011). However, since students in a web-based inquiry project usually work together in small groups, it is vital to pay attention to the collaboration process and students (socially) shared regulation, in addition to processes on an individual level (Spada & Rummel, 2005; Vauras, Iiskala, Kajamies, Kinnunen, & Lehtinen, 2003). In this respect, this study focuses on both the individual and collaborative processes playing when students learn by means of a web-based inquiry science project and deal with information problems in an authentic classroom setting. Moreover, it is questioned whether the quality of collaboration and shared regulation can be improved by integrating a collaboration script and subsequently, if this also leads to higher individual learning outcomes. Providing students with a collaboration script is put forth as a way to support computer-supported collaborative learning (CSCL) since collaboration scripts facilitate social and cognitive processes of collaborative learning by shaping the way learners interact with each other (Kobbe et al., 2007; Kollar, Fischer & Slotta, 2007). The script developed for this study particularly focused on the aspect of roles and the mechanisms task distribution and sequencing aiming at prompting the regulatory skills that critical information problem solving on the web entails (BrandGruwel, Wopereis, & Walraven, 2009). The one student was the executer assigned to operate the computer and typing the answers, the other student was the web detective assigned to critically supervise the online search activities. Students were prompted to switch roles during the project. The effects of the integration of such a collaboration script are investigated through a quasiexperimental field study. In total 220 students from 13 different secondary school classes were involved in this study. The intervention consisted of the implementation of a web-based collaborative inquiry project lasting 4 sessions of 50 minutes. During the first session, secondary students completed an individual pretest and were introduced to the Web-based project provided on the Web-based Inquiry Science Environment (WISE) (Slotta & Linn, 2009). They were free to choose their partner and started the first introductory activity of the WISEproject in dyads. Students worked in the same dyads during the whole intervention and navigated through the sequence of inquiry activities using the inquiry map in the online learning environment. During the project, students were asked to write their answers down in reflection notes. Finally, all students completed an individual posttest. Seven classes were provided with a collaboration script embedded in the WISE-project (experimental condition, N = 97 students), six classes were not provided with this collaboration script (control condition, N = 107 students).
Data and Methods

The study used a mixed-methods approach in which both quantitative and qualitative sources of evidence are triangulated. The quantitative part of the study consisted of pretest-posttest-comparisons regarding the outcome measures knowledge acquisition and metacognitive awareness during information problem solving on the Web. The latter variable was measured by facing students with an unfamiliar information problem in pretest and posttest, more specifically a scientific controversy. They were assigned to take up a particular position that they needed to justify with appropriate evidence from the web to support their claim. After performing this IPS-task students were asked to fill out an adapted version of the Metacognitive Awareness Inventory (MAI) (Schraw & Dennison, 1994). This self-report inventory was used to measure students perception about their metacognitive and strategic activities while performing the task. Moreover, students answers during the web-based projects were scored and summed to result in a process score for every dyad. Quality of collaboration was measured by means of a self-perception questionnaire filled out by the students after finishing the project. In addition, the collaborative processes of a subset of these students (35 dyads) were videotaped and the quality of collaboration was rated based on the scheme of Meier, Spada and Rummel (2007). In this study the students worked together in small groups within classes, and as such, the problem under investigation has a clear hierarchical structure. In this respect, the analysis of test data at an individual level raises a methodological issue frequently discussed in research on group learning and collaborative problem-solving (e.g. Cress, 2008; De Wever, Van Keer, Schellens, & Valcke, 2007). Because of this, the Hierarchical Linear Modeling is used for testing main effects and interaction effects of predictor variables on different levels. The independent variables that were taken into account were gender, achievement level and perceived quality of collaboration at student level; group composition based on gender, group composition based on achievement and students process score at group level; and academic track (science track vs. general track) and condition (collaboration script present vs. absent) at class level.
ISLS
119
ICLS2012
Volume 2: Symposia
The software MLwiN 2.23 for multilevel analysis was used to analyze the hierarchical data. A threestep procedure has been followed to analyze the effects of the presented explanatory variables on the dependent variables. The first step consists of the estimation of a four-level conceptual null model, which serves as a baseline model. This unconditional null model without any predictor variables provides both the overall pretest score and the overall learning gain for all students across all groups and classes. Moreover this null model will, by means of the Intra-Class-Correlation (ICC), answer the question if the outcome measures vary among students, across groups and across classes. The second step concerns the input of the main explanatory variables in the fixed part of the model and allows cross-level interactions between student and class characteristics. This will give insight in the differential effects for different groups of students with different student and class characteristics. Finally, in the third step, the aggregated characteristics based on gender and achievement level, i.e. group composition, was added to the model. The results of these quantitative multilevel analyses will be presented and discussed at the conference in addition with qualitative analyses on the collaborative inquiry activities of the 35 different groups. These in-depth analyses will deepen the quantitative results and will give insight in how students deal or deal not with the embedded collaboration script and how this influences their learning performances and outcomes.
Paper 2: Using Small Group and Classroom Scripts to Foster High School Students Acquisition of Online Search Competence
Ingo Kollar, Christof Wecker, Sybille Langer, & Frank Fischer
Introduction
An important strategy in developing well-grounded positions in societal debates on science issues, such as whether pre-implantation diagnostics should be allowed or not, is to search the Internet for relevant information. However, finding information that is relevant, impartial, credible and scientifically sound is a challenging task for high school students. One promising approach to foster high school students online search competence is (web-based) collaborative inquiry learning. Yet, without appropriate scaffolding, inquiry learning is likely to fail (de Jong, 2006). This contribution looks at the effects of two kinds of scaffolds that can be used during collaborative inquiry learning. First, it investigates small group collaboration scripts that specify, sequence and distribute learning activities among roles that are filled by the members of a small group (e.g., Kollar, Fischer & Hesse, 2006). With respect to online search, such a script may for example have one learner suggest certain search terms, while her learning partner is asked to anticipate what unwanted search results will likely be triggered by these terms. Similar task distributions could be realized for further steps of the online search process (e.g., selecting links from a hit list or finding information on a selected website). Although prior research has shown that small group collaboration scripts can be designed to foster quite some variety of domain-general skills (e.g., argumentation; see Stegmann, Weinberger, & Fischer, 2007), it is still an open question whether they can also foster students online search competence. As a second kind of scaffold, we look at classroom scripts that distribute learning activities over the different social planes of the classroom (Dillenbourg & Jermann, 2007). With respect to online search, one could imagine a classroom script that first has the teacher model successful online search behavior (plenary activity), followed by dyadic and then individualized Internet search. We compared the effects of two classroom scripts (variant 1: all search activities were carried out on the small group level, i.e. in dyads, vs. variant 2: search activities changed between the plenary level, i.e. through teacher-student or student-student modeling, and the small group level) that were either enriched or not enriched by a small-group collaboration script. With respect to classroom scripts, we expected that the script that included plenary (i.e. modeling) in addition to small group activities (variant 2) would work better than the script that located all search activities on the small group level (variant 1). We also expected that when all search activities are conducted on the small group level (variant 1 of the classroom script), a small group collaboration script would effectively scaffold learners acquisition of online search competence. Most effective, however, should be the combination of a small-group collaboration script and a classroom script that included search activities on the plenary and the small-group level to be.
Data and Methods

Participants were 174 9th graders from eight classes from urban high schools (90 female, 84 male). Within regular Biology lessons led by the students regular Biology teachers, who were trained to implement the experimental condition, their class was assigned to one of the four experimental conditions of a 2x2 design (see below). Learners were equipped with laptops and worked on a web-based inquiry learning curriculum unit on Genetic Engineering. The main task was to develop a well-grounded position towards the question whether Genetic Engineering should be allowed or not. The learning phase spanned seven regular Biology lessons and involved considerable time to search the Internet for arguments and evidence. During these search phases, we realized a 2x2-factorial design with the factors small-group collaboration script (present vs. not present) and type of classroom script (small-group level only vs. small-group and plenary level). In all groups, we
ISLS
120
ICLS2012
Volume 2: Symposia
implemented collaborative browsing, i.e., during their online search processes, the laptops of the two co-present partners of a dyad were connected so that the browser displayed identical web pages on both screens, no matter who of the two learners had accessed them. The small-group collaboration script was implemented as a browser plug-in that segmented the browser window into two frames (a scaffolding frame and the regular browser frame). The scaffolding frame displayed prompts related to the actual step in the search process (e.g., Google start page, Google hit list, chosen web site) and distributed them between the two learning partners. For example, accompanying a Google hit list, the scaffolding frame displayed prompts such as Suggest a link which from your perspective is likely to contain valuable information to one learner and Listen to your partners suggestion and estimate whether the link s/he suggested is (a) impartial, (b) relevant and (c) scientifically grounded to his/her learning partner. In the conditions without the small-group collaboration script, no prompts were displayed. The type of classroom script was also manipulated during the online search phases. In the small-group level only classroom script, online search was to be conducted in dyads. In the classroom script with both the small-group and the plenary level, single steps were modeled in front of the class before this activity was to be conducted on a dyadic level. Online search competence was measured by the students performance in an individual test that asked them to describe in as much detail as possible how they would use the Internet to form a position concerning the question whether nuclear power plants should be abandoned. 15% of the data included in the analysis was analyzed by two independent coders. Coding was based on a scheme that captured adequate steps and important quality characteristics during successful online search (e.g., judge the credibility of a website). Intra-class correlation was sufficient (ICC = .83). An analogous test on a different topic was used as a pretest (ICC = .51).
Results and Discussion

Results indicated that as expected, when no small-group collaboration script was provided, the classroom script with online search activities both on the plenary and the small group level led to higher levels of online search competence than the classroom script that located all search activities on the small group level, F(1,66) = 23.89, p < .01, partial Eta = .27. Also confirming our expectations, the small group collaboration script proved more effective than no small group script, when all search activities were located on the small group level, F(1,90) = 10.06, p < .01, partial Eta = .10). Adding to these results, first results from further in depth process analyses revealed that both the small group collaboration script and the classroom script comprising activities on both the plenary and the small group level led to higher quality levels of online search behavior than no such support. In contrast to our hypothesis, however, the combination of the small group collaboration script and the classroom script that alternated plenary and small group search phases was not the most effective concerning the learners acquisition of online search competence. When the classroom script covered both the small-group and the plenary level, adding the small-group collaboration script had a slight negative effect, F(1,74) = 4.27, p < .05, partial Eta = .06. Thus, it appears that modeling as part of a classroom script can be regarded as a viable alternative to scripting small group collaboration (see also Rummel & Spada, 2005).
Paper 3: Scripting Collective Inquiry in High School Physics

Mike Tissenbaum & James D. Slotta
Introduction
Our work focuses on developing a smart classroom for knowledge communities where all students are actively involved in the production, aggregation, and assessment of science content (Ulrich et al., 2008; Ito et al., 2009). Having students contribute their own educational content provides opportunities for them to understand connections amongst often disparate pieces of information within a domain. The most common way of creating such connections is by assigning meta-data, or tags, to individual content (Mathes, 2004; Wiley, 2000)., allowing individuals to assign descriptors without needing to know about every other piece of content that shares the same designation. They can rely on the emergence of a collective data set, guided by the tags, to reveal meaningful connections and increased usefulness of content elements (Hayman & Lothian, 2007). To define and coordinate the flow of materials, activities and interactions within such complex collaborative inquiry, learning scientists have advanced the notions of scripting and orchestration (Dillenbourg, Jarvela & Fischer, 2009; Dimitriadis, 2001), where specified learning and interaction designs (i.e., the script) are enacted (the orchestration). The script can be seen as a formalism that captures the pedagogical structure of a learning design (e.g., student must upload two relevant videos) and collaboration patterns between students, their peers, and the teacher. When user-contributed materials are introduced, the script becomes more openended, and any design must be left somewhat unbound allowing specific themes, directions or content collections to emerge during its enactment (Peters & Slotta, 2010). Technology-supported inquiry learning environments offer a proven means of scaffolding student learning, encouraging reflection, providing timely access to tools and materials, and engaging learners collaboratively (Slotta & Linn, 2009).
ISLS
121
ICLS2012
Volume 2: Symposia
This present paper reports on our efforts to formalize scripts, in terms of the roles, groupings, learning contexts, technology, materials, and interaction patterns for inquiry and knowledge communities. Towards developing a formal concept of scripting, we conducted a series of quasi-experimental studies, beginning with investigations of small, controlled micro scripts, followed by more complex macro scripts, where students contribute content and actively use the emerging aggregate knowledge base. Below, we report on two studies of micro scripts, and a larger macro script to support high school physics students as they collectively upload content, solve, tag and explain physics problems, then re-use those resources in scripted inquiry activities. Students were scaffolded in all aspects the script by a smart classroom technology framework (Slotta, 2010).
Data and Methods: Three Successive Designs of Collaborative Inquiry Scripts

Our first iteration investigated a collaborative inquiry script called Tag, Answer, and Reflect (TAR) where students uploaded their responses to homework problems, including their selection of physics principle tags, then reflected on the aggregate set of tags and explanations. We investigated the following questions: (1) Did TAR help students make connections between physics principles? (2) How did different technologies (large format displays, customized views) help the teacher to understand students knowledge, for adjustments to the script before, and orchestration of the class during its enactment? (3) Did a visualization of student work help students make connections between physics principles? Two different comparisons were examined: First, individuals were compared with small groups as they examined aggregated responses from the community; second, the effect of large-format displays of a groups work vs. a single shared laptop was examined in terms of improved student performance. Overall, groups faired significantly better at solving problems (97% overall accuracy) than individuals working at home (80% overall accuracy), with t=2.02, and p<0.05. The connections made by the groups (80.94%) were also more accurate than individuals (76.57%), although only marginally significant (p=.081). The large-format displays were seen to aid the teacher in orchestrating class activities, allowing the detection of student errors, and supporting spontaneous short lectures and discussions with small groups. Our second iteration sought to extend the TAR script, in order to better aid students in their reflections and investigate the impact of the aggregated knowledge on group performance. Our quasi-experimental study compared two enactments: one where students were provided with access to the aggregated individual responses during their group reflection phase, and one where no aggregates were provided. Groups that received the aggregated responses outscored those who did not (t=4.13, p<0.01), as measured by a rubric developed with the teacher. These outcomes further our ideas of TAR, including the timing of supports for the teacher. The third study builds on the findings from the previous two, establishing a macro script to address a persistent, year-long curriculum engaging students in tagging and reflecting about homework problems while also uploading and tagging found examples from everyday life, including photos, Web URLs, and physics problems. We thus introduce a notion of a macro script specifying longer-term patterns, supported by micro scripts like TAR. The macro script in this study, called the collective inquiry cycle, (CIC) involves five steps: (1) Students submit individual inquiry items to the knowledge base; (2) Collectively (at home or in class) students examine and tag peers work, adding comments to explanations; (3) Teacher reviews the community knowledge, to prepare a discussion; (4) In-class activity engages students with collective knowledge artifacts chosen by teacher; (5) Students reflect individually. Macro scripts must allow for adaptations by the teacher, who inserts new materials and activities in response to micro script data (e.g., students uploads or TAR data). The visualization of student work was expanded to represent student ideas as a complex interconnected web of social and semantic relations, allowing students to filter the information to match their own interests and learning needs - making the knowledge not only collective but also personal. The visualization also became the focal point of real-time smart classroom activities where students leveraged the products of the class in creating physics challenge problems. Customized views allowed the teacher to gain an understanding of students collective knowledge and adjust class activities to better address student needs. Tools were also developed to allow the teacher to assess student work, review individual student progress, and add new materials to the script. We are currently collecting data from two grade 11 physics classes (n=20, n=25), over a six-month curriculum (Sept, 2011 March, 2012), divided into three units: Kinematics; Force and Motion; and Energy, Work, and Power. The units are thematically connected, allowing content to carry over between each. We are orchestrating CIC and TAR, including design-based variations of activities and technologies, within each unit. Our final paper will present a pre- post comparison of students reasoning about physics principles, an analysis of student contributions to the knowledge base (e.g., the challenge problems they create), and progress in student explanation patterns over time. We will connect these analyses to our notions of a scripting framework.
Conclusions
These studies have addressed two key aspects of a scripting framework for collaborative inquiry. First is the notion of a macro-script for coordinating student activities and teacher interventions over a substantial curricular scope. This includes the capability for student created artifacts, peer exchange, voting and discourse, and
ISLS
122
ICLS2012
Volume 2: Symposia
teacher adaptations such as additional resources (ie. questions, examples, assessments), as well as the role for emergent artifacts as resources within the script. Second is the development of micro-scripts (e.g. TAR) that can be supported with a smart classroom infrastructure to orchestrate and coordinate student inquiry activities that make use of, and contribute to the community knowledge base These studies have informed our progress toward a scripting framework for engaging students in a knowledge community while providing teachers with the tools to adjust the script in response to the emergent ideas within that community. Further we have begun to formalize an understanding of the informational needs of the teacher in the execution of these scripts.
Paper 4: Development of a Mobile Scripting Application to Support Students Collection and Analysis of Scientific Data
Vanessa L. Peters & Nancy Butler Songer
Introduction
Research in the learning sciences suggests that even with carefully-designed curricular materials, students develop only a partial understanding of scientific inquiry. While students are able to recognize certain features of inquiry thinking, few students can systematically demonstrate the difference between claim, evidence and reasoning for making predictions or constructing explanations about science (Jeong, Songer, & Lee, 2007). Previous research suggests that more cohesion and scripting are needed to support students development of complex reasoning skills when collecting and interpreting data for scientific investigations (Metz, 2000). The U.S. College Board has identified the following data-related practices as belonging to an essential skill set for answering questions about scientific problems: justifying the selection of the data type that are needed, designing a plan for data collection and documentation, and evaluating the quality and sources of data (College Board, 2009). Pedagogical scripts can support aspects of scientific inquiry by specifying parts of the learning process such as work phases, task distribution, and the sequencing of activities (Dillenbourg & Jermann, 2007; Peters & Slotta, 2010). Scripts can be communicated through initial instructions by the teacher, integrated into the learning environment (Kollar, Fischer, & Slotta, 2007), or integrated into the design of learning technologies. This study builds on the strong foundation of empirically-tested activities developed for BioKIDS: Kids Inquiry of Diverse Species (Songer, 2006) with the development of a new iPod Touch application for supporting students collection and analysis of scientific data. The goal of the BioKIDS program is to support elementary and middle school students development of complex reasoning and technological fluency in science through curricular activities that fuse scientific content knowledge with scientific practices. In the BioKIDS curricular units, students work in small groups to collect data on animal or plant species from several different areas in their schoolyard. Students take part in recording field observations that are later used for constructing scientific explanations and making predictions that use their collected data as evidence.
Data and Methods

Working with a technology developer, we developed a customized iPod Touch application that scripts students recordings of field observations when collecting and classifying data about plant and animal species (Figure 1). The script guides students in data collection through a sequence of panels that asks for specific information about the characteristics of their observed plant or animal, including the habitat, energy role and location of the species. The applications intuitive and icon-driven interface was well-suited for middle school students, including those whose first language was not English. The BioKIDS application did not require that students enter data in one panel before progressing to the next, thus, it was possible for them to omit entering certain information. After students recorded their field observations, the data were aggregated in a repository that organized the information according to biodiversity richness and abundance. Data collected with the BioKIDS application were synched automatically to an online database; data collected with the clipboard were added manually to the repository using the same organizational structure as the iPod Touch application. The participants in this study were 41 seventh graders from an urban middle school in the Midwestern United States. Students were divided into two groups for data collection: one group used the iPod Touches (n = 21), and the other group used printed worksheets and a clipboard (n = 20). Both groups were taught by the same teacher and completed the same activities. Curriculum implementation differed only in the method students used for recording field observations in their schoolyard. Both groups collected data on the same day and under the same conditions. The teacher was provided with instructions on using the iPod Touch application prior to the study; students were not provided with any such instruction. All field observations from both the iPod Touch and clipboard were analyzed for their completeness and usefulness for providing data-based evidence that could be used for answering scientific questions.
ISLS
123
ICLS2012
Volume 2: Symposia
Figure 1. Panel sequence in BioKIDS iPod Touch application for scripting students observations of field data.

Both the iPod Touch and clipboard groups recorded approximately the same number of field observations (79 and 77, respectively), however, there were marked differences between the two groups in terms of the completeness and volume of the collected data. Students using the scripted BioKIDS application noted the habitat and energy role for 100% of their observations, while the clipboard group noted this information for only 5.19% of their observations. Fishers Exact Test was used to compare the frequency of complete observations for each organism class. An observation was considered complete if it included both the abundance (number of animals/plants) and the richness (type of animal/plant) of the observed organism. Between the two groups, there were significant differences in the recorded observations for mammals (p = .0001), birds (p = .0004), anthropods (p = .0119) and plants (p = .0001). Only the gastropods and annelids (p = .4286) were found to be non-significant between the two groups. Aside from the novelty and fun-factor, the BioKIDS iPod Touch application presented a significant advantage over the clipboard as a method for data collection when engaging students in inquiry learning. To be meaningful, students need to collect and analyze data in sufficient volume in order to use it productively as evidence for supporting scientific claims. A field observation in this activity was useful if included both the type of animal or plant observed (e.g., prairie falcon, shrub) and the number observed. Unlike traditional pencil-andpaper approaches, technology makes it easy to illustrate the power of large data sets, providing students with enriched opportunities for data analysis that would otherwise be difficult or impossible with smaller sets of data. In the iPod group, the scripted application made it easier for students to record their observations when collecting data, which resulted in a more voluminous and complete repository of field data. One of the goals of BioKIDS is to develop strong learning tools, including technological tools, that support students when collecting and analyzing data, providing them with opportunities for using data-based evidence for answering scientific questions (Songer, 2006). The findings from this study demonstrate the value of technology scripts for supporting a systematic approach to data collection and analysis, and have implications for both technology developers and learning scientists. New technologies bring with them new ways of interacting and understanding different forms of knowledge. Future development work will consider additional scripting capabilities for supporting students data collection and learning with mobile technologies.
References
Brand-Gruwel, S., Wopereis, I., & Walraven, A. (2009). A descriptive model of information problem solving while using internet. Computers & Education, 53(4), 1207-1217. doi: 10.1016/j.compedu.2009.06.004 College Board. (2009). Science: College board standards for college success. New York, NY: The College Board Press. Cress, U. (2008). The need for considering multilevel analysis in CSCL research-An appeal for the use of more advanced statistical methods. International Journal of Computer-Supported Collaborative Learning, 3(1), 69-84. doi: 10.1007/s11412-007-9032-2 de Jong, T. (2006). Technological advances in inquiry learning. Science, 312, 532-533. De Wever, B., Van Keer, H., Schellens, T., & Valcke, M. (2007). Applying multilevel modelling to content analysis data: Methodological issues in the study of role assignment in asynchronous discussion groups. Learning and Instruction, 17(4), 436-447. doi: DOI 10.1016/j.learninstruc.2007.04.001 Dillenbourg, P. & Jerman, P. (2007). Designing integrative scripts. In F. Fischer, I. Kollar, H. Mandl, & J. M. Haake (Eds.), Scripting Computer-Supported Collaborative Learning: Cognitive, computational and educational perspectives (pp. 275-301). New York: Springer. Dimitriadis, Y. (2011). Supporting teachers in orchestrating CSCL classrooms. Research on E-Learning and ICT in Education, (September), 33-40. Gijlers, H., van Dijk, A. & Weinberger, A. (2011). How Can Scripts and Awareness Tools Orchestrate Individual and Collaborative Drawing of Elementary Students for Learning Science? In H. Spada, G. Stahl, N. Miyake, & N. Law (Eds.), Connecting Computer-Supported Collaborative Learning to Policy and Practice: CSCL2011 Conference Proceedings (Vol. III, pp. 1065-1072). Hong Kong: ISLS.
ISLS 124
ICLS2012
Volume 2: Symposia
Hayman, S., & Lothian, N. (2007). Taxonomy directed folksonomies: Integrating user tagging and controlled vocabularies for australian education networks. World Library and Information Congress: 73RD IFLA General Conference and Council, (August) 1-27. Ito, M., Baumer, S., Bittanti, M., Boyd, D., Cody, R., Herr-Stephenson, B., et al. (2009). Hanging out, messing around, and geeking out: Kids living and learning with new media (john D. and catherine T. MacArthur foundation series on digital media and learning) (1st ed.) The MIT Press. Jeong, H., Songer, N. B., & Lee, S.Y. (2007) Evidentiary competence: Sixth graders understanding for gathering and interpreting evidence in scientific investigations. Research in Science Education, 37(1), 75-97. Kobbe, L., Weinberger, A., Dillenbourg, P., Harrer, A., Hmlinen, R., Hkkinen, P., & Fischer, F. (2007). Specifying computer-supported collaboration scripts. Computer Supported Collaborative Learning, 2, 211-223. Kollar, I., Fischer, F., & Hesse, F. W. (2006). Computer-supported collaboration scripts - a conceptual analysis. Educational Psychology Review, 18(2), 159-185. Kollar, I., Fischer, F. & Slotta, J. D. (2007). Internal and external scripts in computer-supported collaborative learning. Learning and Instruction, 17(6), 708-721. Lai, K.W. (2008). ICT supporting the learning process: The premise, reality, and promise. In J. Voogt & G. Knezek (Eds.), International handbook of information technology in primary and secondary education. Part one. (pp. 215-230). New York: Springer. Mathes, A. (2004). Folksonomies - cooperative classification and communication through shared metadata. Computer Mediated Communication, , 1-13. Meier, A., Spada, H., & Rummel, N. (2007). A rating scheme for assessing the quality of computer-supported collaboration processes. International Journal of Computer-Supported Collaborative Learning, 2(1), 63-86. doi: DOI 10.1007/s11412-006-9005-x Metz, K.E. (2000). Young children's inquiry in biology: Building the knowledge bases to empower independent inquiry. In J. Minstrell & E. van Zee (Eds.), Inquiring into inquiry in science learning and teaching (pp. 371-404). Washington, DC: American Association for the Advancement of Science. Peters, V. L., & Slotta, J. D. (2010). Scaffolding knowledge communities in the classroom: New opportunities in the Web 2.0 era. In M. J. Jacobson & P. Reimann (Eds.), Designs for learning environments of the future: International perspectives from the learning sciences (pp. 205-232). New York, NY: Springer. Raes, A., Schellens, T., & De Wever, B. (2011). Multiple modes of scaffolding to enhance web-based inquiry. In H. Spada, G. Stahl, N. Miyake, & N. Law, N. (Eds.), Connecting Computer-Supported Collaborative Learning to Policy and Practice: CSCL Conference Proceedings, Vol I. ISLS. Rummel, N., & Spada, H. (2005). Learning to collaborate: An instructional approach to promoting collaborative problem solving in computer-mediated settings. Journal of the Learning Sciences, 14(2), 201-241. Schraw, G., & Dennison, R. S. (1994). Assessing Metacognitive Awareness. Contemporary Educational Psychology, 19(4), 460-475. Slotta, J. D. (2010). Evolving the classrooms of the future: The interplay of pedagogy, technology and community. In K. Makitalo-Siegl, F. Kaplan, J. Zottmann & F. Fischer (Eds.), The classroom of the future orchestrating collaborative learning spaces (pp. 215-242). Rotterdam: SensePublisher. Slotta, J. D., & Linn, M. C. (2009). WISE Science, Web-Based Inquiry in the Classroom. New York: Teachers College Press. Songer, N. B. (2006). BioKIDS: An animated conversation on the development of complex reasoning in science. In R. Keith Sawyer, (Ed.) Cambridge handbook of the learning sciences (pp. 355-369). New York: Cambridge University Press. Stegmann, K., Weinberger, A., & Fischer, F. (2007). Facilitating argumentative knowledge construction with computer-supported collaboration scripts. International Journal of Computer-Supported Collaborative Learning, 2(4), 421-447. Vauras, M., Iiskala, T., Kajamies, A., Kinnunen, R., & Lehtinen, E. (2003). Shared-regulation and motivation of collaborating peers: A case analysis. Psychologia, 46(1), 19-37. Ullrich, C., Borau, K., Luo, H., & Tan, X. (2008). Why web 2.0 is good for learning and for research : Principles and prototypes. International Conference on World Wide Web, 705-714. Weinberger, A., Reiserer, M., Ertl, B., Fischer, F., & Mandl, H. (2005). Facilitating collaborative knowledge construction in computer-mediated learning environments with cooperation scripts. In R. Bromme, F. W. Hesse, & H. Spada (Eds.), Barriers and biases in computer-mediated knowledge communication (pp. 15-38). Boston: Kluwer. Weinberger, A., Stegmann, K., & Fischer, F. (2010). Learning to argue online: Scripted groups surpass individuals (unscripted groups do not), Computers in Human Behavior, 26, 506-515. Wiley, D. A. (2000). Connecting learning objects to instructional design theory: A definition, a metaphor, and a taxonomy. Learning Technology, 2830(435), 1-35.
ISLS
125
ICLS2012
Volume 2: Symposia
Learning as Identity Formation: Implications for Design, Research, and Practice

Jochen Rick, EduTech, Saarland University, Campus C5 4, Saarbrcken, D-66123, j.rick@mx.uni-saarland.de Ben DeVane, University of Florida, P.O.Box 115810, 101 Norman Gym, Gainesville, Florida 32611-5810, ben@digitalworlds.ufl.edu Tamara Clegg, College of Information Studies, University of Maryland, 2117A Hornbake, South Wing, College Park, MD 20742, tclegg@umd.edu Vanessa L. Peters & Nancy Butler Songer, University of Michigan, 610 East University Avenue, Ann Arbor, MI 48109, {vlpeters,songer}@umich.edu Susan R. Goldman, Learning Sciences Research Institute (MC 057), 1007 W. Harrison Street, Room 2048, University of Illinois, Chicago, IL 60607-7137, sgoldman@uic.edu Cindy E. Hmelo-Silver, Dept. of Educational Psychology, Rutgers University, 10 Seminary Place (Room 320), New Brunswick, NJ 08901, cindy.hmelo-silver@gse.rutgers.edu Abstract: Increasingly, research is acknowledging the importance of identity for understanding and supporting learning. Identity is a particularly useful concept for highlighting extraordinary individual experiences, interpreting rich social contexts, and viewing learning at multiple timescales, from an instant reaction to lifelong psychosocial development. However, identity is complexit is conceptualized differently by various academic traditions. This inherent diversity can lead to a confusing or murky notion of identity. In this symposium, we clarify the concept, integrating the relevant theoretical perspectives of social and psychological identity to assess its implications to design, research, and practice. We provide concrete examples from our own research of the benefits of viewing learning as identity formation and the challenges to conducting such research.
Learning as Identity Formation

The learning sciences are manifold, acknowledging contributions from numerous academic traditions (education, psychology, computer science, design, etc.) and recognizing multiple perspectives on learning, both their power and limitations. Bransford and Schwartz (1999) argue that having a more fine-grained notion of transfer has powerful consequences for how we understand and promote learning. In his keynote at ICLS 2002, Roy Pea noted the limitations of the learning model imposed by the then-new No Child Left Behind legislation: that learning gains can be quantified through testing and thereby directly compared. In this symposium, we examine the perspective of learning as identity formationpeople use their personal experience to actively construct their identity in relation to ideas, others, and to themselves. We provide concrete examples from our own work on the value of this perspective to design, research, and practice. As with any lens, there are limitations: While there is significant overlap, learning is not the same as identity formation (Nasir & Cook, 2009). On the other hand, important aspects of learning come into clearer focus. In the preface of his seminal book Mindstorms: Children, Computers, and Powerful Ideas, Seymour Papert (1980) describes his early fascination with the differential gear. I believe that working with differentials did more for my mathematical development than anything I was taught in elementary school (p. xviii). He viewed the multiplication tables as gears. His first brush with two variable equations (e.g., 3x + 4y = 10) immediately evoked the differential. The gears of his childhood served him as useful model to assimilate the world and to form his identity as a mathematician. While the cognitive component was significant, Papert credits more power to the affective component (his love of the gear) in shaping his life. Problematically, such a profound example of learning cannot be easily understood through conventional means: A pre- and posttest at age two would have missed them (p. xx). How can we study such experiences and use our understanding to facilitate them? To better understand the challenges involved, it is worthwhile noting the peculiarities of Paperts experience. First, the learning took place largely outside a classroom. Second, the full story takes place over a long time, spanning from playing with toy trucks at age two to using his relationship with gears to inform his studies with Piaget and ultimately his theories of how the computer can support learning. Third, it was a highly individual story of identity formation. Papert admits that others would not have taken away the same things from those experiences. Fourth, it takes place in a larger social context, where he was able to use his passions to inform his school studies and his professional career. Characterizing learning in a larger social context has been well represented inside the learning sciences. In particular, communities of practice has been an influential framework to incorporate social aspects of learning. That theory originated with observations of how traditional methods of career training (e.g., tailors, butchers) have substantially different properties than school-based education. Lave and Wenger (1991)
ISLS
126
ICLS2012
Volume 2: Symposia
document how successful implementations of legitimate peripheral participation can allow newcomers to join a community of practice and progress to being central to that community. Wenger (1998) further theoretically expands the implications to identity formation through the twin processes of becoming and belonging. Over time, an apprentice will learn new skills and take over more responsibility to become a tailor. Concurrently, both the apprentice and the tailoring community will increasingly acknowledge that he belongs to that community. There is both an individual and a social component. There has been substantial work on studying the social components of identity in evocative situations, pointing out the key differences to classroom learning, and then integrating the positive aspects into other learning settings (Gee, 2000). For example, Bryant, Forte, and Bruckman (2005) studied how contributors to Wikipedia became part of that community. They found that a particularly powerful incentive for people to freely contribute their time was that others would see their work (i.e., that it made a difference to a sizable audience). This contrast sharply with standard classwork, which is usually only viewed by the teacher. To exploit the advantage of an audience, Forte and Bruckman (2006) had learners construct a publically available wiki based on their in-class learning and independent research. Studies of identity in meaningful social contexts can help us reflect on existing research and practice. When studying how high schoolers played dominoes in their free time, Nasir (2005) found that players were not just trying to win the game; they also used sophisticated and subtle strategies, often bending the rules of the game, to scaffold weaker players. This stands in contrast to Barrons (2003) work documenting common problems in group-work (e.g., ignoring contributions by a peripheral group member). It suggests that children do have the inherent ability to support each other given the right social context. The design challenge is creating the right context. In another study, Nasir and Cook (2009) focused on learning in after-school athletic activities, such as track and basketball. As voluntary activities, there was a different social impetus and individual commitment to these activities. Coaches explicitly facilitated learning as identity formation. One track coach informed a skeptical student, you are a hurdler. That statement indicated a specific learning trajectory: While she was struggling at the time, she would become a successful hurdlera position of value in this context. Similar sentiments (e.g., you are a scientist) are much more rare in a conventional classroom. Independent of the social and skill-based aspects of identity, there is a core identityan identity so core to the person that it is important in all situations (Erikson, 1963). In Freudian psychology, this core identity roughly equates to the ego. Whether it be in outfitting a home (Csikszentmihalyi & Rochberg-Halton, 1981), writing a computer program (Turkle, 1984), participating in a MUD (Bers, 2001; Turkle, 1995), or working on a simple worksheet (Wenger, 1998), people consciously or unconsciously play with their core identity. The tools we provide learners are thus not neutral in that regard. Bers (2001) has explicitly aimed at creating identity construction environments, technologies and technologically-rich psychoeducational interventions that support explorations of self. She designed the Zora virtual community to allow children to create and share different things (virtual places, objects, heroes, and villains) that reflect their core values. Bers (2006) later expands the term to include an explicit focus on promoting positive youth development. Positive youth development focuses on promoting six aspects of identity for youths: competence (cognitive abilities and healthy behavioral skills), positive bonds with people and institutions, character (integrity and moral centeredness), confidence, caring, and contribution to a civil society. Such vital components of psychosocial development are often neglected in school curricula and by evaluations that focus on cognitive aspects of learning.
Complimentary Perspectives on Identity Formation

This symposium brings together insights from four separate research projects to demonstrate how learning as identity formation can inform the design of learning environments, learning sciences research, and educational practice. Rick will present how different notions of identity overlap and how, to understand the rich learning in such complex situation, certain theories may be more applicable to one persons experience than anothers. Devane and Clegg will provide two perspectives on how to study identity formation at different temporal scales, from the instance of an aha moment to lifelong psychosocial development. Peters will present on the role of disciplinary identity in science education researchhow the knowledge, methods, and culture associated with specific academic domains shapes both our approach to research (i.e., the practice) as well as the products and outcomes that educational research seeks to achieve.
Presentation 1 A Confluence of Identity

Jochen Rick Personal home pages are outgrowing their playful beginnings to serve serious purposes. At the forefront of this emergence is academia, where they are becoming a meaningful way for researchers to engage each other. In computer science, it is common for researchers to visit their colleagues home pages to find research articles and contact information. Additionally, a personal home page acts as an informal curriculum vitae or an berbusiness card. Graduate students are advised to maintain a professional page. Many include a link to their home
ISLS
127
ICLS2012
Volume 2: Symposia
page when applying for a faculty position. Likewise, some faculty members polish their pages before promotion. Yet, the medium is still in its infancy: The medium, its adopters, and their practices are unduly constrained by current technology. To better study the meaning and use of personal home pages in academia, I created a better personal-home-page system to loosen these constraints. That system applies wiki technology to facilitate easy editing, to enable interaction, and to focus the user on content creation. To honor its origins in the WikiWikiWeb, the software was named AniAniWeb. Whereas wiki wiki means quick in Hawaiian Creole (the quickest way to create a website is to ask any visitor to be an author), aniani means mirror (an appropriate metaphor for a tool to write about oneself). Mirrors, literal and metaphorical, play an important role in human development. In literature, music, visual art, or computer programming, they allow us to see ourselves from the outside, and to objectify aspects of ourselves we had perceived only from within. (Turkle, 1995, p. 155) The dissertation research presents a case study of six graduate students and their experience with AniAniWeb over a period of two years (Rick, 2007). Their practices are viewed through three analytical lenses: media theory, communities of practice, and core identity theory. When combined, these frameworks led to a rich understanding of personal home pages in academia. This work is not unique in combining these frameworks to understand a new medium. Turkle (1995) combines aspects of each of these perspectives to understand identity in MUDs (i.e., text-based multiplayer real-time virtual worlds). As a clinical psychologist, her main focus was on the individualhow does the technology affect the individual? Meyrowitz (1985) combines Goffmans (1959) framework on social interaction with a McLuhan (1964) perspective on technology to understand television. His main focus is on understanding the societal changes that television has caused. Going into the case study, I hypothesized that these frameworks would be useful in explaining the use and meaning of personal home pages in academia. The results have confirmed that. All three frameworks contribute to a better understanding of the whole. At times, the frameworks are orthogonalthey explain different things. For some adopters, their experience hardly reflects their core identity; AniAniWeb was simply a useful tool for them. For others, AniAniWeb does act as a mirror allowing adopters to play with who they are. At other times, the frameworks are complementaryilluminating different aspects of the same phenomenon. Graduate students are on a learning trajectory inside of the academic community of practice. When they use their home page to address that audience, the meaning is necessarily social in nature. However, their reflection on how successful they have been in that goal can strongly impact their core identity. As self-presentation environments, authoring a personal home page is naturally about presenting oneself to others. But, who are these others? At times, such as in the middle of a job search, there is a clear primary audience and the home page can be tailored to that audience. More frequently, multiple audiences are simultaneously served by the same home page. A home page can be visited by friends, family, colleagues, fellow academics trying to find out more about you, potential employers, your boss, etc. Compounding the problem, AniAniWeb so facilitated content creation that several adopters primarily used the technology for themselvesto keep notes or to record their lives. Authoring a personal home page that serves these different audiences to the author's satisfaction is the multiple audience problem. Authors in asynchronous media only have a hazy awareness of who their audience is; often they even fail to realize that they have one (Forte & Bruckman, 2006). Authoring a personal home page without useful feedback on how others receive the page is the audience awareness problem. For prolific users, such as those studied in this work, negotiating these two problems can be a challenge. In face-to-face conversation, we can (and do) change how we present ourselves based on our audience (Goffman, 1959). A waiter acts differently while in the kitchen than in the dining room. People act differently at home and at work. When these barriers break down (e.g., introducing your boss to your parents at your wedding), figuring out how to integrate the multiple roles can be tricky. People who simultaneously inhabit multiple spheres, such as teleworkers (Nippert-Eng, 1996), must continually negotiate which roles they play. Electronic technologies have a tendency to change such interpersonal dynamics (McLuhan, 1964; Meyrowitz, 1985). They can radically alter our self epistemologieshow we view ourselves. Because the virtual persona is not tied to an actual body, the Internet is a particularly fertile ground for playing characters radically different than your physical self. In a MUD, an able-bodied man can convincingly play a crippled woman in a chat room (Van Gelder, 1985). Turkle (1995) concludes that electronic media foster a fragmented self epistemology, where it is possible to be radically different in different settings; however, there are several electronic media where it is unacceptable and uncommon for the presented self to be substantially different from the physical self. Personal home pages are one such case. As only one site comes at the top of a Google query for your name, it is nearly impossible to separate content to different audiences. Personal home pages encourage an integrating self epistemology, emphasizing a core self that exists across multiple aspects of the adopter's life.
ISLS
128
ICLS2012
Volume 2: Symposia
Presentation 2 Identity in Time: Theoretical and methodological approaches

Ben Devane Studies in the learning sciences focused on the importance of identity to social learning have increased in number and prominence (e.g., Bers, 2001; Sfard & Prusak, 2005; Barab et al., 2007; Nasir & Hand, 2008). At the same time that identity has become an increasingly important construct in much of learning science research, what learning science researchers mean when they discuss identity can often be indistinct, uncertain or vague. In the broader social science research literature, two major conflicting theories of identity predominate, developmental (Erikson, 1959) and interactionist (Mead, 1934). Learning sciences research often uses the term identity without making clear what theory of identity is intended to be understood, leading to unnecessary confusion and disagreement. In this presentation, I argue that these two theories of identity do not necessarily represent conflicting views of social identity. Instead I argue that these differing theories are the result of disciplinary methods that focus on distinct timescales, the chronological units used to examine social processes (Lemke, 2000). In doing so, I briefly review the major social sciences literature on the two theories of identity, describe the major methodological differences between the two approaches, and outline the relevance of timescales to the particular methodological approach. This presentation pays particular attention to the relationship between short, moment-to-moment timescales of identification, commonly called identity enactment or performance, and longer timescales of identification, known as identity formation or development.
Perspectives on identity interactionist vs. developmental

Developmental and interactionist perspectives on identity are each associated with a formative scholar for the theory, Erik Erikson and G.H. Mead respectively. Developmental perspectives typically understand identity as a normally singular and continuous process of mental and emotional development, emphasizing the importance of both social institutions and social groups like families in said developmental process (see Erikson, 1959). Erikson wrote that identity connotes both a persistent sameness within oneself (selfsameness) and a persistent sharing of some kind of essential character with others (1959, pg. 102). In the learning sciences, developmental perspectives on identity have examined the relationship between learning and long-term, core aspects of identification like sexuality, ethnicity, morality and self-esteem (Halverson, 2009; Kirshner, 2008). Rooted in Pragmatist scholarship in psychology and sociology, interactionist perspectives on identity place emphasis on a persons unique social experiences and social roles. G.H. Mead, often recognized as a key figure in symbolic interactionist school, wrote that identity resulted from a dialogue of an individuals unique set of experiences with the importation of the social process from society as a whole (Mead, 1934, p. 186). Identity in this understanding results from a negotiation between social norms, social context and social structure, and is often referenced as being performed or enacted (e.g. Goffman, 1956). In the learning sciences, interactionist perspectives often interpret identity as learning to perform of a knowledgeable social role, like developing an identity as a scientist (e.g., Brown et al., 2005; Barab et al., 2007).
Timescales, method and identification

The differences between these two approaches to identity can be understood in terms of their methodological relationship to collecting and interpreting data at a given timescale. Interactionist paradigms focus on understanding identity at smaller timescales. Research in interactional sociolinguistics, for example, may use methods from discourse analysis or conversation analysis examine a few minutes of talk to understand how a person is enacting a particular ethnic, gender or role-based identity. Interactional perspective in the learning sciences may use microethnography to examine the development of a given role-based identity, say as a scientist or mathematician, over a period of months in a classroom. Conversely, identity research from a developmental perspective may use ethnographic methods to look at identity development of the course of a year or more. The different theories of identity are rooted in the distinct analytic primacy of their methodological approaches, and not conflicting phenomena (see Penuel & Wertsch, 1995). Research on identity needs analytic methods that put data on identity enactment at shorter timescales into dialogue with data on identity formation at longer timescales. Going further, this paper will examine methodological approaches focused on understanding the relationship between timescales and processes of identificationsuch as connective ethnography (Leander & McKim, 2003), ethnographic discourse analysis (Gee & Green, 1998) and microethnography (Erickson & Schutz, 1982)and discuss the theoretical perspectives connected to each.
Presentation 3 Understanding Identity Development From Day-to-Day Experiences In Kitchen Science Investigators
Tamara Clegg Extending from the previous presentation, this presentation also looks at identity across timescales, but with respect to science learning. It is different in that our goal is to promote science learning and identity
ISLS 129
ICLS2012
Volume 2: Symposia
development. Also drawing from socio-cultural perspectives of identity (Wenger, 1998), we analyzed learners day-to-day experiences using Gees (2000) Discourse framework. In order to understand how learners day-today experiences are becoming more stable aspects of their identity, we analyzed learners scientific dispositions. We define disposition as values of, ideas about, and ways of participating in a particular discipline (in this case, scientific reasoning) that come frequently, consciously, and voluntarily (Gresalfi & Cobb, 2006; Katz, 1993). Simply put, scientific disposition is the initiative learners take to use scientific practices. We see disposition in the increased amount and complexity of learners scientific practices and in their use of those practices in different contexts (Bereiter, 1995). In this presentation, I look at learners development of disposition in the context of Kitchen Science Investigators (KSI). KSI is an after-school or summer camp program we designed to help learners begin to have the types of day-to-day experiences that would promote their scientific identity development. We designed KSI to promote development of scientific disposition through using science to make and perfect recipes. In this afterschool program, participants learn science content and design experiments to learn more about ingredients and how to use them to accomplish cooking and baking goals. In such a context, there are opportunities to learn much about chemistry, biology, physics, and arithmetic and to develop competence in a variety of scientific practices. In this presentation, I look at the case studies of two KSI participants in a 9-month implementation of KSI who began to develop scientific dispositions as a result of KSI experiences. We analyzed those cases to identify how that development progressed, and the components of the learning environment (and other experiences in their lives) that promoted that development. We analyzed learners scientific participation in KSI and their reports of scientific participation outside of the program, in other settings of their lives. Specifically, we found the disposition development process was initiated as learners used science to accomplish personally meaningful goals in the context of making and eating tasty dishes. As learners began to use science to accomplish their goals, they began to take on more scientific roles. As learners had more scientific experiences and as they continued to participate scientifically, their values shifted. While learners experiences, participation, and values were different, each of these aspects of their participation became more scientific, leading to their development of more scientific dispositions, or stable aspects of themselves. In this presentation, I will describe these shifts in each case across the contexts of learners lives (e.g., school, home, KSI). I will then present aspects of the learning environment that promoted that development.
Presentation 4 Blurring the boundaries: Understanding disciplinary identities in science education research
Vanessa L. Peters & Nancy Butler Songer In STEM education, improvements to teaching and learning depend on the collective efforts of scholars who bring disciplinary expertise to research collaborations. To date, research studies have provided limited understanding of how specific individuals such as learning scientists and natural scientists exchange and synthesize expertise towards project deliverables and outcomes. For STEM education research to be truly transformative, scholars must collaborate deeply and effectually not only within and across academic domains, but also with other learning organizations including schools, community programs and industry. In this presentation, I report on a study that examined the knowledge exchange and creation process of an interdisciplinary science education research team, and describe how disciplinary identities shaped the interactions and work practices of collaborators. Interdisciplinarity: Are we there yet? Promoting effective interdisciplinary collaboration has been a top priority for policy makers and federal agencies. In 2005, the US National Academy of Sciences (NAS) published a comprehensive report that summarized the current scope of interdisciplinary scholarship, with the goal of providing findings and recommendations to funding organizations and academic institutions for how such research could be facilitated. The findings of the report discussed at length the institutional-level challenges facing interdisciplinary collaboration, such as the disciplinary-specific rules that govern hiring, promotion and the allocation of resources. Yet the top challenge reported by researchers about interdisciplinary work involved the development of effective strategies for enhancing communication among project members in disciplinary disparate research teams. Guidelines for communication have yet to be developed, pending an analysis of the interactions and discourse strategies used during interdisciplinary exchanges. As Leshner (2011) observes, The need for interdisciplinary approaches has increased tremendously although we have been discussing it for 40 years, collectively we never seem to get it right. If we could come up with a series of distilled lessons learned, principles, and action steps that count be taken, then I think we could make tremendous progress (AAAS, 2011). Interdisciplinarity has been defined in many ways. Although the terms inter-, multi-, cross- and transdisciplinary all refer to pluralistic disciplinary approaches, there are differences in how the disciplines are
ISLS
130
ICLS2012
Volume 2: Symposia
combined. These differences are perhaps best described has having some placement on a continuum of disciplinary integration. At one end of the continuum is monodisciplinary, the traditional approach that employs the theories, methodologies and epistemologies of a single domain (Yang, 2011). Next are multi- and crossdisciplinary approaches that consider the dimensions of different disciplines in parallel, with little attempt to synthesize theories, methods or research findings (Klein, 2010). Interdisciplinary the most broadly used term is a problem-oriented approach that seeks to establish a common terminology and epistemological framework among the disciplines for academic inquiry (Knights & Willmott, 1997). At the far end of the continuum is transdisciplinary, an emerging approach that transcends disciplinary boundaries through the integration of knowledge and the fusion of methodologies, often with the aim of developing new theoretical paradigms (Hadorn, Pohl, & Bammer, 2010). Characterizing interdisciplinarity This research used a case study approach to examine how knowledge is exchanged and synthesized in a large interdisciplinary research team collaborating on curriculum development work for science education. Data were collected from team research meetings (including both in-person and videoconferencing), interviews and project documents. Project meetings are a fundamental component of research collaborations, and often the only forum in which all members participate simultaneously. Thus, their inclusion as a data source is necessary for an indepth understanding of the complexities of interdisciplinary collaboration. In this presentation, I discuss the role of disciplinary identities in science education researchhow the knowledge, methods and language associated with traditional academic domains shapes both our approach to research (i.e., our practice) as well as the products and outcomes that educational research seeks to achieve. I will describe a model of communicative behavior that explains the process of disciplinary knowledge exchange in educational research, and provide recommendations for a research approach that promotes transdisciplinary perspectives over traditional domainbased approaches.
Participants (alphabetical order)

Tamara Clegg Presenter 3 Tamara Clegg received her Ph.D. from Georgia Techs School of Interactive Computing specializing in the Learning Sciences. She is currently a postdoctoral fellow at the University of Maryland with the Computing Innovations Fellows program. Her work currently focuses on developing technology to support life-relevant learning environments where children engage in science in the context of achieving goals relevant to their lives. She is using participatory design with children to design these new technologies and new uses of existing technologies. Her work currently includes creating new life-relevant learning environments to understand how identity development happens across these environments. From this analysis, she aims to draw out design guidelines for life-relevant learning activities and technology in various contexts (e.g., sports, gardening). Ben DeVane Presenter 2 Ben DeVane is assistant professor of Digital Arts & Sciences in the University of Florida Digital Worlds Institute. Trained at the University of Wisconsin Games, Learning & Society initiative, his dissertation research was a four-year ethnographic study of identity and learning in a game-based learning community for workingclass youth. His design research has focused on developing and assessing learning games about topics like public health, environmental science and financial literacy. His scholarship been published in E-Learning, Games & Culture, the International Journal of Learning and Media, and Theory into Practice. Susan R. Goldman Discussant Susan R. Goldman (PhD., University of Pittsburgh) is Distinguished Professor of Liberal Arts and Sciences, Psychology, and Education and Co-Director of the Learning Sciences Research Institute at the University of Illinois at Chicago. She conducts research on subject matter learning, instruction, assessment, and roles for technology, especially in literacy and mathematics. A particular focus of her current research is on understanding the literacy demands in different disciplinary contexts and the implications of these demands for supporting learning. She is Executive Editor for Cognition & Instruction and Associate Editor for Journal of Educational Psychology. Goldman is a board member and President of the International Society of the Learning Sciences (20112012). Cindy E. Hmelo-Silver Session Chair Cindy E. Hmelo Silver is Professor of Educational Psychology at Rutgers University. She received an M.S. in Educational Technology from SUNY at Stony Brook and a Ph.D. in Cognitive Studies from Vanderbilt University and served postdoctoral fellowships at Georgia Institute of Technology and the University of Pittsburgh's Learning Research and Development Center. She is Editor-in-Chief of the Journal of the Learning Sciences.
ISLS
131
ICLS2012
Volume 2: Symposia
Vanessa L. Peters Presenter 4 Vanessa is a postdoctoral research fellow at the University of Michigan, where she is collaborating with an interdisciplinary research team to develop curricular units for teaching middle and high-school students about the ecological impacts of climate change. Her work on the project has focused on the development of SPECIES, an online learning environment with an embedded predictive distribution modeling tool that has been repurposed for younger audiences. Vanessa completed her Ph.D. at the University of Toronto, where she conducted a design-based study on knowledge community and inquiry practices in secondary science education. Her research interests include scripted collaboration, discourse processes in teaching and learning social computing. Jochen Jeff Rick Presenter 1 Jeffs research interests are in designing innovative applications for leading-edge technologies to support collaborative learning through active inquiry, exploration, and construction. As a design-based researcher, he has found it particularly useful to employ qualitative methods (e.g., ethnographic-style interviews, detailed video analysis) to understand the relation that an individual or a specific learning group has to a system. He received a Ph.D. in Computer Science (specialization of Learning Sciences and Technology) from Georgia Tech in 2007. From 2007 to 2010, he worked as a research fellow on the ShareIT project, investigating how new shareable interfaces (e.g., interactive tabletops) can support co-located collaboration. Since then, he has been on the faculty of the Department of Educational Technology, Saarland University. His latest research focuses on two students working with one iPad; he is particularly interested in how children combine physical gestures and verbal communication to collaborate.
References
American Association for the Advancement of Science. (2011, April). Experts at U.S. symposium urge efforts to encourage and support interdisciplinary research News archive. Retrieved August 6, 2011 from http://www.aaas.org/news/releases/2011/0418science_on_fire.shtml. Barab, S., Dodge, T., Thomas, M., Jackson, C., & Tuzun, H. (2007). Our designs and the social agendas they carry. Journal of the Learning Sciences, 16(2), 263305. Barron, B. (2003). When smart groups fail. Journal of the Learning Sciences, 12(3), 207359. Bereiter, C. (1995). A dispositional view of transfer. In A. McKeough, J. Lupart & A. Marini (Eds.), Teaching for transfer: Fostering generalization in learning (pp. 21-34). Mahwah, NJ: Lawrence Erlbaum Associates. Bers, M. U. (2001). Identity construction environments: Developing personal and moral values through the design of a virtual city. Journal of the Learning Sciences, 10(4), 365415. Bers, M. U. (2006). The role of new technologies to foster positive youth development. Applied Developmental Science, 10(4), 200219. Bransford, J. D., & Schwartz, D. L. (1999). Rethinking transfer: A simple proposal with multiple implications. Review of Research in Education, 24, 61100. Brown, B. A., Reveles, J. M., & Kelly, G. J. (2005). Scientific literacy and discursive identity: A theoretical framework for understanding science learning. Science Education, 89(5), 779802. Bryant, S. L., Forte, A., & Bruckman, A. (2005). Becoming wikipedian: Transformation of participating in a collaborative online encyclopedia. In Proceedings of GROUP 05, pp. 110. New York: ACM Press. Csikszentmihalyi, M., & Rochberg-Halton, E. (1981). The meaning of things: Domestic symbols and the self. Cambridge, UK: Cambridge University Press. Erickson, F., & Schultz, J. (1981). When is a context? Some issues and methods in the analysis of social competence. In J. Green & C. Wallat (Eds.), Ethnography and language in educational settings (pp. 147159). Norwood, NJ: Ablex. Erikson, E. (1956). The problem of ego identity. Journal of the American Psychoanalytic Association, 4(1), 56. Erikson, E. H. (1963). Childhood and society (second ed.). New York: W. W. Norton & Company. Forte, A., & Bruckman, A. (2006). From Wikipedia to the classroom: Exploring online publication and learning. In Proceedings of ICLS 2006, pp. 182188. ISLS. Gee, J. P. (2000). Identity as an analytic lens for research in education. Review of Research in Education, 25, 99125. Gee, J., & Green, J. (1998). Discourse analysis, learning, and social practice: A methodological study. Review of research in education, 23(1), 11963. Goffman, E. (1959). The presentation of self in everyday life. New York: Anchor Books. Gresalfi, M. S., & Cobb, P. (2006). Cultivating students' discipline-specific dispositions as a critical goal for pedagogy and equity. Pedagogies: An International Journal, 1(1), 4957.
ISLS
132
ICLS2012
Volume 2: Symposia
Hadorn, G. H., Pohl, C., & Bammer, G. (2010). Solving problems through transdisciplinary research. In R. Frodeman, J. T. Klein, C. Mitcham, & J. B. Holbrook (Eds.), The Oxford handbook on interdisciplinarity (431452). Oxford, UK: Oxford University Press. Halverson, E. (2005). InsideOut: Facilitating gay youth identity development through a performance-based youth organization. Identity, 5(1), 6790. Katz, L. (1993). Dispositions as educational goals. ERIC Clearinghouse on Elementary and Early Childhood Education: ERIC Digest. Kirshner, B. (2008). Guided participation in three youth activism organizations: Facilitation, apprenticeship, and joint work. Journal of the Learning Sciences, 17(1), 60101. Klein, J. T. (2010). A taxonomy of interdisciplinarity. In R. Frodeman, J. T. Klein, C. Mitcham, & J. B. Holbrook (Eds.), The Oxford handbook on interdisciplinarity (pp. 1530). Oxford, UK: Oxford University Press. Knights D., & Willmott, H. (1997). The hype and hope of interdisciplinary management studies. British Journal of Management, 8(1), 922. Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge, UK: Cambridge University Press. Leander, K., & McKim, K. (2003). Tracing the everyday sitings of adolescents on the Internet: A strategic adaptation of ethnography across online and offline spaces. Education, Communication & Information, 3(2), 211240. Lemke, J. L. (2000). Across the scales of time: Artifacts, activities, and meanings in ecosocial systems. Mind, Culture & Activity, 7(4), 273290. McLuhan, H. M. (1964). Understanding media: The extensions of man. Cambridge, MA: MIT Press. Meyrowitz, J. (1985). No sense of place: The impact of electronic media on social behavior. New York: Oxford University Press. Nasir, N. S. (2005). Individual cognitive structuring and the sociocultural context: Strategy shifts in the game of dominoes. Journal of the Learning Sciences, 14(1), 534. Nasir, N. S., & Cooks, J. (1999). Becoming a hurdler: How learning settings afford identities. Anthropology & Education Quarterly, 40(1), 4161. Nasir, N. S., & Hand, V. (2008). From the court to the classroom: Opportunities for engagement, learning, and identity in basketball and classroom mathematics. Journal of the Learning Sciences, 17(2), 143179. National Academy of Sciences (2005). Facilitating interdisciplinary research. Washington, DC: National Academies Press. Nippert-Eng, C. E. (1996). Home and work. Chicago: The University of Chicago Press. Papert, S. (1980). Mindstorms: Children, computers, and powerful ideas (second ed.). New York: Basic Books. Penuel, W., & Wertsch, J. (1995). Vygotsky and identity formation: A sociocultural approach. Educational Psychologist, 30(2), 8392. Rick, J. (2007). Personal home pages in academia: The medium, its adopters, and their practices. Unpublished doctoral dissertation, Georgia Institute of Technology, Atlanta, GA. Sfard, A., & Prusak, A. (2005). Telling identities: In search of an analytic tool for investigating learning as a culturally shaped activity. Educational Researcher, 34(4), 1422. Turkle, S. (1984). The second self: Computers and the human spirit. New York: Simon & Schuster. Turkle, S. (1995). Life on the screen: Identity in the age of the Internet. New York: Simon & Schuster. Van Gelder, L. (1985, October). The strange case of the electronic lover: A real-life story of deception, seduction, and technology. Ms. Magazine, XIV(4), 94124. Wenger, E. (1998). Communities of practice: Learning, meaning, and identity. Cambridge, UK: Cambridge University Press. Yang, A. S., (2011). Interdisciplinary and critical inquiry: Visualizing at the Art/Bioscience interface. Interdisciplinary Science Reviews, 36(1), 4355.
Acknowledgements
This symposium originated at the early career workshop of CSCL 2011. There, we discovered our mutual interest in identity formation and that we had different insights to contribute. We would like to thank the organizers and mentors of that workshop for encouraging us to collaborate on an ICLS 2012 symposium.
ISLS
133
ICLS2012
Volume 2: Symposia
Building (Timely) Bridges between Learning Analytics, Educational Data Mining and Core Learning Sciences Perspectives
Ido Roll, University of British Columbia, 6224 Agricultural Road, Vancouver, BC V6T 1Z1, ido@phas.ubc.ca Vincent Aleven, Kenneth R. Koedinger, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213 {aleven, koedinger}@cs.cmu.edu Matthew Berland, University of Texas at San Antonio, 1 UTSA Circle San Antonio, TX 78249, matthew@berland.org Taylor Martin, Tom Benton, Carmen Petrick, University of Texas at Austin, 1 University Station, Austin, Texas, 78712 {taylormartin, tbenton, cpetrick}@mail.utexas.edu Arnon Hershkovitz, Michael Wixon, Ryan Baker, Janice Gobert, Michael Sao Pedro 100 Institute Rd, Worcester Polytechnic Institute, Worcester, MA {arnonh, mwixon, rsbaker, jgobert, mikesp}@wpi.edu. Bruce Sherin, Northwestern University, 2120 Campus Drive, Evanston, IL, bsherin@northwestern.edu Paulo Blikstein, Marcelo Worsley, 520 Galvez Mall, Stanford, CA, 94305 {paulob, marcelo.worsley}@stanford.edu, Paulo Blikstein, Organizer and Chair Roy Pea, Stanford University, Discussant Abstract: Despite the exponential growth of the research on Learning Analytics (LA) and Educational Data Mining (EDM) over the last few years, the work has been still distant from the core Learning Sciences methods, theoretical constructs, and literature. At the same time, over the last 15 years, Learning Sciences as a field has been quite innovative, eclectic, and effective in incorporating new methodological stances, such as micro-genetic methods, microethnographies, and design-based research. It seems that the time has come to build sound connections between these traditions. The goal of this symposium is to bring together researchers coming from different academic perspectives, to explore and examine common LA/EDM methodological and theoretical threads with wide applicability within the Learning Sciences. The papers presented explore text mining in clinical interviews, moment-bymoment learning curves and traces, data mining of programming logs, and cognitive tutors, representing the main perspectives and methodological approaches in the field.
Overall focus of the symposium

Despite the exponential growth of the research on Learning Analytics (LA) and Educational Data Mining (EDM) (Baker & Yacef, 2009) over the next few years, the work has been still distant from the core Learning Sciences methods, theoretical constructs, and literature. At the same time, over the last 20 years, the Learning Sciences as a field has been quite innovative, eclectic, and effective in incorporating new methodological stances, such as micro-genetic methods (Siegler & Crowley, 1991), micro-ethnographies (Nemirovsky, 2011), and design-based research (Cobb, Confrey, diSessa, Lehrer, & Schauble, 2003; Confrey, 2005; Edelson, 2002). However, in the last years, as computational technologies became truly ubiquitous, the possibilities of data collection opened up incredible new opportunities, such as the capture of students interaction with computer software, websites, as well as gesture, speech, position, mood, etc. There is clear interest in these techniques in the Learning Sciences for various reasons for one, those data seem like a somewhat natural complement to micro-genetic methods, since they allow for an even greater sampling rate, and to design-based research, since they are accepting of unstructured learning activities and rapid-prototyping. Second, many learning scientists themselves have a computer science background, so they are familiar with computational modeling tools. However, by its very nature, data-mining and machine learning tend to be agnostic of mechanism in many fields of application (outside of education), gargantuan datasets and fast computers provide researchers with answers oftentimes much more advanced than the theoretical or explanatory models they havewhat matters are correct predictions. This approach, however, is problematic in education, where understanding the causal chain is crucial for prescribing sound novel policies or educational designs. Thus, we are at a crossroad in the history of both fields, one in search of new methods, and another searching for sound theoretical models to explain its findings. The question posed for the LS community is
ISLS
134
ICLS2012
Volume 2: Symposia
how to make a productive connection and avoid big data from drifting towards a-theoretical, data-oriented work? The goal of this symposium is answer that questions by bringing together researchers coming from different academic perspectives, to explore and examine common LA/EDM methodological and theoretical threads with wide applicability within the Learning Sciences. The papers presented explore text mining in clinical interviews, data mining of programming logs, cognitive tutors, moment-by-moment learning traces, coming from scholars from different traditions but with a strong presence in the LS community.
Presentations
Hershkovitz et al. examine disengaged behavior and how to identify it using an automated detector. They will also relate that data with data from the Patterns of Adaptive Learning Scale and try to find out if different behavioral and epistemological issues relate to disengaged behavior. Overall, they show that students characterized by mastery or performance goal orientation have (on average) double the probability of carelessness as compared to students characterized by low scores for these goal orientations. Berland et al. present a study with students in which they use visual programming to create virtual robots. Patterns of collaboration affected students success at programming students who shared complex code created better programs immediately afterwards. The study suggests that sharing is beneficial in the short term, but over the entire implementation, the average sharing measure was not correlated with code quality. Blikstein et al. paper follows a group of undergraduate students learning computer programming, as snapshots of their code are stored in a central server. Students fill a questionnaire about programming style and previous experience. The researchers use machine learning techniques to find patterns in how more and less experienced learners progressed towards expertise. Roll et al. demonstrate the potential of using students moment-by-moment traces to offer domain-level support in scientific inquiry tasks, suggesting that adapting the task can be a productive mean for giving support in a constructivist environments. Log and assessment data from an evaluation of the Invention Lab in six grade nine classes (N = 92) was used to validate the modeling approach of the lab. Roll suggests that students trace data can be used to offer adaptive, individualized support in a manner that does not reduce critical elements of inquiry learning. Sherins paper examines techniques from statistical natural language processing (SNLP) and their use to analyze data produced by clinical interviews with middle-school students about scientific topics. In particular, he introduces Latent Dirichlet Allocation as an alternative to the more popular Latest Semantic Analysis, and uses it to segment transcripts of students conceptions in openended interviews about the seasons, verifying the algorithm against human coders.
Using Dynamic Time Warping and Cluster Analysis to Analyze the Learning of Computer Programming
Paulo Blikstein, Marcelo Worsley, Stanford University Educational data mining and Learning Analytics (EDM; Amershi & Conati, 2009; Baker, Corbett, Koedinger, & Wagner, 2004) has grown rapidly over the last years, profiting from the vast availability of logfile data from elearning systems, instrumented computer applications (Blikstein, 2009, 2011, Berland, 2008), computer vision systems, and web logs. The majority of the work focuses on standardized tasks delivered by cognitive tutors or clickstreams of online courses (Baker & Yacef, 2009). However, there is increasing interest in promoting types of learning activities and outcomes that are non-standardized, engaging students in constructionist (Papert, 1980) open-ended tasks such as designing robots or programming computers. In this work, we present a machine-learning-based framework to predict the performance and level of expertise of students based on their coding style. We will show results from a study in which we capture and analyze logs of undergraduate students (n=150) learning to program in an introductory Java course. In addition to logfiles, we also had students fill a questionnaire about their programming style, motivation, and previous experience. As students learn to write program, they develop their own distinctive style of coding (top-down vs. bottom-up, planners vs. tinkerers, Turkle & Papert, 1991). The capture of these styles and their evolution during a computer science course could point to cognitive changes as well, and point to optimal points of intervention. Therefore, we analyze code fragments written by students for various assignments, based on fundamental concepts such as recursion, functional programming, object-oriented programming, error-handling, looping, etc. The code fragments committed to a global repository, giving us access to the each incremental time-stamped step in the process. First, we calculate differences between each successive commits (or diff statistics) by a student, in terms of the number of lines and characters added/deleted/reordered/modified, control-flow blocks added/deleted, and comments added. This gives us an approximate idea of the increments made by the student in each successive step and the time interval between steps. The individual code fragments are separately compiled and executed to capture their runtime errors and compiler errors. This results in several time-series of various diff statistics at each commit step, along-with the compiler or run-time errors. Using Dynamic Time Warping to calculate the distance between the time-series, we hierarchically cluster the timeseries data of students, and use silhouette values to decide the cutoff for the number of clusters. We then analyze
ISLS 135
ICLS2012
Volume 2: Symposia
the clusters for certain underlying coding trends which all students in that cluster share. The overall idea is to first identify the features influencing the coding style of a student, and then use them as features to group students into clusters. Finally, we cross those results with self-reported questionnaires on style, motivation, and experience. Preliminary results show distinctive changes in the style and learning pattern for self-declared topdown and bottom-up programmers, as well as connections between the evolution of coding styles and expertise.
AMOEBA: Mining how students learn to program together

Matthew Berland, University of Texas at San Antonio Taylor Martin, Tom Benton, Carmen Petrick, University of Texas at Austin This work details findings from AMOEBA, a new system to describe from a large dataset how students learn to program collaboratively. Work such as Hancock (2003) and Guzdial (2003) have shown key processes in how students learn to program through collaboration, but human observation is limited in the scale at which process data can be collected, the time frame over which it can be considered. Based on a single 90-minute session with students ranging from beginning to more experienced programmers, we were able examine student programming at a fine grain to produce a meaningful description of how students collectively learned to program, where similarities arose, and how students programs differed. To explore student patterns of programming, we built the IPRO programming environment to collect those fine-grained data. The IPRO programming language was designed to foster quick programming on the fly (tinkering); it utilizes a restricted library of sensors, logical operators, mathematical operators, and actions. IPRO programs are immediately executable, and no syntax errors are possible in the environment. We designed IPRO with these features so that: student programs can be easily categorized and analyzed; it is easy to start programming with no background or prior knowledge; and students can share code easily. Berland (2008) and Hancock (2003) both describe how features such as reusable code and easy testability engage novices by allowing them to learn as they make rewarding progress on their programs. We deployed the system to 53 students across 16 teams (of 3-4 virtual robots each) on the first day of a girls programming summer camp. Each student made between 100-500 edits to her programs over the course of 90 minutes. The new programs produced by each edit were automatically evaluated by the system in terms of how successful that program was at scoring goals. Metrics were created to evaluate similarity of any two edits as well as uniqueness of a program at a particular time. To investigate these relationships, we built AMOEBA. AMOEBA explores the log data by visualizing patterns of similarity across programs. AMOEBA tracks the similarity of all programs of each of the 53 students on 16 teams over the course of 90 minutes. In short, our similarity metric reports how many subcomponents of the program are found to be in common with all of the other students' programs in existence at that moment during the class. Similarity is scored by the number of components in common times the inverse frequency of that subcomponent in the corpus as a whole (similar to tf-idf as per Salton, 1989). AMOEBA in Figure 1 (left), below, shows connections between students where a connection is any similarity between two programs that is unlikely to be random. Those connections are marked with the most unlikely subcomponent that the two students' programs have in common. Note that the students are groups by team in the diagram, to highlight when similarity is more common on teams and when it is not. We also report, as per Figure 1 (right), a running correlation of similarity to quality.
Figure 1 AMOEBA system (left) and running correlation of similarity to program quality (right) Our results show that students routinely wrote remarkably similar code at the same time. As seen in Figure 2, the correlation between similarity and quality also decreased. That is, the benefit gained by writing similar program code to your peers (through sharing or through mutual understanding of a topic) decreased as the situation became more complex. By matching AMOEBA log data to video, we noted that most of the
ISLS 136
ICLS2012
Volume 2: Symposia
unlikely similarities were not due to students sharing much program code. Indeed, this is borne out by the data, which show few working programs replicated in their entirety across the classroom in the last half of the class. These results have two interesting implications for learning. The first implication is that not all sharing and collaboration are created equal: that collaboration needs to be situated in necessity (as per Brown, Collins, & Duguid, 1989) appears to bear out even at a fine-grained level. Second, students learning to program in a complex system can react to that system in ways that are similar enough as to be very unlikely.
Student Attributes and Carelessness in Science Microworld

Arnon Hershkovitz, Michael Wixon, Ryan Baker, Janice Gobert, Michael Sao Pedro, Worcester Polytechnic Institute Student attributes including student goals, attitudes, and beliefs play a key role in their learning outcomes. These attributes might impact learning by creating different forms of disengagement. One disengaged behavior is carelessness, i.e., when a student fails to answer an item correctly despite possessing the required skills (Clements, 1982). We have operationalized carelessness using an automated detector of contextual slip, i.e., the probability that the student performed incorrectly at a specific time despite knowing the needed skill (Baker, Corbett, & Aleven, 2008). The notion of contextual slip matches previous carelessness definitions, but is easier to apply than previous operational definitions. Our detector uses a log-based machine-learned model, hence can be scaled without being overly time-consuming. We study carelessness in demonstrating science inquiry skills (e.g., control for variable strategy) in the context of a phase change activity using microworlds. Participants are 148 eighth grade students, aged 12-14 years old, from a public middle school in Central Massachusetts. Before taking the activity, the participants took the Patterns of Adaptive Learning Scale (PALS) survey. All students fine-grained actions were logged and then analyzed at the clip level; a clip is a consecutive set of a students actions describing activity in its context. Carelessness detector was developed based on 73 features describing the clips (N=2114), and using REPTree, a regression tree classifier (resulted in a 6-fold cross-validation correlation of r=0.63). Overall, mean carelessness was low with a value of 0.05 across clips (SD=0.16) and 0.06 (SD= 0.05) across students (N=130). Students were then clustered based on their PALS measures, resulting in three clusters: Learning Goals, Performance Goals, and Lack of Goals. Surprisingly, it was found that mean carelessness in the Lack of Goals cluster was significantly lower from its mean in both Learning Goals and Performance Goals clusters. Furthermore, we studied how carelessness is being manifested over three consecutive activities. Overall, carelessness increased over all trials (as students have more practice opportunities for the same skill), however there were important differences among the clusters: In the Learning Goal cluster mean carelessness does not significantly increase from activity 1 to 2, nor from activity 2 to 3, but it is significantly higher in activity 3 compared to activity 1; in the Performance Goal cluster mean carelessness significantly increases between activities 1 to 2, but does not significantly increase between activities 2 to 3; and in the Lack of Goals cluster there is no significant differences were found in mean carelessness for either pair of activities. This work shows that students characterized by mastery or performance goal orientation have (on average) double the probability of carelessness as compared to students characterized by low scores for these goal orientations. It is possible that students with higher amounts of mastery or performance goals succeed in learning and correspondingly become more confident, and following that they tend to carelessness despite their goal orientation. Differences in manifestation of carelessness over consecutive trials demonstrate important individual differences which might imply on timely scaffolding for those students who need to be encouraged to be more focused during learning. The methodology used in this research demonstrated the strength of using student log files and data mining methods, as the current operationalization of carelessness might be relatively easily (under limitation of, e.g., transferability) large-scaled.
Computing student science conceptions with Latent Dirichlet Allocation

Bruce Sherin, Northwestern University Computationally-based analytic methods are becoming part of the standard toolkit employed by researchers in the learning sciences, and education more broadly. But, as a field, we have only scratched the surface of what is possible. Some techniques, such as Latent Semantic Analysis, have seen extremely wide use (Landauer, Foltz, & Laham, 1998). But the popularity of these techniques might be accidental; they were employed, with great success, by a few researchers (e.g., Graesser, Wiemer-Hastings, & Wiemer-Hastings, 2000; Magliano, WiemerHastings, Millis, Munoz, & McNamara, 2002; Wade-Stein & Kintsch, 2004). This initial success then led to their wide adoption. But we should be aware that there are many other existing methods, that are sometimes more appropriate to the task at hand. Here I will discuss one such method Latent Dirichlet Allocation (LDA), that can
ISLS
137
ICLS2012
Volume 2: Symposia
be applied to solve problems that have been solved by LSA (Blei, Ng, & Jordan, 2003). Although these two methods can be applied to some similar tasks and although they both have the word Latent in their names they are actually quite different. Furthermore, in some cases, LDA is more powerful and more appropriate. In the remainder of this summary, I will first explain the research context. Then I will briefly explain LDA and how it can be employed in this context. Over the last few decades, there have been literally thousands of research studies devoted to the study of students alternative conceptions in science (Duit, 2009). In many cases this research proceeds as follows: (1) We interview some students about a target domain or phenomenon. These interviews are videotaped. (2) We transcribe the videos. (3) We somehow extract a set of student conceptions from the transcripts. This generally requires repeated reading of the transcript and viewing of the video. And it usually involves some type of hand coding of the transcripts. What I would like to do is to employ learning analytic techniques to perform the tasks in step (3); I want to give transcripts to a computer algorithm, and have it extract student conceptions. In particular, the data I will discuss is drawn from a corpus of clinical interviews in which middle students were asked to explain the Earths seasons. Thus, I would like my learning analytic algorithms to extract students seasons-related conceptions from transcripts of these interviews. There are two additional complications. First, I adopt a knowledge-in-pieces perspective (diSessa, 1993). This means that I do not assume that students have existing models of the seasons. Instead, in many cases, they construct models of the seasons from fragments of knowledge. Thus, I actually want my analysis to identify these fragments, rather than full-blown models. Second, I expect that the fragments that a given student draws on may change as an interview unfolds. For this reason, I will not want to analyze transcripts, viewed as a whole. Instead, I must break each transcript into segments and determine the knowledge fragments associated with each segment. LDA is well-suited to this task. Although the computational algorithms that perform LDA are quite complicated, the underlying conception is straightforward. In LDA, a corpus of text is modeled by a set of topics. In my case, these topics are the fragments. Each of these topics/fragments is associated with a probability distribution over a set of words. If a particular fragment is active, then it will generate words according to this probability distribution. Finally, each segment of a transcript is modeled as a mixture of the fragments. This means that, within the segment, a particular fragment is chosen with a probability determined by this mixture. Then a word is generated according to the probability distribution associated with the fragment that was selected. This imaginary process iterates, once for each word, generating a text. This leads to a computational problem that must be solved. We have to discover the topics/fragments, the probability distribution of terms associated with each fragment, and the mixture of fragments associated with each segment of text. This is a difficult computational problem, but it is one that has been solved for us (Blei, et al., 2003). The bottom line, as I will describe in my talk, is that the application of LDA to the seasons corpus produces an analysis that is sensible, and that aligns extremely closely with the work of human analysts; it discovers similar fragments, and it also produces a sensible account of the fragments of knowledge that are drawn on as each interview unfolds. I will conclude with a comparison to a similar analyses I performed using LSA.
Automated Task Adaptation to Support Students inquiry Learning

Ido Roll, University of British Columbia, Vincent Aleven, Kenneth R. Koedinger, Carnegie Mellon University. Constructivist instructional activities hold the promise of helping students acquire transferable knowledge (Duffy & Cunningham, 1996; Tobias & Duffy, 2009). However, research often shows that students fail to acquire the desired learning goals when learning from constructivist instruction (Kirschner, Sweller, & Clark, 2006; Tobias & Duffy, 2009). The challenges of learning from constructivist instruction are most apparent in exploratory learning environments and scientific inquiry activities. In these activities students are expected to reveal an underlying model that governs the behavior of given data or simulation (de Jong & van Joolingen, 1998). One source of difficulty in scientific inquiry tasks is the relative lack of explicit support at the domain level. Since students are expected to discover the deep structure of the domain and learn by a process of exploration, inquiry environments usually withhold domain knowledge. For example, the Invention Lab (Figure 2) asks students to invent a method for calculating spread based on given data (Roll, Aleven, & Koedinger, 2010). Students are not given direct instruction on how to use standard deviation (or other alternatives), as this will short-circuit the reasoning process. At the same time, in the absence of explicit guidance, students often default to solutions that are partial at best. For example, many students simply settle for range (that is, max minus min), to measure spread (Roll, Aleven, & Koedinger, 2011).
ISLS
138
ICLS2012
Volume 2: Symposia
A common solution to the lack-of-support problem is often given in the form of process scaffolding or the use of cognitive tools. For example, students are often prompted to raise hypotheses and test them. However, it seems that process support alone, without complementary domain-level support, is insufficient for novice students (Mulder, Lazonder, & de Jong, 2009). Two main challenges limit the ability to give domain-level support during scientific inquiry tasks. First, as suggested above, explicit support is likely to short-circuit the desired sense-making processes. Second, given the high agency that students have in scientific inquiry tasks, students often explore the domain in different directions. Thus, support cannot be one size fits all, and canned responses cannot address many of the situations in which students are in need for support.
Figure 2: A Typical Task in the Invention Lab asks students to invent a general method for calculating the variability of the given data. In this talk we demonstrate the potential of using students moment-by-moment traces to offer domainlevel support in scientific inquiry tasks. First, we suggest that adapting the task can be a productive mean for giving support in a constructivist environment. By adapting the task to students demonstrated difficulties one can create tasks that are within students ZPD, thus addressing the need for explicit support in overly-complex situations. Second, adapting the task does not reduce the autonomy and agency of the learner, thus adhering to the constructivist principles of inquiry learning. Last, task adaptation does not short-circuit the reasoning behavior. Putting it more figuratively, while explicit support helps learners reach the goalpost, task adaptation brings the goalpost closer to the learner. Second, we suggest that the combination of constrained based modeling (Mitrovic, Koedinger, & Martin, 2003) and symbolic modeling (Anderson et al., 2004) is best suited for adapting the task to individual students. Specifically, the Invention Lab analyzes students invented models in real time. Since no two models are identical, the Invention Lab looks for pre-defined desired features in these models. For example, does the model uses all the given data? (While range uses only the extreme data points, using all data points is essential to capture the true distribution of the data). By identifying the shortcomings of students invented models, the Invention Lab creates new sets of data for students to analyze. The new data targets these knowledge gaps one by one. For example, students who invent range may be asked to compare two data sets with distinct distributions and a common range (e.g., {1,4,4,5,5,8} vs {1,2,3,6,7,8}). This iterative process of adapting available data to students invented methods is assumed to achieve two goals. First, it breaks down the problem and allows students to tackle the deep concepts of the domain one-by-one. Second, the iterative invention using adaptive data sets encourages students to integrate the different features of the data into a unified schema. In addition, the invention lab uses its knowledge of students moment-by-moment actions to offer feedback on students general inquiry behaviors. Log and assessment data from an evaluation of the Invention Lab in six grade nine classes (n = 92) was used to evaluate the modeling approach of the lab. Overall, this work suggests that students trace data can be used to offer adaptive, individualized support in a manner that does not reduce critical elements of inquiry learning.
References
Anderson, J. R., Bothell, D., Byrne, M. D., Douglass, S., Lebiere, C., & Qin, Y. (2004). An integrated theory of the mind. Psychological Review, 111(4), 10361060
ISLS
139
ICLS2012
Volume 2: Symposia
Baker, R.S.J.d., Corbett, A.T., Aleven, V. (2008). More accurate student modeling through contextual estimation of slip and guess probabilities in Bayesian Knowledge Tracing. In Proceedings of the 9th International Conference on Intelligent Tutoring Systems, 406-415. Baker, R.S.J.d., Yacef, K. (2009) The State of Educational Data Mining in 2009: A Review and Future Visions. Journal of Educational Data Mining, 1 (1), 3-17. Berland, M., Martin, T., Benton, T., & Petrick, C. (2011). Programming on the move: design lessons from IPRO. Proceedings of the 2011 annual conference extended abstracts on Human factors in computing systems (pp. 21492154). Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet Allocation. Journal of Machine Learning Research, 3(4/5), 993-1022. Blikstein, P. & Worsley, M. (2011). Learning Analytics: Assessing Constructionist Learning Using Machine Learning. Paper presented at the Annual meeting of the American Research Education Association, New Orleans, LA. Blikstein, P. (2009). An Atom is Known by the Company it Keeps: Content, Representation and Pedagogy Within the Epistemic Revolution of the Complexity Sciences. Unpublished PhD. dissertation, Northwestern University, Evanston, IL. Blikstein, P. (2011). Using learning analytics to assess students behavior in open-ended programming tasks. Paper presented at the I Learning Analytics and Knowledge Conference. Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18 (1), 32-41. Bruner, J. (1999). Postscript: Some reflections on education research. In E. C. Lagemann & L. S. Clements, M.A. (1982). Careless errors made by sixth-grade children on written mathematical tasks. Journal for Research in Mathematics Education, 13, 136-144. Cobb, P., Confrey, J., diSessa, A. A., Lehrer, R., & Schauble, L. (2003). Design experiments in educational research. Educational Researcher, 32(1), 9-13. Confrey, J. (2005). The evolution of design studies as methodology. The Cambridge handbook of the learning sciences, 135-151. de Jong, T., & van Joolingen, W. R. (1998). Scientific discovery learning with computer simulations of conceptual domains . Review of Educational Research, 68, 179-201. diSessa, A. A. (1993). Toward an epistemology of physics. Cognition and Instruction, 10(2 & 3), 165-255. Duffy, T. M., & Cunningham, D. J. (1996). Constructivism: Implications for the design and delivery of instruction. In D. H. Jonassen (Ed.), Handbook of research for educational communications and technology. (pp. 170-98). New York: Macmillan. Duit, R. (2009). Bibliography: Students' and Teachers' Conceptions and Science Education. Kiel, Germany: Leibniz Institute for Science Education. Edelson, D. C. (2002). Design research: What we learn when we engage in design. The Journal of the Learning Sciences, 11(1), 105-121. Graesser, A. C., Wiemer-Hastings, P., & Wiemer-Hastings, K. (2000). Using Latent Semantic Analysis to Evaluate the Contributions of Students in AutoTutor. Interactive Learning Environments, 129-147. Guzdial, M. (2003). A media computation course for non-majors. Proceedings of the 34th ACM SIGCSE Technical Symposium on Compuer Science Education. Hancock, C. M. (2003). Real-time programming and the big ideas of computational literacy. MIT. Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75-86. Landauer, T., Foltz, P. W., & Laham, D. (1998). An introduction to latent semantic analysis. Discourse Processes, 25(2-3), 259-284. Magliano, J. P., Wiemer-Hastings, K., Millis, K. K., Munoz, B. D., & McNamara, D. (2002). Using latent semantic analysis to assess reader strategies. [Empirical Study]. Behavior Research Methods, Instruments & Computers, 34(2), 181-188. Mitrovic, A., Koedinger, K.R., Martin, B.: A comparative analysis of cognitive tutoring and constraint-based modeling. In: Brusilovsky, P., Corbett, A.T., de Rosis, F. (eds.) UM 2003. LNCS (LNAI), vol. 2702, pp. 313322. Springer, Heidelberg (2003) Mulder, Y. G., Lazonder, A. W., & de Jong, T. (2009). Finding out how they find it out: An empirical analysis of inquiry learners' need for support. International Journal of Science Education, 1-21. Nemirovsky, R. (2011). Episodic Feelings and Transfer of Learning. The Journal of the Learning Sciences, 20(2), 308-337. Resnick, M., Maloney, J., Monroy-Hernndez, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., et al. (2009). Scratch: programming for all. Communications of the ACM, 52(11), 6067.
ISLS
140
ICLS2012
Volume 2: Symposia
Roll, I., Aleven, V., & Koedinger, K. R. (2010). The invention lab: Using a hybrid of model tracing and constraint-based modeling to offer intelligent support in inquiry environments. In V. Aleven, J. Kay, & J. Mostow (Eds.), Proceedings of the international conference on intelligent tutoring systems. (pp. 11524). Berlin: Springer Verlag. Roll, I., Aleven, V., & Koedinger, K. R. (2011). Outcomes and mechanisms of transfer in invention activities. In L. Carlson, C. Hlscher, & T. Shipley (Eds.), Proceedings of the 33rd Annual Conference of the Cognitive Science Society (pp. 2824-2829). Austin, TX: Cognitive Science Society. Salton, G. (1989). Automatic Text Processing: the Transformation, Analysis, and Retrieval of Information by Computer. Boston: Addison-Wesley Longman. Shulman (Eds.), Issues in education research: Problems and possibilities (pp. 399-409). San Francisco: Jossey-Bass Publishers. Siegler, R. S., & Crowley, K. (1991). The microgenetic method: A direct means for studying cognitive development. American Psychologist, 46(6), 606-620. Tobias, S., & Duffy, T. M. (2009). Constructivist instruction: Success or failure? New York: Taylor & Francis. Wade-Stein, D., & Kintsch, E. (2004). Summary Street: Interactive computer support for writing. Cognition and Instruction, 22(3), 333-362.
Acknowledgments
For Automated Task Adaptation to Support Students inquiry Learning: This work was supported by the Pittsburgh Science of Learning Center, which is supported by the National Science Foundation (#SBE0836012), and by the University of British Columbia through the Carl Wieman Science Education Initiative.
ISLS
141
ICLS2012
Volume 2: Symposia
The Use of Game Design, Social Learning Networks, and Everyday Expertise to Engage Youth with Contemporary Science
Philip Bell, Leah A. Bricker, Katie Van Horne, Theresa Horstman University of Washington, 1100 NE 45th St., Suite 200, Seattle, WA 98105, Email: pbell@uw.edu, lbricker@uw.edu, katievh@uw.edu, thorst@uw.edu Nichole Pinkard (Discussant), DePaul University, 243 South Wabash Avenue, Chicago, IL 60604 npinkard@cdm.depaul.edu
Abstract: Developments in information technology and digital media offer the opportunity to dramatically reorganize the curricular experiences available to students. This symposium explores elements of a next-generation high school biology course design, development, and implementation project that utilizes a social networking/media platform, facilitates youth participation in contemporary science, and networks youth with scientists and professionals. Specifically, we aim to study aspects of the platform that allow us to investigate learning related to: (a) the integration of game mechanics into the design of instructional materials, (b) youths engagement in contemporary scientific practices, and (c) platform features that make youths interests, expertise, and experiences visible so that they can be leveraged in instruction. We explore how to promote generative conditions for expertise development within these technology-mediated social learning networks and the effects of those experiences on student participation, identification, and learning of knowledge, skills, and practices.
Introduction
This symposium explores elements of a next-generation high school biology course design, development, and implementation project that utilizes a social networking/media platform, facilitates youth participation in contemporary science, and networks youth with scientists and other relevant professionals. Disciplinary experts give students feedback on their projects and share their career and educational trajectories. In this symposium, researchers present conceptual frameworks associated with the various aspects of the project model, as well as preliminary data, using the courses infectious disease and genetics units as a case study. Specifically, we aim to study aspects of the social networking/media platform (henceforth referred to as the learning platform) that allow us to investigate learning related to: (a) the integration of game mechanics into the design of instructional materials, (b) youths engagement in contemporary scientific practices, and (c) platform features that make youths interests, expertise, and experiences visible so that they can be leveraged in instruction. After an overview of the project writ large from which this case study stems, we present three elements of the case study units, including the associated conceptual frameworks guiding design and development, and some preliminary analyses. The first presentation examines the conceptual terrain of a badge system designed for both the infectious disease and genetics units. The second presentation examines the conceptual terrain associated with youths engagement in contemporary scientific issues and practices and presents a preliminary analysis of participating youths explorations in this arena.. The third presentation examines the conceptual terrain associated with design strategies that attempt to bridge youths out-of-school interest-driven pathways with their in-school mandate-driven trajectories and how the learning platform can serve as a bridging object.
Developing and Studying a Social Learning Network Model in the Context of Next Generation High School Courses
Developments in information technology and digital media offer the opportunity to dramatically reorganize the curricular experiences available to students (United States Department of Education, 2010). Collectives of students and teachers can engage in sustained, collaborative projects through cultivated social learning networks. Students can use social media to more easily connect to disciplinary experts and receive feedback and mentoring. They can use information technology to meaningfully participate in contemporary disciplinary pursuits, allowing for more participatory forms of project-based learning (e.g., citizen science, participatory youth models). Digital media technologies allow students to consume and produce a broad variety of rich media sources that relate to their learning investigations. Students can also use pervasive technologies to extend their learning experiences across the settings of their lives. We are developing an educational enterprise that integrates these learning affordances into three next-generation courses in the subject areas of biology, English language arts, and algebra. We have leveraged and customized the Remix platform to provide a cloud-based technological infrastructure for our social learning network (cf. Nichole Pinkard, 2007,
ISLS
142
ICLS2012
Volume 2: Symposia
http://remixlearning.com/). In this symposium we use two units in the biology course (infectious disease and genetics) as case studies to explore theoretical terrains associated with some of our design principles. Building upon a history of research on leveraging Internet technologies in instruction in the science classroom (e.g., Linn, Davis & Bell, 2004), we approach the cultivation, study, and refinement of social learning networks as design-based research (Bell, 2004) that sets the stage for large-scale design-based implementation research in the near future (see Penuel, et al., 2011). We pursue a sociocultural learning perspective to attend to the development of everyday expertise across social settings and networks of actors over developmental timescales (Bell et al., in press). We build upon theoretical traditions that highlight sophisticated learning as participation in repertoires of practice (Gutirrez & Rogoff, 2003) and apprenticeship processes, which occur within a nexus of structures of critical social practice (Dreier, 2009; Lave & Wenger, 1991). Youth are positioned as developing experts who express agency through extended projects designed to overlap with practices and interests of their home communities. We leverage the sense-making practices of youth and relate them to the practices of disciplinary fields. Youths are brought into sustained social interaction with networks of disciplinary experts who model expertise and guide youths project work. Youth authentically engage in interdisciplinary intellectual work and report their results to peers, teachers, disciplinary experts, and community members through multiple mechanisms.
Research Program and Scholarly Significance

A multifaceted research program explores: (a) how to promote generative conditions for expertise development within these technology-mediated social learning networks across classrooms and (b) the effects of those experiences on student participation, identification, and learning of knowledge, skills, and practices. We leverage the affordances of the technology platform to develop case studies that document and help visualize relevant phenomena, supplemented with video recording of classroom life and artifacts associated with the project work of youth. Through these efforts, we hope to develop an educational model for promoting sustained disciplinary learning across settings that builds on the social and intellectual capital of youth and extends their educational pathways.
Paper 1: Designing Badges for Use in a Project-Based Learning Curriculum Facilitated by a Social Media Platform
The use of video games, games for learning, and game-based education is increasing in formal and informal educational settings (Salen, 2008; Gee, 2009). Also garnering more attention is the related gamification movement, using game mechanics in typically non-game learning environments to increase motivation and participation (Deterding, et al. 2011). Though games and game mechanics have prompted innovative applications in education (most recently the Mozilla Open Badges https://wiki.mozilla.org/Badges and Digital Media and Learning competition http://dmlcompetition.net/Competition/4/badges-competition-cfp.php), the practices for how to do so successfully remain elusive and can benefit from further investigation. Designing games for learning is a complicated endeavor that leverages the practices of game design and instructional design combined to create engaging opportunities for students to learn without compromising the depth of instruction or the authenticity of the disciplines. Though this sounds ideal, actual implementation is filled with complex challenges. This paper examines one such combination of game and instructional design through the conceptual design and partial implementation of a badge system in a project-based curriculum facilitated by a social networking/media platform. This paper attempts to address issues which may contribute to answering the following questions: (a) How can we successfully design a badge system for educational settings?, and (b) What are the strategies for successfully reconciling differences between game and instructional design without compromising the qualities of game play or the opportunities for students to make meaningful contributions to contemporary scientific issues?
Research Focus
To answer these questions, the focus of this research examines three key components, which informed the conceptual design of the badge structure. First, we detail an analysis of existing badging systems in order to glean the appropriate design strategies to meet the needs specific to the curriculum. This includes an account of military and scout badges and the structure of achievement systems and badges in different video games to better understand the purpose of badges in specific settings. Second, we explicate a detailed framing of the instructional strategies and learning objectives that the infectious disease and genetics units are designed to address. In addition, the project has a commitment to provide students learning opportunities that put the learners at the forefront of contemporary scientific issues and knowledge production by positioning learners as developing experts. These components need to be accounted for in the design of the badge system and the question that remains is how. Lastly, we examine
ISLS
143
ICLS2012
Volume 2: Symposia
constraints and affordances of the social media platform itself including the technical capabilities as it informs the design of the badge system relative to the curricular demands.
Conceptual Framework and Study Design

The conceptual framework for this paper is rooted in game design (Crawford, 2003; Rollings & Adams, 2003; Salen & Zimmerman, 2003; Schell, 2008). This sets the foundation for considering the tensions created by combining game design with instructional design practices in designing a badge system. Though there are many methods and multi-faceted purposes to both game design and instructional design, this conceptual framework is limited to considering general purposes of each and the implications to design decisions. A comparison of the general purposes of game and instructional design leads to discussions on approaches to systems design, the tension between innovation and standardization, and designing a badge system that serves the unique needs of the curriculum. The study design is a combination of design based research (Design Based Research Collective, 2003) and conceptual analysis. The research consists of iterative cycles of integrating game design theory with instructional design practices in designing and partial creation of a badge system and then enacting it in the infectious disease unit and genetics unit. The infectious disease unit is a five-week unit that was piloted twice; in both the Spring and Fall of 2011. In the unit students designed and carried out investigations of local or global transmission of infectious disease using contemporary scientific software tools and the learning platform. The genetics unit is a six-week unit piloted in the Winter of 2012. The conceptual analysis includes a closer examination of the anticipated purposes of the badge system (inspire, mark interest and student identification, provide diagnostic feedback, acknowledge accomplishments, and make learning pathways visible).
Preliminary Data and Badge System Specifications to Date

This paper presents preliminary data and system specifications given the unit and badge system are still in enactment at the time of writing. The conference paper will expand on this analysis and discuss preliminary results. The design of the current badge system in development strategically awards badges to students (a) at significant moments in project progression, (b) when they make connections across disciplines, and (c) during instances of practice related to learning platform use and out-of-school community involvement. At the core are four main badges that represent culminating moments in the curriculum. For example, the scientific investigator badge is awarded to students for participating and completing project work in either global epidemic modeling or local social network analysis. To earn this badge, students must generate a testable question and propose a research design to progress in their investigation and reflect on how they would pursue future studies at the conclusion of the unit. The culminating scientific argumentation badge involves incremental levels of badges awarded during all aspects of argumentation work built into the unit. These levels include: (a) identifying and describing how argumentation is built into an activity with which they claim they have deep expertise, (b) identifying the rules of argumentation in their self-identified activity, and (c) utilizing the class-generated set of scientific argumentation rules to craft a scientific argument related to his/her project research and findings. There are four additional badges that act as support structures to these core badges and reinforce learning platform and community involvement. Two additional badges requiring work above and beyond the coursework are designed to be particularly difficult to acquire. With the exception of these last two difficult badges, all groups of badges consist of mandatory tasks (required to achieve the culminating badge) and optional badges (as opportunities to pursue additional interests). Other badge attributes such as hidden, automatic, peer-awarded, teacher-awarded, expert-awarded, repeatable, unlocking, emergent, limited, and expired are distributed through the badge infrastructure. Iterative design and conceptual rework of the badge system as we move through enactment cycles will continue to inform the system design.
Implications for Designing Educational Badges

We anticipate the process of interpreting previous badge systems, instructional framing of learning objectives through core ideas and practices in contemporary science, and the technical support and restrictions of the platform as central to understanding the success of a badge system. At the conclusion of enactment and through the iterative design of the badge system refined by student feedback, analysis of student work and badge artifacts, and design analysis, we will report upon the following conditions for designing educational badges: Defining a clear purpose of the badge system, determining which pieces of student activity are badge worthy, complicating the learning space to bridge connections not otherwise visible to learners, and lastly, examining design consequences of keeping within the technical restrictions of the platform functionality. Calling attention to specific attributes of learning through the incentive of a badge system offers a unique opportunity to make explicit learning that may otherwise go unnoticed. It is also an opportunity to shape and influence the way in which students engage in the curriculum both for individual activities and with the curriculum unit as a whole. However, it is important to recognize that neither game design nor instructional
ISLS
144
ICLS2012
Volume 2: Symposia
design approaches, independent of each other, meet the design needs of an educational badge system. It is the integration of both approaches, unique to each context that will most likely produce the desired results.
Paper 2: Networked Learning and Engagement in Contemporary Scientific Practices

Student participation in contemporary scientific practices, such as global epidemic modeling, requires intensive engagement within networked learning contexts. Learning analytics provide a set of tools to allow educators to trace students learning pathways as they gain competence with the scientific practices. The field of learning analytics represents a new analytic direction in education, its roots stemming from web analytics and business intelligence (Shum & Ferguson, 2011). Researchers collect data such as learners interests, learning preferences, interaction dynamics with resources, disciplinary experts, educators, and fellow learners, and sense-making strategies and routines (e.g., Retalis et al., 2006). Because this is a new analytic direction in education, few researchers in K-12 education are working in this arena to date. We highlight the conceptual terrain of a learning analytic approach used to study student learning of contemporary scientific practices in a six-week infectious disease unit and a six-week genetics unit. In addition, we present preliminary analysis guided by two research questions: (a) How does participation in the social learning network provide opportunities for deepening participation in disciplinary practices? (b) How is exploration, sharing, and deepening of interests evident in the social learning network? (Nacu & Pinkard, 2011)
Theoretical Framework and Methodological Approach

Emphasizing mastery of contemporary scientific practices helps foster deep understanding of the construction of scientific knowledge (NRC, 2011). Learning trajectories represent the available sets of learning experiences students can engage in to gain expertise relative to their interests. Exploration of trajectories within online learning environments allows for scaffolding integration of students experiences and expertise while making visible pathways for students to increase their domain-specific competencies and at the same time resourcing those pathways by connecting students to disciplinary experts, literature, materials, and tools (Pinkard & Schmidt, in preparation). This framework emphasizes the need to instructionally integrate a focus on scientific practices with possible learning trajectories in order to understand how to support learners engagement in communities of contemporary scientific practices. The six-week infectious disease unit, in which students investigated local or global transmission of infectious disease, was piloted twice; in both the Spring and Fall of 2011. In the six-week genetics unit piloted in Winter of 2012 students investigated and identified species using DNA barcoding and participated in a Foldit protein folding contest. This study reports on fieldnotes of classroom activity, criterion-based analysis of student products, artifacts of student expert communication, and metrics tracking students interactions with online and offline technologies, their peers, and curricular resources. We employ both qualitative accounts and quantitative analytics to examine learning trajectories over the unit.
Conceptual Constructs of Learning Trajectories & Preliminary Analysis

In order to map the conceptual terrain, we lay out constructs specifically developed to direct learning analytics research on students expertise development in interdisciplinary science. Given a focus on fostering student engagement in contemporary scientific practices with the goal of producing knowledge and artifacts that make meaningful contributions to the disciplines, the constructs outline possible ways to account for student participation and learning in social learning network contexts. We hope to contribute to thinking about the applications of these constructs to the analysis of learner behavior on social learning networks specifically related to disciplinary science learning over relatively long timescales (months to years). The development of these constructs was modeled after Nacu & Pinkards (2011) conceptual framework for analyzing social learning networks that support youths practices of critical consumption and active production of digital media. Here we focus on some similar constructs and other constructs that are uniquely associated with science learning including: interest, participation in practices, knowledge-linked expertise, identity, engagement in learning, networked learning, production and contribution. In the preliminary analysis of the infectious disease unit, students engaged in a practice of iterative model building and simulation execution using software designed to model global disease spread. They drove this practice through use of online resources, scientific journal articles, and consultation with peers, educators, and experts. In addition, they began to see biology as a predominately interdisciplinary science (NRC, 2009) and through participation in these practices and interfacing with experts, students recognized career opportunities that require expertise at the intersection of disciplines.
ISLS
145
ICLS2012
Volume 2: Symposia
Two representative examples demonstrate student engagement in the intellectual work of contemporary science as they interface with disciplinary experts in ways that direct their present and future learning trajectories. First, Julie talked about her interests in mathematics and computer science. She was on the robotics team at the high school and used her involvement with the team to develop her computer programming skills. She reported that her mother, a biologist, often talked with her about pursuing a career in the biological sciences. Julie did not see the connection between biology and her interests in computer science and mathematics until she began to work on computational modeling of infectious disease spread across the globe. Through iterative development and tweaking of her teams epidemic models, she worked to identify the underlying mathematical expressions that fuel the software-generated simulations. Her daily project work always focused on understanding how her teams research about the effects of airline travel on disease spread could be explained mathematically. She guided her team in comparing their infection curves generated from the modeling work with published data from research at the Centers for Disease Control she found online. After interacting with a computer scientist who uses computational models to study infectious disease scenarios, Julie told the researchers about how her mom has always pushed her toward biology but she did not see the connection between biology and her interests in programming until she participated in this unit. Through deep engagement with technological tools, Julies learning trajectory was driven by her interests and resourced by the curriculum, educators, and disciplinary experts in ways that allowed her to access knowledge about extended learning pathways that lead into careers at the intersection of biology and computer science. Second, the unit allows students to engage in collaborative knowledge production at the edges of the current research in the disciplines through access to the software tools and expert users of those tools. During the Spring 2011 unit enactment, a team chose to study the effects of poverty on the spread of infectious disease but the softwares design did not offer a straightforward way to accomplish this. Instead, the team decided to alter parameters in the model to simulate poverty. They shared this work with the principal investigator and designer of the modeling software, and he told the team that this was an excellent choice of study design given the limitations of the system. He also let them know that his team is addressing this limitation and is working on building this capability into the software. During his final review of their research he commented, The inclusion of economic and health infrastructure indicators in large-scale simulations is an interesting direction at the forefront in the research field. The technology platform allows students to connect with disciplinary experts and resources as they deepen their participation in contemporary scientific practices while tackling relevant problems in the field.
Implications for Sustained Disciplinary Learning

Learning analytics have affordances for researchers, educators, and learners. Online learning technologies that foreground social networking capabilities allow for data collection and analysis related to learning in ways that are powerful for (a) documenting learning, and (b) formatively giving insight to educators and learners about the learning process and personal progress. This is especially important in learning environments that support complicated and complex participation in contemporary scientific practices. In addition, aggregate learning analytics will highlight patterns and can be used to understand differences in learning influences based on instructional supports and specific learning cultures. These patterns will help identify the variety of pathways students can pursue towards expertise in contemporary scientific practices which in turn can lead to improved design and implementation of learning experiences embedded in social learning networks. Additionally, it is the first-step toward tracing the development of student identities by understanding how social learning networks connect students to scientists and other professionals and resources that provide them social capital and degrees of freedom to pursue learning pathways related to developing contemporary and interdisciplinary scientific expertise.
Paper 3: Leveraging Youths Everyday Expertise in Service of Engagement with Contemporary Scientific Practice: The Case of Argumentation
This paper highlights how researchers used the learning platform to surface youths interests, experiences, and expertise, which researchers then leveraged in the infectious disease and genetics units in service of engaging youth in scientific practices, such as the construction of scientific arguments. As previously discussed, one goal of the biology course design effort from which the papers in this symposium stem is to engage youths with contemporary scientific practices. One such practice is argumentation given its knowledge-shaping function in the sciences (e.g., Latour and Woolgar, 1986; Toulmin, Rieke, and Janik, 1984). In the infectious disease and genetics units, youths construct written products (e.g., a scientific research abstract, a proposal for funding) in which they argue scientifically using various rhetorical forms common in the sciences. Researchers working on this aspect of the biology units have a history of engaging youth with how to argue scientifically, as well as documenting youths everyday argumentative practices (see Bricker and Bell, 2011). Scholars have reported that it is quite difficult for youth to engage in scientific argumentation (e.g.,
ISLS
146
ICLS2012
Volume 2: Symposia
coordinate evidence with theory) (e.g., Kuhn, 1993) and in the first iteration of the infectious disease unit, we saw evidence of this difficulty. When writing their research abstracts, youths had no difficulty constructing claims relative to their research but they did not always utilize applicable evidence from their research to support their claims. Given our research on youths everyday argumentation, we have argued elsewhere (Bricker and Bell, 2008) that a helpful design strategy might be to surface youths everyday argumentation practices and then help them code switch in order to craft scientific arguments (cf. Gumperez and Hymes, 1986). We designed opportunities into the infectious disease and genetics units to test our assertions relative to engaging youth with scientific argumentation through the use of their everyday argumentation expertise. In this paper, we outline the conceptual terrain associated with these design strategies and report preliminary data.
Using the Learning Platform to Surface Interest and Everyday Expertise

At the beginning of both the infectious disease and genetics units, youths created About Me profiles. They selected an avatar to represent themselves (e.g., a symbol, photographs that represented a particular interest, drawings that the youths themselves created). In addition to whatever they wanted to write about themselves, we asked them to address the following question when creating their profiles: What are you an expert at doing and/or what do you do often as a hobby/hobbies? Youth noted activities such as playing sports, playing instruments, reading, writing, socializing with family and friends, watching TV, and playing videogames. In prior research an ethnography of youths science and technology learning across settings and timescales (see Bell, et al., 2006) once we identified youths areas of significant interest and expertise, we asked them about argumentative practices that were embedded within their activity. The youth participating in the ethnography identified specific argumentative strategies that they utilized within any given activity context. For example, youth identified strategies for arguing with parents in order to obtain a desired object or outcome (tell the parents only the affordances of the object or outcome and leave out any cons). In another example, they identified strategies for argumentation in their sports play (e.g., make certain that argumentation contains elements of critique so that it serves a learning function in service of helping self and others improve). In addition, we gave ethnography participants digital cameras and asked them to document the various meanings they associated with argumentation-in-context-of-everyday-expertise (cf. Clark-Ibanez, 2004). We embedded many of these same activities into the design of the infectious disease and genetics units. Using areas of interest and expertise culled from youths About Me profiles, we asked youth to explicate the rules of argumentation, particularly rules related to discourse, as argumentation takes place within their areas of everyday expertise. We also asked them to photograph images related to their areas of everyday expertise and the argumentation within those areas. Once we (the community of learners participating in the unit youth, teachers, and researchers) fully explicated the details of our everyday argumentation practices as they are situated within specific activity, we created rules for scientific argumentation, which we coded as simply another type of argumentation practice situated within a specific activity (science). We utilized the platform for this activity as well by using it to communicate with participating scientists and other professionals about their scientific argumentation practices. We report preliminary findings as part of this paper.
Implications
Utilizing youths interests, expertise, cultural practices, and experiences in the designs of learning environments is a well-documented learning strategy (e.g., Gonzlez, Moll, and Amanti, 2005; McIntyre, Rosebery, and Gonzlez, 2001; Nasir, et al., 2006). The use of social networking and media platforms enable learners to easily surface, share, compile, and trace everyday expertise. Learners can then utilize this as a springboard to learn complex disciplinary ideas and practices.
References
Bell, P. (2004). On the theoretical breadth of design-based research in education. Educational Psychologist, 39(4), 243-253. Bell, P., Bricker, L.A., Lee, T.R., Reeve, S., & Zimmerman, H.T. (2006). Understanding the cultural foundations of childrens biological knowledge: insights from everyday cognition research. In S.A. Barab, K.E. Hay, and D.T. Hickey (Eds.), Proceedings of the Seventh International Conference of the Learning Sciences (pp. 1029-1035). Mahwah, NJ: Lawrence Erlbaum Associates. Bell, P., Bricker, L.A., Reeve, S., Zimmerman, H.T., & Tzou, C. (in press). Discovering and supporting successful learning pathways of youth in and out of school: Accounting for the development of everyday expertise across settings. In B. Bevan, P. Bell, & R. Stevens (Eds.), Learning about out of school time (LOST) learning opportunities. New York: Springer. Bricker, L.A., & Bell, P. (2008). Conceptualizations of argumentation from science studies and the learning sciences and their implications for the practices of science education. Science Education, 92(3), 473498.
ISLS
147
ICLS2012
Volume 2: Symposia
Bricker, L.A., & Bell, P. (2011). Argumentation and reasoning in life and in school: Implications for the design of school science learning environments. In M.S. Khine (Ed.), Perspectives on scientific argumentation: Theory, practice, and research. New York, NY: Springer. Clark-Ibanez, M. (2004). Framing the social work with photo-elicitation interviews. The American Behavioral Scientist, 47(12), 1507-1527. Crawford, C. (2003). Chris Crawford on game design. Indianapolis, IN: New Riders. Deterding, S., Sicart, M., Nacke, L., OHara, K., & Dixon, D. (2011, May 7-12). Gamification: Using game design elements in non-gaming contexts. Workshop presented at the ACM CHI Conference. Retrieved September 2, 2011, from http://gamification-research.org/wp-content/uploads/2011/04/01-DeterdingSicart-Nacke-OHara-Dixon.pdf Design Based Research Collective. (2002). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5-8. Dreier, O. (2009). Persons in structures of social practice. Theory Psychology, 19(2), 193-212. Gee, J. P. (2009). Digital media and learning as an emerging field, Part I: How we got here. International Journal of Learning and Media, 1(2), 13-23. Gonzlez, N., Moll, L.C., & Amanti, C. (Eds.). (2005). Funds of knowledge: Theorizing practices in households, communities, and classrooms. Mahwah, NJ: Lawrence Erlbaum Associates. Gumperz, J.J., & Hymes, D. (Eds.) (1986). Directions in sociolinguistics: The ethnography of communication. Oxford, UK: Basil Blackwell. Gutirrez, K., & Rogoff, B. (2003). Cultural ways of learning: Individual traits or repertoires of practice. Educational Researcher, 22(5), 19-25. Kuhn, D. (1993). Connecting scientific and informal reasoning. Merrill-Palmer Quarterly, 39(1), 74-103. Latour, B., & Woolgar, S. (1986). Laboratory life: the construction of scientific facts. Princeton, NJ: Princeton University Press. Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. New York, NY: Cambridge University Press. Linn, M. C., Davis, E. A., & Bell, P. (2004). Internet environments for science education. Mahwah, NJ: Erlbaum. McIntyre, E., Rosebery, A., & Gonzlez, N. (2001). Classroom diversity: Connecting curriculum to students lives. Portsmouth, NH: Heinemann. Nacu, D., & Pinkard, N. (2011) iRemix platform: Reports and visualizations. A Progress Update for the Bill and Melinda Gates Foundation. Nasir, N.S., Rosebery, A., Warren, B., & Lee, C.D. (2006). Learning as a cultural process: Achieving equity through diversity. In R.K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (pp. 489504). Cambridge, UK: Cambridge University Press. National Research Council. (2009). A new biology for the 21st century. Washington, DC: National Academies Press. National Research Council. (2011). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas. Washington, DC: National Academies Press. Penuel, W.R., Fishman, B.J., Cheng, B.H., & Sabelli, N. (2011). Organizing research and development at the intersection of learning, implementation, and design. Educational Researcher, 40(7), 331-337. Pinkard, N. (2007). Preparing urban youth to be multiliterate. Spotlight on Digital Media and Learning, from http://spotlight.macfound.org/blog/entry/nichole-pinkard-youth-multiliterate-learners Pinkard, N., & Schmidt, R. (in preparation). Using role based pathways to ignite student learning identities. Retalis, S., Papasalouros, A., Psaromiligkos, Y., Siscos, S., & Kargidis, T. (2006). Towards networked learning analytics: A concept and a tool. In S. Banks et al. (Eds). Proceedings of the Fifth International Conference on Networked Learning 2006 (pp. 1-8). Lancaster, UK: Lancaster University. Rollings, A., & Adams, E. (2003). Andrew Rollings and Ernest Adams on game design. Berkeley, CA: New Riders. Salen, K. (2008). Toward an ecology of gaming. In K. Salen (Ed.), The ecology of games: Connecting youth, games, and learning (pp. 1-20). Cambridge, MA: The MIT Press. Salen, K., & Zimmerman, E. (2003). Rules of Play: Game Design Fundamentals. Cambridge, MA: The MIT Press. Schell, J. (2008). The Art of Game Design: A book of lenses. Waltham, MA: Morgan Kaufmann. Shum, S.B., & Ferguson, R. (2011). Social learning analytics. United Kingdom: Knowledge Media Institute, The Open University. Toulmin, S.E., Rieke, R., & Janik, A. (1984). An introduction to reasoning (2nd ed.). New York, NY: Macmillan Publishing Company, Inc. United States Department of Education. (2010). Transforming American education: Learning powered by technology. Washington, D.C.: Government Printing Office.
ISLS
148
ICLS2012
Volume 2: Short Papers
Developing Primary Students Argumentation Skills in Inquiry-Based Mathematics Classrooms

Jill Fielding-Wells, Katie Makar, The University of Queensland, Australia Email: j.wells2@uq.edu.au, k.makar@uq.edu.au Abstract: Most educational research on argumentation comes from science, with argumentation in mathematics tending to focus on proof. We contend that argumentation can be used productively in learning mathematics even at the primary level. A research study was designed to explore childrens development of argumentation in an Australian primary mathematics classroom. The classroom of 23 children (aged 9-10) had regularly used an inquiry-based approach to address extended, complex, ill-structured problems. The childrens discussions and use of evidence is reported as they considered contentious media claims. The results of the design-based research study suggest that the children became proficient with Toulmins argument framework (simplified). They were able to use this framework to plan, implement and defend the outcomes of a mathematical investigation they designed to provide evidence for or against the media claims. The paper highlights benefits and challenges with which student grappled while making and substantiating their final claims.
Why Study Argumentation in Mathematics?

There is no natural or logical necessity to the state of mathematical knowledge at present. For as long as our results are useful, appropriate and consistent within our current understanding of the term, and given the existential position of our experience, we call them true. At any particular time, however, this is controversial, as there are generally rival theories claiming truth. What characterizes mathematics is the activity: engaging in interesting problems; making imaginative conjectures; testing; reflecting; examining results informally; formalizing and testing results formally; publishing ideas for criticism and development by the mathematical community, etc. (Lerman, 1990, p. 55). Contrast Lermans description of mathematical practice with the way in which mathematics is largely taught in schools. New understandings routinely come through the repetition of increasingly complex concepts and learnt procedures as students demonstrate growing competence; there are few opportunities for the practical application of mathematics through addressing ill-structured problems in real-life contexts. It is unsurprising then that students report that they fail to see the sense in mathematics or find it difficult to connect it to their lives (McPhan, Moroney, Pegg, Cooksey, & Lynch, 2008). In response to these issues, a focus on mathematical reasoning has been incorporated into many curriculum documents (ACARA, 2012; National Council of Teachers of Mathematics, 2000) with investigative and inquiry approaches frequently suggested as a way forward. Mathematical inquiry is used to address ill-structured problems, those in which the problem statement or solution strategies contain ambiguities (Makar, 2010; Reitman, 1965). Their inherent ambiguity allows for multiple interpretations of a question, a range of pathways, and numerous solutions with varying degrees of efficiency, applicability and elegance. This requires students to focus on decision making, analysis and justification. Rather than a correct answer or strategy, there is a claim which requires evidence, explanation and defensein short, an argument. Potential benefits to argumentation in the classroom have been welldocumented in science education where inquiry-based learning (IBL) has been promoted for some time. For example, explanation and argumentation provide students with opportunities to develop high levels of subject specific literacy (Jimenez-Aleixandre & Erduran, 2007) and enculturate them into the practices of the subject (Duschl, 2007). Research informs us however, that students must be explicitly taught the skills of argumentation (Kuhn, 1991) through suitable instruction, task structuring and modeling (Jimnez-Aleixandre & Erduran, 2007). Skills to be taught include knowing the meaning of argument, positioning, justifying with evidence, constructing arguments, evaluating arguments and reflecting on the argument process (Simon & Richardson, 2009). While argumentation is not a new concept in mathematics, argumentation research in the discipline has largely been associated with mathematical proof (see, for example, Conner, 2007; Lampert, 1990). Lampert (1990, p. 40) for example, focused on students use of hypothesis testing, asserting that it is the strategies used for figuring out, rather than the answers, that are the site of the mathematical argument. However, the approach to argumentation adopted in this paper differs from this, and describes a process by which students use IBL to examine a contextualized problem, use mathematics to obtain evidence, and then make and justify claims within
ISLS
149
ICLS2012
the question context. In IBL, numerous solutions are possible and therefore the answer itself must be justified and defended, as well as the choice of solution strategies. The nature of the problem is such that the students are supported to determine their own method(s) of approaching and solving their problems; the mathematical solutions and strategies employed serve to provide the grounds and justification for their claims.
Structure and evaluation of arguments

One of the mostly widely used frameworks for examining argument structure is that of Toulmin, Rieke and Janik (1984). Their argument framework describes multiple elements of an argument, among which are claim, grounds, and qualifiers. The claim is the initial assertion that identifies the stance and position taken. The grounds provide the underlying support, or primary evidence, that is required to enable the claim to be accepted as reliable and valid. In practice, claims are often based on imperfect evidence and it is necessary to indicate the strength of the argument through the use of a qualifier which communicates the strength of the link between the grounds and the claim. Toulmin et al. describes breakdown in the grounds or warrants as fallacy, resulting in arguments that can seem persuasive despite being unsound (1984:132). Toulmin et al. identify broad categories of fallacy which can be adopted to assist in the determination of common fallacious thinking, for example, missing grounds, defective grounds or irrelevant grounds. Sampson and Clark (2006) took a broader approach and provided five criteria for teachers to assess the quality of students scientific arguments: (1) The nature and quality of the claim (Are students able to coordinate claims with available evidence?); (2) How far the claim is justified (Was evidence provided? Was it the right kind of evidence?); (3) If the claim accounts for all the available evidence (Was the evidence complete and comprehensive?); (4) How the argument attempts to discount alternatives (Were multiple perspectives considered?); (5) How epistemological references are used to coordinate claims and evidence (How did students gather and interpret data? Did they consider the design or methodology when weighing the evidence?). Further research into argumentation practices in IBL in primary mathematics is warranted; much of the research into argumentation and IBL has come from school and tertiary science, which cannot be assumed to be directly transferrable to learning in mathematics. IBL has been introduced relatively recently as a pedagogical approach in mathematics and, as such, there has been little research into its development in primary mathematics and still less into the development of argumentation within this context. Likewise, there is little reported in the literature in terms of what is possible in developing argumentation with young children or what supports and structures may facilitate this. This paper attempts to explore what children may be capable of in terms of the development and use of evidence in argumentation as well as supports that may assist them.
Design and Method

While learners often initially respond to inquiry questions with rapid, intuitive responses that are typically unsupported, scaffolding them through a cycle of collection, and organization and analysis of evidence, enables them to plan increasingly sophisticated statistical investigations (Fielding-Wells, 2010). This was achieved by focusing students on the inquiry question and having them collaboratively envisage evidence they might need and conclusions that could result. The current study adopted a design-experiment approach (Cobb, Confrey, diSessa, Lehrer, & Schauble, 2003) as a methodological stance because of its aim to develop theory through multiple iterations of reflective-prospective cycles of improvement. The focus in this paper is on one cycle of a project which seeks to deepen understanding of primary age childrens developing argumentation practices. Participants were from a class of twenty-three 9-10 year olds in a large, co-educational, suburban primary school in Australia who had been involved in learning mathematics through inquiry for one and a half years. A mathematical inquiry unit designed and taught by the first author provided a topical and contentious context for the use of statistics. As a result of ongoing drought, the state water authority had spent $1.7M to purchase and issue shower timers to all households in an attempt to encourage decreased water usage through limiting shower duration, a move which received both public support and criticism. The five-week unit was implemented with approximately two 60 minute sessions per week. Prior to this unit, specific modeling and discussion of a simplified structure of an argument was undertaken. The model used by the students incorporated claim, grounds and qualification. The learning sequence was videotaped, students work samples were collected, and research logs maintained. The results reported here focus on excerpts from the students final written arguments. An adapted grounded theory approach (Corbin & Strauss, 2008; Flick, 2009) was taken to analyze the student artifacts. Each argument was subjected to open coding to identify unanticipated insights (Corbin & Strauss, 2008), sensitize the researcher to overall patterns and nuances in the data, and enable development of codes to classify aspects of the study (Flick, 2009). Next, each argument was analytically deconstructed into its individual components using Toulmin et al.s (1984) structure of argumentation. Toulmin et al.s fallacy schema was used to identify the frequency and nature of categories of fallacy. This level of deconstruction enabled an overall view of the developing abilities of the students as well as a closer consideration of any patterns in fallacious thinking that would indicate areas for the development of additional supports. Finally, students
ISLS
150
ICLS2012
responses were compared with Sampson and Clarkes (2006) criteria for quality arguments to further suggest areas in which students were generally strong and weak in their presented arguments.
Results
In the initial phase of the inquiry, students were challenged to determine whether a tax-payer funded distribution of shower timers to all households was a cost effective decision, particularly in light of media criticism of their quality. The students collectively negotiated and refined the inquiry question: Are the [state] water commission shower timers accurate to within 10% of 4 minutes? The class designed a statistical investigation in which thirty shower timers were tested five times each. Working in collaborative groups, students timed a small number of shower timers with the results recorded, collated and distributed among the class for analysis. Their results found that 18 of the 30 shower timers came within 10% of four minutes (3:36 4:24) on all five tests. The students were generally able to successfully plan and conduct the inquiry and interpret their data accurately. This was anticipated in consideration of the students extensive prior experiences with inquiry learning. In this section, we focus on students final arguments of the shower timer inquiry. After completing their data collection, students were asked to write a letter to the water commissioner and report what they had found. The extent to which an individual students argument exhibited a claim, evidence and qualification was categorized as Evident, Developing-Inconsistent and Not Evident. As distinguishing factors, the main codes from the analysis (Table 1) frame our reporting and discussion around student work. Table 1: Categorization of claims, evidence and qualifications by students (n = 23) Evident Claim Clearly stated position Consistent throughout Foregrounded Addresses the inquiry question Ground within the scope of the inquiry question Grounds support the claim Ground acceptable within the community of practice Indicates the strength of the grounds 21 20 19 6 19 13 13 3 Developing Inconsistent 1 2 2 15 3 9 8 3 Not Evident 1 1 2 2 1 1 2 17
Grounds
Qualification
Claim
Few students experienced difficulty in producing an argument in which their claim was foregrounded, which positioned the reader clearly and which maintained consistency of position throughout the argument (Table 1). Below, we illustrate the diversity of responses from students across the more challenging area of reporting claims. In terms of students addressing the inquiry question when making their claim, some students did articulate a response specifically related to the inquiry question: Shana: The four minute shower timers produced to help introduce the campaign are faulty. Leticia: Shower timers are clarey (sic) faulty. However, it was notable that the majority of students experienced difficulty in doing so. Without exception, each of the students assigned to the Developing-Inconsistent category made a claim consistent with the broader context of the inquiry. The specific inquiry question related to the accuracy of the timers, however, and these students claims addressed tangential contextual issues that were not directly addressed, nor could be supported through the inquiry: Dominica: The 1.7 million dollars spent on shower timers was a waste of money and only helped saving water a bit. In order to justify such a claim, it would be necessary to have data on the financial benefit of the water saved and the actual amount of water saved as a direct result of the shower timers. Dominicas response was included in the Developing category as it was situated within the broader context of the inquiry rather than considering only the data gathered to address the refined inquiry question.
Grounds
Most students were able to provide sufficient grounds from the inquiry findings to enable the reader to be satisfied that their claim was justified, reasoned and supported. Notably, these students were able to demonstrate
ISLS
151
ICLS2012
that their selection and application of mathematical processes was appropriate and efficient, and that they were able to interpret the resultant answers. Dominica: My class has provided grounds from this test. We tested every shower timer 5 tim es we worked out that 18/30 were accurate and ran in the range of 3:36 4:24. Fo r the other 12 shower timers they were faulty because they were out of the range [or] they stopped during timing. Not all students were proficient in providing evidence mathematically. Despite all students having an amalgamated list of times, some students incorrectly interpreted the mathematics, or relied on non-mathematical sources for their evidence. These arguments were further examined using Toulmin et al.s (1984) categorization of fallacy to identify areas of difficulty. The fallacious grounds put forward by students were segregated fairly evenly between the provision of irrelevant grounds (Leanne, Samuel) or defective grounds (Geneva, Blain): Leanne: Shower timers help the environment a lot becauseif you have a hot shower you could use up all the hot water. Samuel: There are different amounts of sand in the timers. Geneva: They would still shorten showers that can go up to 30 minuites (sic). [No evidence was provided or discovered that gave 30 minutes as a time for a shower]. Blain: One of the timers we tested was 2:00-2:25 over 4:00. [Focused on a single outlier rather than aggregate results]
Qualification
Finally, consideration was given to whether the students had used a qualifier to indicate the strength of their findings, specifically based on the statistical acceptability of inferring numbers for a population of 1.1 million from a sample of 30. Several of the students recognized that this was a limitation despite this being a concept that is not expected to be appreciated this early in formal schooling. Andrea: From the evidence that has been collected Im not completely sure because we only timed thirty shower timers out of 1100000. There is (sic) 3666.6 groups of 30 in 1100000 and we tested one group.
Discussion and Conclusion

In this paper, we explored potential benefits of adopting argumentation practices for primary children working in a classroom which engages in mathematical inquiry. In particular, we sought to understand the ways in which argumentation structures could be adapted into the context of mathematics education. Because mathematics is often considered to be a discipline which eschews ambiguity and lacks contention, we saw important opportunities for creating student learning experiences which might dispel this perspective. In this study, children as young as nine years old were able to take advantage of argumentation structures to develop understanding in multiple areas of mathematics. In this particular inquiry, students worked with concepts in measurement, data collection and analysis, financial mathematics and logical reasoning. Although students still found many aspects of formal argumentation challenging, what they were able to accomplish was encouraging. As an exploratory study, we were interested in what students were able to achieve as well as what they found challenging. Building on previous work (Wells, 2010), it appeared that students found the simplified Toulmin et al. framework generally useful when structuring and articulating the process of inquiry. Conversely, experience with inquiry provided the familiarity with gathering evidence and drawing conclusions that enabled argumentation practices to be developed. In designing their inquiry question, for example, the students found it beneficial to consider from the outset both the potential claims that could emerge and the evidence that would be needed in order to support and defend those claims. These were practices that had been emphasized throughout their experiences with conducting mathematical inquiry. In considering Sampson and Clarks (2006) criteria for quality arguments, many of the students were at least partially able to appropriately coordinate their claims with the available evidence, go beyond reliance on beliefs and emotion in justifying their findings, and recognize the need to include aspects of their methodology into the reporting of findings. Being able to acknowledge limitations or qualify their findings were areas that students across the board found difficult and may be an area that is more appropriately addressed in later years. The results also suggest that students had some difficulty in stating relevant claims that addressed the specific inquiry question, an issue that several researchers have argued students find difficult when working with data in mathematics (Hancock, Kaput, & Goldsmith, 1992; Lavigne & Lajoie, 2007). However, all students that made claims that went beyond the inquiry question remained focused within the broader context of the inquiry. Four elements of the design likely contributed to students success in engaging with argumentation practices in mathematics. First, the students were enculturated in an inquiry-based environment that valued discussion, meaning-making and multiple perspectives. Second, the topic of the inquiry was designed around a
ISLS
152
ICLS2012
contentious issue which challenged and provoked the students into providing an effective argument. Third, students were asked to respond to an authentic audience beyond the school. This aspect likely raised the need to be explicit about their method and inclusive of detail as the audience was unaware of their processes and procedures (Berland & Forte, 2010). Finally, students were scaffolded through the inquiry and the argumentation process by the teacher to ensure that they were explicitly taught foundational concepts of argumentation. This point has been raised by multiple researchers working in argumentationthat these skills need to be explicitly taught (Simon & Richardson, 2009). This study provides initial evidence that the benefits of argumentation further extend practices valued in mathematical inquiry. Argumentation works well to enculturate learners into a community of practice which values negotiation, making connections and applying mathematics beyond a procedural level. Finally, embedding argumentation into mathematical inquiry strengthens the conception of mathematics as more than a discipline of certainty, particularly when using mathematical concepts within authentic contexts.
References
ACARA. (2012). Australian Curriculum: Mathematics v3.0. Retrieved from www.australiancurriculum.edu.au. Berland, L. K., & Forte, A. (2010). When students speak, who listens? Constructing arguments in classroom argumentation. Paper presented at the 9th International Conference of the Learning Sciences, Chicago. Cobb, P., Confrey, J., diSessa, A., Lehrer, R., & Schauble, L. (2003). Design experiments in educational research. Educational Researcher, 32(1), 9-13. Conner, A. (2007). Student teachers' conceptions of proof and facilitation of argumentation in secondary mathematics classrooms. Ph.D. dissertation, The Pennsylvania State University, Pennsylvania. Dissertations & Theses: Full Text database. Corbin, J. M., & Strauss, A. (2008). Basics of qualitative research: Techniques and procedures for developing grounded theory. Thousand Oaks, CA: SAGE. Duschl, R. A. (2007). Quality argumentation and epistemic criteria. In S. Erduran & M. P. Jimenez-Aleixandre (Eds.), Argumentation in science education. New York: Springer. Fielding-Wells, J. (2010). Linking problems, conclusions and evidence: Primary students early experiences of planning statistical investigations. Paper presented at the Eighth International Conference on Teaching Statistics, Ljubljana, Slovenia. Flick, U. (2009). An introduction to qualitative research (4th ed.). London: SAGE. Hancock, C., Kaput, J. J., & Goldsmith, L. T. (1992). Authentic inquiry with data: Critical barriers to classroom implementation. Educational Psychologist, 27(3), 337-364. Jimenez-Aleixandre, M. P., & Erduran, S. (2007). Argumentation in science education: An overview. In S. Erduran & M. P. Jimenez-Aleixandre (Eds.), Argumentation in science education (pp. 3-27): Springer. Kuhn, D. ( 1991). The skills of argument. Cambridge: Cambridge University Press. Lampert, M. (1990). When the problem is not the question and the solution is not the answer: Mathematical knowing and teaching. American Educational Research Journal, 27(1), 29-63. Lavigne, N. C., & Lajoie, S. P. (2007). Statistical reasoning of middle school children engaged in survey inquiry. Contemporary Educational Psychology, 32(4), 630-666. Lerman, S. (1990). Alternative perspectives of the nature of mathematics and their influence on the teaching of mathematics. British Educational Research Journal, 16(1), 53-61. doi: 10.1080/0141192900160105 Makar, K. (2010). Teaching primary teachers to teach statistical investigations: the uniqueness of intial experiences. Paper presented at the Eighth International Conference on Teaching Statistics, Ljubljana. McPhan, G., Moroney, W., Pegg, J., Cooksey, R., & Lynch, T. (2008). Maths? Why not? AAMT: Canberra. National Council of Teachers of Mathematics (2000). Principles and standards for school mathematics. Reston, VA: author. Retrieved from www.nctm.org/standards/content.aspx?id=4294967312. Reitman, W. (1965). Cognition and thought: An information-processing approach. New York: Wiley. Sampson, V., & Clarke, D. (2006). Assessment of argument in science education: A critical review of the literature. Paper presented at the International Conference of the Learning Sciences, Indiana. Simon, S., & Richardson, K. (2009). Argumentation in school science: Breaking the tradition of authoritative exposition through a pedagogy that promotes discussion and reasoning. Argumentation, 23(4), 469. Toulmin, S., Rieke, R., & Janik, A. (1984). An introduction to reasoning (2nd ed.). New York: Macmillan. Wells, J. (2010). Developing argumentation practices in inquiry based mathematics classrooms. PhD Confirmation Document. School of Education. University of Queensland. Brisbane.
Acknowledgments
This research was funded by the Australian Research Council (LP0990184) in partnership with Education Queensland and The University of Queensland. The first author is in receipt of an Australian Postgraduate Award Scholarship and also wishes to acknowledge the financial support of the Commonwealth Government.
ISLS
153
ICLS2012
Language Formality, Learning Environments, and Student Achievement

Emily Schoerning, Brian Hand, University of Iowa, N297A Lindquist Center, Teaching and Learning, Iowa City, IA 52242 Email: emily-schoerning@uiowa.edu; brian-hand@uiowa.edu Abstract: This study examines learning environments in traditional elementary science classrooms and elementary science classrooms that utilize Argument Based Inquiry (ABI) via the Science Writing Heuristic (SWH). The degree of formality used in classroom spoken language is characterized, and its impacts on learning environments and student outcomes are explored. It is found that distinctly different styles of language characterize traditional vs. SWH classrooms, and proposed that informal language has a positive impact on student achievement.
Major Issue Addressed

This study seeks to examine and characterize the linguistic styles utilized in traditional and Science Writing Heuristic (SWH) elementary science classrooms, with a particular focus on spoken language formality. Characteristics utilized to describe language formality include fluidity of dialog, prevalence of honorifics, use of slang and popular expressions, presence of non-verbal relaxation markers, summarization of others speech during dialog, as well as several others. Both students and teachers use of spoken language are examined. Once the language styles typical of traditional and SWH classrooms have been characterized their relationship to the classrooms learning environment will be explored. Connections between language formality, learning environments, and student outcomes will be identified via quantitative and qualitative methods.
Potential Significance of the Work

It has been seen that Argument Based Inquiry (ABI) and specifically the SWH increases student achievement on both tests of basic skills and critical thinking tests (Author, 2007; Lord, 1999, Mergendoller et al., 2006; HlemoSilver 2007; Dochy et al, 2003). This study will contribute to our theoretical understanding of SWH. The question of why the SWH increases student achievement, and specifically student achievement in hard to reach groups such as students with special needs, has yet to be fully unpacked (Author, 2009). It is theorized that one of the reasons the SWH is so effective is related to the learning environment it generates. Characterization of this learning environment is ongoing. As we examine the differences between SWH and control classrooms in terms of argument structures we have begun to consider that one element may underlie many of the differences we see: linguistic formality (Author, 2010). The use of formal and informal language may be different in SWH and traditional classrooms. While there are occasional mentions of language formality as a possible contributor to learning environments in the literature, little has been done to characterize what makes language formal or informal (Piirto, 2000; Carrell & Willmington 1996; Rubin, 1982; Krashen, 1976). An instrument to accomplish this goal has been developed for this study. In the course of this study the instruments validity and reliability will be established to allow its use by other researchers. Language formality greatly impacts our perceptions and our actions. Allowing clear characterization of this element of the classroom environment will allow for exploration of such topics as discourse space and non-threatening learning environments. The generation of a reliable and valid instrument for the characterization of linguistic formality is potentially significant to those researchers who are interested in exploring learning environments. The studys contributions to our understanding of what factors underlie the effectiveness of inquiry-based teaching and learning would also be a potentially significant in terms of theory development.
Theoretical and Methodological Approaches Pursued

Both quantitative and qualitative methods were utilized in this research. The qualitative work consists of indepth interviews with teachers and detailed notes regarding classroom observations made at five schools: three treatment and two control. Quantitative assessment of language formality was also done during classroom observations using the Spoken Language Style Characterization Tool (SLSCT), which consists of 18 Likertscale measures assessing elements of student and teacher language. Face validity of this instrument has been determined by a panel of experts. The reliability of the instrument has yet to be determined due to ongoing data collection; internal reliability will be determined once a sufficiently large data set has been generated. If reliability is insufficient the instrument will be revised and the process repeated until reasonable reliability is
ISLS
154
ICLS2012
achieved. Quantitative assessment of student achievement will be accomplished via two instruments; the Cornell Critical Thinking Test (CCT) and the Iowa Test of Basic Skills (ITBS). These instruments are both well-characterized and have been shown to have clear correlations with student achievement, (Ennis et al., 2005; Hoover et al., 2003). Classrooms SLSCT scores will be compared to their ITBS and CCT scores to determine if correlations exist between language formality and student achievement. SLSCT scores will also be analyzed in the context of observation notes and teacher interviews to qualitatively describe the effect of language formality on learning environments.
Preliminary Findings, Conclusions, and Implications

Preliminary findings indicate that there are clear differences in the language styles of traditional and SWH classrooms. Teachers in traditional classrooms are more likely to emphasize the acquisition and use of formal vocabulary and require that students make formal requests to speak. Dialog between students is rare, and both students and teachers infrequently summarize each others contributions before adding to the dialog. These characteristics all indicate formal language use, while SWH classrooms can be characterized by their utilization of aspects of informal language. Dialog interchange is more often frequent and fluid, both students and teachers are likely to restate or summarize others thoughts before beginning their own, and all participants in the classroom are more likely to display non-verbal relaxation markers such as smiling, laughter, or relaxed body language. Importantly, dialog between students emerges as a valid mode of discourse within SWH classrooms. Language is the medium through which we explore and engage in science. Differences in spoken language style both reflect and form distinctly different learning environments. The clear characterization of these language styles allows us to describe and quantify differences in learning environments that before we may only have intuited or vaguely felt. Through this characterization we may be able to unpack why the SWH and other forms of ABI are beneficial to so many types of students that traditionally struggle to perform in science class (Author, 2009). It is possible that language informality gives traditionally disadvantaged students both the comfort and the confidence that allows them to meaningfully engage in science education (Duran, 1998; Rakow & Bermudez, 1993). The use of a language style common to their community and everyday lives may reduce their perceptions of science as an exclusive field, increasing participation. Implications of this study include suggestions for practitioners in regards to what language styles contribute to learning environments that foster student achievement. The study will be of interest to researchers who want to understand why and how factors in the classroom environment influence student learning, particularly as these relate to sociocultural practices, language, and science-related modes of discourse such as argumentation.
References
Akkus R., Gunel M., Hand B. (2007). Comparing an Inquiry-based Approach known as the Science Writing Heuristic to Traditional Science Teaching Practices: Are there Differences? International Journal of Science Education. 29(14), 1745-1765. Choi, A., Notebaert, A., Diaz, J, Hand, B. (2010). Examining Arguments Generated by Year 5, 7, and 10 Students in Science Classrooms. Research in Science Education. 40:149-169. Carrel, L.J, Willmington, S.C. (1996). A Comparison of Self-Report and Performance Data in Assessing Speaking and Listening Competence. Communication Reports. 9(2), 185-191 Dochy,F., M. Segers, P. Van den Bossche, D. Gijbels. (2003). Effects of problem-based learning: A metaanalysis. Learning and Instruction. 13: 533-568. Duran, B.J. (1998). Language minority students in high school: the role of language in learning biology concepts. Science Education, 82(3), 311-341. Ennis, R.H, Millman, J. (1985) Cornell critical thinking tests level X and level Z manual. Pacific Grove: Midwest Publications. Hand, B., Norton-Meier L., Staker J. (2009). Negotiating Science: The Critical Role of Argument in Science Inquiry, Grades 5-10. Portsmouth, NH: Heinemann. Hoover, H.D. et al. (2003). The Iowa tests: guide to research and development. Itasca, Illinois: Riverside Publishing Company. Hmelo-Silver, C.E. (2007). Scaffolding and achievement in Problem-Based and Inquiry Learning: A response to Kirschner, Sweller, and Clark (2006). Educational Psychologist, 42: 99-107. Krashen, S.D. (1976). Formal and Informal Linguistic Environments in Language Acquisition and Language Learning. TESOL Quarterly. 10(2), 157-168. Lord, T. R. (1999). A comparison between traditional and constructivist teaching in environmental science. Journal of Environmental Education, 30(3), 2228.
ISLS
155
ICLS2012
Mergendoller, J.R, N.L. Maxwell, Y. Bellisimo. (2006). The effectiveness of problem-based instruction: A comparative study of instructional method and student characteristics. Interdisciplinary Journal of Problem-based Learning. 1: 49-69. Norman G.R, H.G. Schmidt. (2000). Effectiveness of problem-based learning curricula: theory, practice and paper darts. Medical Education. 34: 721-728. Piirto, J. (2000). Speech: An Enhancement to (Technical) Writing. Journal of Engineering Education. 21-23. Rakow, S.J., & Bermudez, A.B. (1993). Science is Ciencia: Meeting the needs of Hispanic American students. Science Education, 77(6), 669-683. Rubin, R.B. (1982). Assessing Speaking and Listening Competence at the College Level: The Communication Competency Assessment Instrument. Communication Education, 31, 19-32.
ISLS
156
ICLS2012
Memetic Processes as Conceptual Framework for Idea Improvement in Knowledge Building

Karsten Krauskopf, Johanna Bertram, Stefan Huber, Knowledge Media Research Center, Schleichstr. 6, 72076 Tbingen, Germany, k.krauskopf@iwm-kmrc.de, j.bertram@iwm-kmrc.de, s.huber@iwm-kmrc.de Ya Ping Hsiao, Open University of the Netherlands, Valkenburgerweg 177, 6419 AT, Heerlen , The Netherlands, amy.hsiao@ou.nlKatherine Panciera, University of Minnesota, 4-192A KHKH, 200 Union St SE, Minneapolis, MN 55455, USA, katpa@cs.umn.edu Nicole Strfling, University of Duisburg-Essen, Forsthausweg 2, 47057 Duisburg, Germany, nicole.straefling@uni-due.de Astrid Wichmann, Ruhr University Bochum, Universittsstrae 150, 44801 Bochum, Germany, astrid.wichmann@rub.de Jan van Aalst, Faculty of Education, University of Hong Kong, 323 Runme Shaw Building, Pokfulam Road, Hong Kong, China, vanaalst@hku.hk Abstract: Idea improvement is a key characteristic of knowledge building where ideas are conceptualized as improvable and epistemic objects to sustained inquiry. However, despite its importance in this theory, little research exists that focuses on the process of idea improvement. In this paper we examine the development of ideas within a community of learners by proposing a conceptual framework to understand ideas as memes and investigate the memetic processes affecting them. We apply this framework to students discourse through a three week unit from a Knowledge Forum data set by following the survival paths (fitness) of memes. We use a mixed methods approach to identify memes, define quantitative indicators to calculate the memes' fitness and elaborate on the improvement of the fittest memes. Benefits and open questions related to our suggested framework are discussed.
Introduction
Knowledge building focuses on the production and continual improvement of ideas of value to a community (Bereiter, 2002). The principle of idea improvement is a feature of knowledge building, where ideas are conceptualized as real, improvable, and epistemic objects; once an idea has been contributed to a shared knowledge space, students can ask about the assumptions underlying the idea, and modify it in various ways (Bereiter, 2002; van Aalst, 2006). One influential aspect in this process lies with the learners and their original ideas. Another influential aspect is independent from their creators: the expressed ideas can also be described as something with an out-in-the-world existence and a public life (Zhang, Scardamalia, Lamon, Messina, & Reeve, 2007). Once an idea has been made public it changes based on the input of the whole community. In short, the development of an idea becomes an evolutionary process with the learners as the main driving force of idea improvement (Hong & Sullivan, 2009). Research on knowledge building has recently been focussing on assessing aspects of idea improvement by observing students learning processes. Zhang and colleagues (2007), for instance, assess idea improvement by judging the scientific acceptability of an idea. The authors identify inquiry threads, which are sequences of notes that address the same problem or topic, and measure scientific levels of ideas. But assessing idea improvement in terms of scientificness does not allow judgement of relevance of ideas for the community. We propose a framework that allows us to focus on the relevance of an idea in a community of learners. This framework conceptualizes ideas as memes and then analyzes the development of these ideas through the learners discourse. The goal of this study is to explore idea development in analogy to the development of memes within the context of Knowledge Forum, an online learning environment (Scardamalia & Bereiter, 2003). Therefore, we examine the development of ideas within a community of learners through the lens of memes and memetic processes. We analyze students discourse by following the survival paths of memes through a three week unit in the Knowledge Forum data. Based on this examination, memes are identified in the students discourse and described by quantitative indicators for these memes fitness. Second, we qualitatively describe the developmental paths for the fittest memes. Finally, benefits, limitations, and open questions related to our suggested framework are discussed.
Ideas and Memetic Processes

Memes, as defined by Dawkins (1976), are units of cultural transmission and imitation that spread within a culture, as an equivalent of genes. Behavior, ideas, knowledge or fashion can be considered memes that propagate from person to person via imitation and variation (Dawkins, 1976). In some ways, the culture itself can be seen as a meme pool. Bereiter (2002) describes the cultural meme-pool as frequently changing over time, while new memes emerge, others mutate or their frequency changes. These processes can be seen as a cultural
ISLS
157
ICLS2012
evolution. This concept of the cultural meme-pool contributes to Scardamalia and Bereiters (2006) description of knowledge as an advancing concept with ideas emerging, others dying out, problems being solved and new problems coming up. Because knowledge can thus be compared to the cultural meme pool which underlies evolutionary processes, idea improvement can be analyzed by comparing it to memes. The quality of an idea can be defined by evolutionary indicators such as its likelihood to survive. The overall survival rate of a meme can be understood as its fitness (Heylighen, 1999). This notion includes, however, that idea survival determined for an expressed meme by its repetition and variation as in evolution, does not necessarily progress but adapts. In analogy, the fitness of a meme cannot be used as a sole measure of quality but is a complementary step for focused qualitative analyses. With regard to the memetic processes co-determining a memes fitness transmission, variation, and the resulting selection pressure are of relevance to our framework. Following the overview by Nye (2011) these can be defined as follows: The core information of a meme is its semantic information. When semantic information changes, the meme has mutated or a new meme has been created. A meme reproduces when semantic information is replicated from one agent to another. [...] Conversely, identical physical transmissions change semantic meaning based on context and interpretation. (Nye, 2011, p. 14). Thus, a prerequisite for considering semantic information to be a meme is its ability to reproduce recursively within the respective environment. First, this implies that a meme must be expressed in behavior, or in the case investigated here, in written language. The transmission of this expressed meme can be considered most important to its reproduction. As Nye (2011, p. 18) puts it: This definition is ontologically complete: semantic information is a meme within a society and environment if and only if it can recursively reproduce in that society and environment. Variation can be affected during expression and/or transmission by external factors, such as time pressure, or internal factors, such as limits to cognitive and motivational resources. Even though following a code, language, or procedure can reduce misunderstandings thanks to a given syntax, reducing the complexity in to a specific form of notation fosters ambiguous statements. This is why semantic variation appears likely to occur for symbolically expressed memes. Overall, this noise included in the transmission of memes and resulting variation puts the memes under a selection pressure and only certain memes survive. Therefore, analyzing the development of ideas, instead of the correctness of ideas, focuses more on the processes involved and less on the content. In sum, we assume that identifying memetic processes in knowledge building is a complementary approach that can help to tap more directly into the central process of idea improvement, i.e. how ideas spread and survive over time in a learners discourse. We expect to (1) quantitatively describe indicators for idea improvement that can be inferred from the notion of memes and memetic processes. Furthermore, we expect, (2) based on this, to be able to qualitatively describe how ideas develop over time and to more directly describe their improvement as a process.
Method
Data
We analyzed a subset of the data from a study conducted by Niu and van Aalst (2009). Two classes of a tenth grade social studies course participated in that study. Each class was divided into groups of eight persons. In a short inquiry unit (three weeks) the students investigated general environmental problems such as pine beetle infestation. For this investigation they used the asynchronous online discourse environment Knowledge Forum (Version 4.5, Scardamalia, 2003, see www.knowledgeforum.com). In Knowledge Forum, students can contribute their ideas to the database in the form of written notes which make up discussion threads. Other students who have access to the database can revise these notes, reply to them, or contribute their own reflection. For this study we used a data set of eight students (one female, seven male). The participants worked on the topic of how to free a forest from a pine beetle infestation. They worked irregularly on the problem over a period of eight consecutive days and five additional posts were added a month later. A total of 128 posts were analyzed, consisting of all the posts from this group.
Data Analysis
The analytic procedure consisted of three steps. First, to identify the memes emerging in the data set a coding scheme was inductively developed and all notes were coded accordingly. Individual notes served as the unit of analysis. Secondly, quantitative information was extracted from the data in order to calculate a fitness score for each meme monitoring when memes were expressed and reproduced. Finally, after calculating and plotting the fitness scores over time, we zoomed in qualitatively on the fittest memes and followed their path in order to describe the development of the complexity and quality of the meme, and therefore, understand its variation.
Qualitative Analysis - Identifying Memes

We based our analysis on the notion that a meme can be expressed in written language, e.g. the notes posted by students. To identify the memes emerging in the students discussion two independent raters (two of the
ISLS
158
ICLS2012
authors) performed a qualitative content analysis following Mayring (2000) to inductively develop categories that would capture the central memes emerging in the data set. These initial sets of categories were discussed by the two raters and converged. Then, a third independent rater (first author) double checked the categories. Disagreements were resolved by discussion resulting in the final 18 categories, example categories/memes are described in Table 1. Finally, all posted notes were coded by applying one or more of these categories. We differentiated between the repetition of the essential meme and variation, i.e. emergence of a different meme. Repetition (the same category applied) was coded when the analyzed note resembled the core idea of the previous note. Variation within a meme was also coded as repetition if the main message had not been changed. If two or more memes were contained in a note, multiple codes were applied. Coders followed the rule to apply the same code again if the content of the note resembled (copied) the same idea as the previous note. Variation (new category applied) was coded when different ideas emerged spontaneously or existing ones were integrated in a way that the original idea did not resemble the final idea. Coders followed the rule to apply a different code if individual ideas mutated (at the group level) or the new change to existing elements was introduced, also if the preceding idea was lost during the developmental process and the following idea did not resemble the preceding idea. Table 1: Coding scheme - examples of memes. Meme Forest ecosystem Pest control Predators Definition Impact of beetle infestation and counter measures on the forest ecosystem, question of balance between environmental costs and benefits Repetition of initial task or problem, the question of how to resolve the beetle infestation is discussed Beetles should be killed by predators, from within (parasites) or outside (woodpeckers) and discussion about these predators
Quantitative Analysis
After we had identified memes in the qualitative coding procedure described above, we extracted the respective notes in which each meme occurred. Because the notes were posted over a period of 8 days at varying points in time, we defined a fixed time interval for which we aggregated the quantitative indicators. We performed the analysis with MS Excel. Due to the fact that the results do not differ meaningfully, we chose a 15 minutes interval, mainly to represent our analyses in an economic way (see Figure 1). The results did not differ meaningfully between smaller time intervals, but contained more specific information than larger time intervals. However, we are aware that the issue of conceptualizing time in asynchronous communication is a complex issue, in part because the time scale is a different one for every participant (Suthers, Dwyer, Medina, & Vatrapu, 2010). Thus, the time intervals created here are defined by our analysis not by the original time line of the data and therefore not readily interpretable. Quantitative Indicators. As mentioned above, we adopted the formula proposed by Heylighen (1999) for the two external stages: expression and transmission (see Formula 1). For each of the two components a value can be computed and then combined into a fitness score for the respective meme a predefined time interval. The simplified formula for our combined fitness measure f has the following form f(m, t) = E(m, t) * T(m, t) (1)
The fitness f of a meme m for the expression stage in time interval t is denoted by E (m, t) (expression) and the fitness in the transmission stage by T (m, t) (transmission). E (m, t) describes expression, or how often a meme has been expressed by saving a new note or changes in a note that contain the respective meme m for time interval t. T (m, t) describes transmission, or how often a posted note containing the respective meme m has been read by others during time interval t. The values of f are not interpretable in an absolute sense but only relative to other memes. Both terms can be larger than 1 and if one the terms reaches 0, the meme has been eliminated. This happens when a meme is not replicated further or when posted messages containing a meme are not read by other students anymore. In our case, however, notes were not deleted and stayed present in the Knowledge Forum database to potentially be read. Therefore, in this study a meme can only be eliminated when T (m, t) equals 0. To get a global fitness measure f , we calculated the mean fitness of a meme (Formula 2). In this formula the fitness of all time intervals is summed up and divided by the number of intervals n:
g
fg(m) =
( E(m, t ) * T(m, t ))
i=1
i i
(2)
ISLS
159
ICLS2012
Results
Quantitative Describing the Paths of Memes
Figure 1 depicts the results for the analysis applying Formula 1 to three sample memes (see Table 1). The last time interval available was excluded from analysis, because reading activity had ceased at the end of the course. What is most prominent in the data is that most of the memes have low fitness values over all the students discourse and that there are three dominant memes, which also repeatedly show higher spikes than any other meme: Forest Ecosystem, Pest Control, and Predators. Towards the end of the discussion, however, after showing two peaks clearly higher than any other meme (time intervals 20 and 24), the forest ecosystem becomes visible as the fittest meme, which is also the only meme that is still alive at the very end. In contrast, in the beginning of the discussion other memes - concerned with describing pine beetles in general and as a threat - show higher fitness values (not included in Figure 1).
Figure 1. Fitness of sample memes within predefined 15 minute time intervals (see sect. quantitative analysis). The average fitness values derived from Formula 2 mirror this data pattern. Here, also forest ecosystem, and pest control show high mean fitness. Additionally, the Predator meme had a high mean fitness, however as shown in Figure 1 it did not survive until the end of the discussion. Overall, we see from the quantitative data that the different solutions to the infestation problem are the fittest memes but rather we find a struggle between the elaboration on the initial task (pest control) and a counter argument; the impact of any counter measure on the forest ecosystem.
Qualitative Interpretation How do the Fittest Memes Develop (Variation)

To qualitatively zoom in on the idea development in our data, we tried to describe the variation within a meme and the interactions between memes. The meme Pest Control was coded 25 times in the data. Different kinds of pest control mechanism were discussed, starting with a list of various pest control options, which were not discussed any further by the students. Some options from this list of measures were discussed and were judged to be no solution to the pine beetle problem, mainly again due to their impact on the forest as a whole (i.e. forest ecosystem). The most prominently discussed solution was the idea of having woodpeckers prey on the beetles (i.e. predators). Furthermore, it was mentioned that these measures can kill the beetles, but can not prevent further epidemics. In other words, it was discussed that long-term solutions must be sought; for the current pest control there will be no solution which not has an impact on other animals or environment - the forest ecosystem. The meme Forest Ecosystem was coded 32 times in the data and it clearly interacts with pest control. The students notes show that the students tried to optimize their ideas regarding the problem of the pine beetles. They not only tried to eliminate the beetles, but also considered the environmental and economic impact of the possible solutions. Every single idea regarding the elimination of the pine beetles was tested against the impact on the environment and most were also tested against the cost of the solution. Thus showing that the students wanted to improve their ideas. Overall, the qualitative results show that the development of the solution of the pine beetle problem was not a straightforward one. The idea for the solution improved from simple versions like kill beetles to sophisticated ones like looking for long-term solutions. Along this development it seems that in particular the initial task of pest control and the counter argument of the impact on the rest of the (forest) ecosystem contend with each other. This contest seems mutually beneficial to the improvement of the solution. As above mentioned, the various pest control ideas had been made public and were changed based on the input of the whole community in this case the aforementioned impact on the forest ecosystem.
ISLS
160
ICLS2012
Discussion
The goal of this paper was to contribute to the study of idea improvement, a core component of knowledge building. Assessing idea improvement is of complex nature. One aspect that has been neglected so far is to determine idea improvement by assessing the relevance of ideas in the community. The current results show that the way of data preparation and analysis can be a powerful tool to complement the summative appraisal of learners discourse. We revealed that the group had broadened their discussion and improved their ideas in the sense of understanding the complexity of the initial problem by relating it to adjacent issues (e.g. forest ecosystem), which here were the fittest in the end. In sum, we found that the few memes which survived the discourse were also more complex ones. So in this case, the fitness of the memes was related to their quality in terms of complexity. However, in respect to the potential of the described framework, there are also some limitations: first, our analyses zoomed in on just one aspect that could help assess idea improvement. In order to gain a complete understanding of what determines idea improvement, various facets need to be taken into account. Hence, future research should focus on combining several aspects (e.g. such as specificness, see Zhang at al., 2007) when assessing idea improvement. A second limitation lies in the reductionist approach taken: evolutionary processes only allow a very limited view on ideas; possibly it even neglects important aspects of ideas that are characteristic for idea development. We do not suggest that our framework alone is sufficient, but it can help to focus the questions for deeper analysis. Overall, we conclude that our framework provides a way to more directly research idea improvement as a process in which the community is a driving force.
References
Bereiter, C. (2002). Education and mind in the knowledge age. Mahwah, NJ: L. Erlbaum Associates. Dawkins, R. (1976). The selfish gene. Oxford: Oxford University Press. Hong, H.-Y., & Sullivan, F. R. (2009). Towards an idea-centered, principle-based design approach to support learning as knowledge creation. Educational Technology Research and Development, 57, 613-627. Heylighen, F. (1999). What makes a meme successful? Selection criteria for cultural evolution. Proceedings of the 15th International Congress on Cybernetics, Namur, p. 418- 423. Mayring, P. (2000). Qualitative content analysis. Forum: Qualitative Social Research, 1(2). Retrieved August 25, 2011, from http://www.qualitative-research.net/fqs-texte/2-00/02-00mayring-e.htm. Niu, H., & van Aalst, J. (2009). Participation in Knowledge-Building Discourse: An analysis of online discussions in mainstream and honors social studies courses. Canadian Journal of Learning and Technology 35(1), http://www.cjlt.ca/index.php/cjlt/article/viewArticle/515/245. Nye, B. D. (2011). Modeling Memes: A memetic view of affordance learning. Publicly accessible Penn Dissertations, paper 336. Retrieved August 25, from http://repository.upenn.edu/edissertations/336. Scardamalia, M. (2003). Knowledge building environment: Extending the limits of the possible in education and knowledge work. In. A. DiStefano, K. E. Rudestam, & R. Silverman (Eds.), Encyclopedia of distributed learning (pp. 269 - 272). Thousand Oaks, CA: Sage Publications. Scardamalia, M., & Bereiter, C. (2003). Knowledge building. In Encyclopedia of education, second edition. (pp. 1370-1373). New York: Macmillan Reference, USA Scardamalia, M., & Bereiter, C. (2006). Knowledge building: theory, pedagogy, and technology. In R. K. Sawyer (Eds.), Cambridge handbook of the learning sciences. (p. 97-115). New York: Cambridge University Press. Suthers, D. D., Dwyer, N., Medina, R., & Vatrapu, R. (2010). A framework for conceptualizing, representing, and analyzing distributed interaction. International Journal of Computer Supported Collaborative Learning, 5(1), 5-42. van Aalst, J. (2006). Rethinking the nature of online work in asynchronous learning networks. British Journal of Educational Technology, 37, 279-288. Zhang, J., Scardamalia, M., Lamon, M., Messina, R., & Reeve, R. (2007). Socio-cognitive dynamics of knowledge building in the work of 9- and 10-year-olds. Educational Technology Research & Development, 55, 117-145.
Acknowledgments
This study resulted from an exploration the authors began during the Summer School Making Sense of Social Media, co-sponsored by the Knowledge Media Research Center and Tuebingen ScienceCampus, August 1-4, 2011. The data were originally collected as part of a New Economy Collaborative Research grant from the Social Sciences and Humanities Research Council of Canada to Marlene Scardamalia (OISE/University of Toronto). The authors thank Hui Niu (Simon Fraser University) for permission to reanalyze the Knowledge Forum database.
ISLS
161
ICLS2012
Unraveling Idea Development in Discourse Trajectories

Iassen Halatchliyski1, Aileen Oeberst1, Martina Bientzle1, Franziska Bokhorst1, Jan van Aalst2 1 Knowledge Media Research Center, Schleichstr. 6, 72076 Tbingen, Germany 2 Faculty of Education, University of Hong Kong, 323 R S Building, Pokfulam Road, Hong Kong, China Emails: {i.halatchliyski, a.oeberst, m.bientzle, f.bokhorst} @iwm-kmrc.de, vanaalst@hkucc.hku.hk, Abstract: With the present paper we want to shed light onto an issue that is central within the knowledge building theory but only little studied the development of ideas in collaborative learning discourse. Starting from the construction of a network of explicit and implicit relations between ideas, we apply a scientometric method to tackle the temporality of collaborative processes based on the structure of successive ideas. The resulting discourse trajectories are shown to give a holistic and also a detailed view on how knowledge advances when their interpretation is combined with a qualitative analysis of the content of the ideas and their relations. The weighted relevance of relations between ideas enables the identification of sub-topics in the discourse, important ideas, and influence or uptake events.
Introduction
How is knowledge about the world created and advanced? Knowledge building is an approach from the learning sciences that builds on contemporary philosophical views and research on expertise (Bereiter, 2002; Scardamalia & Bereiter, 2006) to engage students in modern knowledge work, including ability to collaborate, deal with novelty, and solve ill-structured problems. At the heart of knowledge building is a computer-mediated collaborative discourse that is oriented toward idea improvement. Following Poppers (1972) theory of objective knowledge, knowledge-building theory considers ideas as real objects that can be critiqued, tested and modified, much like how real objects like bicycles undergo these processes. Hence, the process of idea development is fundamental for understanding knowledge building. However, despite this acknowledged role, there is a dearth of analytical approaches for investigating the dynamic development of ideas in knowledge-building discourse. Therefore, the main goal of this paper is to present a methodological technique from sceintometrics for studying what we call a discourse trajectory, i.e. the genuine process characteristics of a discourse based on idea development over time.
Related Research
Most studies of knowledge building have followed a content analysis approach to studying a discourse process (Gunawardena, Lowe, & Anderson, 1997; Henri, 1992), where qualitative data is segmented into idea units and these are coded for their cognitive, social and other aspects. The frequency of the assigned codes is then statistically regarded for comparisons of different students, discussion groups or phases. Students contributions can also be categorized as different discourse activities, on the basis of which a sequential analysis (Jeong, 2005) of their temporal ordering can reveal patterns and facilitate deeper insight into the collaborative discourse as a process. However, as is becoming increasingly clear in CSCL research, the coding and counting technique neglects some important discourse qualities as it takes statements out of their context and generally addresses actions of individuals instead of the group as a whole (Strijbos & Stahl, 2007). Stahl, Koschmann and Suthers (2006) defined collaborative learning as an interactive process of shared meaning-making in a group and pleaded for exploratory and interpretative conversation analysis of case study narratives. Henri (1992) addressed interactivity in a discourse process distinguishing independent from implicit interactive and explicit interactive contributions depending on if and how they refer to other contributions in the discourse. Later, Gunawardena et al. (1997) extended this view noting that in a knowledge constructing discourse all contributions can be linked to one or more other contributions and to the discussed topic. This marks a change from understanding interactivity as reference between contributions to treating it as diffusion of knowledge and other more general forms of uptake or influence. Suthers, Dwyer, Medina and Vatrapu (2010) introduced and employed contingency graphs for uptake analysis with the goal to identify and map also very subtle kinds of contribution uptakes that often remain unnoted. As noted by Lipponen, Rahikainen, Lallimo and Hakkarainen (2003) a discussion thread structure conceals the majority of semantic and conceptual relations between contributions. Another approach to studying interactivity in discourse processes presents the social network analysis (SNA), which is a well established methodology in the CSCL research field (e.g. de Laat, Lally, Lipponen, & Simons, 2007; Reffay, & Chanier, 2002). It has been used for studying relations between persons embedded in a network determining cohesiveness of the learning group and students relative positions. Following the reasoning of Stahl (2006) on intersubjective meaning-making in CSCL in terms of networks of references the discourse process can also be approached from the perspective of a network of collaboratively created artifacts.
ISLS
162
ICLS2012
There are accordingly a few notable examples of studies of automatically detected semantic networks of related contributions (Sha, Teplovs, & van Aalst, 2010). Because network analysis is based on indicators of relations that lack deeper meaning in the discourse, it should be applied in combination with other in-depth and content related methods. In sum, an appropriate methodology for studying discourse as a process has to be multi-faceted and address: the temporal dimension of development; the interactivity between contributions and between participants; the content of the discourse (Arvaja et al., 2007).
Main Path Analysis of Discourse Trajectory

The main goal of our study is to provide a primer for a novel methodology that tackles the trajectory of a discourse process analyzing the collaborative development of ideas over time. The approach presented here is rather simple and grounds on a network analysis of interrelated ideas. Based on a set of identified relations a main path analysis algorithm (Hummon & Doreian, 1989) assigns different weights to the relations according to their position in the network. This weighted network can be interpreted in order to identify the most influential ideas, idea paths, i.e. successions of related contributions, discourse trajectory, i.e. the overall structure of the paths over time, etc. The procedure of main path analysis stems from the scientometric research tradition that deals with citations between scholarly publications (e.g. Carley, Hummon, & Harty, 1993). In the present application of the analysis again relations between authored content are of interest. To the best of our knowledge, the present work is a pioneering attempt of applying scientometric methodology in the field of CSCL, and knowledge building in particular. Main path analysis calculates a weight for each relation in a citation network according to the number of times the relation is used while tracing all chronological connections between all possible pairs of contributions (De Nooy, Mrvar, & Batagelj, 2005). So, a relation on idea paths is more important, the more often it serves as a link between preceding and subsequent ideas. Contributions that establishing such important relations often synthesize old ideas, add new knowledge and represent a basis for developing new ideas. A main path in the discourse trajectory is produced by the links with the highest weights. De Nooy et al. (2005) reason that networks of scientific citations represent systems of knowledge flows. This fits our goal to analyze development of ideas in a knowledge building discourse. The resulting discourse trajectory is comparable to the development of a scientific field and can be analyzed for instances of integration, fragmentation, specialization or paradigmatic changes (De Nooy et al., 2005). Moreover, we see very promising possibilities for combining it with in-depth and content analysis methods.
Method
Data Set
We reanalyzed Knowledge Forum data from a study of two Grade 10 classes one regular and the other honors who investigated aspects of environmental issues; see Niu & van Aalst (2009) for details. The original study showed that both high and low-achieving students can sustain a knowledge building discourse. The students worked on the Knowledge Forum software in groups of approximately eight over a period of three weeks. For the present study we compared the discourses of one group from each class. Both groups separately discussed ideas for handling the problems of the nuclear accident at Chernobyl. The honors group wrote 60 notes and the regular group 164 notes on this issue. We analyzed the log data and the content of the computer notes; the log files contain writing (saving a note) and reading (opening a note) events, which were recorded with a timestamp while using Knowledge Forum.
Analysis
As we were interested in collaborative learning at the level of idea development in discourse it was important to take all ideas and all paths of influence over time into account. We adopted Lipponens (2000, p.185) definition that an idea is a set of propositions that formed a coherent unit of meaning and determined that a single note represents a suitable unit of analysis for our data (cf. Rourke, Anderson, Garrison, & Archer, 2001). Even though a note may present new information taken from an external source, almost always it is also connected to some previously stated ideas, as put forward by Gunawardena et al. (1997). Direct links between notes in a thread capture only a small part of all existing relations between notes, however. Thus, in order to capture relations between ideas, we made use of both explicit links and implicit connections by analyzing data from two different sources, the log files of the software as well as the content of the contributed notes. Based on the timestamps two independent raters considered all the notes each student had read before writing a new note. The raters then compared the content of each note the student had read and the content of her own new note. Any disagreements between the raters about the evidence of connection between notes were discussed and solved unanimously. Implicit relations in the discussions of the Chernobyl issue were found for example between notes that addressed the same point: the fault of the designer of the reactor vs. that of the
ISLS
163
ICLS2012
technical operator; the radiation outburst; the increase in the incidence of cancer; the old covering of the remains of the reactor; financial support for the Ukraine; and many more other points. The complete qualitative analysis of the content relations between the notes is still in progress. The discourse trajectory results reported here are only based on the detection of connections. We then applied the main path analysis procedure (Hummon & Doreian, 1989; Carley, Hummon, & Harty, 1993) to the resulting networks of both group discourses and obtained weights for each link based on its relative importance for all paths of idea development. This was done with a standard procedure for main path analysis in the Pajek software (Batagelj & Mrvar, 1998) for network analysis.
Results
Figures 1 and 2 depict the results of the main path analysis of both group discourses. The vertically layered view illustrates idea development during the different days of activity in class and at home. The numbered points represent different notes. Arrows represent some of the relatively important relations between notes with thicker arrows denoting more important relations, i.e. higher weights calculated with the main path analysis. The arrows are directed from an older to a newer note. It is important to mind that the isolated notes positioned on the left and on the right of the figures were also found by the raters to be related to other notes, but these relations received very low weights in the main path analysis. In order to identify the main idea paths in the discourse trajectories more clearly, Figure 1 and 2 show only relations over the arbitrary threshold weight of 0.05 for normalized weights between 0 and 1. Both figures display single connected main idea paths, because there were no disparate discussions around different topics in the groups. However, both figures show different discourse trajectories at a first glance that remain to be characterized with the help of the contributed content. Honors Group (Figure 1) The notes from the first day on (top of the figure) generate concurrent idea paths interrelating and stimulating one another; addressing background information on causes and on effects of the accident. On April 4th the development of these ideas seems to get focused into one path dealing with technological issues based on the causes of the accident. After April 4th the idea paths in the honors group separate resulting in four largely independent lines of inquiry that are pursued until the last days of the course. The thickest path deals with futuristic solutions of neutralizing radiation and inspires the largest amount of participation. The remaining three paths discuss the covering of the destroyed reactor, the use of nuclear energy in general and the politics behind the accident.
April 1 | --April 2 -home --April 3 -home | --April 4 | --April 4 -home | --April 5 --April 6 --| April 6 -home | --| April 7 --April 11 -home --| April 15 --April 18 -home
Figure 1. Discourse trajectory of the honors group: notes and main idea relations
ISLS
164
ICLS2012
Regular Group (Figure 2) All the initial idea paths merge into a single note, 316, on April 1st that brings up the need for solutions after the basic facts are known. Further tracking of the discourse trajectory suggests that this pattern repeats within the regular group. Concurrent idea developments are short-lived and continuously meld into a single note on the same or the next day. On April 4th note 488 proposes a solution for the polluted soil around the old reactor, i.e. digging it and disposing of it in outer space. Then, again, this central idea gives rise to various related idea paths concerning difficulties of the solution most importantly the money problem. Additional solutions and refinements emerge on April 5th and are then put to a vote by note 686. Note 843 bundles the focus to the solution of building a new covering for the old reactor. The game of proposing solutions and recognizing difficulties continues yielding tightly interrelated idea paths.
March 31 | | | --April 1 | --| April 4 | --| April 5 | --| April 6 | | --| April 7 --April 13 -home --| April 14 | | --April 15
Figure 2. Discourse trajectory of the regular group: notes and main idea relations
Conclusion
The short comments on the results illustrate some initial ways of interpreting discourse trajectories obtained through main path analysis. With the present paper we pursued the goal to open up a field of possibilities for studying collaborative learning processes as we introduced a new method to the field. We showed that it handles the temporal perspective of idea development very well providing an objective measure of the relevance of ideas and their relations. These can then be examined more closely with regard to their contents in a mixed methods approach (see also Carley et al., 1993). The obtained discourse trajectory also provides a holistic view on the collaborative process. Although both analyzed groups were successful in knowledge building, they showed very different styles of collaboration regarding the convergence-divergence polarity (Halatchliyski, Kimmerle, & Cress, 2011). The regular group produced a large number of notes, and their main ideas were very tightly interrelated; convergence was maintained over the whole discourse trajectory. The honors group achieved a selforganized division of labor by building up a common understanding of the problems and then following divergent solution paths. It remains to be shown to what extend this is due to individual differences. Our next goal is to complete the qualitative analysis of the data set in order to determine what kind of contributions and relations receive higher or lower weights, i.e. are more or less important in the discourse.
References
Arvaja, M., Salovaara, H., Hkkinen, P. & Jrvela, S. (2007). Combining individual and group-level perspectives for studying collaborative knowledge construction in context. Learning & Instruction, 17,
ISLS
165
ICLS2012
448-459. Batagelj, V., & Mrvar, A. (1998). Pajek Program for Large Network Analysis. Connections, 21, 47-57. Bereiter, C. (2002). Education and mind in the knowledge age. Mahwah, NJ: Lawrence Erlbaum Associates. Carley, K. M., Hummon, N., & Harty, M. (1993). Scientific Influence: An Analysis of the Main Path Structure in the Journal of Conflict Resolution. Knowledge: Creation, Diffusion, Utilization, 14, 417-447. de Laat, M., Lally, V., Lipponen, L., & Simons, R.-J. (2007). Investigating patterns of interaction in networked learning and computer-supported collaborative learning: A role for Social Network Analysis. International Journal of Computer-Supported Collaborative Learning, 2, 87-103. de Nooy, W., Mrvar, A., & Batagelj, V. (2005). Exploratory Social Network Analysis with Pajek, Cambridge University Press, 2005. Gunawardena, C. N., Lowe, C. A., & Anderson, T. (1997). Analysis of A Global Online Debate and The Development of an Interaction Analysis Model for Examining Social Construction of Knowledge in Computer Conferencing. Journal of Educational Computing Research, 17, 397-431. Halatchliyski, I., Kimmerle, J., & Cress, U. (2011). Divergent and convergent knowledge processes on Wikipedia. In H. Spada, G. Stahl, N. Miyake, & N. Law (Eds.), Connecting Computer-Supported Collaborative Learning to Policy and Practice: CSCL2011 Conference Proceedings (Vol. II, pp. 566570). Hong Kong: International Society of the Learning Sciences. Henri, F. (1992). Computer Conferencing and Content Analysis. In: A. Kaye (Ed.), Collaborative Learning through Computer Conferencing: The Najaden Papers (pp. 117-136). Berlin: Springer-Verlag. Hummon, N. P., & Doreian, P. (1989). Connectivity in a Citation Network: The Development of DNA Theory. Social Networks, 11, 39-63. Jeong, A. (2005). A guide to analyzing message-response sequences and group interaction patterns in computermediated communication. Distance Education, 26, 367-383. Lipponen, L. (2000). Towards knowledge building: from facts to explanations in primary students computer mediated discourse. Learning Environments Research, 3, 179-199. Lipponen, L., Rahikainen, M., Lallimo, J., & Hakkarainen, K. (2003). Patterns of participation and discourse in elementary students computer-supported collaborative learning. Learning and Instruction, 13, 487-509. Niu, H., & van Aalst, J. (2009). Participation in knowledge-building discourse: An analysis of online discussions in mainstream and honours social studies courses. Canadian Journal of Learning and Technology 35(1). http://www.cjlt.ca/index.php/cjlt/article (Retrieved November 5, 2011). Popper, K. R. (1972). Objective knowledge: An evolutionary approach. Oxford, UK: Clarendon Press. Reffay, C. & Chanier, T. (2002) Social Network Analysis used for modelling collaboration in distance learning groups, In S. A. Cerri, G. Gouardres, and F. Paraguau (Eds.): ITS 2002 Lecture Notes in Computer Science, 2363, 3140. Rourke, L., Anderson, T., Garrison, D. R., & Archer, W. (2000) Methodological issues in the content analysis of computer conference transcripts. International Journal of Artificial Intelligence in Education, 11, 8-22. Scardamalia, M., & Bereiter, C. (2006). Knowledge building: Theory, pedagogy, and technology. In R. K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (pp. 97-115). New York, NY: Cambridge University Press. Sha, L., Teplovs, C., & van Aalst, (2010). A visualization of group cognition: Semantic network analysis of a CSCL community. In K. Gomez, L. Lyons, and J. Radinsky (Eds.) Proceedings of the 9th International Conference of the Learning Sciences - Volume 1 (ICLS '10) (pp. 929-936). Stahl, G. (2006). Group cognition: Computer support for building collaborative knowledge. Cambridge, MA: MIT Press. Stahl, G., Koschmann, T., & Suthers, D. (2006). Computer-supported collaborative learning: An historical perspective. In R. K. Sawyer (Ed.), Cambridge handbook of the learning sciences. Cambridge, UK: Cambridge University Press. Strijbos, J.-W., & Stahl, G. (2007). Methodological issues in developing a multi-dimensional coding procedure for small group chat communication. Learning & Instruction. Special issue on measurement challenges in collaborative learning research, 17, 394-404. Suthers, D. D., Dwyer, N., Medina, R., & Vatrapu, R. (2010). A framework for conceptualizing, representing, and analyzing distributed interaction. International Journal of Computer-Supported Collaborative Learning, 5(1), 5-42.
Acknowledgments
This study resulted from an exploration the authors began during the Summer School Making Sense of Social Media, co-sponsored by the Knowledge Media Research Center and Tuebingen ScienceCampus, August 1-4, 2011. The data were originally collected as part of a New Economy Collaborative Research grant from the Social Sciences and Humanities Research Council of Canada to Marlene Scardamalia (OISE/University of Toronto). The authors thank Hui Niu (Simon Fraser University) for permission to reanalyze the database.
ISLS
166
ICLS2012
Investigating Emergent Dynamics to Understand Interactions in Small Professional Development Groups

Susan A. Yoon, University of Pennsylvania, Philadelphia, PA, yoonsa@gse.upenn.edu Abstract: In this study a dynamic systems approach and analyses is used to identify dynamics that constrained the collaboration of a group of teachers to complete a reform-oriented curriculum task in a professional development setting. A central goal was to identify challenges to achieving emic or self-organized structure necessary for sustaining innovative educational programs. The study found that a local social and emotional dynamic influenced the emergence of a strong global dynamic of social support that ultimately produced the contextual outcome of low productivity. The paper asserts that teachers need intentional training to gain skills for working autonomously. Helping teachers become aware of the dynamics needed for successful collaboration would assist in achieving self-organized participation. An important contribution of this study is the application of a dynamic systems lens, which enabled the location of levels of micro and macroscopic influences that can be used to inform future professional development activities.
Introduction
The research reported here follows on a series of studies that have investigated the complex or non-linear nature of teacher interactions in professional development activities (Yoon et al., 2010). Previous research has revealed differences in the way that teacher groups display more or less adaptive collaborative activity (e.g., Horn & Little, 2010) and how collective approaches in professional development influence teacher growth and student learning (Desimone, 2009). Some research has suggested that differences between how teacher groups function may be attributed to several variables which include a lack of understanding of how to self-organize effectively as a team (Main, in press). However, the general consensus in the literature about teacher collaborative groups is that we still know every little about how they operate and the interactional processes that lead to problem solving and decision making (Havnes, 2009; Meirink et al., 2007; Scribner et al., 2007). The often non-linear nature of team formation and development (Ito & Brotheridge, 2008) coupled with the need to promote collectivity suggests that investigating the interaction of teacher groups requires methods that can accommodate these characteristics and capture critical impacts on collaborative activities as they emerge. To date there appear to be few methodological tools that can do this systematically. Thus, in this small group case study, I use a dynamic systems approach as a methodological tool to investigate the interactions of six teachers as they collaborated to construct reform-oriented curricula. I wanted to examine the utility of this systematic process in revealing the complex nature of interactions that enabled or constrained self-organization. I was also interested in understanding the developmental dynamics that shaped the group's ability to accomplish a systemic reform task.
A Dynamic Systems Approach to Understanding Group Activity

Dynamic systems are complex organizations of interacting parts that work together to give rise to patterns of behavior over time (Churchill, 2007). Through these lower level interactions that are initially unstable, more stable patterns emerge and solidify into attractor states that the entire system tends toward (Granic et al., 2007). For example, Martin et al. (2005) have studied how children's play partners, while initially variable, organize into coherent friendship clusters due to individual differences in personalities and other behavioral characteristics. While perturbations can shift the configuration of the system temporarily (e.g., a new person monopolizes a friends attention), depending on the strength of the attractor (e.g., friendship bond), the system will form very stable behavioral states (e.g., friendship resumes). How attractor states form and settle over time are known as self-organization and emergence in dynamic systems and can be tracked over time to explain the evolution of system trajectories (Lewis, 2000; Steenbeek & van Geert, 2007; Thelen & Smith, 1994). The self-organization or emic nature of dynamic systems aligns well with the local or peer-to-peer goal for our professional development activities. Methods for which to understand these dynamics also connect well with this studys focal interest in how teacher interactions can shape outcomes. For example, Farmer et al. (2007) and Lichtwarck-Aschoff et al. (2008) identify developmental factors that work together as a system of correlated constraints. They discuss how system components mutually influence each other and become aggregated in macroscopic patterns to constrain or control the entire system activity. A coherent model that categorizes such local and global dynamics with contextual outcomes can be found in Arrow et al. (2000) in which small groups are viewed as dynamic systems. The case study in the present research uses this model and defines local dynamics as rules of activity for parts of the system, i.e., the teachers, and global dynamics as rules
ISLS
167
ICLS2012
of activity for system-level properties that emerge out of local dynamics. These are the patterns of behavior or system attractors that stabilize in the professional development group and influence contextual outcomes.
Methods
Participants and Context
The study investigates the interactions among a group of six teachers who worked together for 10 months. The group was comprised of three males and three females from six different high schools in a large urban school district. Teaching experiences ranged from 526 years. Teachers taught across different science disciplines and all high-school grades. More details about selected participants follow. Teachers participated in a ten-day summer PD workshop in August of 2010 (50 hours), and then in monthly meetings from September 2010 May 2011 (35 hours). Their task was to create a publishable high school science curriculum book using units they already constructed for a larger project on 21st century problem-based learning and digital participation. Teachers were selected due to their demonstrated commitment in the larger project, their pedagogical skills in delivering their curricular units, their perceived abilities for leadership, and their perceived ability to collaborate. Teachers were told that the project was ultimately aimed at producing curriculum that was vetted and constructed by teachers for teachers for implementation in real-world urban classrooms thereby increasing the potential that new teachers would be able to use the curriculum successfully. The goals of teacher ownership, collaboration and decision making were greatly emphasized and teachers were given the message that selforganization was a major expectation, i.e., that there would be little input from researchers as to how the book would be constructed.
Data Sources, Data Coding and Analyses

Data sources and analyses included: 1) Initial surveys of participant demographics that collected experiences and goals for participation; 2) Six individual interviews at the end of the summer workshop 2010; three individual interviews at the end of May 2011. Interview questions asked how participants felt about the collaborative effort, what their role in the book project was, and reasons for why particular project and group outcomes emerged; 3) Sixty hours of recordings from both the summer workshop and monthly sessions documenting group activities and dynamics; and 4) Five surveys of collaboration rankings during the summer workshop in which participants were asked to rank other members of the group from 1-5 in terms of who was the most central person in the collaboration task, a rank of 1 being most central. Initial surveys were used to establish an understanding of each participant's unique set of qualities and experiences that potentially influenced local dynamics such as having less teaching experience than others in the group. Interviews and recordings of group meetings were qualitatively mined by the researcher and two doctoral research assistants to look for local and global dynamics and contextual outcomes as they emerged over time. To validate the findings, the global dynamics and contextual outcomes were reviewed with participants in the May 2011 interviews. For collaboration rankings, an in-degree score for each teacher was calculated from the average of teachers' collaboration rankings for each time sample. In-degree scores can be used to represent an actors prestige or status in a system (Wasserman & Faust, 1994). In this study, the in-degree score was used to determine who might have influence on group dynamics at particular points of the study. The lower the average number a teacher received, the higher the degree of centrality. A sample of the findings is presented below.
Results
Local dynamics (rules of activity for the teachers): Table 1 shows the in-degree scores of the participants in the study. Participant trajectories illustrate who had influence in the system at specific points. Don ranked near the top or at the top during the first week of the workshop while Isabels ranking was at the middle or below. During the second week, Isabel and Dons rankings switch on Day 7 where Isabel had the highest in-degree score. At the end on Day 9, Isabel returned to her normal middle spot while Dons position fell to nearly the bottom.
Table 1: Teacher's in-degree scores based on a ranking of like-mindedness Ranking In-degree Score In-degree Score In-degree Score In-degree Score In-degree Score Day 1 Day 3 Day 5 Day 7 Day 9 1 2 3 4 5 6 Carol (1.8) Don (2.0) Isabel (3.0) Stan (4.3) Shelley (4.4) Bill (4.5) Don (2.6) Carol (3) Bill (3.3) Isabel (4.0) Stan (4.0) Shelley (5.0) Don (3.0) Bill (3.0) Carol (3.4) Isabel (3.6) Stan (4.2) Shelley (4.8) Isabel (1.8) Carol (2.3) Don (3) Bill (3.3) Shelley (5.6) Stan (5.6) Bill (1.8) Carol (2.2) Isabel (2.4) Stan (4.2) Don (4.3) Shelley (5.5)
ISLS
168
ICLS2012
A review of their individual qualities, experiences and goals for participation helps to make sense of these trajectories over time as detailed in the next section. Don was the oldest teacher in the group who came to teaching after a series of jobs in the corporate world because he wanted to make a difference in society. His goals for participation were to help other teachers feel comfortable with new pedagogical approaches. At the time of the summer workshop, he was working on a Masters degree in family therapy and intended to make another career change from teaching. In the workshop interview, he described his contribution as keeping things light and social to ensure that the group had fun. He often discussed the difficult bureaucratic and social issues that impacted students and teachers lives in the district. In contrast, Isabel was the youngest teacher in the group with five years of experience. Her goals for participation included an interest in contributing to science education and to impact the practice of other teachers by constructing good curricula. She expressed an interest in eventually doing a doctorate in education. She wanted to participate in the summer workshop to collaborate with different teachers. In the May 2011 interview, she said she felt that she had many good ideas to share but didn't feel that she had the authority to push them in the group.
Global dynamics (rules of activity for the group): To investigate the global dynamics that emerged at
the system level, workshop recordings were mined to identify patterns in group interactions that may have been influenced by the above local teacher dynamics. Since the task required them to make decisions about how the curricular book would be constructed the analysis concentrated on instances in the discourse where the group had to make a decision. From this analysis, three global dynamics emerged, i.e., social support, anti-work; and dont rock the boat. Evidence of the first dynamic is presented below.
The social support dynamic: This dynamic was about group members gaining moral support from each other and sharing experiences about the challenges they faced as teachers working in a dysfunctional urban school district. Dons personal dynamics and goals being aligned with social and emotional support, continually influenced others in the group to participate in social sharing rather than completing the curriculum construction. Don also had a strong and forceful personality, which he immediately exerted. These dynamics can be observed in the following excerpt of Day 1 discourse. Before this excerpt, the group was asked to begin making decisions about what activities they wanted to include in the curriculum. Bill started on a line of discussion about getting parents on board. He talked about how some of his students went home after an impactful demo and told their parents.
Excerpt 1: August 2, 2010, Day 1 hour 1: 1:10:031 1. Bill: So the parents [were interested] 2. Don: [That is a great demo] 3. Bill: [Right. Parents were] 4. Don: [that really is.] 5. Bill: Parents were interested in what was going on so that told me right then 6. that the kids went home and told their [parents] 7. Don: [EXACTLY.] 8. Bill: well were going to be doing this. It was something different and 9. And when my parents came in on parents night, Id say, you know well, 10. if your children dont know exactly what field they want to go into, 11. this maybe a field that your children could go into 12. so by putting in a hook with your parent that made them interested which made 13. the children a little more interested [in whats going on] 14. Don: [So if we are going] to do that 15. at least now we cant speak for you because you are outside the school district, 15. 16. Carol: [Right] 17. Don: thinking about us, we have the first week, Tuesday and Wednesday 18. ((knocking the table)) were off for two days, then we have four days and that, 19. and that Thursday night, thats back to school night. So if were going to do it 20. Bill: I mean thats a, [I mean] 21. Don: [I mean] its a good idea 22. Bill [Right and] 23. Don: [but we gotta] do it early. 24. Bill: Right wed have to, wed have to do [something like that early.] 25. Don: [Because we literally have] 26. 6 days until back to school night.
ISLS
169
ICLS2012
Then Stan explained that he continued to have trouble getting his students to sign study consent forms. From there he talked about how his teaching roster had also changed mid-semester which prompted Shelley to share her own issues with the teaching roster. By this time, the conversation had spiraled into one that moved away from decision making about what activities to include in the curriculum. From the discourse excerpt, the tone of Dons comments were cautionary due to the fact that teachers needed to stay in line with the districts often irrational decisions (such as the timing of back to school night). This opened the floor for Stan and Shelley to discuss their professional issues to garner social support from the group. This global dynamic of social support often dominated the discourse throughout the workshop and in combination with other emergent global dynamics, constrained the groups ability to complete the curriculum construction. This strong attractor was initiated and sustained by Dons local dynamics as evidenced from the discourse and his high degree of centrality in the first week. Isabel, whose local dynamics represented contribution and collaboration, rarely participated in social sharing. In the discussions she often attempted to redirect the group toward focusing on the curriculum construction task. Despite her efforts, several decision-making episodes Isabel initiated were left unresolved. When asked to reflect on the kinds of group dynamics that emerged over the 10 month collaboration during the May 2011 interview, she said the group socialized a lot and hypothesized that this dynamic in part emerged due to the research facilitators inability to manage Dons dynamics. Its important to note from Table 1 that the day Isabel ranked #1 in the in-degree scores of the summer workshop (Day 7), the project PI spent a great deal of time in the morning discussing with the group, the limited time they had to finish and urged them to work more efficiently and to make greater strides in the remaining part of the week. However, due to the stability of the strong social support attractor, Isabels position fell again two days later (Day 9). These dynamics ultimately produced two contextual outcomes, i.e., a lack of self-organization and incomplete curriculum, which caused the project to fall short of its goals.
Discussion and Implications

Similar to previous studies (e.g., Havnes, 2009), this research investigated conversational processes that emerge in groups to understand the dynamics that impact collaboration (Scribner et al., 2007) and productivity. Using Arrow et als (2000) framework the findings demonstrate how Dons local dynamics influenced the emergence of a strong global dynamic of social support that ultimately produced the contextual outcomes of the lack of self-organization and incomplete curriculum construction. By following the dynamics that emerged over time and by using in-degree rankings this method located potential source(s) of influence, evaluated the comparative strength of the influence and hypothesized some reasons why the influence persisted. Between the two focal teachers in this paper, Isabel who represented the dynamics of contribution and collaboration clearly had the disposition and goals that were ideal for participation. However, her local dynamics did not influence the group where Dons appeared to dominate. Differences between the two teachers in terms of their experience and personalities may have contributed to their abilities to influence the group. Another important contributor as revealed in Isabels interview was the fact that teachers appeared to rely on the researchers to exert control, despite the strong emphasis on leadership, collaboration, teachers as decision makers, and ownership. But what is the reason for why this group operated in the manner that it did and how can knowing about this help to achieve self-organization in teacher groups? The interactional patterns shown in the groups discourse can be explained through a dynamic systems lens. Initially, varying unstable local dynamics interacted and the confluence of variables gave rise to patterns of behavior over time (Churchill, 2007). Dons passionate personality coupled with the expectation of external control solidified into attractor states (e.g., the social support dynamic) that the entire system tended toward (Granic et al., 2007). Although perturbations (e.g., the PI urging completion) had a temporary effect, the strength of the social support attractor continued to win out. From this assessment, it seems that the perturbation needs to be stronger than the stable attractors the system settles into. Other implications pertain to creating professional development structures that simultaneously address the local dynamics and needs of teachers and goals of the project. We saw that teachers wanted social and emotional support of the collaborative group, but this function did not fulfill the curriculum construction activity. A collaborative teacher group should succeed on both professional and personal levels to ensure positive membership and productivity. Another implication of the findings is to provide opportunities for teachers to become skilled collaborators. Teachers need intentional training and modeling on collaboration to gain self-organization skills. This point supports assertions made by Main (in press) in that teachers may lack understanding of how to work effectively as a team. Helping teachers become aware of the dynamics needed for successful collaboration, as Scribner et al. (2007) suggests, would assist in achieving the goal of self-organized participation. In order to identify these dynamics, an important contribution of this study is the application of a dynamic systems approach and analysis that revealed local influences on global dynamics that can ultimately inform future professional development activities.
ISLS
170
ICLS2012
Endnotes
(1) Transcript convention used in excerpts follow the Jefferson Transcription System [ ]: start and end of overlapping speech ((laugh)): gestures or comments Underlining: emphasis in speech (.): hearable micro pause (2): seconds of pause in speech CAPS: rise in volume
References
Arrow, H., McGrath, J., & Berdahl, J. (2000). Small groups as complex systems. Thousand Oaks, CA: Sage Publications. Desimone, L. (2009). Improving impact studies of teachers professional development: Toward better conceptualizations and measures. Educational Researcher, 38(3), 181-199. doi: 10.3102/0013189X08331140 Farmer, T., Farmer, E., Estell, D., & Hutchins, B. (2007). The developmental dynamics of aggression and the prevention of school violence. Journal of Emotional and Behavioral Disorders, 15(4), 197-208. doi:10.1177/10634266070150040201 Granic, I., OHara, A., Pepler, D. & Lewis, M. (2007). A dynamic systems analysis of parent-child changes associated with successful real-world interventions for aggressive children. Journal of Abnormal Child Psychology, 35, 845-857. doi: 10.1007/s10802-007-9133-4 Havnes, A. (2009). Talk, planning and decision-making in interdisciplinary teacher teams; a case study. Teachers and Teaching: theory and practice, 15(1), 155-176. doi: 10.1080/13540600802661360. Horn, I.S., & Little, J.W. (2010). Attending to problems of practice: Routines and resources for professional learning in teachers workplace interactions. American Educational Research Journal, 47(1), 181-217. doi:10.3102/0002831209345158 Ito, J.K., & Brotheridge, C.M. (2008). Do teams grow up one stage at a time? Exploring the complexity of group development models. Team Performance Management, 14(5/6), 214-232. doi:10.1108/13527590810898491 Lewis, M. (2000). The promise of dynamic systems approaches for an integrated account of human development. Child Development, 71(1), 36-43. doi:10.1111/1467-8624.00116 Main (in press). Effective middle school teacher teams: A ternary model of interdependency rather than a catch phrase. Teachers and TeachingTheory and Practice, 18(1). Martin, C., Fabes, R., Hanish, L., & Hollenstein, T. (2005). Social dynamics in the preschool. Developmental Review, 25, 299-327. doi:10.1016/j.dr.2005.10.001 Meirink, J.A., Meijer, P.C., & Verloop, N. (2007). A closer look at teachers individual learning in collaborative settings. Teachers and Teaching: theory and practice, 13(2), 145164. doi:10.1080/13540600601152496 Scribner, J.P., Sawyer, R.K., Watson, S.T., & Myers, V.L. (2007). Teacher teams and distributed leadership: A study of group discourse and collaboration. Educational Administration Quarterly, 43(19), 67100. doi:10.1177/0013161X06293631 Steenbeek, H., & van Geert, P. (2007). A theory and dynamic model of dyadic interaction: Concerns, appraisals, and contagiousness in a developmental context. Developmental Review, 27, 1-40. doi:10.1016/j.dr.2006.06.002 Thelan, E., & Smith, L. (1994). A Dynamic Systems Approach to the Development of Cognition and Action. Cambridge: MIT Press. Wasserman, S. & Faust, K. (1994). Social Network Analysis: Methods and Applications. Cambridge: Cambridge University Press. Yoon, S., Liu, L., & Goh, S. (2010). Convergent adaptation in small groups: Understanding professional development activities through a complex systems lens. Journal of Technology and Teacher Education, 18(2), 319-344. doi: p/31362.
Acknowledgment
This research is supported by the National Science Foundation under Grant No. 0741659.
ISLS
171
ICLS2012
What a Long Strange Trip Its Been: A Comparison of Authors, Abstracts, and References in the 1991 and 2010 ICLS Proceedings
Victor R. Lee, Lei Ye, & Mimi Recker, Department of Instructional Technology and Learning Sciences, Utah State University, 2830 Old Main Hill, Logan, UT 84322-2830 USA, {victor.lee, lei.ye, mimi.recker}@usu.edu Abstract: We examine differences in authorship, word usage, and references in full papers from the 1991 and 2010 ICLS proceedings. Through a series of analyses, we observe that, while authors largely hail from the US, national and regional participation in the LS community has broadened. Word usage suggests a shift in emphasis from cognitive issues to ones that are both cognitive and cultural. Reference analysis indicates a shift in core literatures and influential authors.
Introduction
While the dawn of an academic discipline is usually not heralded by a birth announcement, 1991 was certainly marked by three signature moments with respect to the publication and presentation of Learning Sciences research: the release of the first issue of the Journal of the Learning Sciences (JLS), edited by Janet Kolodner; the first proceedings of ICLS, which was held in Evanston, Illinois (USA), and the first Computer-Supported Collaborative Learning (CSCL) workshop in Carbondale, Illinois (USA) (as stated on the ISLS website). Today, JLS continues to thrive as a highly influential education research journal. ICLS continues as a respected conference venue, with its present iteration taking place in Sydney, Australia. A vibrant CSCL community continues to grow with a series of ongoing conferences and, most recently, the creation of another high-impact academic journal (ijCSCL) that began printing in 2004. Considering those research outlets alone, there are now at least four formally recognized venues for publishing innovative work related to the Learning Sciences. In this paper, we focus on changes in one of those publication venues (ICLS proceedings) at two points in time (Birnbaum, 1991; Gomez, Lyons & Radinsky, 2010). We adopt this more narrow focus for reasons of tractability and systematicity. On the one hand, we were eager to explore whether fairly simple tabulation procedures could offer us a glimpse into the nature of our field. At the same time, and considering there are only a limited number of printed copies of the 1991 proceedings and no public electronic versions, we were well aware that doing systematic counts of selected items within the proceedings would require a great deal of data preparation. However, we believe that this endeavor was appropriately timed and the two texts were well selected, as the two conferences were in the same metropolitan area (and thus should have enabled comparable geographic participation) and the time span was over the equivalent of a human generation. Moreover, these proceedings were also the oldest and most recent data points from a venue that has maintained the same name, even when additional relevant publication venues (such as ijCSCL) have emerged and established shared, but still distinct identities.
Analytical Precedents
Within the past decade, members of the Learning Sciences community have used tools from the information sciences to better understand participation in relevant journals and conferences. For example Kirby, Hoadley & Carr-Chellman (2005) conducted a citation analysis of six Learning Sciences (LS) and Instructional Systems Design (ISD) publications published through 2001. They sought to determine if overlap existed between two fields that have been understood by some as pursuing similar goals. In their study, they found that very few scholars (less than 0.5%) published in flagship journals for both fields and that cross-citations between LS and ISD publications did not exceed 0.5% of total references in either direction. Hoadley (2005) extended this work in an analysis of CSCL conference participation from 1995 to 2003. That study identified disciplinary and national affiliations of CSCL paper presenters and international collaborations over time. Kienle & Wessner (2006) provided another analysis of CSCL conference proceedings that included 2003 and 2005. There analysis showed greater international diversity and collaboration over time. For ICLS, however, the picture is less clear. Kirby, et als work included analysis of ICLS proceedings but that analysis has not considered the five meeting since 2000. All analyses of later conferences have maintained CSCL as the focus. While the contributions of scholars involved in the CSCL community is central to the growth of the Learning Sciences, it still remains the case that the individuals participating and the topics discussed have some areas of individual distinction. We intend to fill the ICLS gap by using some of the same analytical tools that were used in studies of CSCL proceedings. Also, we are considering the simple metric of word frequency as a potentially telling attribute for a proceedings analysis.
ISLS 172
ICLS2012
Simply stated, we wanted to know which words were used most often in ICLS papers. While frequency alone can be a fairly crude measure for characterizing text contents, simple word frequencies from text corpora have still been recognized in high profile journals as a surprisingly powerful tool through which one might understand changes within cultures and communities (Michel et al., 2011). Our hope is to elucidate the topics and issues that were of primary concern at ICLS meetings.
Questions, Data Sources, and Methods

We ask four questions about the two sets of proceedings: 1) Which authors were contributing to these two conferences? 2) What nations were being represented? 3) What topics were being presented?, and 4) What sources were most cited? To answer these questions, we obtained copies of all full papers (up to 8 pages each) from both the ICLS 1991 and 2010 proceedings. We excluded posters, symposia, and keynote abstracts in order to maintain comparable data sets (i.e., the 1991 proceedings had no listed posters and keynote abstracts were highly variable in their lengths). In total, there were 58 full papers in the 1991 proceedings and 149 from 2010. The first page, with author information and abstract, and the complete reference lists were manually scanned from printed copies or downloaded from the ACM digital library. From those, we extracted the following information: Paper authors. Unique authors for each paper were extracted, segmented, and tallied. First authors Geographic Location. Because of the high cost of data preparation and variability of presentation, only the first authors geographic location (at the time of publication) was considered. Abstracts. Abstracts are meant to be concise and comprehensive descriptions of the contents of an article. We used these as a proxy for paper content that could help avoid possible inflation of word frequencies due to excessive word repetition within the paper. We excluded common stop words (such as the or it) and words that are generic to research papers (e.g., results, study). Referenced works. The complete reference lists were automatically parsed to extract the full set of authors, the date of publication, and the publication venue (i.e., journal titles).
Results & Analysis

Which authors were contributing to these conferences?
The 2010 proceedings contained almost three times as many papers as the 1991 proceedings (2010: 149 papers, 1991: 58 papers) from more unique authors (2010: 364 authors, 1991: 113 authors). Papers in the 2010 proceedings overall had a greater tendency to have more co-authors (2010: 2.44 co-authors, 1991: 1.95). Perhaps surprising is the observation that there were only four individuals who were listed as contributors to full papers in both conferences: Katerine Bielaczyc, John M. Carroll, Kenneth Koedinger, and Janet Kolodner. From firsthand observations by the authors of this current paper, we remain certain that there were more than four individuals who participated or attended both conferences. Most likely, these individuals had a change in status with respect to their conference involvement. They were presenters within symposia or posters, serving as discussants, participating in workshops, or simply interested community members who were attending and meeting with friends and colleagues. Comparable changes in participation status at conferences were documented by Hoadley (2005) with respect to five CSCL conferences.
What nations were being represented?

Because of the relatively high cost of data preparation (scanning, converting, editing, etc.) and a large amount of variability in how authors listed their information (thus making it difficult to automatically parse all author affiliations), a geo-analysis of authorship was conducted only on first authors. While we are aware of the limits associated with considering only one contributor to a paper, we did expect that this would be telling in that first authors are often the presenters and primary attendees for conference papers. With respect to national representation at ICLS in both meetings, there was a clear majority of first authors who came from North America (Figure 1), and in particular the United States (69% in 1991 and 62% in 2010). In some respects, this is not surprising as conference location influences who will submit papers and ultimately attend (Kienle & Wesser, 2006). The most visible changes in national participation took place beyond North America. The 1991 proceedings had first author representation in Europe and Australia only. In 2010, Asia represented 15% of the first authors and the other regions declined. In neither year were there any first authored papers from South America. An additional analysis of countries of percent representation of first authors in specific nations was conducted. These analyses suggest a general increase in the number of nations represented in 2010 in comparison to 1991 (2010: 18 nations, 1991: 8 nations). The greatest percent increase appeared from
ISLS
173
ICLS2012
Germany (2010: 1.7%, 1991: 8.7%) and the greatest percent decrease coming from the United Kingdom (2010: 8.6% 1991: 0.7%).
Figure 1: Percent of first authors by world region in 1991(left) and in 2010 (right) In making sense of these differences and changes over time, it is important to note that when ICLS was first held in 1991, it was actually organized as a special session of a conference normally held by the Artificial Intelligence in Education community, a research community where a number of Learning Scientists had been originally trained. Thus, we should not be too surprised that there are differences. Still, the increase in representation from Asian nations is also paralleled by a greater global prominence that has been noted over the past two decades. Yet, even with this global shift, there is a striking pervasiveness of papers from the US (over 60% in both years) in comparison to CSCL. This can be understood as partially due to the fact that the conferences were held in the same metropolitan area. However, as is the case for many large nations, a variety of regions and a number of institutions comprise that large percentage. Given the large number of contributions from US authors, we chose to analyze the distribution of US-based first authors by state. Three of the more populous states, California, Illinois, and Pennsylvania, were highly represented in 1991 (CA = 20%, IL = 28%, PA = 15%). These states have been known to have prominent institutions conducting research related to Artificial Intelligence (AI). Those three states continued to have a relatively high percentage in 2010 (CA = 19%, IL = 9%, PA = 9%)., but were also accompanied by other states such as Indiana (6%), Maryland (6%), Washington (5%), and Wisconsin (9%). Sixteen states previously unrepresented in 1991 had first authors in 2010 and two states represented in 1991 (Connecticut and New Mexico) were not represented in 2010. To determine if population was the biggest predictor of author location, we extracted U.S. population data by state from the 1990 and 2010 censuses. A Spearman rank order correlation between first author location and U.S. state population was not significant in neither the 1991 (rs = .36) nor the 2010 (rs = .31) proceedings. Note also that paper contribution rates by state across conference proceedings were significantly correlated (rs = .70, p < .05), showing similar rates across time periods. Thus, state participation appears to be broadening, but it seems to be highly dependent on the presence and location of particular individuals and institutions with research resources (e.g., the LIFE Center with University of Washington as a partner institution, the GLS group at University of Wisconsin) rather than a uniform change due to demographic shifts.
What topics were being presented?

We used word frequency in abstracts as a proxy for the content that was being presented in each paper. We deliberately conflated the counts of words that would be the same except for small variations, such as plural form (e.g., case and cases), change in tense (e.g., model and modeled), or comparable adjective and noun forms (e.g., mathematics and mathematical). While these conflations could have led to groupings where there are subtle nuances in senses of words that reflect different research agendas, there were enough overlaps that consolidation was deemed appropriate by the authors. Full tables of words and word frequencies were produced, but due to space limitations, we present word clouds of the top 20% of words that appeared in the abstracts (Figure 2). In the word clouds, larger and darker fonts represent higher frequencies of occurrence.
ISLS
174
ICLS2012
Figure 2: Word clouds from 1991 (left) and 2010 (right). In 1991, the most frequently used words were: model, cognition, conceptual, domain, environment, model, strategy, training, and tutor. The most frequently used words in 2010 were: case, conceptual, epistemic, examine, inquiry, interaction, mathematics, practices, representation, and strategies. The intersection of this set includes: case, cognition, conceptual, representation, theory, and strategy. While used frequently, case was often used in 1991 to refer to cases as used in case-based reasoning (Kolodner, 1993) and in 2010 it was used for case-based learning and for case studies. Representation had been used in 1991 often to refer to knowledge representation, and increasingly in 2010 to refer to external representations and representational practices (such as creating inscriptions). 1991 involved unique terminology that involved information processing models and constructs such as training, tutor, and instruction. Unique terminology in 2010 suggested a contingency of scholarship geared toward sociocultural constructs such as discourse, participants, and practices. While sociocultural constructs emerged and gained prominence, cognitive terminology such as cognition and conceptual still appeared in both conferences, often associated with research related to tutoring systems, conceptual change, and artificial intelligence.
What sources were being used?

Our final question in this investigation related to the sources in particular, the authors and reference sources that appeared in the two sets of proceedings. We were hoping to find out what journals figured most prominently (e.g., did JLS indeed play a prominent role in this community after it was established?) and whose writings were considered influential. As there were many more references than there were number of papers or contributing authors in the proceedings (2010: 24.9 refs/paper, 1991: 12.7 refs/paper) and given the aforementioned difficulties of data preparation, we focused strictly on automated analyses of references. While we ran several such analyses, we presently report just on journals and authors (due to space limitations). In analyzing the journals that were cited, we chose to focus on the journals that comprised at most the top 20% of cited journal articles. Only two journals (2.2% of all unique journals) were the source of 20% of journal articles cited in 1991. These included Cognitive Science (13.3%) and Artificial Intelligence (6.9%). In 2010, four journals served as the source of 20% of cited articles. These included: Journal of the Learning Sciences (7.1%), Cognition & Instruction (4.9%), Journal of Research in Science Teaching (4.7%), and Science Education (3.3%). Cognitive Science was still cited in 2010 but it made up a much smaller percentage of the total share (1.6%). The most highly cited AI journal in 2010 was the International Journal of Artificial Intelligence in Education (0.1%). Based on the journal share percentage, the foundational literature emphasis changed substantially as newer journals tuned to issues of learning and instruction were established and more widely cited. Finally, we considered authors of cited works, regardless of publication type. Although selfcitations may have played a role, we did not exclude instances of self-citation. Our underlying assumption was that, when an author was cited so heavily, even had s/he cited a lot of their own work, s/he were still likely to have been cited in at least some articles in which s/he were not an author. Space limitations prevent us from showing a longer list. Therefore, we list simply the top five cited authors (or organizations, as is the case with the US National Research Council) and the frequency of their name in the entire corpus of references. diSessa, A. A. was the only author to appear in both top-five lists. Table 3: Most cited authors and percent frequency within each set of proceedings.
1991 Author Anderson, J.R. Schank, R.C. Breuker, J.A. diSessa, A.A. Papert, S. % 3.1% 2.5% 1.2% 1.2% 1.2% 2010 Author US NRC Scardamalia, M. Hammer, D. diSessa, A. A. Brown, A. L. % 1.1% 1.0% 0.8% 0.8% 0.7%
ISLS
175
ICLS2012

What sort of journey has it been from Evanston, Illinois in 1991 to Chicago, Illinois in 2010? In terms of distance and distance, it was roughly 19 miles over 19 years. However, the field of Learning Sciences, as reflected in its conference proceedings, has shown a number of shifts. Participation generally broadened across geographic locales. A new core body of journals served as the source of a peer-reviewed knowledge and the authors whose work was acknowledged and cited largely changed. The research emphasis appeared to have moved from a cognitive approach in which training with systems using Artificial Intelligence was used to provide instruction to a view of learning that involved enculturation into disciplinary practices and a variety of interactions within a complex, socially-mediated learning ecology. Cognitive Science and Artificial Intelligence continue to play a role in the Learning Sciences, but that body of research is now accompanied with investigations of inquiry, science and mathematics learning, and a growing interest in issues of external representation. Undoubtedly, these findings are limited in their generalizability due to the sampling strategies and selected analytical techniques. More analyses and even more sophisticated techniques could have been used. However, these snapshots are still useful because they can serve as touchstones to help us reflect upon the ongoing intellectual trip that has been and continues to be the Learning Sciences.
References
Birnbaum, L. (Ed.). 1991 The International Conference on the Learning Sciences: Proceedings of the 1991 Conference, Association for the Advancement of Computing in Education, Charlottesville, VA. Gomez, K., Lyons, L., & Radinsky, J. (Eds.). Learning in the Disciplines: Proceedings of the 9th International Conference of the Learning Sciences (ICLS 2010). International Society of the Learning Sciences: Chicago. Hoadley, C. M. (2005). The shape of the elephant: Scope and membership of the CSCL Community. Paper presented at the 2005 Conference on Computer Supported Collaborative learning. Kienle, A., & Wessner, M. (2006). The CSCL community in its first decade: Development, continuity, connectivity. International Journal of Computer-Supported Collaborative Learning, 1(1), 9-33. Kirby, J., Hoadley, C., & Carr-Chellman, A. A. (2005). Instructional systems design and the learning sciences: A citation analysis. Educational Technology Research and Development, 53(1), 37-48. Kolodner, J. (1993). Case-Based Reasoning. San Mateo: Morgan Kaufmann. Michel, J.-B., Shein, Y. K., Aiden, A. P., Veres, A., Gray, M. K., The Google Books Team, . . . Aiden, E. L. (2011). Quantitative analysis of culture using millions of digitized books. Science, 331, 176-182.
Acknowledgments
We gratefully acknowledge Min Yuan, Jon Thomas, and Anne Diekema for their comments and assistance. A slightly longer version of this paper with additional figures and analyses can be obtained from the lead author.
ISLS
176
ICLS2012
Scaffolding Collaborative Sensemaking during Critique of Explanations in Technology-Enhanced Science Curriculum

Elissa Sato, University of California at Berkeley, 4523 Tolman Hall, Berkeley CA 94720, elissa.sato@berkeley.edu Vanessa Svihla, University of New Mexico, 1 University of New Mexico, Albuquerque, NM 87131, vsvihla@unm.edu Abstract: This qualitative study extends a comparison study investigating critique and feedback in a technology-enhanced sixth-grade earth science curriculum unit. The unit was designed to elicit and scaffold collaborative sensemaking of criteria for explanations in science. Quantitative analysis suggested the activity had limited success in terms of prompting revision, but the qualitative analysis illustrated different dimensions of engagement in critique, demonstrating the rich potential of critique for collaborative sensemaking. Implications for instructional design are discussed.
Introduction
Generating explanations is central in science (National Research Council, 2007) and important in learning (Chi et al. 1994; NRC, 2007), yet non-trivial for students learning complex topics. Typical instruction does not support students in the practice of generating explanations, yet prior research has shown the value of scaffolding students to construct and evaluate explanations (e.g., McNeill, Lizotte, Krajcik, & Marx, 2006; Sandoval & Reiser, 2004). We situate the generation of explanations within the knowledge integration (KI) framework (Linn & Eylon, 2006), which identifies four processes: eliciting ideas, adding new ideas, distinguishing among ideas, and refining the repertoire of ideas through reflection. From this perspective, learning to generate quality explanations involves, in addition to synthesizing relevant domain knowledge, making sense of and applying criteria for what constitutes a good explanation. We conceptualize scaffolded critique of explanations as generative activities in that they support students in a) distinguishing among criteria in the context of specific explanations, and b) sorting through and refining the ideas captured by the targeted explanations. We investigate how critiquing explanations might impact students ability to distinguish and refine their repertoire of ideas. We present two cases to highlight ways students engaged with a particular critique activity, examining how students make sense of commonly used, yet vague criteria for good scientific explanations.
Methodological Approach Building on Prior Work

This paper presents findings that extend a prior comparison study investigating the source of feedback (i.e., from the teacher or a peer) and the quality of the explanation being critiqued (i.e., high or low KI score, cf. Linn, Lee, Tinker, Husic, & Chiu, 2006) and their impact on students revised explanations. Quantitative analysis found a significant advantage for instructor feedback on gains after controlling for the quality of the original explanation. A comparison of instructor and peer feedback revealed that instructor feedback was more prompt-specific than peer feedback, although students still struggled to apply specific feedback during revision (for a more detailed discussion of the study findings, please refer to Sato and Linn, 2011; in preparation). Although this may suggest that the activity did not help students distinguish between their ideas, our video data suggest otherwise and indicate elements in activity design that have implications for instruction. In this qualitative analysis, we present two cases that illustrate the kinds of engagement during critique that were observed in the data corpus as a whole. Although their orientations were neither one-dimensional nor fully consistent throughout the critique activity, the orientations were generally representative of each dyads engagement with the task. Due to space limitations, we primarily focus on one case.
Context
A sixth-grade technology-enhanced earth science curriculum unit, Global Climate Change (GCC), was developed using the Web-based Inquiry Science Environment (WISE, Linn, Davis, & Bell, 2004) based on the KI perspective (Linn & Eylon, 2006). Students worked in pairs throughout the unit. They generated an explanation for a phenomenon, then their ideas about criteria were elicited in a first critique of two sample explanations chosen from student responses in a previous implementation of the project (Table 1). We focus on this activity as it affords insight into how students make sense of the criteria presented in the task. Because students tend to focus on surface features rather than the underlying ideas, explanations were selected to represent these: one was stylistically sound in terms of spelling and grammar, but sparse in terms of science content; the other had imperfect spelling and grammar, but described the phenomenon in detail. Students critiqued the two in succession. Our goal was to implicitly prompt students to compare the two and help them distinguish between their existing ideas about what makes a good explanation. Students rated each explanation
ISLS
177
ICLS2012
for both surface (spelling and grammar) and science content (e.g., the response needs more evidence; the science ideas are wrong or vague; the science ideas could be described in more detail) from a list of criteria, then explained their choice for science content. Our goal was to prompt students to discuss criteria commonly encountered during instruction; there was no correct choice per se among the list of science content criteria, nor were they mutually exclusive. By asking students to choose the criterion they felt best captured the science content in the explanation, we hoped to motivate students to engage in sensemaking of the criteria. Table 1: Preselected Explanations Critiqued by Students during the First Critique Prompt: What happens to global temperature in an environment with low albedo? What happens to solar radiation (SR) in the model in step 4.2 that supports your answer? Characteristics Preselected Explanation Good style (spelling/grammar), It changed a lot. It went down then bounced. vague science content Imperfect style (spelling/grammar), THe globle tempeture went down when the albedo was low, like for the detailed science content ocean, or when the albeod reflected only 5 percent of the solar radiation. When the soler radiatoin was reflecting, it coud not change into heat energy it just went back to space.
Case 1: The Sense-Making Pair

Kostas and Ted studied the peer feedback version of the GCC unit. Though they began the unit in the bottom quartile of their treatment group based on pretest measures, they were both highly motivated students who engaged with the unit and held extended discussions before deciding on responses to explanation prompts. It was common for the two to raise questions and objections if one did not share the others opinion or understanding of a particular activity. Although they did not generate detailed feedback, their discussions illustrated rich instances of both surface and content criteria sensemaking throughout critique. Kostas and Ted negotiated whether accurate spelling and grammar can be judged independently of science content during surface critique of both explanations. Kostas appeared to grapple with the distinction between stylistic clarity and its impact on the clarity of science content. In the transcript below, the pair has begun surface critique of the first explanation (It changed a lot. It went down then bounced, Table 1). Ted asserted his rating (good: few errors), and after an initial assent, Kostas had reversed his position. T: K: T: K: T: K: T: What would you give it? Out of three? On a scale //of// //One?// One? ((incredulously)) Yeah, this is like, so bad. ((sweeps finger across the explanation)) No, its not Its not good English, you know? Like, It changed a lot. It //bounced and//((reading out loud in a singsong)) //Thats in// scientific content and clarity. ((pointing to next critique prompt for science content and clarity)) Thats what it means. Here, the science idea could be described in, some ideas are wrong or vague. ((reading some of the science content criteria; K leans in toward the monitor as T reads)) Like, what changed, though? ((pointing to It changed portion of first explanation))
K:
Although Kostas did not specify his misgivings other than that they were not about spelling, he elaborated that the explanation had bad English (lines 7-8). Ted countered that Kostas was referring to science content and clarity, not spelling and grammar (lines 9-12), and oriented Kostas to the next critique prompt and its list of criteria targeting science content and clarity. By asking Ted, what changed? (line 14) Kostas seemed to have been trying to identify what was bothering him on a more specific level. Given that the explanation did not specify what it was that changed, this problem can be viewed simultaneously as an issue of vagueness in language and of lack of clarity in content. Here, Kostas was focusing on the vagueness of the language. In effect, Kostas may have been alluding to sophisticated ways in which the surface and content criteria can be considered to overlap. Although they eventually chose good as their response, the discussion re-emerged during the surface critique of the second explanation, indicating that making sense of the surface and content criteria was important for Kostas. Kostas and Ted continued to engage in sensemaking discussions during the science content critique for both explanations. In the example below, they had formed initial impressions about whether each criterion was applicable to the particular explanation before reaching the last criterion on the list,
ISLS
178
ICLS2012
the science ideas could be described in more detail. The ensuing discussion illustrates the rich conversations they had as they negotiated the meanings of the criteria: K: This is... This is it. ((points to needs more detail criterion)) In more detail... 'cause [the explanation] just said, "It changed a lot. It went down then bounced." So it needs like, more support... evidence [another criterion], and like, more detail. Yeah, it's more evidence. It's... No, it doesn't need more detail. 'Cause, I mean... [it] has a LOT of detail. Well, not that... well((sighs)) It has like, a 4 out of 5 detail. It said, "It went down, then bounced." That's kind of describing it in detail. But it's not describing it in like, proper grammar. I think this is right. ((points to needs more detail)) I think no. This is need more detail. I think its needs to have evidence. Im really confused between both of those. ((points to needs more detail and needs to have evidence criteria)) I dont know. [inaudible] I think what you chose is right. I dont think theres, like, a wrong answer in this question.
T:
K: T: K:
T:
Although the target explanation prompt explicitly asked for evidence from a previously explored model (Table 1), Kostas and Ted did not discuss whether the explanation contained explicit references to the model. Instead, they discussed how to distinguish evidence from detail. Ted struggled to clarify why he believed that needs to have evidence was the more appropriate critique over needs more detail (lines 19-21). Teds concluding remark, But its not describing it in like, proper grammar (line 23), is notable in light of the fact that they had already evaluated the grammar and spelling of the explanation to be good. It may suggest that he had some ideas about what counts as evidence in science explanations as opposed to simply descriptive detail. His struggle is similar to Kostas earlier attempt to distinguish between surface stylistic clarity and content clarity in that Ted also struggles to articulate why he believes one criterion is more applicable than the other (lines 19-21). In this case, however, Kostas and Ted did not pursue this line of thought to the same extent as they did previously for Kostas. After Kostas professed his uncertainty about what evidence and detail meant (line 27-28), Ted responded that any of the criteria may be acceptable answers (line 31). Unable to pin down what exactly they felt was lacking in the explanation, they eventually decided on the science ideas are wrong or vague and wrote, We think that the student needs to add more detail and support his answer because we can't understand the answer, it is vague and needs to become clearer. Whereas the first explanation was intended to serve as an exemplar for good style and vague science content, the second was intended to serve as an example for poor style but detailed science content. As such, the second explanation was rife with spelling and grammatical errors. It may therefore seem a straightforward task to score the spelling and grammar, perhaps even more so than the first explanation due to the obvious errors. However, the confusion between stylistic and content criteria reemerges in their exchange. In the transcript below, Kostas and Ted have just finished identifying numerous spelling errors in the second explanation and were about to choose a criterion for spelling and grammar. As before, they disagreed, but this time, Kostas used the first explanation as a point of reference when questioning Teds assessment of Not so good. This prompted Ted to repeat his earlier attempt to orient Kostas toward science content criteria to address his concerns with the explanation. K: T: T: K: T: T: K: K: T: T: K: T: //U::h.// //U::h.// N::::not good. ((selects Not so good criterion with pointer)) No, uh, it's, //it's good.// ((points to Good criterion with finger)) //It has many spelling.// Has //many// spelling. //Ohh.// Uhh. Yeah, but this like, so you're saying the, the one before that has like one sentence, is "good... few spelling and //the"//((points to first explanation, then to Good criterion)) //That's// for grammar and spelling. ((traces grammar and spelling in critique prompt; K drums fingers on desk)) I like this, the idea that it was, the ((points to second explanation)) Yeah, youoh, oh. ((nods, points to science content and clarity criteria list)) I like, for science clarity, response, uh, needs evidence to explain, so((points at criterion))
ISLS
179
ICLS2012
And the::n, "ideas could be elaborated." I think... I think yeah. In questioning Teds not so good surface critique, Kostas drew attention to the brevity of the first explanation relative to the second (lines 39-41). This indicates that Kostas was still struggling to distinguish between surface and content criteria. From this, his earlier dissatisfaction with the first explanation for not good English seemed to have remained unresolved. In response, as in the first case, Ted repeated his assertion that Kostas issue with the first explanation pertained to science content and not spelling and grammar (lines 41-42). This time, Kostas nodded and pointed to the science content criteria list as if to accept and affirm Teds claim. For Kostas, critiquing two explanations with contrasting dimensions seemed critical for making progress in teasing apart surface and content issues. Although Kostas and Ted engaged in sensemaking discussions about both surface and content criteria, their rich conversations and oral critique were not fully captured in their written feedback. Following the discussion, they wrote, We chose this answer because we thought that his/her explanation could describe his/her evidence in more detail. However, their active engagement with the critique tasks indicates that the critique activity provided them with multiple opportunities for beginning to make sense of and distinguish among the criteria.
Case 2: The Task-Oriented Pair

Tammy and Giulia studied the instructor feedback (low KI) version of the GCC unit. They began the unit at the top quartile of the treatment group and were highly motivated students who remained on-task throughout. They typically approached prompts from what might be characterized as task completion-oriented remarks (e.g., We could say We could put) when beginning to formulate their responses. Although they were able to identify issues during critique, their sensemaking discussions were brief. During critique, Tammy and Giulia discussed the criteria until they reached agreement. When choosing the criterion, one of the pair would immediately suggest a candidate, generally with no rationale and without subsequent disagreement, as the following example of content critique illustrates: T: T: G: T: Score this response for SCIENCE content and clarity. ((Reads list of criteria out loud (Figure 3))) I like the last one. [The science ideas can be described in more detail.] Yeah. ((T checks this criterion)) Explain your choice for SCIENCE CONTENT and CLARITY and give an example from the response to help the student improve the explanation.
Tammy immediately declared that she liked the last criterion (line 3) but without stating why. Giulia agreed without providing rationale (line 4), and the pair moved on to the next prompt (line 5). Similar episodes were observed for both critiques. Thus, if one student did not find cause to disagree with the others proposal, there was no further discussion, and the pair moved on to the next task without elaborating on their decisions. There were instances that could have led the dyad to explore a criterions meaning. However, they did not pursue those opportunities, instead rapidly coming to agreement and moving on to the next task. It is possible that they knew each other well and felt so aligned with each other that they were not motivated to explicate their reasoning. Another explanation is that they did not engage in sensemaking beyond what was necessary to complete the task. They achieved consensus about which criterion to select, but it is unclear whether their rationales for their opinions actually aligned because neither of them explicated the basis for their respective opinions. Similar episodes during their critique tasks were observed, suggesting that, while Tammy and Giulia both engaged with the critique tasks and collaborated with each other, they missed several opportunities to delve more deeply into the criteria.
Cross-Case Comparison
Our findings from the comparison study seemed to suggest that the activity of generating feedback for a peer had little impact. The case studies indicate that a lack of detail in the written responses does not de facto indicate a lack of sensemaking. The case studies highlight the range of engagement during a critique activity (Table 2), and potential tradeoffs between sensemaking and efficient task completion. Although Kostas and Ted did not generate detailed feedback, their discussions illustrated rich instances of criteria sensemaking that are arguably at least one of the essential first steps students must take before they can engage productively in critique and generate good feedback. In contrast, although Tammy and Giulia were similarly highly engaged with the tasks as a team, their discussions did not lead to rich sensemaking. Unlike Kostas and Ted, who engaged in crossexplanation comparisons in their discussions, Tammy and Giulia seemed to approach each explanation critique as an isolated, unconnected task; this limited the activitys potential to help the students develop integrated understanding. The dyad seldom leveraged opportunities to delve deeply into the criteria and try to understand
ISLS
180
ICLS2012
the nuanced distinctions to the same extent as Kostas and Ted. Meager responses belie the diversity of sensemaking about both science content and evaluative criteria students engaged in. This speaks to the findings of the comparison study: few students provided detailed, actionable feedback to their peers, yet the activity of providing critique may have provoked sensemaking. The task was specifically designed to encourage students to struggle with commonly-encountered vague criteria; the degree to which students made sense of these criteria is tied to a tradeoff between efficient completion of isolated tasks, and more time-consuming negotiation and consensus building. Table 2: Summary of Cases with Dimensions of Engagement Case Sensemaking Pair Task-Oriented Pair Students Kostas and Ted Tammy and Giulia Engagement as a Team Yes Yes Criteria Sensemaking Yes No Task Completion Orientation No Yes
Implications
These findings provide insight into the affordances and constraints of peer critique and identify promising avenues for scaffolding productive critique activities in technology-enhanced instruction. Although students encounter abstract criteria for what makes a good science explanation in instruction, findings indicate that students would benefit from opportunities to reflect on and distinguish between these criteria in the context of specific explanations. Providing students with more than one peer explanation to serve as contrasting cases during critique may increase the activitys potential for helping students refine their criteria for explanations. Forcing students to select among non-mutually exclusive criteria prompted some students to engage in sensemaking of the criteria. The case study findings indicate the potential value of explicitly problematizing the criteria selection process to more effectively engage students in criteria sensemaking. However, the case studies also point to the need for designing critique activities that prompt students to contrast explanations and distinguish their ideas across multiple explanations instead of considering each explanation in isolation during critique.
References
Chi, M. T. H., de Leeuw, N., Chiu, M., & LaVancher, C. (1994). Eliciting self-explanations improves understanding. Cognitive Science, 18(3), 439-477. Linn, M. C., Davis, E. A., & Bell, P. (Eds.). (2004). Internet Environments for Science Education. Mahwah, NJ: Lawrence Erlbaum Associates. Linn, M. C., Lee, H.S., Tinker, R., Husic, F., & Chiu, J. L. (2006). Teaching and assessing knowledge integration. Science, 313, 1049-1050. McNeill, K. L., Lizotte, D. J., Krajcik, J., & Marx, R. W. (2006). Supporting students construction of scientific explanations by fading scaffolds in instructional materials. Journal of the Learning Sciences, 15(2), 153-191. Sandoval, W. A., & Reiser, B. J. (2004). Explanation-driven inquiry: Integrating conceptual and epistemic scaffolds for scientific inquiry. Science Education, 88, 345-372. Sato, E., & Linn, M.C. (2011, April). Developing Criteria for Explanations in Science: Scaffolding Peer Critique and Feedback in Technology-Enhanced Instruction. Poster presented at the 2011 American Educational Research Association Annual Meeting, New Orleans, LA. Svihla, V., Gerard, L., Ryoo, K., Sato, E., Visintainer, T., Swanson, H., Linn, M.C., Lee, H-S., Liu, O.L., & Dorsey, C. (2010, June). Energy across the Curriculum: Cumulative Learning Using Embedded Assessment Results. Paper presented at the International Conference of the Learning Sciences, Chicago, IL. National Research Council. (2007). Taking science to school: learning and teaching science in grades K-8. Washington, DC: National Academy Press.
ISLS
181
ICLS2012
Collaborative Reading Comprehension with Communication Robots as Learning Partners

Jun Oshima, Ritsuko Oshima, Shizuoka Univ., 3-5-1 Johoku, Naka-ku, Hamamatsu-shi, Japan, joshima@inf.shizuoka.ac.jp, roshima@inf.shizuoka.ac.jp Naomi Miyake, Univ. of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, Japan, naomiyake@gmail.com Abstract: We are developing a learning environment where communication robots play critical roles to support pre-service teachers active engagement in collaborative learning. Robovie-W, a communication robot, was employed as a learning partner in a jigsaw activity. Four students in a group with one Robovie-W collaboratively explained their ideas from different articles about learning environments, and discussed how to integrate these ideas. Our analysis suggests that the effectiveness of robots is influenced by the humans expertise in collaboration. On the basis of these results, we discuss hypothetical design principles for collaborative learning with robots.
Theoretical Background and Purpose of Research

Studies on intelligent agents in learning environments have been gradually making it clear that artificial agents can facilitate learners active engagement in learning. Following from previous theoretical studies related to learning by teaching, Schwartz et al. (2009) discuss how higher levels of metacognitive agency are achieved through teaching activities than that through ordinary learning. In the social interaction of teaching others, learners are expected to gain a deeper conceptual understanding of study topics as well as metacognition of how to learn. Bettys Brain is a teachable agent-based learning environment for learners to engage in learning by teaching an intelligent agent in collaboration with another intelligent agent that acts as a mentor (Biswas, Katzlberger, Bransford, & Schwartz, 2001; Biswas, Leelawong, Schwartz, & Vye, 2005; Blair, Schwartz, Biswas, & Leelawong, in press; Chin et al., in press). Four core design principles have been identified for effectively facilitating learners interactions with agents in the environment: (1) an explicit and well-structured visual representation in agents minds is needed, (2) agents should be enabled to take independent actions, (3) productive learner behaviors should be modeled, and (4) environments that support teaching interaction should be created. Based on these four core design principles of teachable agent-based environments, further studies have been conducted to answer several research questions such as how learners manage feedback from agents and how learners responsiveness is related to their learning and task performance. The study reported here takes a new perspective on learners social interaction with artificial and intelligent agents. We examine the effectiveness of communication robotsartificial but physical agentsin collaborative learning (Miyake, Ishiguro, Kanda, & Shirouzu, 2011). Collaborative learning has been practiced widely in the community of learning scientists, producing relatively small to profound changes in the quality of learning achievements (Bransford, Brown, & Cocking, 1999; Sawyer, 2006). To improve the design of such practices within a wider variety of cultural settings, stronger scientific evidence is needed for such success, as well as a more precise explanation of the mechanisms of how and why it works (c.f., Miyake, 2008). One motivation for proceeding in this direction is the need to obtain a stronger set of evidence for the power of a collaborative, learner-centered orientation in education, in order to work effectively with policy makers. Humanrobot interaction research has been actively conducted for years in robotics engineering (e.g., Goodrich & Schultz, 2007), and androids have been developed to communicate with humans in a variety of everyday settings. In our view, robots can play two roles that allow us to conduct more powerful research on collaborative learning: assisting collaborative learning and assessing the processes involved in such learning. As a learning partner, robots can present prepared explanations to their human partners, comment on others ideas, and exchange questions and answers, at least to the extent that those exchanges could be either prepared prior to the classes or provided by human operators. As a research partner, robots can run a controlled experiment during a collaborative learning process in a classroom setting. One example of such an experiment is to have a robot in each discussion group and to have it deliver the same information to different groups, so that we can explore the effects of a particular discourse in collaboration. Funded by the Ministry of Education, Culture, Sports, Science and Technology (MEXT), Japan, this study was conducted as a part of a collaborative research project in which robotics engineering researchers, cognitive scientists, and learning scientists work toward the goal of articulating core design principles for effective collaborative learning supported by artificial agents.
ISLS
182
ICLS2012
Research Design
Context
We implemented communication robots as learning partners in a summer intensive course curriculum and instruction in the teacher certificate program at a Japanese public university. In the year of the study, twelve students from engineering and informatics department took the course. They were required to participate in the course as groups in which they discussed their ideas related to the course content and constructed their shared understanding of the learning environment (Bransford et al., 1999) and its application to classroom practices. This study was conducted in the first module of the course (the first one and a half day). In the first module, students were involved in collaborative reading comprehension activities in which they were expected to construct their conceptual understanding through explaining their responsible articles to one another and integrating ideas from different articles.
Collaborative Reading Comprehension in a Jigsaw Activity

Collaborative reading comprehension is an activity structure based on Jigsaw method for learners to deeply engage in their collaborative knowledge building based on their understanding of multiple document resources (Oshima & Oshima, 2011). In this study, students were first placed in expert groups. In each expert group, three students collaboratively read and constructed their understanding of a different article that they would explain to others afterward in jigsaw groups. We implemented the reciprocal teaching approach (Brown & Palincsar, 1989; Miyake & Shirouzu, 2006) for their collaborative reading such that each student took different roles and was in charge of different sections of the article. Through their collaboration, each student produced a summary in our prepared Microsoft Word template that was used as a handout for the explanation in jigsaw groups.
Figure 1. Participatory Structure of the Jigsaw Activity for Collaborative Reading Comprehension (left) and Jigsaw Group Activity with Robovie-W (right). Three jigsaw groups were then formed consisting of one student from each expert group and one communication robot. Students in the jigsaw groups worked to integrate ideas from five different articles explained by the responsible students and a robot (see Figure 1). Communication robots were in charge of explaining article #3. After discussing the five articles, the students reported how ideas from the articles were related to one another and interpreted them with reference to the basic framework of learning environments in a CSCL system.
Communication Robots as Learning Partners

Robovie-W, a communication robot, was developed at ATR, Japan. Robovie-W was placed on the desk and remotely operated by teaching assistants. Robovie-W could communicate with students in the following ways. First, it could speak a pre-set script to explain the article for which it was responsible (#3). The script was written by the first author and Robovie-W spoke the script by dividing it into three segments. Second, it could speak what the operators typed on their laptop computers at the time. After speaking each segment of the script, Robovie-W asked students whether they had questions about its explanation. Several scripts for answering predictable questions such as the meaning of specific terms in the explanation were prepared beforehand. However, when students asked questions for which we had not prepared answering scripts, operators who had taken the course before gave their opinion or prompted students to ask the instructor (e.g., Thats a good but difficult question. Why dont we ask Prof. O about it?). Third, Robovie-W could move his head and arms to provide nonverbal signs to students, such as moving its face toward students who were speaking (see Figure 1).
ISLS
183
ICLS2012
We explored humanrobot interaction in the setting of collaborative reading comprehension based on video-recordings of students collaborative activities, their note-taking activities on handouts, and written discourse on the CSCL environment. By examining each students expertise in collaboration (explaining and listening), we attempted to characterize each group activity and identify the effectiveness of Robovie-W as a learning partner.

For examining how the implemented communication robots influence students collaboration in the collaborative reading comprehension, we conducted two exploratory analyses: (1) the comparison of students use of understanding from explanations by humans and a robot; and (2) dynamics of collaboration based on individual differences in learners expertise as explainer and listener.
Collaboration with a Human vs. Collaboration with a Communication Robot

For evaluating the effectiveness of learners collaboration with communication robots, the quality of written discourse in students reports on the CSCL system was analyzed. In their reports, students cited multiple articles including article #3 by Robovie-W to explain ideas related to the framework of learning environments that they had learned in the brief lecture. The quality of citations was evaluated as being at one of the two following levels: (1) citation only, and (2) citation with reasoning. When students cited some segments from the article summary but did not add any reasoning of why they thought the cited segments were related to the framework of learning environments, their discourse was categorized as citation only. If they added some reasoning to their citation segments, the discourse was categorized as citation with reasoning. A Chi-square analysis of the frequencies of citations with different qualities across five articles showed no significant differences (2 = 0.098, df = 4, p > 0.05), suggesting that students in this study had learned from their collaboration with communication robots in the same way as they had from collaborating with their human partners (see Table 1). Table 1: Frequencies of different types of citations in students written discourse on a CSCL environment. Article #1 Citation Only Citation with Reasoning 16 14 Article #2 10 19 Article #3 15 6 Article #4 22 14 Article #5 17 16
Dynamics of Collaboration with Communication Robots

We attempted to characterize each students expertise as explainer or listener in collaborative reading comprehension based on her/his behavior during group work. For the purpose of characterization, we referred to the theoretical framework of adaptive expertise (Hatano & Inagaki, 1986; Schwartz, Bransford, & Sears, 2005). In collaborative reading comprehension, students needed to contribute to constructing their shared understanding of ideas from the five articles and further develop their own perspectives on designing learning environments. The context required students with adaptive expertise for collaborating with others as an explainer and as a listener. We thought that adaptive explainers might engage in cognitive activities to construct their understanding from reading their articles (e.g., Pressley & Afflerbach, 1995). In their explanation, they do not only discussed the content of their created summary, but also modified their explanations by monitoring listeners responses and restructuring their scripts. For instance, adaptive explainers repeatedly paraphrase what they speak to help listeners understand, elaborate their ideas by monitoring listeners understanding, and give metacognitive comments to encourage listeners to engage actively in constructing their shared understanding with explainers, e.g., instructing listeners to read specific parts in their summary before giving an explanation. Routine explainers, on the other hand, only tell their summary. They do not frequently monitor how listeners respond to their explanations but instead look at their handouts. We also thought that adaptive listeners might actively participate in construction of shared understanding by supporting adaptive explainers cognitive activities. They respond to explanations by making eye contact with the explainer, nodding or raising their eyebrows in doubt to signal their understanding, and taking notes on the handouts being discussed. They further pose questions seeking clarification to their explainers, and raise issues for all members to share in constructing their understanding. Thus, adaptive listeners make an effort to support their explainers by engaging in collaborative discourse rather than merely listening to explainers solo discourse. Routine listeners do not frequently express their understanding in a manner enabling their explainers to monitor whether their explanations are comprehensible, and are more likely to look at their handouts during the explainers solo discourse.
ISLS
184
ICLS2012
Based on transcription of video recordings, we coded students discourse related cognitive activities when explaining their responsible articles: (1) elaboration, (2) paraphrase, and (3) metacognitive comment, and evaluated each students adaptive expertise of explanation on a continuum from routine to adaptive explainer. We also evaluated each students expertise of listening by examining her/his behavior to respond to explainers cognitive activity and identifying use of six different note-taking strategies (i.e., memo, emphasis, drawing, underline, box, and arrow). Before coding procedure, we selected one student as the benchmark student. This student was deemed to have an ordinary level of collaboration expertise. Then, after coding, we decided the other students positions on the collaboration expertise matrix in relation to the benchmark student. Our coding agreement between the first and the second authors was 90%, and the disagreement was resolved through their discussion. We also identified the position of Robovie-W as seen in Figure 2 for the following reasons. First, regarding its expertise in explaining, we evaluated it as not adaptive but better than typical routine explainers because it provided an appropriate explanation script and asked if human listeners had questions after speaking every segment. Second, regarding its expertise in listening, we also evaluated it as not adaptive but slightly better than typical routine listeners because it attempted to move its head toward human explainers and make eye contact with them.
Group A
Group B
Group C Figure 2. Characterization of Jigsaw Groups Based on Students Expertise in Collaboration. Our exploratory analysis of students expertise in collaboration and their interaction with Robovie-W produced the following results. First, as expected, there were a large variety of differences in group characterization of collaboration expertise. In Group A the majority of students (three) were identified as adaptive explainers and listeners. Group C had the completely opposite characterization. Group B exhibited large individual differences. The group differences in characterization gave us an opportunity to examine how human collaboration expertise influenced the interaction with artificial agents as learning partners. The style of students interactions with Robovie-W was quite similar to that shown in their interactions with human collaborators. Adaptive listeners attempted to actively communicate with Robovie-W during its explanation, engaging in collaborative discourse as if communicating with a human explainer. While giving an explanation, adaptive explainers were likely to monitor the response (movement) of Robovie-W, although it simply moved its head to face them. Thus, interaction with Robovie-W was more productive when human collaborators were adaptive collaborators. Our results suggest that we need to consider designing further supports, particularly for less adaptive collaborators, so that students can productively communicate with robots as learning partners. There are a
ISLS
185
ICLS2012
couple of hints for the design found in adaptive explainers and listeners behaviors. First, we could identify several strategies used by the adaptive explainers. They were more engaged in metadiscourse or metacognitive comments for welcoming listeners to collaborative discourse. The adaptive explainers also attempted to regulate group learning activities by asking listeners to read their prepared summary, telling them what to read and when and how to read it. Use of these types of instructional scripts should be considered depending on the levels of learners collaboration expertise. Second, we think that adaptive listening behaviors could be improved in Robovie-W. In the present study, it just moved its head toward speakers. To augment this behavior, we could further prepare question scripts for articles based on our analysis of how students understood the ideas contained in them. Robovie-W might help human listeners comprehend explanations more deeply by asking appropriate questions to explainers as well as listeners.
References
Biswas, G., Katzlberger, T., Bransford, J., & Schwartz, D. (2001). Extending intelligent learning environments with teachable agents to enhance learning. In J. D. Moore, C. L. Redfield, & W. L. Johnson (Eds.), Artificial Intelligence for Education 01 (pp. 389-397). Amsterdam: IOS Press. Biswas, G., Leelawong, K., Schwartz, D., & Vye, N. (2005). Learning by teaching: A new agent paradigm for educational software. Applied Artificial Intelligence, 19, 363-392. Blair, K., Schwartz, D. L., Biswas, G., & Leelawong, K. (in press). Pedagogical Agents for Learning by Teaching: Teachable Agents. Educational Technology. Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds.) (1999). How people learn: Brain, mind, experience, and school, Washington, DC: National Academies Press. Brown, A. L. & Palincsar, A. S. (1989). Guided, cooperative learning and individual knowledge acquisition. In L. B. Resnick (Ed.), Knowing, learning, and instruction: Essays in honor of Robert Glaser. Lawrence Erlbaum, Mahwah, NJ. Chin, D. B., Dohmen, I. M., Cheng, B. H., Oppezzo, M. A., Chase, C. C., & Schwartz, D. L. (in press). Preparing Students for Future Learning with Teachable Agents. Educational Technology: Research and Development. Goodrich, M. A. & Schultz, A. C. (2007). HumanRobot Interaction: A Survey. Foundations and Trends in HumanComputer Interaction, 1(3), 203275. Hatano, G. & Inagaki, K. (1986). Two courses of expertise. In H. A. H. Stevenson, & K. Hakuta (Eds.), Child development and education in Japan (pp. 262272). New York, NY: Freeman. Miyake, N. (2008). Conceptual change through collaboration. In S. Vosniadou (Ed.), Handbook of research on conceptual change. London, Taylor & Francis Group. Miyake, N., Ishiguro, H., Kanda, T. & Shirouzu, H. (2011). Robotics for CSCL. Preconference Workshop at CSCL2011, Hong Kong. Miyake, N. & Shirouzu, H. (2006). A collaborative approach to teach cognitive science to undergraduates: The learning sciences as a means to study and enhance college student learning. Psychologia, 18(2), 101113. Oshima, R. & Oshima, J. (2011, April). Knowledge building for pre-service teachers through collaborative reading comprehension. Paper presented at the Annual Meeting of the American Educational Research Association, New Orleans, LA. Pressley, M. & Afflerbach, P. (1995). Verbal Protocols of Reading: the Nature of Constructively Responsive Reading. Hillsdale NJ: Lawrence Erlbaum. Sawyer, K. R. (Ed.) (2006) The Cambridge handbook of the learning sciences. New York: Cambridge UP. Schwartz, D. L., Bransford, J. D., & Sears, D. (2005). Efficiency and innovation in transfer. In J. Mestre (Ed.), Transfer of learning: Research and Perspectives. Information Age Publishing. Schwartz, D. L., Chase, C., Chin, D. B., Oppezzo, M., Kwong, H., Okita, S., Biswas, G. (2009). Interactive Metacognition: Monitoring and Regulating a Teachable Agent. In D.J. Hacker, J. Dunlosky, & A.C. Graesser (Eds.), Handbook of Metacognition in Education. New York, NY: Routledge.
Acknowledgments
This study was supported (partly) by MEXT Japan Grant-in-Aid for Scientific Research on Innovative Areas No. 4101-21118001 (granted to Naomi Miyake, U. of Tokyo).
ISLS
186
ICLS2012
Multi-touch technology to support multiple levels of collaborative learning in the classroom

Emma Mercier, Steve Higgins, Elizabeth Burd, Andrew Joyce-Gibbons Durham University School of Education, Leazes Road, Durham, DH1 1TA, UK. emma.mercier; s.e.higgins; liz.burd; andrew.joyce-gibbons @durham.ac.uk Abstract: While much research on collaborative learning has focused on what goes on within groups to lead to success, in this paper, we consider the classroom context within which collaborative learning occurs. Drawing on concepts of classroom orchestration and repertoires of collaborative practices we discuss how multi-touch technology can be used to support collaborative learning in classroom settings. Data drawn from three classes of sixteen students, organized in four groups of four, is examined to identify how the movement between small group and whole class interaction supports the collaborative engagement, finding that the whole-class discussion prompts a move to higher levels of reasoning within the groups.
Introduction
While research has repeatedly shown the value of collaboration to support knowledge building, problem solving and learning (see, Barron & Darling-Hammond, 2008) evidence indicates that this form of learning is rarely, if ever, used in actual classrooms (Baines, Rubie-Davies, & Blatchford, 2009). However, the development of multi-touch technology (Dillenbourg & Evans, 2011; Higgins, et al., 2011) combined with efforts to understand the importance of fully integrating technology into the classroom environment suggests the possibility of more easily supported collaboration in classrooms, leading to more frequent use of this pedagogic strategy. Current research into the use of multi-touch technology for learning has largely been limited to an exploration of groups working on a single table, rather than the interaction of multiple devices in a classroom. This research indicates that multi-touch tables can lead to more task focused, and less process focused talk than single-touch tables (Harris et al., 2009), that they can both better support the development of a joint problem space (Higgins et al., in press) and increased communication and cohesion (Evans, et al. 2011). These findings suggest that multi-touch technology can be used effectively to support the interaction of groups during a learning activity, and the potential of such tools to be integrated into formal learning environments. While much research has focused on the role of new technologies to support learning, rarely has the work considered the implications of the changes these technologies can bring to classroom communities, and the necessity of more widespread change to the classroom norms to take full advantage of these new tools. Slotta, (2010) suggests that the relative lack of uptake in new forms of pedagogy, despite consistent research that shows it is beneficial to learners, is due to the profoundly disruptive effect these pedagogies have on classroom communities. He proposed that new forms of technology could be used to support the transition to more revolutionary approaches to learning and teaching in the classroom, with technology specifically designed to support some of the complexity of the learning environment. Dillenbourg & Jermann (2010) defined 14 features of classroom orchestration that need to be taken into account when embedding new technologies into classrooms. In our work, we consider the potential disruptive effect that both technology and collaborative learning can have on traditional classroom learning, and examine how this technology can be developed in such a way as to support the teachers orchestration of the classroom, while at the same time supporting the integration of this form of learning. The orchestration of collaborative learning requires attention to multiple levels of activity, including the individual participant, the interpersonal interactions, both in relation to the problem and relational spaces, the context of the learning and the community practices (Barron et al., 2009). In our project, we have designed a multi-touch integrated classroom, with four student multi-touch tables, a multi-touch interactive whiteboard and a teachers multi-touch desk which can be used to control the student tables and interactive whiteboard (see figure 1); the room is also equipped with multiple cameras and microphones. This allows us to examine the individual learner, the interaction within groups, the between group and whole class interactions and how the teacher manages the classroom. In this paper, we focus on how the movement between small group and whole class discussion influences the interaction and conversation within the groups.
Method
Three schools participated in this study of the multi-touch classroom. Sixteen students from each school consented to participate after completing introductory activities in their classroom. Students were in their final year of primary school, and were between 10 and 11 years old. Each school group came to the lab on a different day and stayed for up to five hours. They were taught by one of two former primary school teachers, who were members of the research team; one teacher taught two of these classes, while the second taught one class. For
ISLS
187
ICLS2012
two schools, students were seated in same-gender groups of four, while the students from the third school sat in mixed groups of two male and two female students in accordance with the overall project research design A number of activities have been designed to help the students become familiar with the tables and how to use them. After completing these activities, the students worked on a number of history and mathematics tasks. The data for this paper is drawn from the history data.
The task
The history task, which is described in detail elsewhere (Higgins et al., in press) was based upon an incident in a coalmine in 18th century north-east England. At the start of the task the teacher read aloud a statement about the accident, in which a 10 year old boy, Robert Dixon, lost his leg. The children then received 16 clues to help them determine what happened to Robert Dixon and who was responsible. This task is designed to encourage divergent argumentation, with multiple possible answers (adapted from Higgins, Baumfield & Leat, 2001) and is an example of a mystery pedagogical strategy (Leat and Higgins, 2002). Figure 1: The SynergyNet Multi-touch Classroom The task was introduced by the teacher who then unlocked the tables and allowed the groups to begin to make sense of the task by reading and connecting the pieces of information. The teacher then orchestrated the activity, moving the groups between small-group conversation and whole-class discussion to facilitate argumentation and problem solving.
Figure 2. The analysis tool
Data
Video was collected for each group, using cameras that are placed in the ceiling; a fish-bowl camera provides a view of the entire room. Each table has a microphone, and the teacher wore a lapel microphone during the class. The audio was transcribed verbatim, using the projects analysis tool (Fig 2). In this tool, transcripts are laid along a time-line, and each participant is represented as a separate line, into which their utterances are transcribed. Analysis is conducted directly onto the transcript, allowing coding to be examined over time.
Coding
The videos were coded using an adaptation of the SOLO taxonomy, which provides a hierarchy of reasoning complexity (Biggs & Collis, 1982; Moseley et al. 2005). This scheme was selected because it allows for an exploration of how the groups are making sense of the mystery, and the levels of relational complexity
ISLS
188
ICLS2012
emerging from the group. The coding scheme, and how it was adapted for use for the mystery, is shown in table 1. Table 1: SOLO taxonomy adapted for Robert Dixon mystery
Color Pre structural Unistructural
Turquoise Blue
Multi-structural Purple Relational Extended abstract Results
Red
Yellow
Definition Reading clues Reading clues, making brief comments in relation to value of the clues Comments on value of clues by referring to other clues within the task (or drawing on prior knowledge) Puts together an explanation of mystery, drawing on clues available Puts together an explanation of the mystery drawing both on clues available and prior knowledge
The history mystery lasted about half an hour (M = 26 minutes, 39 seconds; SD = 3 minutes, 7 seconds). In each class, the teacher introduced the activity and sent the clues to the four tables. The teacher stopped the groups twice during the task for a discussion and brought the class to an end with one final group discussion. In each instance, the focus of the first discussion was about how the group was going about the task, while the second class discussion was focused on what explanation the students were finding for the problem. Table 2 shows the highest level of reasoning reached within each group in each of the small-group discussion phases of the task. While the data shows that there were some groups who did not appear to engage with the task (e.g. Yadstone Yellow or Shadbrook Yellow), eight of the 12 groups moved onto a more complex form of reasoning during the task, reaching higher levels of reasoning in the final group time session. Five of the groups were able to offer an explanation who was responsible for Robert Dixons injuries, showing relational and extended abstract thinking. Table 2: Highest level of reasoning in each group in each time period Group Time 1 Group Time 2 Yadstone Red Prestructural Prestructural Yadstone Blue Multi-structural Multi-structural Yadstone Green Unistructural Unistructural Yadstone Yellow Unistructural Unistructural Benbrook Red Benbrook Blue Benbrook Green Benbrook Yellow Shadbrook Red Shadbrook Blue Shadbrook Green Shadbrook Yellow Unistructural Multi-structural Relational Unistructural Multi-structural Prestructural Multi-structural Unistructural Unistructural Multi-structural Relational Unistructural Multi-structural Prestructural Unistructural Unistructural Group Time 3 Unistructural Relational Multi-structural Prestructural Extended Abstract Extended Abstract Relational Unistructural Relational Unistructural Multi-structural Unistructural
In the examples that follow, segments of the coded transcript are displayed, to show how the coding scheme was applied, and identify the changes in discussion over time. One group from each school was selected to show the different patterns of change in their reasoning through the task (note the participants are identified by a letter and number at the start of their transcript line; in order to show as much group conversation, the introduction and final discussion have not been included in the images below). In the first example, Yadstone Blue, the students spend some of the first group time reading the clues aloud, without making substantial comments about their value (indicated by the turquoise pre-structural code). Towards the end of this first group time, we see one comment that is coded as multi-structural (purple) and one that is coded as unistructural (blue), indicating the group are beginning to make sense of the clues in relation to the problem. During the second group time, this group initially pick up some of their ideas from before the class time, but then go back to reading more clues, beginning to reason about them in more complex ways (identified by the purple codes) towards the end of group time two. This group struggle with some distraction at the beginning of the third group time, reading the timetable for the day off the whiteboard behind them, and discussing what time they have to return to school.
ISLS
189
ICLS2012
However, they return to the topic, reaching the relational level of reasoning (shown in red) in the last minute before the teacher calls the class to an end. The Benbrook Blue group show a more complex pattern of engagement than the Yadstone Blue group, with more students involved in the discussion (this can be seen from the more scattered pattern of colored utterances, as well as a greater density of utterances as a whole). As with most groups, some time is spent just reading the clues, but the Benbrook Blue group also begin discussion of the relevance and inter-relatedness of the clues during the first few minutes, as can be seen in the unistructural utterances (blue), made by pupil b27 at the beginning of group time 1. By the end of the first group time, pupils b26 and b27 are making multi-structural comments (purple), which b28 shared with the class during the class time between group time 1 and 2. During the second group time, the conversation is more balanced within the group, with pupils b25 and b28 contributing comments that were identified as multi-structural (purple). The group also returned to reading some of the clues to further build their argument (turquoise) and evaluated the new clues in light of their reasoning about the task (blue and purple codes). Again, b28 contributes to the whole class discussion; although it is not clear in this view, b28 puts forward a relational statement in answer to the teachers questions during the whole-class discussion. In the final, short group time, b27 brings together all the comments, making an extended abstract statement. Example 1: Yadstone Blue
Group Time 1 Example 2: Benbrook Blue
Group Time 2
Group Time 3
Group Time 1
Group Time 2
Group Time 3
Discussion
The process of collaboration using multi-touch surfaces in a classroom environment was explored in this paper. Drawing on the repertoires of collaborative practices framework (Barron et al, 2009), collaboration was viewed as taking place on a number of planes simultaneously; the analysis begins to look at the movement between two of these planes the interpersonal and the context and investigate how they might influence each other. The results indicated that eight of the twelve groups moved to higher levels of reasoning throughout the task, while two remained at the unistructural level throughout the task, one remained at the relational throughout and a final group and one moved from unistructural to prestructural in the final group time. The groups who did not improve during the task can be explained by examining the content of their interactions. Benbrook Green began discussing the task at a high level, moving quickly to the relational level, and continuing to return to that level of discussion across the three group sessions. In contrast, neither Benbrook Yellow or Shadbrook Yellow moved above the unistructural level, while Yadstone Yellow worked at the unistructural level for the first two group sessions, finishing at the prestructural level. When examined, all of these groups struggled with relational interaction issues, arguing over turn taking, levels of participation, and issues unrelated to the task, rather than attending to the activity beyond the most superficial of reading and beginning to make sense of the clues. The progressive improvement was expected, as students learned more about the task, however, the specific changes during and after the whole class interactions showed the influence of the classroom context on the groups. In almost all cases the students remain at the same level of engagement after the first whole-class discussion, where the teacher mainly focused on procedural issues. However, during the second whole-class discussion, the teachers asked the groups to describe what they thought had happened to Robert Dixon, and whose fault it had been. By projecting one of the tables content to the large whiteboard in the room, the groups explored whether they were making similar decisions to the other groups in their class. The projection also gave the groups a common reference to discuss their progress, and created a source of joint attention for the class.
ISLS
190
ICLS2012
The two examples provided a different perspective on how the groups functioned within the classroom. While there is more to be explored in terms of the interpersonal interactions within the groups, and the types of contributions made by each group member, the patterns of each group time, and how the groups are influenced by the whole-class discussion should be considered. The Yadstone Blue group make very few contributions to the whole class discussion, and yet reach the highest level of reasoning of any group from that school, and were likely the most on-task of the Yadstone groups throughout the task. They appear to be distracted from task when the whole-class discussions occur, reaching higher levels of reasoning towards the end of each group time, but not picking up at the same level after the whole-class discussion. In contrast to Yadstone Blue, Benbrook Blue appear to have been able to use the whole class time to move their thinking forward, particularly as their self-selected spokesperson, b28, used the whole-class discussion as a place to report higher levels of reasoning than he had actually contributed within the group. This group also showed a more balanced range of contributions from all group members, with everyone making comments that were coded as unistructural and multi-structural at some stage during the activity. Across the twelve groups, the whole-class discussion proved to have both a positive and negative effect on the groups. At times, stopping the group task causes disruption to the group process, that the groups could not immediately recover from, while in other instances, the whole class discussion appears to have prompted the groups to think about the task in more depth, pushing them into a higher level of reasoning, either during the class discussion, or when they returned to their group conversation. The data from this paper indicates a complex interaction between the small groups and whole class discussions, that is mediated, supported by, and possibly disrupted by, the teachers ability to lock the tables and project the content of one table to the whole class. The technology provides equal access to the content, although, as can be seen from the three examples, as well as the range of reasoning levels reached by the twelve groups, it does not guarantee equal participation or engagement with the task. While this paper has focused on the interpersonal and context levels of analysis of the groups, one feature of the data is what appear to be differences between the school groups. The groups from Benbrook reached higher levels of reasoning than the students in from the other two schools, suggesting that the students may have been bringing community practices of engagement in collaborative learning to the lab classroom, which altered how they interacted with each other, the technology and the task.
References
Baines, E., Rubie-Davies, C., & Blatchford, P. (2009). Improving pupil group work interaction and dialogue in primary classrooms: results from a year-long intervention study. Cambridge Journal of Education, 39(1), 95117. Barron, B., & Darling-Hammond, L. (2008). How can we teach for meaningful learning? In L. DarlingHammond (Ed.), Powerful Learning: What we know about teaching for understanding. Jossey-Bass. Barron, B. et al. (2009). Repertoires of collaborative practice. Proceedings of the 9th international conference on Computer supported collaborative learning - CSCL09, 25-27. ISLS. Biggs JB & Collis KF (1982) Evaluating the quality of learning - the SOLO taxonomy. Academic Press. Dillenbourg, P., & Evans, M. (2011). Interactive tabletops in education. International Journal of ComputerSupported Collaborative Learning, 6(4), 491-514. doi:10.1007/s11412-011-9127-7. Dillenbourg, P., & Jermann, P. (2010). Technology for classroom orchestration. In M. S. Khine & I. M. Saleh (Eds.), New Science of Learning (pp. 525-552). New York, NY: Springer New York. Evans, M. a, Feenstra, E., Ryon, E., & McNeill, D. (2011). A multimodal approach to coding discourse: Collaboration, distributed cognition, and geometric reasoning. International Journal of ComputerSupported Collaborative Learning, 6(2), 253-278. doi:10.1007/s11412-011-9113-0. Harris, A., Rick, J., Bonnett, V., Yuill, N., Fleck, R., Marshall, P., & Rogers, Y. (2009). Around the table: are multiple-touch surfaces better than single-touch for childrens collaborative interactions? Proceedings of the 9th international conference on Computer supported collaborative learning-Volume 1. ISLS Higgins, S.E., Mercier, E., Burd, E., & Hatch, A. (2011). Multi-touch tables and the relationship with collaborative classroom pedagogies: A synthetic review. International Journal of Computer-Supported Collaborative Learning, 6(4), 515-538. doi:10.1007/s11412-011-9131-y. Higgins, S.E., Mercier, E., Burd, E. & Joyce-Gibbons, A. (in press) Multi-touch tables and collaboartive learning. British Journal of Educational Technology. doi:10.1111/j.1467-8535.2011.01259.x Leat, D. & Higgins S. (2002) The role of powerful pedagogical strategies in curriculum development The Curriculum Journal 13.1 pp 71-85 ISSN 0958-5176. :doi:10.1080/09585170110115286. Moseley, D., Baumfield, V., Elliott, J., Higgins, S., Miller, J. and Newton D. P. (2005) Frameworks for thinking: a handbook for teaching and learning Cambridge: Cambridge University Press Slotta, J. (2010). Evolving the classrooms of the future: The interplay of pedagogy, technology and community. In K. Makitalo-Siegl, J. Zottmann, F. Kaplan, & F. Fischer (Eds.), Classroom of the Future: Orchestrating collaborative spaces (pp. 215242). Sense Publishers.
ISLS
191
ICLS2012
The SunBay Digital Mathematics Project: An Infrastructural and Capacity-Based Approach to Improving Mathematics Teaching and Learning at Scale
Charles F. Vanover & George J. Roy1, Zafer Unal, Vivian Fueyo University of South Florida St. Petersburg vanover@mail.usf.edu, royg@mail.usf.edu, unal@mail.usf.edu, vfueyo@usfsp.edu Phil Vahey, SRI International, philip.vahey@sri.com Abstract: This paper discusses the first year of the SunBay Digital Mathematics Project (SunBay Math) and the projects implications for educational reform. It reviews the research literature on the SimCalc replacement unit Managing the Soccer Team and describes the units implementation in a large, urban district. Findings from professional development experiences, teacher observations, and student gain scores are shared. Similar to the results published in Roschelle, Schectman, et al. (2010), there was a large and significant increase in mathematics understanding for SunBay Math students who learned in classrooms implementing Managing the Soccer Team in contrast to students in Texas control classrooms.
Introduction
This paper describes the first phase of a multi-year effort to raise the quality of middle grades mathematics instruction at scale in a large and diverse urban district. Its purpose is to describe the partnership of a college of education (the College), a nonprofit research organization (the Nonprofit), and a large, urban school district (the District) to provide intensive professional development (PD) to middle school mathematics teachers using computer-based, dynamic curricular materials designed to increase students abilities to engage in complex mathematics, particularly algebra, and to change the way District teachers teach. The SunBay Digital Mathematics Project (SunBay Math) is organized around an infrastructural and capacity-based perspective on instructional improvement. We define infrastructure as the tools teachers use to deliver instruction (Kaput & Hegedus, 2007; Roschelle, Tatar, & Kaput, 2008). These tools include the textbooks and paper workbooks that have become a familiar part of mathematics education in many classrooms around the world, as well as innovative technologies such as dynamic simulations and networks of handheld devices. We understand capacity as the human and organizational resources that enable school professionals to use these tools skillfully (Cohen & Ball, 1999; Massell, 2000). Capacity, in our perspective, exists in the systems of instructional leadership, networks of professionals, and other organizational resources supporting high quality teaching and learning (Fishman, 2006; Resnick, 2010; Stein & Coburn, 2008). SunBay Maths long term goal is to change what and how teachers teach by using dynamic technology and well-researched replacement units to augment the Districts middle grades mathematics curriculum and build the capacity to enable every teacher to use this infrastructure well.
The Research Supporting SunBay Maths Infrastructure

SunBay Maths major reform strategy is to replace critical parts of the local curriculum with replacement instructional units that address core mathematics content (see Tatar et al., 2008; Vahey, Roy, & Fueyo, in press). These units employ computer-supported dynamic representations of real world phenonmena (Kaput, 1994; Kaput & Hegedus, 2007), such as races between people, designed to support students efforts to learn algebra and other complex mathematics. The representations are executable (Hegedus, Moreno, & Dalton, 2007) and, consistent with recommendations from Hiebert and Grouws (2007), support students efforts to understand the meaning of the mathematics they study. The replacement learning unit discussed in this paper, Managing the Soccer Team (MST) (Knudsen, de Frondeveille, & Rafanan, 2003), organizes dynamic representations from SimCalc Mathworlds into lesson segments designed to foster understanding of ratio and proportionality. Algebraic symbols are introduced after students engage with graphs and other representations of motion phenomena in order to help students connect these big ideas with formal, mathematical notation (Vahey, Knudsen, Rafanan, & Lara-Meloy, in press). Field studies show MST is both effective and robust (Roschelle, Shechtman, et al., 2010; Tatar, et al., 2008). By effective, we mean capable of producing measurable gains in student learning and other important educational outcomes. By robust we mean the unit shows benefits for student learning when deployed consistently to a wide variety of students, teachers, and settings. (Roschelle, Tatar, Shechtman, & Knudsen, 2008, p. 151). The seventh grade experiment described by Roschelle, Shechtman, et al. (2010) recruited a randomly selected group of 140 teachers in 73 schools in Texas. Participants were divided, after attrition, into a Treatment group of 48 teachers, who used MST, and a Control group of 47 teachers, who taught similar content using their districts curriculum in a delayed-treatment design (Slavin, 2002). MST and Control teachers
ISLS
192
ICLS2012
received a two-day workshop focused on rate, proportionality, and other demanding mathematic topics. MST teachers were then given three extra days of PD on how to use the replacement unit and a fall workshop devoted to implementation planning. Gain scores taken from the pre- and post-tests showed an effect size of .63 favoring the Treatment group (those classrooms that used MST). There were no significant differences between the gains of girls, Hispanic students, and free-lunch students compared to the gains of other students; all achieved high gains (Vahey, Lara-Meloy, & Knudsen, 2009). Low achievers, however, did show lower gains compared to other students when, as Roschelle, Pierson, et al. (2010) show, these at-risk students were placed in classrooms where the teacher produced low average gains. Importantly, there was no difference in gains generated by classrooms that started out with low average pre-test scores compared to classrooms that began with high pre-test scores. Thus, given sufficient teacher quality, MST was effective in classrooms with many low-achieving students. In the next section we detail the first phase of SunBay Maths efforts to implement MST and improve the Districts mathematics instruction at scale. SunBay Maths theory of improvement begins by creating a new infrastructure for mathematics instruction in local classrooms and the College. We then build capacity outwards by improving teachers knowledge of mathematics instruction and implementing dynamic replacements units in breadth and depth (see SRI International & University of South Florida St. Petersburg, 2010).
Method
The investigation presented in this paper was conducted according to the principles of design-based research (DBR) (e. g. Cobb, Confrey, diSessa, Lehrer, & Schauble, 2003; Penuel, Fishman, Cheng, & Sabelli, 2011). In the DBR approach, emphasis is placed on examining a particular change program using both quantitative and qualitative methods in accords with a conceptual frame. Data are then mixed (Creswell, 2003) to deepen team members understanding of the projects larger research questions and to guide participants choices as the project unfolds. The goal is to construct a shared, empirically-based perspectivealbeit one with varying levels of agreementabout the phenomena under investigation, the most beneficial ways to implement and improve the design, and the implications of findings for reform and theory-building. Sources of data collected during the first year of SunBay Math, instruments used, and findings team-members generated are discussed in SRI International and University of South Florida St. Petersburg (2010).
Building Capacity by Improving Teaching and Learning: Findings from SunBay Maths First Year
The book does it that way, SimCalc does it this way. This way is better. Teacher discussing SimCalc software at August 2009, SunBay Math PD . In June 2009, the SunBay Math partners began working to create a comprehensive plan to implement digital mathematics replacement units in local schools, refine the teams understanding of specific needs within the District, and build implementation capacity at the College. The first unit the partners chose for implementation was MST. Following a suggestion in Fishman et al. (2009), a team of Florida experts was convened to ensure that the unit was aligned with Floridas state standards. The experts recommended mostly cosmetic changes and thus the version of MST used in SunBay Math is comparable to the unit Roschelle, Shechtman, et al. (2010) evaluated in Texas.
District Demographics and Teacher Recruitment

The District is located in a diverse urban area and is one of the largest in the State of Florida. It serves more than 100,000 students, of which more than 21,000 are in grades 6-8. Approximately 54% of the Districts students live in poverty and about 40% are members of minority groups primarily Black (19%), and Hispanic (9%). The District manages 21 middle schools and employs approximately 250 middle school mathematics teachers. On June 2, 2009, the Districts secondary mathematics supervisor sent emails to principals at 10 District middle schools asking leaders to select two 7th grade teachers from each school to participate in SunBay Math. Three of the original schools ultimately declined and were replaced with larger and lower SES schools. Fifteen 7th grade teachers at 7 District schools ultimately participated in SunBay Maths initial, summer PD session. All teachers received District PD credits and compensation for their time. Of the 15 trained teachers, 13 implemented the unit. The participating teachers varied in age from 26 to 61 years and ranged in experience from two to 24 years teaching. All were certified to teach mathematics, although most had previous careers outside of education and had received certification through alternative routes. Some were mathematics leaders within their schools, 3 were department heads, others said their principals volunteered them for the program.
August 2009: 3 Day Professional Development for MST

We have a hard time teaching children why is the slope like this, or why is the slope like that. Now, I can say what that means. Teacher at August 2009, SunBay Math PD.
ISLS
193
ICLS2012
Professional developers from the Non-Profit joined faculty from the College to help deliver a 3-day PD organized similar to the Texas PD experience described in Roschelle, Shechtman, et al. (2010). Teachers were asked to experience the unit as learners, and they worked with the professional developers to solve the mathematical problems posed by the unit. As discussed in Roy, Vanover, Fueyo, and Vahey (in press), SunBays initial PD trainings were organized around 3 norms. These were: Do the math. Make bold conjectures. Think about your classrooms. Teachers completed a workshop evaluation form and rated the PD on a Likert scale of 1 to 5 on characteristics such as pacing, organization of materials, and the extent the PD prepared them to teach the digital mathematics units. Teachers reported high levels of satisfaction; the average rating was 4.6 out of 5. During an opportunity for the teachers to discuss the workshop at the end of the third day, teachers praised the hands-on nature of the trainings and power of the dynamic unit. One teacher said, Ive been to a lot of trainings where I couldnt see it. [SunBay Math] helped me do it. I could see what I was doing and could imagine what I might do in my classroom. Another said, Whats great about this, is I dont have to develop all the parts of the lessons myself. SimCalc is already built. I can concentrate on teaching the lesson rather than doing the lesson.
Fall 2009: Monthly Evening Professional Development

To support implementation, teachers were provided with a 3-hour PD session once a month from September through December. Notes show that a majority of time during these trainings was spent working on key lessons, discussing the mathematical content embedded in the software and workbook, and brainstorming how best to use tools embedded in the units to support student achievement. Each session began with a warmup mathematics question designed to get teachers thinking deeply about rate and proportionality. Teachers provided an average rating over all workshops of 4.4 out of 5. In phone interviews conducted in December and January, teachers appreciated the professional community that grew out of these sessions. One teacher said, I liked seeing where the unit was coming from. Going deeper with questioning strategies. Another said, I liked the fact that they wereopening us up to ideas and strategies that I had forgotten or never learned. I didnt learn stuff just about SimCalc, but something about being a better teacher.
Implementation of MST: October-December 2009

SunBay Math team members from the College provided technical assistance for the 13 teachers to implement MST in their classrooms in the Fall of 2009. One key form of support was surveying the 7 schools technology capabilities, meeting with technology and media staff, and working with the teachers to ensure that computers and SimCalc MathWorlds software was available for the unit. College personnel logged more than 150 hours of time calling principals and technology coordinators ensuring teachers had access to computer labs or laptop carts, installing software, and troubleshooting technology dilemmas. Twelve of the 13 teachers were observed during one of MSTs major lessons, On the Road. The lessons implementation might be described as variation within a theme. All teachers used the workbook to guide instruction. For example, while some engaged in pre-lesson reviews and examples, all had students read On the Roads framing scenario, and then move through the workbook problems in order. There was wide variation, however, within this orchestration. Teachers varied in the pace of activities. Some, particularly those who had been identified by the team as less skillful technology users, moved quite slowly through the materialin these classrooms, just passing out computers and getting software loaded could take 10 minutes. The team also observed a wide variety of classroom structures, both in terms of physical layout and the types of activities students engaged in when using computers. Some teachers tended to group students in pairs or small groups focused on a particular computer; others focused predominantly on whole-class activities and projected MST materials to the class using a LCD projector. While it may intuitively make sense that particular computer uses lead naturally to particular types of instruction, our observations show that this is not necessarily the case. Some teachers led inquiry-based whole class conversations using a single laptop and a LCD projector. Conversely, other teachers passed out enough laptops to enable students to work in pairs around a single computer, but used that infrastructure to lead the class in a directed way with little mathematical conversation. This variability replicates the Nonprofits experience in Texas, where teachers adapted the use of the software to the mode they had available or with which they were most comfortable.
Results: Student Learning from MST and Comparisons to Texas

We compared the learning gains of the District students who used the MST unit (n = 246) with the learning gains of those students in the Texas study (n = 825) who used a slightly different version of the unit using the assessment instrument from Roschelle, Shechtman, et al. (2010). Figure 1 shows there was no statistically significant difference between the average pretest score of the SunBay students and the students in the Texas study (both Treatment and Control). Figure 2 shows the spread of mean gain scores for three
ISLS 194
ICLS2012
populations of students: Texas Control students (who did not use MST); Texas SimCalc students; and SunBay SimCalc students. Gain scores were calculated by subtracting students pretest score from their posttest scores. The SunBay gain scores were almost identical to the Texas SimCalc gain scores, and both are significantly greater than the Texas Control gain scores. There was a large and significant main effect size for SunBay students in contrast to Texas Controls. Figure 2 shows teachers who did not use the Digital Mathematics materials had limited learning gains (indicated by the large number of Control teachers grouped to the left of the graph). This graph also shows that all teachers who used MST had higher learning gains than half the control teachers, and all Florida teachers who used MST had higher learning gains than approximately two-thirds of the Controls. Other analyses detailed in SRI International and University of South Florida St. Petersburg (2010) indicates that MST is effective regardless of student ethnicity or prior math knowledge. Figures 1 & 2: Texas Pretests and Gain Scored Compared to SunBay Math.
Discussion: Infrastructure First

Proposals to improve U. S. schools commonly emphasize the importance of increasing capacity before change occurs in classrooms. The logic of that approach appears to be that if only districts and schools would hire higher quality teachers, create coherent systems of assessment and instructional guidance, improve the practice of local leadership, or engage in sustained content-aligned PD, then classroom teaching might improve (e. g. Bryk, Camburn, & Louis, 1999; Desimone, Porter, Garet, Yoon, & Birman, 2002). SunBay Math illustrates an alternative approach, in which classroom practice is improved as a first step in increasing district capacity. Using resources available in almost any U.S. district3 days of summer PD, monthly trainings, technical assistance from a college of education, and a well-researched replacement unitthe partners were able to effectively increase student learning in core content specified by state standards. Our findings show that, within this approach, MST was highly robust. Teachers used MST in classrooms in which every student had a net-book computer, and teachers used MST in classrooms in which the SimCalc MathWorlds software was projected on a screen from a laptop students never touched. Even the SunBay Math teacher with the lowest gain scores outperformed more than 66% of the control teachers in Texas, and 6 SunBay Math teachers outperformed the top 99% of the Texas Controls. Rather than waiting for a pool of reform-oriented teachers to develop in order to achieve the goals outlined in current national mathematics standards, SunBay Math is using infrastructure from SimCalc and a college of education to build this capacity one teacher at a time.
Endnotes
(1) Co-first authors.
References
Bryk, A., Camburn, E., & Louis, K. S. (1999). Professional community in Chicago elementary schools: Facilitating factors and organizational consequences. Educational Administration Quarterly, 35(December), 751-781. Cobb, P. A., Confrey, J., diSessa, A. A., Lehrer, R., & Schauble, L. (2003). Design experiments in educational research. Educational Researcher, 32(1), 9-13. Cohen, D. K., & Ball, D. L. (1999). Instruction, capacity and improvement (CPRE Research Report Series No. RR-43). Philadelphia, PA: Consortium for Policy Research in Education: University of Pennsylvania. Creswell, J. W. (2003). Research design: Qualitative, quantitative, and mixed methods approaches (2nd ed.). Thousand Oaks: Sage Publications. Desimone, L., Porter, A., Garet, M., Yoon, K., & Birman, B. (2002). Does professional development change teachers' instruction? Results from a three-year study. Educational Evaluation and Policy Analysis, 24, 81-112. Fishman, B. J. (2006). It's not about the technology. Teachers College Record, 2006.
ISLS
195
ICLS2012
Fishman, B. J., Penuel, W., Hegedus, S., Moniz, R., Dalton, S., Brookstein, A., et al. (2009). What happens when the research ends? Factors related to the sustainability of a research-based innovation. Melrose Park, CA: SRI International. Hegedus, S., Moreno, L., & Dalton, S. (2007). Technology that mediates and participation in mathematical cognition. Paper presented at the 5th Congress of the European Society for Research in Mathematics Education (CERME). Hiebert, J., & Grouws, D. A. (2007). The effects of classroom mathematics teaching on students learning. In F. K. Lester (Ed.), Second handbook of research on mathematics teaching and learning. Charlotte, NC: Information Age. Kaput, J. (1994). Democratizing access to calculus: New routes using old roots. In A. Schoenfeld (Ed.), Mathematical thinking and problem solving. Hillsdale, NJ: Erlbaum. Kaput, J., & Hegedus, S. (2007). Technology becoming infrastructural in mathematics education. In R. Lesh, E. Hamilton & J. Kaput (Eds.), Foundations for the Future in Mathematics Education. Mahwah, NJ: Lawrence Erlbaum Associates. Knudsen, J., de Frondeveille, T., & Rafanan, K. (2003). Managing the soccer team: A 7th grade unit on rate and proportionality. Melrose Park, CA: SRI International. Massell, D. (2000). The district role in building capacity: Four strategies (No. CPRE Policy Briefs: RB-32September 2000). Philedelphia, PA: Graduate School of Education, University of Pennsylvania. Penuel, W. R., Fishman, B. J., Cheng, B. H., & Sabelli, N. (2011). Organizing research and development at the intersection of learning, implementation, and design. Educational Researcher, 40(7), 331-337. Resnick, L. B. (2010). Nested systems for the thinking curriculum. Educational Researcher, 39(3), 189-197. Roschelle, J., Pierson, J., Empson, S., Schectman, N., Dunn, M., & Tatar, D. (2010). Equity in scaling up SimCalc: Investigating differences in student learning and classroom implementation. Paper presented at the 9th International Conference of the Learning Sciences, Chicago. Roschelle, J., Shechtman, N., Tatar, D., Hegedus, S., Hopkins, B., Empson, S., et al. (2010). Integration of technology, curriculum, and professional development for advancing middle school mathematics: Three large-scale studies. American Educational Research Journal, 47(4), 833878. Roschelle, J., Tatar, D., & Kaput, J. (2008). Getting to scale with innovations that deeply restructure how students come to know mathematics and science. In E. A. Kelly & R. Lesh (Eds.), Handbook of design research in mathematics, science and technology education. Mahwah, NJ: Erlbaum. Roschelle, J., Tatar, D., Shechtman, N., & Knudsen, J. (2008). The role of scaling up research in designing for and evaluating robustness. Educational Studies in Mathematics, 68(2), 149-170. Roy, G., Vanover, C., Fueyo, V., & Vahey, P. (in press). Providing professional support to teachers that are implementing a middle school mathematics digital unit. Contemporary Issues in Technology and Teacher Education. Slavin, R. (2002). Evidence-based education policies: Transforming educational practice and research. Educational Researcher, 31(7), 15-21. SRI International & University of South Florida St. Petersburg. (2010). SunBay Digital Math: Pilot year 1 final brief. Menlo Park, CA: SRI International. Stein, M. K., & Coburn, C. (2008). Architectures for learning: A comparative analysis of two urban school districts. American Journal of Education, 114, 000. Tatar, D., Roschelle, J., Knudsen, J., Shechtman, N., Kaput, J., & Hopkins, B. (2008). Scaling up technologybased innovative mathematics. Journal of the Learning Sciences, 17, 248286. Vahey, P., Knudsen, J., Rafanan, K., & Lara-Meloy, T. (in press). Curricular activity systems supporting the use of dynamic representations to foster students deep understanding of mathematics. In C. Mouza & N. Lavigne (Eds.), Emerging Technologies for the Classroom. New York: Springer. Vahey, P., Lara-Meloy, T., & Knudsen, J. (2009). Meeting the needs of diverse student populations: Findings from the Scaling Up SimCalc Project. Paper presented at the Proceedings of the 31st annual meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education. Vahey, P., Roy, G., & Fueyo, V. (in press). Sustainable use of dynamic representational environments: Toward a district-wide adoption of SimCalc-based materials. In S. Hegedus & J. Roschelle (Eds.), Democratizing Access to Important Mathematics through Dynamic Representations: Contributions and Visions from the SimCalc Research Program. New York: Springer.
Acknowledgments
This paper is based on work supported by the Helios Education Foundation, the Pinellas County Schools, and the Pinellas Education Foundation. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the funders. We also recognize Susan Holderness and Nicole Collier at the University of South Florida St. Petersburg and Jennifer Knudsen, Teresa Lara-Meloy, and Ken Rafanan at SRI International for their many hours of work on SunBay Math.
ISLS
196
ICLS2012
Measuring Sophistication of Epistemic Beliefs Using Rasch Analysis

Johan L. H van Strien, Monique Bijker, Saskia Brand-Gruwel, Henny P. A. Boshuizen Centre for Learning Sciences and Technologies, Open University of the Netherlands Valkenburgerweg 177 6419 AT, Netherlands Johan.vanStrien@ou.nl, Monique.Bijker@ou.nl, Saskia.Brand-Gruwel@ou.nl, Els.Boshuizen@ou.nl Abstract: Measuring epistemic beliefs is challenging from a conceptual as well as a methodological point of view. Conceptually, it is hard to distinguish between nave and sophisticated beliefs, and methodologically, common statistical techniques yield inconsistent results. In response to these challenges, a new instrument to measure high school students epistemic beliefs was designed and validated. The 55-item instrument contained three scales (absolutism, multiplism, evaluativism), that reflected different stages in epistemic development (Kuhn, 1991; 1999), moving from nave to sophisticated beliefs. Rasch analysis was used to analyze the data. The analysis confirmed the existence of the three scales with moderate reliability coefficients.
Introduction
Epistemic beliefs, or learners beliefs about knowledge and knowing (Hofer & Pintrich, 1997), have received much attention in the learning sciences. The influence of epistemic beliefs on learning is well-documented (e.g., Muis, 2007; Schommer, 1990), but measurement of these beliefs is a challenge, from a conceptual as well as a methodological point of view. It has been argued that disappointing results should lead researchers to question not only the instruments they use, but also the underling theoretical frameworks as well as assumptions of psychometrics (Duell & Schommer-Aikins, 2001). Aim of this study was to do just that. A new measure of epistemic beliefs for high school students was designed and validated using a modelling technique with promising properties.
Conceptual issues: Sophistication of epistemic beliefs

An important issue in studies on epistemic beliefs concerns the level of sophistication of such beliefs. In many theories of personal epistemology, sophisticated learners are seen as independent and active information seekers who make up their minds independently from authorities. However, most judgments about knowledge claims are not based on personal experience, but rather on information acquired from other sources. Therefore, relying on experts does not necessarily reflect less sophisticated beliefs (Bromme, Kienhues, & Stahl, 2008; Elby & Hammer, 2001). Indeed, previous work has shown that believing in knowledge as personal construction rather than relying on experts may not always be an adaptive strategy in complex reading tasks, especially when readers lack prior knowledge (Brten, Strms, & Samuelstuen, 2008). Others have argued that believing in scientific knowledge as tentative and evolving which is typically considered to reflect sophistication as well does not correspond to reality in some scientific domains (Elby & Hammer, 2001). Developmental, or stage models, of epistemic beliefs focus on the development from nave to sophisticated beliefs and may therefore provide a solution to the challenges with respect to the issues of sophistication described above. An example is the model by Kuhn (1991), in which three stages are distinguished: Absolutism, multiplism, and evaluativism. People at the absolutist level see knowledge as certain and absolute, either transferred by experts or from direct observation. At the multiplist level, which becomes prevalent at adolescence (Kuhn, 1999), people become aware of the uncertain and subjective nature of knowledge and knowing. However, this awareness overrules any objective standards for the evaluation of information. As a consequence, all opinions are deemed equally valuable and right. At the evaluativist level, both previous stages come together. Uncertainty is acknowledged, but without ignoring the importance of evaluation of knowledge claims. That is, two positions can both be right, but one can be better supported by evidence and argumentation, making it more valuable. This makes that evaluativists are considered to have the most well developed epistemic beliefs (Nussbaum, Sinatra, & Poliquin, 2008).
Methodological issues: Instruments and analysis

These conceptual issues are directly related to measurement of epistemic beliefs. In paper-and-pencil instruments (e.g., Schommer, 1990) high scores on measures are interpreted to reflect sophisticated beliefs, regardless of stage of epistemic development. More importantly, it has been argued that research on epistemic beliefs is not based on a strong psychometric foundation (DeBacker, Crowson, Beesley, Thoma, & Hestevold, 2008). Validation of epistemic beliefs questionnaires is typically done with factor analysis of Likert-scale data (Wood & Kardash, 2002), but it has proved difficult to replicate the factor structures of widely used instruments
ISLS
197
ICLS2012
across samples (Clarebout, Elen, Luyten, & Bamps, 2001; DeBacker et al., 2008). Replication and validation studies have also shown large amounts of error variation and low reliability coefficients (DeBacker et al., 2008). Therefore, new instruments that can distinguish between sophisticated and nave epistemic beliefs are necessary. However, stage models of epistemic development are typically based on interviews (Hofer, 2004). Such epistemic interviews have the advantage of being capable of capturing beliefs regarding the source of knowledge and justification for knowing, with which existing questionnaires generally have difficulties (Hofer, 2004). However, although such interviews can provide valuable and detailed insights into learners epistemic development, the use of paper-and-pencil instruments enables researchers to include larger numbers of participants in their studies and to link epistemic beliefs to learning outcomes (Hofer & Pintrich, 1997). An existing paper-and-pencil instrument that does distinguish between absolutists, multiplists, and evaluativists was designed by Kuhn, Cheney, and Weinstock (2000). This instrument consists of 15 items in which two contrasting statements are described. Participants have to answer the question whether only one could be right (absolutist), that both can be equally right (multiplist), or that both can be right, but one more than the other (evaluativist). However, this answering structure makes it difficult to assess nuances.
Research aim
Although others have attempted to resolve measurement problems by coming up with new or adapted instruments as well (see DeBacker et al., 2008), use of accurate modelling techniques is equally essential. Therefore, a novel approach was taken validating the instrument with a more advanced model than factor analysis: The Rasch model (Rasch, 1960). The Rasch model can assist in modelling invariant, test- and sample free, interval person and item measures. If raw ordinal person responses on sets of items composed of Likert scales fit the Rasch model, the model can mathematically transform the ordinal measures into interval measures for persons and items. The model positions persons and items on the same interval scale, making a distinction between items that are easy or difficult to endorse. As such, this can provide additional information on what epistemic beliefs make individuals more sophisticated and which ones do not. These are important advantages over factor analysis, making Rasch modelling a very accurate analysis method.
Method
Participants
Participants were 509 high school students from 11th grade (257 boys; 252 girls, Mage = 16.64 years, SD = 0.68) from five schools for pre-university education in the South of the Netherlands.
Instrument
A Dutch-language questionnaire was developed based on the stages of absolutism, multiplism, and evaluativism. It was aimed at dealing with (conflicting) scientific claims and information, and with scientists in general. Items were partly based on existing materials (Conley, Pintrich, Vekiri, & Harrison, 2004; Schommer, 1990; Schraw, Bendixen, & Dunkle, 2002; Wood & Kardash, 2002), either in original form or modified, but also featured newly developed content based on theories of epistemic development models (e.g., Kuhn, 1991), and a pilot study among Dutch high school students. The questionnaire consisted of 55 items, of which 19 were hypothesized to reflect absolutist beliefs, 19 multiplist beliefs, and 17 evaluativist beliefs. Participants rated their agreement or disagreement on a 6-point Likert scale (1 = completely disagree, 6 = completely agree). Table 1 presents a number of sample items for each scale. Table 1: Sample items per level, translated into English from Dutch Absolutistist Multiplistist By now, scientists know Personal opinions are just as everything there is to know valuable as scientific knowledge I find it unreliable when scientists give contradictory information on a particular topic Studies that do not provide clear conclusions are useless A single study is enough for scientists to get a good impression of a particular issue Scientists will never know anything for sure, no matter how hard they try I do not believe that something can be proven indisputably Scientists will never agree with one another Evaluativist I always wonder on which information scientists base their conclusions You only have the right to speak when you know very much about a particular topic What is right in one situation, does not need to be so in another I attach much more value to research data than to personal experiences and ideas
ISLS
198
ICLS2012
Procedure
Participants filled in the paper-and-pencil questionnaire during regular class hours at their schools. They were given a full class hour to complete the questionnaire, and were explicitly requested to fill it in individually and honestly. Participation was made compulsory by the schools in order to avoid convenience samples.
Analysis
The Rasch polytomous "Grouped response-structure" model (Rasch, 1960; Linacre, 2009b) was employed to construct Rasch interval measures for the epistemic beliefs. Data analysis was performed in WINSTEPS (Linacre, 2009a).
Results
Scale characteristics
First, the responses on the three sets of items of absolutist, multiplist, and evaluativist items were analyzed in the Rasch group response model. Scale diagnoses displaying category infit and outfit mean square statistics (model quality indicators), and measure-to-category and category to measure statistics (category measurement quality indicators) revealed that a good quality interval scale could only be obtained if the items in the absolutist and multiplist scales were recoded into dichotomous scales, and if the items of the evaluativist scales were recoded into a three-point scale. However, the structure of the two dichotomous scales, the absolutist and multiplist scales, was different. In the absolutist scale, the original six categories 1-3 and 4-6 were collapsed into a 1-4 dichotomous scale. The original categories 1-3 and 4-6 of the multiplist scale were collapsed into a 3-5 dichotomous scale. Finally, in the evaluativist scale, the original categories 1-3 were collapsed in category 3, category 4 remained, and the categories 5-6 were collapsed into category 5. The scales of absolutism, multiplism, and evaluativism were construct validated in the Rasch model. Of the original 55 items, nine items did not fit the group structure model (three absolutist items, one multiplist item, and five evaluativist items). As Table 2 shows, Cronbachs alphas are moderate for the three separate dimensions (.60-.62), and reasonable for the scale as a whole (.73). Item reliabilities were very high (.99-1.00), indicating that the ordering from items that are relatively easy to endorse to items that are more difficult to endorse is very stable and that this ordering should be replicable across other, comparable samples (Bond & Fox, 2007). Table 2: The three Rasch validated dimensions of epistemic beliefs Description Number of Cronbachs Item reliability items Absolutist Multiplist Evaluativist 16 18 12 .62 .62 .60 .99 .99 1.00 Dichotomous 1-4 Dichotomous 3-5 Three-point scale 3-4-5 Scale structure
Table 3 displays the means and standard deviations in logits (the unit of measurement in the Rasch model). One outlier was removed from the analysis. The absolutist scale is the most difficult one to endorse, whereas the evaluativist scale is relatively easy to endorse. Table 3: Means and standard deviations (n = 508) Dimension Mean Absolutist Multiplist Evaluativist Total -0.58 0.07 0.44 -0.06
SD 0.45 0.48 0.78 0.28
Zero-order correlations showed that the absolutist and multiplist scales were independent (r = .07, ns), and that the absolutist and evaluativist (r = .14, p < .01), and multiplist and evaluativist scales (r = .19, p < .01) were weakly correlated.
ISLS
199
ICLS2012
Item ordering
As mentioned, Rasch analysis positions persons and items on the same interval scale, making a distinction between items that are more difficult and easier to endorse. As such, it is possible to see which items are more indicative of high performance on a certain scale. For the absolutist scale, the item most difficult to endorse was By now, scientists know everything there is to know (measure = 1.11). As a result, this item is indicative of a high level of absolutism and thus, only students with high levels of absolutism would endorse it. By contrast the absolutist items easiest to endorse were I find that people have to choose explicitly whether they are for or against something (-0.78) and I find it unreliable if scientists give contradictory information on a particular topic (-.0.75). For the multiplist scale, the most difficult items to endorse were Things are never as complex as scientists want you to believe (0.79) and What is true, is only a matter of opinion (0.75), making these most indicative of multiplist beliefs. The multiplist item easiest to endorse was I am perfectly capable of acquiring reliable knowledge on new topic on my own (-0.89). Finally, the evaluativist items most difficult to endorse and thus reflecting a high level of sophistication were I always deliberately search for information that does not correspond with my beliefs (2.31), and I attach much more value to research data than to personal experiences and ideas (1.54). The easiest evaluativist items were Knowledge can always change as a result of new insights (-2.12), and What is right in one situation, does not need to be so in another (-1.68).
Conclusion and discussion

The analysis showed that the three hypothesized dimensions of absolutism, multiplism, and evaluativism could be distinguished. Reliability was moderate for the three separate dimensions, and satisfactory for the entire epistemic beliefs scale. However, in the current study the person internal consistency measures of scales with only two or three categories came up to similar levels as previous instruments with more categories. Fewer categories always lead to lower person reliability estimates. Hence, even though internal person consistency measures are still modest due to category loss, the current study with the Rasch model has provided the insight that respondents do not distinguish more than two or three different categories in the items. Consequently, although person reliability coefficients are not yet sufficiently high, the use of Rasch measurement rather than exploratory factor analysis is a step forward with respect to measurement and construct validity. It should also be kept in mind that internal consistency does not necessarily imply unidimensionality, and that reliability estimates may also have been lowered due to the relatively homogenous sample in the present study (cf. Wood & Kardash, 2002). However, item reliabilities were very high, suggesting replicability of the item ordering across comparable samples. There was a weak correlation between the multiplist and evaluativist scales, and between the absolutist and the evaluativist scales. This implies that the different levels may not necessarily reflect separate stages, as suggested by Kuhn (1999) and other stage models, but rather reflect a certain degree of multidimensionality (cf. Schommer, 1990). However, these findings seem to support the notion that evaluativist beliefs are theoretically related, in part, to the absolutist and multiplist levels. Analyses of the scale structures revealed that participants did not use the whole 6-point Likert scale. In fact, both the absolutist and the multiplist scales provoked dichotomous response patterns, whereas the evaluativist scale provoked responses on a 3-point scale. This is an interesting finding, because the items were formulated in a pronounced way (i.e., no use of words and phases such as sometimes, in most cases, and often) so as to prompt participants to express possible nuances in their responses. The absolutist scale was the most difficult to endorse, whereas the evaluativist scale was relatively easy to endorse, and might even reflect more socially desirable responses. In fact, compared to studies using other stage models, the high school students in the present study were far more evaluativist than the average college student in these other studies (e.g., King & Kitchener, 2002). However, previous studies were conducted in the 1970s and 1980s, and cultural differences are likely. On the other hand, the standard deviation of the evaluative scale is fairly large, which might indicate that the persons are in a transition stage regarding their epistemic beliefs. As a matter of fact, previous work has indicated that students from senior years in secondary school increasingly hold predominantly evaluativist epistemic beliefs with regard to knowledge claims on social and physical truth (Kuhn et al., 2000).
Implications
The instrument described in this paper provides a first step toward accurate measures of epistemic beliefs that can distinguish between sophisticated and less sophisticated beliefs. Future development of the instrument will focus on filling the gaps items that constitute the separate dimensions. In particular, the evaluativist scale needs items that are more difficult to endorse, whereas the absolutist scale needs items that are easier to endorse. In its current form, the multiplist scale can already sufficiently distinguish between high and low multiplist students, and the evaluativist scale comes close. The absolutist scale requires more effort, but it can be argued that this may be due to the population used in this study, which is very likely beyond the absolutist stage (cf. Kuhn, 1999). This makes it difficult for these student to endorse such items. Therefore, future work should be directed
ISLS
200
ICLS2012
at studying epistemic beliefs in groups that are more heterogeneous regarding their epistemic beliefs than the groups participating in the present study. In addition, the instrument will be included in empirical studies on information processing to investigate whether epistemic beliefs are determinants of such processing.
References
Bond, T. G., & Fox, C. M. (2007). Applying the Rasch model. Fundamental measurement in the human sciences. Mahwah, NJ: Lawrence Erlbaum Associates. Brten, I., Strms, H. I., & Samuelstuen, M. S. (2008). Are sophisticated students always better? The role of topic-specific personal epistemology in the understanding of multiple expository texts. Contemporary Educational Psychology, 33, 814840. Bromme, R., Kienhues, D., & Stahl, E. (2008). Knowledge and epistemological beliefs: An intimate but complicate relationship. In M. S. Khine (Ed.), Knowing, knowledge and beliefs (pp. 423-441). New York, NY: Springer. Clarebout, G., Elen, J., Luyten, L., & Bamps, H. (2001). Assessing epistemological beliefs: Schommers questionnaire revisited. Educational Research and Evaluation, 7(1), 5377. Conley, A., Pintrich, P., Vekiri, I., & Harrison, D. (2004). Changes in epistemological beliefs in elementary science students. Contemporary Educational Psychology, 29, 186-204. DeBacker, T. K., Crowson, H. M., Beesley, A. D., Thoma, S. J., & Hestevold, N. L. (2008). The challenge of measuring epistemic beliefs: an analysis of three self-report instruments. The Journal of Experimental Education, 76, 281312. Duell, O. K., & Schommer-Aikins, M. (2001). Measures of peoples beliefs about knowledge and learning. Educational Psychology Review, 13, 419449. Elby, A., & Hammer, D. (2001). On the substance of a sophisticated epistemology. Science Education, 85, 554 567. Hofer, B. K. (2004). Epistemological understanding as a metacognitive process: Thinking aloud during online searching. Educational Psychologist, 39, 4355. Hofer, B. K, & Pintrich, P. (1997). The development of epistemological theories: Beliefs about knowledge and knowing and their relation to learning. Review of Educational Research, 67, 88-140. King, P. M., & Kitchener, K. S. (2002). The reflective judgment model: Twenty years of research on epistemic cognition. In B. K. Hofer, & P. R. Pintrich (Eds.), Personal epistemology: The psychology of beliefs about knowledge and knowing (pp. 3761). Mahwah, NJ: Lawrence Erlbaum Associates. Kuhn, D. (1991). The skills of argument. Cambridge, UK: Cambridge University Press. Kuhn, D. (1999). A developmental model of critical thinking. Educational Researcher, 28, 16-25. Kuhn, D., Cheney, R., & Weinstock, M. (2000). The development of epistemological understanding. Cognitive Development, 15, 309328. Linacre, J. M. (2009a). Winsteps (Version 3.69.0). [Computer Software]. Beaverton, OR: Winsteps.com. Linacre, J. M. (2009b). A user's guide to Winsteps Ministeps Rasch model computer programs. Retrieved from www.winsteps.com/winpass.htm Muis, K. R. (2007). The role of epistemic beliefs in self-regulated learning. Educational Psychologist, 42(3), 173190. Nussbaum, E.M., Sinatra, G.M., & Poliquin, A. (2008). Role of epistemic beliefs and scientific argumentation in science learning. International Journal of Science Education, 30(15), 19771999. Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Chicago, IL: University of Chicago Press. Schommer, M. (1990). Effects of beliefs about the nature of knowledge on comprehension. Journal of Educational Psychology, 82, 498504. Schraw, G., Bendixen, L. D., & Dunkle, M. E. (2002). Development and validation of the Epistemic Belief Inventory. In B. K. Hofer & P. R. Pintrich (Eds.), Personal epistemology: The psychology of beliefs about knowledge and knowing (pp. 103118). Mahwah, NJ: Lawrence Erlbaum Associates. Wood, P., & Kardash, C. (2002). Critical elements in the design and analysis of studies of epistemology. In B. K. Hofer & P. R. Pintrich (Eds.), Personal epistemology: The psychology of beliefs about knowledge and knowing (pp. 231260). Mahwah, NJ: Lawrence Erlbaum Associates.
ISLS
201 5
ICLS2012
Establishing a Mathematical Practice in a Middle School Classroom

Marta Kobiela, Richard Lehrer, Vanderbilt University, Department of Teaching & Learning 0230GPC, 230 Appleton Place, Nashville, TN 37203-5721 Email: marta.a.kobiela@vanderbilt.edu, rich.lehrer@vanderbilt.edu Abstract: In this paper, we investigate how the practice of mathematical defining was established in one middle school classroom. Our investigation of establishment involves a close look at the co-constitution of practice with communal knowledge. To do so, we employ a theoretical framework that characterizes forms of participation in defining, and illustrates how these aspects contributed to developing a mathematical system of definitions. The teacher, taking the role of a disciplinary representative, supported this co-establishment by modeling aspects of definitional practice, by positioning students as participants in practice, and by positioning definitions at the forefront of discussion. As students began to appropriate definitional practices, they, in turn, became supporters of their collective enterprise of defining. These characterizations of definitional practices and supports may provide resources for developing similar classroom environments.
Introduction
Recently, reform efforts in mathematics education have attempted to provide students with opportunities to participate in mathematics in ways that more closely reflect practices in disciplinary mathematics (Lampert, 1990). Central to these efforts is how such practices are established within classrooms. In this paper, we attend to the establishment of one particular practice, mathematical defining. Our focus on mathematical defining is motivated by the fact that in many classrooms, definitions are often treated in ways that are counter to how they are treated in the discipline of mathematics. Historically, mathematicians have participated in the coconstruction of definitions, and defining often emerged from proving (Lakatos, 1976). Some scholars have thus suggested that we instead engage students in defining as a practice, by providing them with opportunities to make sense of and construct definitions themselves, and, in turn, become authors of definition (e.g., de Villiers, 1998; Zandieh & Rasmussen, 2010). Although such studies provide examples of students engagement in the practice of defining, very little has been done to show how the practice is established. In this paper, we investigate how the practice of defining was established in one middle school mathematics classroom. We take the view that a practice is a recurrent activity structure governed by normative expectations about appropriate forms of participation. Practices are tied to the production of knowledge. The practice of defining, in particular, is tied to (a) the production of definitions, (b) the close examination of the properties of the objects being defined, and (c) the network of relations by which new definitions build on established definitions. Thus, our investigation of establishment involved a close look at the co-constitution of the practice of defining with communal knowledge. Accordingly, we were interested in the following two questions: 1) How are knowledge and the practice of defining co-constituted? and 2) How do participants in the community contribute to, or support, this co-constitution? We were particularly interested in the teachers role in initially supporting emergent forms of definitional practice and how students, in turn, became participants in the practice. To attend to these questions, we first present a framework for characterizing the practice of defining in classroom communities. We then use this framework to illustrate how the co-constitution of defining and knowledge was established in three excerpts of classroom interaction.
Characterizing Defining as a Practice

To describe the lens we used to examine defining as a practice in classrooms, we begin by describing from a disciplinary perspective what we mean by mathematical definitions and defining. We then outline a framework for characterizing forms of participation in defining in classrooms, what we refer to as aspects of definitional practice. The first author created this framework by reviewing 19 studies in which researchers instigated and/or studied students engagement in defining as a practice. These studies varied in content, context and in the age of the students. The aspects of practice were developed through a method of iterative refinement, using the lens of disciplinary perspectives on definitions and defining to determine what constituted an aspect of definitional practice.
Disciplinary Perspectives on Definitions and Defining

A mathematical definition is a description of the properties of a mathematical object (such as a geometric shape) and the relations among those properties (Polya, 1957). Mathematical definitions are distinct from other mathematical entities questions, conjectures, axioms, lemmas, theorems or corollaries because they are the negotiated grounds for mathematical work. Unlike axioms, definitions are contested rather than taken for
ISLS
202
ICLS2012
granted and unlike lemmas, theorems or corollaries, definitions cannot be proven. In order to characterize defining as a mathematical practice, we draw upon the work of Imre Lakatos (1976), who analyzed how mathematics developed historically in the profession. Essentially, mathematicians create systems of mathematical objects and relations between objects. Defining serves several functions in creating these systems. It contributes to the refinement of proof and to the development and refinement of other definitions. For instance, in Lakatoss example of the Euler Characteristic, defining polyhedron led to a counterexample that, in turn, spurred discussions about the definition of polygon and, later, the definition of edge. Defining is also a form of argument, in that it arises out of contest about the meaning of particular objects motivated by the need for members of the mathematical community to communicate and develop a shared understanding.
A Framework for Analyzing Defining in Classrooms: Aspects of Definitional Practice

In reviewing the literature, we identified multiple aspects of definitional practice. These aspects characterize how students from previous studies (of all ages) have participated in defining in ways representative of, yet distinct from, professional mathematicians. We created this framework to provide a lens for investigating students participation in defining in classrooms. Although there are more, we briefly describe five of the aspects of definitional practice below that are most relevant to the results discussed here. While we describe these aspects separately, in reality, defining entails their collective functioning. Definitional arguments and explanations are used to justify (a) inclusion or exclusion of a definition, (b) inclusion or exclusion of an example of a definition, (c) aspects of qualities of the object being defined, or (d) whether conditions in a definition are minimal. For example, Lehrer and colleagues (Lehrer, Jacobson, Kemeny, & Strom, 1999) describe one childs argument for the inclusion of an example during her classs construction of a definition for triangle. The child had constructed a triangle with 3 paper strips, one of which was curved. When the class rejected her example as a triangle, she disagreed, appealing to their collectively constructed definition of 3 corners, 3 sides: No. It doesnt matter. Look [gesturing to the board], it has three corners [gesturing to each vertex] and three sides [gesturing to each strip of paper] (p. 78). This type of argument is emblematic of those within the discipline of mathematics (Lakatos, 1976) because it takes as evidence agreed-upon definitions. Arguments and explanations may take similar forms, but, as illustrated in the example above, unlike explanations, arguments arise from contest and are used to resolve that contest. This distinction is significant because historically the need to resolve disagreements led to advancement in the field (Lakatos, 1976). Defining also involves the construction and/or evaluation of examples and/or non-examples of the object being defined, where evaluation involves determining whether or not a case should be included as part of the set in question. Constructing and evaluating examples is significant to the practice of defining because it helps students consider what the class of objects being defined should include and provides a set of objects to describe (see, for example, Zandieh & Rasmussen, 2010). Defining may also involve revising definitions to serve the needs of the mathematical classroom community. Revision often results from definitional arguments or from evaluating examples or non-examples. When revising, definitions are sometimes expanded to include additional properties or relations while, at other times, reduced to become more minimal. In other instances, definitions are instead modified, mainly to improve their correctness. Proposing definitions about the properties to include in a definition is another aspect of defining. Proposed definitions may then be tested, for instance, against examples of the object being defined, and possibly revised. Finally, defining involves asking questions about definitions or about the qualities, properties or relations of the objects being defined. Some definitional questions are general (e.g., what is a polygon?). Others are more about particular qualities of an object, often asked in the process of trying to make sense of examples (e.g., Will this still be a rectangle if I make these sides longer and longer and these shorter and shorter? from Lehrer, Randle, & Sancilio, 1989, p. 166-167). Questions may also be asked about which properties of an object are necessary and/or sufficient for inclusion in the definition, such as, Does it really guarantee that if a triangle has two equal angles then it is isosceles? (Borasi, 1992, p. 34).
Methods
We present data from video records of whole class activity where sixth-grade students created and refined mathematical definitions of geometric objects. Our instructional design capitalized on students everyday experiences and conceptions of space, especially bodily motion, and on everyday forms of argument, especially propensities to categorize and classify. For example, we anchored students learning about polygons to paths that they walked (Abelson & diSessa, 1980; Lehrer et al., 1989) and related familiar properties of polygons, such as straight sides, to experiences of unchanging direction while walking. Working from these embodied forms of activity, we cultivated students dispositions toward posing questions and making conjectures. We privileged forms of explanation that were oriented toward the general and that appealed to mathematical system.
ISLS
203
ICLS2012
Although our focus on spatial mathematics was informed by the schools grade-level standards for mathematics, the conduct of any particular class was informed by our interpretations of students questions and by our judgments of their current levels of understanding.
Participants, Setting and Data Collection

Participants (n=18, 10 male, grade 6, ethnically diverse) attended an urban school serving primarily underrepresented youth in the southeastern region of the United States. Half of the students came from traditional classrooms that emphasized procedural mathematics. The other students had been with the classroom teacher the year before, and had engaged in some conversations about definitions related to mathematical symmetries. Despite this, we still considered the context to be good for studying establishment of practice because (a) norms surrounding participation in practice still needed to be established for new members, (b) old members varied in their participation in practice, and (c) the content was not trivial to students. Our participants came from a contained classroom, that is, they remained in the classroom with one teacher for all their core academic subjects. The second author served as a visiting classroom instructor for mathematics during the school year, and the regular classroom teacher occasionally interacted as well. The first day of instruction occurred during the second week of school, after a week working within a Connected Mathematics Project (CMP) curriculum unit on polygons. When the visiting mathematics instructor (who we shall from here on refer to as the teacher) first visited, he intended simply to have a conversation with the students about what they had learned. It was only after this first class, when it was clear that students ideas about polygons were still developing, that he decided to continue to teach mathematics. Mathematics class was conducted twice each week, for 1.5 hours per class. We videotaped each lesson and then digitally rendered the video for further analysis. We also took field notes of whole group interactions in order to contextualize the video recordings, serve as a platform for reflection, and guide the next days instruction. Students also wrote summaries of their thinking at the end of every lesson and took periodic assessments, and both were additional sources of data.
Analysis
For our analysis, we traced initial explorations that emerged as students pursued the question, What is a polygon? We focused on the first six days of instruction because the activity largely involved defining and because it allowed us to see how initial forms of definitional practice arose and were supported. To do so, we divided the data into definitional episodes segments of (possibly overlapping) time in which the class participated in making sense of one particular object (e.g., polygon or side). We limited definitional episodes to whole class discussion in order to capture collective activity. When creating definitional episodes, we identified three 10-minute excerpts of class discussion for careful analysis of the establishment of the practice of defining. We chose the excerpts (from days one, four and six) because they were similar in activity structure (open-ended construction of definitions) and topic (all began with the question, what is a polygon?) and served as good representations of shifts in classroom interaction. We wanted the excerpts to be long enough to span multiple definitional episodes, in order to see the development of the mathematical system, but short enough to look carefully at interaction. The excerpts were then transcribed, taking into account both talk and gesture. We then conducted four phases of analysis. First, we created a representation of the development of collective knowledge as a mathematical system. To create this representation, we looked across neighboring definitional episodes to identify moments of talk, gesture and inscription about interrelationships between mathematical objects and/or qualities of objects. For instance, defining polygon created the need to establish what a side was, suggesting a link between polygon and side. Our intention in making this representation was not to make claims about what individuals were thinking, but rather to represent the terrain investigated by the class. Second, using our theoretical framework, we coded when a member of the classroom community (teacher or student) participated in an aspect of definitional practice, using one or more speaker turns as the codable unit. Third, we mapped uses of aspects of definitional practice onto the representation of the mathematical system. Finally, we characterized patterns of interaction within each excerpt in relation to the map between the coded aspects of definitional practice and the mathematical system, and then looked for shifts in these patterns across the excerpts. In particular, we considered the roles taken on by students and the teacher in these interactions. Our choices for determining their roles were guided by the lens of participant frameworks (Goffman, 1981), and in particular OConnor & Michaels (1996) framing of revoicing as positioning. We chose to use this framework because we were interested in how the classs activity might be positioned as defining and how participants might be positioned as definers. To do so, we looked at how participants (both teachers and students) used talk and gesture to position their collective activity and roles within that activity.
Establishing the Co-Constitution of Practice and Knowledge

Here, drawing upon the three excerpts, we highlight a few ways in which defining and the construction of a mathematical system co-emerged and how that emergence was supported. First, definitional questions served to
ISLS
204
ICLS2012
encourage the investigation of new and related mathematical objects, and thus supported development of a mathematical system. That is, when a new object was introduced, the teacher asked the students for the definition of the new object. For instance, when the class was making sense of a definition containing angle, the teacher asked, What makes an angle again? The teacher often further highlighted the importance of new objects by writing the names of the objects on the board. Later, on the fourth day, students began to appropriate these types of questions. For example, after students revised their definition of polygon to include not only sides and angles, but also closed, the teacher asked, if we take this definition, can there be a polygon with two sides? One student, Kate, suggested that as long as the two sides were connected, it was possible, and then suggested an oval as an example. When Kates example caused many in the class to protest, a group of students asked their peers, Whats a side, people? By asking definitional questions, students were beginning to take on the role of supporting one another in their collective activity. The teacher also played a large role in modeling aspects of definitional practice. As time progressed, the teacher modeled different aspects in order to serve the emergent needs of the community. Initially, as noted above, the teacher modeled the asking of definitional questions that supported development of the mathematical system. Later, he also modeled constructing definitional arguments, and, in doing so, encouraged preciseness in students definitions. For instance, when students defined a polygon as having sides and angles, the teacher drew three connected, but not closed lines, and said, I want to know what makes something a polygon. I know it has sides and it has angles SOthis then is a polygon right? In making his argument, he positioned the counter-example in relation to their definition, and, in turn, caused students to revise their definition to include the property of connectedness. In the last class, the teachers modeling of definitional practice shifted to address new mathematical relations. For instance, the teacher asked a new type of definitional question, one that encouraged students to think about the economy of their definition: Can you make any closed figure with sides that does NOT have angles? At the same time, students continued to appropriate forms of participation that the teacher had been modeling. For instance, in response to the teachers question, one student, Ned, constructed the example of a football-shaped figure. When asked to explain his thinking, he pointed to the lines and noted, two sides, then pointed to the vertices and said, no angles. He continued, They cant be angles cause an angle has to be a straight line, two straight lines make an angle. What is noteworthy about Neds definitional argument is that it appealed to his conceived definition of angle in a similar manner as had been earlier modeled by the teacher. Finally, the teacher also played a large role in positioning both students and content. Initially, the teacher positioned students as participants in aspects of definitional practice. For instance, when one student suggested that a polygon has the same angles and the same length of uh, same lengths of sides, the teacher revoiced the students utterance as a claim, thereby positioning his activity as proposing definitions. Another student, in response, suggested, all regular polygons. The teacher referred to this suggestion as an amendment, in turn positioning her contribution as participating in revising definitions. Later, as the class developed a need to remember their agreed upon definitions, the teacher positioned definitions at the forefront. For instance, when students proposed definitions, he wrote them on the board, and when those definitions were revised, he indicated those changes as well. He also often also requested that students write agreed definitions in their notebooks.
Discussion
In this paper, we provided an illustration of the initial establishment of a mathematical practice, defining. We do not mean to claim that by the end of the six days, the practice was fully established. Rather, we illustrate how in establishing this practice, the roles of the teacher and the students were constantly shifting as the students gained more authority and began to appropriate forms of participation. Our analysis suggests the importance of the teacher in modeling aspects of definitional practice, in initially positioning students as participants in those aspects, and in positioning definition at the forefront of discussion. As students began to appropriate particular forms of participation, the teacher in turn modified what he modeled and positioned to fit the new goals of the community and to support investigation of new mathematical properties and relations. Controversies about definition led to elaboration of mathematically important ideas such as side, angle, polygon, and straight that contributed to the development of a mathematical system. These ideas were then taken up and used during the remainder of the year. Figure 1 illustrates the relation between students engagement in aspects of practice, teacher supports and the development of a mathematical system. Our paper has two contributions. First, the use of our framework of aspects of definitional practice illustrates a potentially significant analytic tool for characterizing student engagement in the practice of defining. This framework has the potential to be refined and expanded as it is used in relation to new classroom environments. Although others have parsed mathematical practices tied to particular content (Cobb, Stephan, McClain, & Gravemeijer, 2001), this paper illustrates how this may be done in regards to an epistemic practice that spans mathematical content. Likewise, the framework, along with the supports we identified, have the potential for supporting teachers interested in developing similar learning environments and supporting students
ISLS
205
ICLS2012
in engaging in the practice of defining. The aspects of definitional practice may allow a teacher to identify what types of activity to model and encourage with her students. We focused on collective activity, but this framework may also be useful for capturing changes in how individual students participate in the practice of defining and develop identities as definers. In our ongoing analysis, we are investigating how roles of individual students shift, taking into account their particular histories within the classroom community. Beginning of Excerpt 1 (Day 1 of Instruction): Definitional Practices (in gray) & Supports (in white)
Model Definitional Questions Proposing Teacher: What is a Polygon? Definitions Victor: A polygon has the same angles and the same lengths of sides. Positioning Utterance Teacher: Victors claim is that all polygons as Participating in have the same length of sides Proposing Definitions and the same angles. Rachel? Rachel: All regular polygons. Teacher: All regular polygons. Do you Positioning Utterance accept her amendment? as Participating in Victor: yeah Revising Definitions
Development of Mathematical System

Polygon
Polygon Same Angles & Sides Regular Polygon
Revising Definitions
Later in Day 1:
Model Definitional Questions One Students Definition for Regular Polygon: Sides and Angles Teacher: What makes an angle again?
Quadrilateral
Polygon Regular Polygon
Angle
Congruent
References
Figure 1. Establishing definitional practice on the first day of instruction. The left side presents transcript from two time points in Excerpt 1. The right shows the mathematical system concurrently developed. Aspects of definitional practice and supports are highlighted in the transcript. Nodes indicate objects that were defined or whose qualities were explored. Solid lines in the system indicate relations discussed between objects.
Abelson, H. & diSessa, A.A. (1980). Turtle Geometry: The Computer as a Medium for Exploring Mathematics. Cambridge, MA: The M.I.T. Press. Borasi, R. (1992). Learning mathematics through inquiry. New Hampshire: Heinemann Educational Books, Inc. Cobb, P., Stephan, M., McClain, K., & Gravemeijer, K. (2001). Participating in classroom mathematical practices. The Journal of the Learning Sciences, 10(1/2), 113-163. de Villiers, M. (1998). To teach definitions in geometry or teach to define? In A. Olivier & K. Newstead (Eds.), Proceedings of the Twenty-second International Conference for the Psychology of Mathematics Education. (Vol. 2, pp. 248-255). Stellenbosch, South Africa: University of Stellenbosch. Goffman, E. (1981). Forms of talk. Philadelphia: University of Pennsylvania Press. Lakatos, I. (1976) Proofs and refutations: The logic of mathematical discovery. Cambridge: Cambridge University Press. Lampert (1990). When the problem is not the question and the solution is not the answer: Mathematical knowing and teaching. American Educational Research Journal, 27, 29-63. Lehrer, R., Jacobson, C., Kemeny, V., & Strom, D. (1999). Building on children's intuitions to develop mathematical understanding of space. In E. Fennema & T. A. Romberg (Eds.), Mathematics classrooms that promote understanding (pp. 63-87). Mahwah, NJ: Lawrence Erlbaum Associates. Lehrer, R., Randle, L., & Sancilio, L. (1989). Learning preproof geometry with LOGO. Cognition and Instruction, 6(2), 159 - 184. OConnor, M. K., & Michaels, S. (1996). Shifting participant frameworks: Orchestrating thinking practices in group discussion. In D. Hicks (Ed.), Discourse, learning and schooling (pp. 63-103). New York: Cambridge University Press. Polya, G. (1957). How to solve it: A new aspect of mathematical method. Princeton, NJ: Princeton University Press. Zandieh, M., & Rasmussen, C. (2010). Defining as a mathematical activity: A framework for characterizing progress from informal to more formal ways of reasoning. Journal of Mathematical Behavior, 29(2), 57-75.
ISLS
206
ICLS2012
Redesigning Classroom Learning Spaces: When technology meets pedagogy and when they clash
Elizabeth S. Charles, Dawson College, 3040 de Maisonneuve W., Canada, Email: echarles@dawsoncollege.qc.ca Nathaniel Lasry, John Abbot College, 21275 Rue Lakeshore, Canada, Email: lasry@johnabbott.qc.ca Chris Whittaker, Dawson College, 3040 de Maisonneuve W., Canada, Email: cwhittaker@place.dawsoncollege.ca Abstract: As educational institutions look to the future, there is growing interest in constructing technology-rich classrooms. Using a three-part study, this research examined the relationship between pedagogy and socio-technological spaces in a college-level physics course. The first part looked at the effect of implementing socio-technological environments on students conceptual change, while comparing implementations made with different pedagogies (active learning vs. enabled-traditional instruction). The second looked at the students perceptions of this new learning space. The last examined the effect of the instructors perception of their instructional approach (teacher-centered versus studentcentered) on students conceptual learning. Findings show that active instructional approaches are an essential condition for success of socio-technological spaces. Students who received an active learning pedagogy achieved greater conceptual gains and were more able to take up the affordances for learning of the environment.
Introduction
Interest in redesigning traditional learning spaces to promote greater student engagement and deeper learning has reached the science classrooms of institutions for higher education. Heralding this commitment to change are projects such as the Peer Instruction at Harvard, and Student-Centered Activities for Large Enrollment Undergraduate Programs (SCALE-UP) at North Carolina State University. Adoption of educational innovations is not without challenge particularly in regard to greater student engagement, i.e., active learning (e.g., Dori & Herscovitz, 2005). But with increasing interest in these socially-based technologically-rich classroom learning environments, new questions arise. We examine the effect of designed technology-rich learning spaces on students conceptual change, comparing the impact of different instructional approaches i.e., structured student-centered active learning instruction vs. enabled-traditional instruction (defined in the upcoming section). Second, we examine how students perceptions of the new learning spaces differed depending on the treatment condition. Third, we examine the effect of teachers perception of their instructional approach (teacher-centered vs. student-centered) on students conceptual knowledge.
Background
Student-centered active learning (referred to as AL hereon) has become a way to describe the type of pedagogy that is derived from both principles in constructivist and social constructivist theories of learning and knowing. It runs counter to traditional views of learners as passive recipients, and instructors as transmitters of information (Keyser, 2000). AL can be summarized as a pedagogical approach that engages students in the process of purposefully thinking, questioning and reflecting on specific aspects of their understanding while engaged in authentic activities that are domain-specific. AL pedagogy acknowledges that it is important to consider the key ideas and practices described above when designing learning activities for a specific domain or field. For instance, in physics it is important for teachers to design activities that account for the difficulties students have when faced with conflicting models of the world, and promote the process of conceptual change (e.g., Chi, 2005). Empirical studies of implementations of AL approaches in physics instruction show benefits such as more meaningful construction of knowledge and deeper understanding (e.g., Dori & Belcher, 2005). Research shows that AL approaches encourage students to take on more meaningful ways of learning, with implications on strategies used in the process of knowledge construction i.e., deep approaches versus surface approaches (e.g., Marton, Hounsell & Entwistle, 1997).
Technology and Learning Spaces

When social constructivist theories inform educational reform not only do classroom activities change, but often, so too do the physical environments. Instead of rows of desks facing the front of the class, students sit and work together in circular pod-like groupings while the teacher circulates around the class, thereby providing students with easy opportunities of collaboration and group work. Such environments are generally rich in
ISLS 207
ICLS2012
technology thereby allowing for knowledge sharing, distribution and visualization e.g., simulations, web access, online collaboration and sharing of notes. In the current study, the condition described as active learning in the socio-technology environment follows the above description. It has long been argued that technology itself is not a substitute for good pedagogy (e.g., Clark, 2001). Recent meta-analyses, however, show that technology can be an effective tool when used in support of learners effort to achieve and not merely to present content (Tamim, Bernard, Borokhovski, Abrami, & Schmid, 2011). So what happens to learners when technology-rich learning spaces are used by teachers whose pedagogical approaches are best characterized as low to moderate in its student-centeredness? (We characterize this type of pedagogy as enabled-traditional because these teachers are willing to use new environments but still mainly use teacher-centered strategies). One possibility is that affordances for different forms of group work (e.g., cooperative or collaborative activities), created by the designed layouts of these new spaces, may of themselves improve learning outcomes. Another possibility is that the new environments can be either under-used or possibly misused if instructors are not fully committed to social constructivist pedagogical approaches - e.g., limited know-how, simultaneously holding opposing views on teaching and learning (more teacher-centered than student-centered beliefs). The current research also looked at teachers with different views of learning to determine whether the affordances provided by socio-tech environments enable them to promote students conceptual change.
Connection Between Teaching and Learning

There is a growing body of evidence suggesting a strong relationship between teaching and learning (Trigwell, 2010). These studies show that the approach to teaching and the design of the learning environment can substantially impact learning outcomes by influencing students approach to learning. Students who perceived their teachers as using a transmission approach (teacher-centered) often use a surface approach, while those who perceived their teachers as using a facilitator (student centered) use a deep approach to learning. The result of these studies was the production of the Approaches to Teaching Inventory (ATI; Trigwell & Prosser, 2004). The inventory is composed of 22-items on two scales (1) conceptual change/student focused (CCSF) and (2) information transmission/teacher focused (ITTF) represented by 11 items each. Studies that have used the ATI show a correlation between the approach to teaching and a teachers repertoire of teaching methods (Gibbs & Coffey, 2004). Those results suggest that to change teachers approach to teaching may mean that we need to change how they understand teaching and learning and depends on how they see their role in the classroom. In the current study we used the ATI to establish where teachers might be positioned along a continuum in their approach to teaching and learning. This allowed us to examine the relationship between these self-reported perceptions and students conceptual learning.
Methods
This three-part study used a mix-method design described below. For simplicity, we refer to the enabledtraditional instruction simply as Traditional instruction and the socially-based technology-rich environment as socio-tech.
Research Designs
Part 1 was a quasi-experimental design assessing conceptual change in introductory physics students using the Force Concept Inventory (FCI; Hestenes, Wells, & Swackhamer, 1992). The FCI is a 30-item multiple-choice test made up of questions designed to reveal levels of conceptual understanding, and non-normative understanding, of Newtonian physics. It is arguably one of the most widely used assessment instrument in physics, which assesses conceptual change on the topic of kinematics and Newtons Laws (McDermott & Redish, 1999). Students were given the FCI at the beginning and at the end of the semester. Pretest-posttest differences on the FCI allowed us to compare between the four groups in a 2x2 comparison: pedagogical approach (Active Learning vs. Traditional) by classroom settings (Socio-tech vs. Conventional). Summary of the research design is shown in Table 1. Table 1. Four categories of learning environments in our study. Pedagogical Approach Classroom Setting Active-Learning Instruction Traditional Instruction Soc-Tech_ AL Soc-Tech_Trad Socio-Tech classroom (n=56) (n= 51) Conv_AL Conv_Trad Conventional classroom (n=49) (n=58) Part 2 was a qualitative investigation based on focus group interviews. We conducted focus group sessions with students during their semester using a semi-structured interview format. Part 3 was a post hoc examination of whether or not there was a correlation between a teachers view of instruction (i.e., student-
ISLS
208
ICLS2012
centeredness or teacher-centeredness as assessed by the ATI) and their students average conceptual gain on the FCI.
Participants and Setting

Participants in Part 1 were first-year science majors at an urban Anglophone college in Quebec. They were generally between the ages of 17 19 years. The ratio of males to females was roughly 50-50. Intact sections of the introductory physics course were the focus of the study from the Fall 2010 and Fall 2008 1. Part 2, we recruited 34 students to take part in focus group interviews in the Fall 2010. Approximately half of the students were from the AL treatment group, the other half made up of students from the classes of the other comparison group teachers i.e., Traditional instruction. The gender spread was near equal with 16 males and 18 females in total. Part 3 participants were six physics instructors. They form a representative sample of the physics faculty. Their teaching experiences ranged from 2.5 to 15 years.
Setting
In all cases, the research setting was the physics classroom the socio-tech or the conventional classrooms. The socio-tech environment used in this study facilitates group work by organizing worktables into pod-like configuration seating four students, with one computer for every two students; and two interactive white boards that allow for knowledge visualizations. Note that in Part 1 the intact section, and its teacher, was assigned based on conditions described in footnote 1. For Part 2, all sections, and the respective teachers, were assigned to the same soc-tech environment.
Analysis and Results

Part 1 environment and conceptual gains
Students end-of-semester FCI scores were compared between groups using an ANCOVA, taking the students beginning of semester FCI scores as a covariate. Results from these analyses allow for comparisons between groups while accounting for prior knowledge differences (i.e., what students know before instruction). In contrast, simple conceptual knowledge gains are calculated as the differences between pre-test and post-test scores. Given that such absolute change is constrained by ceiling effects, this study also used a normalized gain (Hake, 1998; normalized gain is calculated as: g = (Post-test% Pre-test%)/ (100% pre-test%)). In the analysis of the FCI we address whether the affordances provided by socio-tech environments enable teachers with different views of learning to equally promote students conceptual change. Figure 1 shows the interaction of pedagogy and technology on students conceptual change. We found a significant between subject effects for Pedagogy (F value 28.808, p<0.001)]. That is, students in AL pedagogies achieve greater conceptual change, as gauged by their FCI gains, than students in traditional pedagogies. However, the sociotech environment on its own had little impact on conceptual learning. Indeed, there is no significant difference between subject effects for Classroom (F=0.201, p=0.7). Our data therefore suggests that students have increased conceptual gains only when taught with a student-centered AL pedagogy. The salient result is that without the accompanying student-center active pedagogy, the socio-tech environment failed to show improvement in learning (see Figure 1). Furthermore, students participating in the student centered AL pedagogy within the socio-tech environment were those that benefitted the most. Hence, the potential affordances to improve learning outcomes, provided by socio-tech environment, were realized only by teachers who adopted socio-constructivists views of learning to promote students conceptual change. Lastly, there is a non-significant trend suggesting that socio-tech environments, when not used with an appropriate pedagogy may be less productive than conventional classrooms. This needs further exploration.
In an effort to maintain authenticity of the design we collected data over the course of several years. Data collection started with the AL teacher before there was a Socio-tech classroom (Fall 2008). It continued with teachers who could genuinely be categorized as unaffected by the zeitgeist of change (Fall 2009), when the room was first in use and there was little talk of pedagogical change. Lastly, data was collected from teachers who had had a couple of semesters to teach in the room.
ISLS
209
ICLS2012
Figure 2. Impact of pedagogy and classroom on FCI normalized gains (error bars = +/- 1 StdErr), N=214.
Part 2 student perceptions of socio-tech environment

In total 34 students who were taught in the new socio-tech environment were interviewed in grouping of 3-4 students. Of those interviewed 19 were taught by a teacher identified as using a high student-centered approach (high CCSF AL group); while the others 15 were taught by teachers identified as using a medium to low student-centered approach (low CCSF comparison group). This classification of teachers was made based on the results of the ATI questionnaire. The interviews were video recorded, transcribed and coded by two raters using a grounded method (Glaser & Strauss, 1967). The unit of analysis was the students unique contribution (i.e., repeating the same statement/sentiment was not counted). Two categories emerged: (1) students perception of the classroom authority structure and role of the teacher (authority), and (2) perception of learning in physics (epistemic perceptions/beliefs). The latter category was composed of three sub-components of perceptions/beliefs about how one learns or uses what is learned in physics: (a) relates to real world phenomena (real world), (b) takes additional work outside of class (requires effort), and (c) requires access to a variety of experiences in the classroom (experientially mediated). Both main categories revealed differences between the AL and comparison groups. Looking at category 1 (Authority), students in the AL group were twice as likely to view themselves as essential to the learning process (29% vs. 15%). These differences are also consistent with our characterization of teachers, which was based on their self-reports using the ATI. Category 2 (Epistemic beliefs) results showed larger differences between the AL groups and comparison groups (see Figure 2). While the AL groups were at least twice as likely to hold these sophisticated views of knowledge construction (epistemic views), these differences were not statistically significant.
Student's Epistemic Beliefs AL group Comp. group
% of total coded statments
6.0 5.0 4.0 3.0 2.0 1.0 0.0
rWorld
rEffort
eMediate
Figure 2. Impact of teaching approach on students epistemic beliefs in Soc-Tech classroom.
Part 3 teaching approach and conceptual gains

We also analyzed the correlation between ATI subscales (CCSF and ITTF scales) as self-reported by instructors and their students FCI gains. Our results show that the ATI scale of an instructors self-reported studentcenteredness (CCSF scale) is very highly correlated with students conceptual knowledge gains on the FCI (R = 0.83). By comparison, an instructors teacher-centeredness (ITTF scale) is poorly correlated with gains on the FCI (R = 0.11). These results suggest that the greater the instructors degree of student-centeredness, the greater
ISLS
210
ICLS2012
the likelihood that students will achieve higher conceptual gains. However, instructors degree of teachercenteredness does not impact (neither positively nor negatively) students conceptual change.
Discussion
Recent studies have reopened the issue of the role of technology in learning (Tamim, et al., 2011). The current study supports findings that show differences in learning is related to how technology is used, which appears to be influenced by a teachers approach to instruction. This approach also has an impact on what students learn and how they perceive their learning experience. Though all students in the Part 2 were taught in a new sociotech environment only those in the high student-centered AL instruction showed significant increases in their conceptual knowledge, compared to those taught by teachers with medium to low student-centered approach. Providing students with opportunities and technological tools for engaging with each other is not enough to promote conceptual change if the teaching itself does not change. Given that student-centered teaching was more effective at using new socio-tech environments, as shown in Part 2 of this study, it may be that teachers who use such approaches may also be more interested in using technology to support and create learning opportunities rather than using it to transmit content. Therefore, while there is still a lot to learn about effective design of these new technology-rich learning environments, that leverage social engagement, one thing stands out, investments in teachers pedagogical knowledge should go hand in hand with investments in these new spaces. Their interest in learning how to use the technology affectively will follow (i.e., techno-pedagogical knowledge). Additionally, the ATI may be an effective and easy way to assess teachers readiness to take on the challenges of teaching in such spaces.
References
Clark, R. E. (2001). Learning from media: Arguments, analysis evidence. Greenwich CT: Information Age Publishers Inc. Chi, M. T. H. (2005). Commonsense conceptions of emergent processes: Why some misconceptions are robust. The Journal of the Learning Sciences, 14(2), 161-199. Dori, Y. J., & Belcher, J. W. (2005). How does technology-enabled active learning affect students understanding of scientific concepts? The Journal of the Learning Sciences, 14(2), 243-279. Dori, Y. J., & Herscovitz, O. (2005). Case-based long-term professional development of science teachers. International Journal of Science Education, 27(12), 1413-1446. Entwistle, N. (2010). Taking stock: An overview of key research findings. In J.C. Hughes & J. Mighty (Eds.), (pp.15-51). Taking Stock: Research on teaching and learning in higher education. Montreal: QC., McGill-Queens University Press. Glaser, B.G. & Strauss, A.L. (1967). Discovery of Grounded Theory. Mill Valley, Ca.: Sociology Press. Hestenes, D., Wells, M., & Swackhamer, G. (1992). Force Concept Inventory. The Physics Teacher, 30(3), 141158. Keyser, M.W. (2000). Active learning and cooperative learning: Understanding the difference and using both styles effectively. Research Strategies, 17, 3544. Marton, F., Hounsell, D., & Entwistle, N. J. (Eds.) (1997). The experience of learning: Implications for teaching and studying in higher education, 2nd edition. Edinburgh: Scottish Academic Press. McDermott, L. C., & Redish, E. F. (1999). Resource letter: PER-1: Physics education research. American Journal of Physics, 67(9), 755-767. Tamim, R. M., Bernard, R. M., Borokhovski, E., Abrami, P. C., & Schmid, R. F. (2011). What forty years of research says about the impact of technology on learning: A second-order meta-analysis and validation study. Review of educational research Trigwell, K. (2010) Teaching and Learning: A relational view. In J. Christensen Hughes and J. Mighty (Eds.) Taking Stock: Research on Teaching and Learning in Higher Education. McGill-Queens University Press: Montreal and Kingston. Trigwell, K. & Prosser, M. (2004). Development and use of the Approaches to Teaching Inventory. Educational Psychology Review 16(4), 409-24.
Acknowledgments
This study was supported by a grant from the Ministere de lEducation et du Loisir et du Sport, province of Quebec, Canada, granting agency PAREA (grant PA2009-005). We thank Steven Rosenfield and Helena Dedic, our colleagues and researchers with the Center for the Study of Learning and Performance (CSLP),
ISLS
211
ICLS2012
Bridging Design and Practice: Towards a Model-based Collaborative Inquiry Science Learning Environment
1
Daner Sun, 2Chee-Kit Looi, Learning Sciences Laboratory, National Institute of Education, Singapore 3 Baohui Zhang, Institute of Education, Nanjing University, China 1. daner.sun@nie.edu.sg, 2. cheekit.looi@nie.edu.sg, 3. baohui.zhang@nju.edu.cn Abstract: The WiMVT system is a web-based science learning environment for secondary school students. In this paper, we describe the theoretical underpinnings that guided the design of WiMVT framework, the components and core features as well as the work flow of WiMVT. We elucidate our development process that supports the development of the system. To examine the functional usability and evaluate its impact on students learning, a pilot study was conducted to show the findings with implications for how to improve the functionalities of the existing, briefly present the learning outcomes of lessons with the system, and provide feedback to the researchers, designers and teachers. Keywords: WiMVT; science learning; inquiry
Background
Learning science through inquiry that incorporates Information and Communication Technologies and scientific practices has been a desired pedagogical approach for science learning. In recent years, a number of computersupported inquiry-based science learning environments have been developed, studied and evaluated, such as WISE, Co-Lab, Inquiry Island, and nQuire. Substantial evidence reported such learning environments could facilitate the development of cognitive and metacognitive strategies in the pupils (Schwarz & White, 2005). Due to benefits for learners and the educational demand for a new science learning environment to effectively support the class and meeting the societal demands on science education, a group of researchers at Learning Sciences Laboratory, National Institute of Education in Singapore endeavours to develop a web-based science learning environment named WiMVT system (Web-based inquirer with Modelling and Visualization Technology, http://www.sstlsl-wimvt.sg/wimvt/). It is designed as an innovative learning environment in which modelling and visualization, inquiry and social interaction brought together and integrated. With a number of features, the system is targeted to support secondary school students to acquire sophisticated understanding of scientific conceptions, develop crucial learning skills (inquiry skills, modelling skills, collaborative learning skills) and reasoning skills, as well as reflective thinking skills. We will present a pilot study of WiMVT system, which happened at the stage of the development of simplified version. It is hoped this study could contribute to narrowing the gap between the design and development of a science learning environment and its actual usage and enactment in the classroom practice.
Theoretical Underpinnings
The pedagogical principles in model-based inquiry serve as the guide for design decisions on the framework of the WiMVT system. Different teaching patterns were formed and demonstrated in relevant studies. White and Frederiksen (2002) proposed an inquiry cycle that consisted of question-predict-experiment-mode-apply. The results indicated that both of lower and higher achieving students benefit from this inquiry model. The inquiry phases in the Inquiry Island were described with questions, hypothesis, plan, investigation, creation and evaluation of models, and the evaluation of models and research processes. It was proposed to facilitate students sociocognitive and metacognitive development (White, et al., 2002). Christina and Gwekwerer (2007) designed an inquiry framework EIMA: Engage-Investigate-Model-Apply. This inquiry pattern encouraged students engaging in the guided inquiry with a focus on creating, using and revising models. In summary, despite of using different teaching patterns, these studies pointed to the necessity of having modelling as an important component in science inquiry. And the model-based inquiry process could mainly include orientation or question, hypothesis, plan, investigation, model, and conclusion (Bell, et al., 2010). With modelling as one of major strategies to visualize and examine students conceptual understanding in science class, and guided by the relevant design and the educational principle: Predict-Observe-Explain (POE) adopted in science class in some Singapore schools (White & Gunstone, 1992), as it has been demonstrated as an effective way to examine students prior knowledge and conceptual changes. Thus, we propose a phase named Pre-model with corresponding Model in the inquiry cycle to probe students conceptual transformation process. Finally, based on the theoretical analysis discussed above, considering other prominent learning environments design, a revised model-based inquiry cycle proceeded with eight phases is proposed: Contextualize, Question & Hypothesize (Q&H), Pre-model, Plan, Investigate, Model, Reflect, and Apply. These eight phases are refined as the components of the WiMVT inquiry cycle. Icons were used to denote each phase of the inquiry cycle in the system.
The Architecture and work flow of the WiMVT System

ISLS 212
ICLS2012
The WiMVT system operates via the Internet and is accessible through a general web browser. The system supports access of administrators, teachers, and students. The main functionalities of each component in the teacher and students module are as follows: 1) profile/mailbox: both of teachers and students are provided profile and mailbox. Profile is used to identify users based on photos, name, nationality and profile description; mailbox is designed to send message to exchange ideas, written materials or other information. 2) teachers subject management: it allows defining subjects and grades for projects. 3) teachers project management: it allows editing content, attaching guided questions, inserting images, videos and simulations, and establishing and managing groups for students. 4) teachers solution review: it allows viewing and examining students artefacts (e.g. written information, pre-models and models); providing feedback through a comment box. 5) teachers simulation library: it allows uploading and executing simulations.6) students project: it allows students to go through inquiry phases to conduct learning activities and complete series of learning tasks.7) students group: it enables students to manage the access to the system and create or find an available group to join. Based on the above functional module, the work flow of WiMVT system can mainly be executed in four stages: 1) the teacher establishes the project: the main tasks involve editing the Home page to give brief description, learning objectives, and tasks of the project (Linn, 2000); defining content of the Contextualize tab. Besides, the teacher is also responsible for assigning inquiry questions, plan, modelling tasks, the reflective task and assignments for students in terms of the inquiry levels. At last, the teacher arranges students groups in appropriate size in the Group Management. 2) students get into the system: after logging into the system with their accounts and passwords, students can access to a work section consisted of four components aforementioned. Thus, general information of the assigned project can be viewed in My Project. 3) students conduct inquiry activities: students choose the project and access to the work session with their group members concurrently. The window of project work session generally consists of four panes: shared workspace (It holds the content or tools associated with each phase, status of group members, name list of group members, and a chat box. Students will experience a series of learning activities based on the available inquiry phases designed by the teacher. 4) the teacher reviews and comments artefacts: the teacher can access to students artefacts (hypothesis, plan, investigation report and reflection content, pre-models and models, as well as Apply content) to review and comment them while travelling in the Review Solution section.
Core Features of the WiMVT System

The system encompasses three kinds of self-developed modelling tools: sketch tool, qualitative modelling tool and quantitative modelling tool. The sketch tool integrated in the Pre-model and Model tab respectively holds basic painting functions. The graphic-based diagram is employed to construct the qualitative and quantitative models in the Model tab. The quantitative modelling tool is building quantitative models via identifying precise mathematical formulas involving variables. A simulation engine is run when the relation has been specified in the form: If A goes up, B goes up when students construct qualitative models. In this way, student-centred modelling thereby can be progressive because they can start from more qualitative models without having to define mathematical formulas and then get into the stage of more quantitative modelling when figuring out the mathematical formulas finally. In addition to sketched models, diagrams of qualitative and quantitative models aforementioned, other representations are allowed to import in the system as the form of images and simulations. For example, interactive simulations can be embedded together with guided questions in the Investigate tab. The hyperlink of images, audio, videos can be established in the tabs as well. The use of multiple external representations is intended to receive motivational benefit but also leads the learner to a deeper understanding of the subject being taught (Ainsworth, 1999). In the system, the noted feature is the shared workspace combined with a chat window embedded in several inquiry phases. The design supports students synchronous (real-time) co-constructive modelling, as well as peer discussions. More specifically, the shared modelling space encourages students to work jointly to create something complex models which are impossible to construct individually (Urhahnea, et al., 2010). On the other hand, the interactive collaboration between teachers and students are established by asynchronous review function, comment box and mailbox. In that way, students are quite possible to look back and revise their pre-models after they receive the quick feedback through the comment box.
Development Process of WiMVT System

The development process consists of five short cycles which mostly involve: design discussion selected features development discussion & usability test redesign and improvement. At each development stage, weekly meetings involving designers, programmers, researchers and teachers are conducted for discussing design decisions, lesson plans, lesson design and research plans, etc. For the system evaluation, our actual pathway involves getting feedback and comments from consultants and collaborators from different research disciplines, such as science education, computer technology, and educational technology. Four levels of studies have been done or to be done in the development process. Level 1 (the usability tests in Cycles 1 and 2) is focused on improving the functionality of the system through standard usability heuristics and groupware
ISLS 213
ICLS2012
heuristics. Level 2 (the usability tests in cycles 3 and 4, we called them pilot studies in this paper) uses classroom-based studies to find evidence that can inform how the software and pedagogical activities can be modified and improved. Level 3 (the educational research in cycle 5) refers to a longitudinal study of our dependent variables (conceptual changes, inquiry skills, reasoning skills and modelling skills) when the system is fully prepared (Buckley, et al., 2006). Such further studies which address what supports are necessary to support the adoption and adaptation of the system in science lessons will be conducted in terms of the data analysis of educational research and feedback. To date, we have completed usability tests in research cycles 2 and 3. In this study, we focus on the findings from the first pilot study of the simplified version in cycle 3 which seek to identify problems in enacting inquiry-based learning activities with WiMVT system in the classroom, so that continuous improvements on design and pedagogy can be made.
Pilot Study on WiMVT Simplified Version

Participants
In this study, 46 students from two secondary 2 classes and s physics teacher with around 9-year teaching experience from a school in Singapore were participated. As a future school, the school had excellent computer facilities and employed full-time ICT coordinators who assisted teachers in the technical aspects of the use of ICT in their teaching. ICT skill was one of the key learning objectives listed in the schools syllabus. Each student owned and used Macbook for daily lessons in the various subjects.
Procedures
The study consisted of two phases, namely, the co-design of WiMVT lessons and WiMVT instruction. In the co-design process, 3 science teachers contributed to the design of instructional contents, researchers and collaborators helped to revise and refine teachers instructional design. Designers and programmers focused on the usability of the system. The classes studied the topic of Current Electricity and D.C. Circuit for around two weeks. The main objectives were that students should be aware of 1) the definitions of current and current flow; 2) the relationship between current, voltage and resistance; 3) the scientific models of analyzing current / resistance in simple series and parallel circuits; 4) modelling skills of creating simple circuits. The topic was divided into 8 lessons of 50 minutes each. Four lessons incorporated the use of WiMVT system. Before class, students tried the system at home to familiarise its functions. As 23 students in each class were divided into 8 groups with heterogeneous, they mainly worked in triads.
Date Source and Data Analysis

The study was aim to examine the relative technological functionalities and their impact on students learning activities. Multiple sources included field notes, observation sheets, screen capture videos, face-to face interviews, onsite videos and audios, learning artefacts were selected for data collections. We used software to capture each screen activity to validate the data analysis. Videotaped recordings of the teacher and students interactions at different points in time, in both whole and small group settings, were used to identify patterns of change for triangulation purposes. One audio recorder was directed at each of 8 groups in both of classes. In the data analysis, all videotapes and audios were transcribed to examine the usability violations and students learning performance on the involvement of learning activities. The results were subsequently verified by crossreferencing collected data. On the other side, we used Studiocode to manage and analyze data from classroom observation and the teacher and students interview to get feedback on both of usability and instruction. The combination of the qualitative and quantitative analysis method was proposed to reveal the relationship between existing technological functionalities and the learning activities.
Findings and Discussions Usability in Learning Activities

We recorded and analysed students main usability violations which referred to the obvious usability problems in learning activities. In this study, the diagnosis of the main violations was based on observing students performance of using the system. This was done to detect the system bugs, and lagging issues. The findings helped us to make design decisions on the supplementary functions, as well as inform the improvement of relevant teaching strategies. Figure 1 depicts the rate of the major usability violations in the class. The rate of violations referred to the rate of students who had the same challenges. For the Contextualize tab, 82.6% of students just glanced over this content. We inferred that the textual information without any images offered little attraction. In the pre-model, students misoperation of the toolbar and inadequate cooperation lead to the high rate of incomplete tasks. In the Model tab, the same problems still existed but receiving low rate, because of more guidance of the cooperation in modelling phase was offered by the teacher. According to the observation, some of students just wasted some time to walk around other tabs. On the other side, more attention should be paid on teachers teaching skills of the WiMVT implementation. Therefore, the proposed design decisions as well as the recommendation on the teaching strategies were refined as follows: 1) design decisions: lock function for tabs that are not required for the particular lesson; improve free hand function of sketch tool to decrease the bandwidth requirement; a reminder for Clear the teams artefacts; popup of tool description when mouse over; move save button at the bottom of the workspace; enlarge the modelling space using scroll bar;
ISLS
214
ICLS2012
reminder message for saving when logging out. 2) teaching strategies: guide students collaboration; integrate appropriate representations into Contextualize; show exemplar models for students; guide and monitor students modelling process; provide appropriate feedback; teaching cooperative skills.
Figure 1. The rate of usability problems in each phase
Students Modelling Performance

In this study, the modelling task was building a model of circuit which used in quiz show for 3 teams of participants, and pointing out the direction of the current flow in the Pre-model phase and revising the premodels in the Model phase. Reflection was then proposed to help students gain further insight into their experience and understanding of the relevant knowledge in Reflect tab. In this paper, the quality of models was used to assess students modelling performance, because the changes of quality of models suggested the changes of students modelling performance. We classified the quality of models into three levels (Grosslight, et al., 1991; Harrison & Treagust, 2000): 1) High Quality Models (HQMs) which contain accurate description of science conceptions or phenomena that involve objects with basic properties, and reflect interaction between objects. 2) Medium Quality Models (MQMs) are the models featuring partially exact description of particular scientific conceptions or phenomena and they take into account some of appropriate components of models. The representations of MQMs are generally at the concrete level. 3) Low Quality Models (LQMs) refer to the models which contain inaccurate description of all modelling components, they are usually at the level of the scribble drawing. Below are the exemplar models of circuits at different quality levels drawn by students in this pilot study, see Figure 2. In this study, we collected a total of 11 models in Pre-model phase, and 14 models in Model phase. The results showed that the MQMs occupied most in both Pre-model and Model phase with the rate of 82% and 71%. For MQMs in Pre-model phase, most of them were constructed without defining the current flow directions. And compared to 3 groups of students drew the wrong current flow directions, only one group made the same mistake in the Model phase. In the Model phase, 3 groups defined the right objects of models; however, they drew the current in the broken circuits. Additionally, the increment of the HQMs in the Model phase reflected students understanding of circuits had improved significantly. Moreover, there was no LQM in the Model phase. In conclusion, students performed better in the Model phase with less usability violations (see Figure 1) and better quality of models. Students positive responses on the process of understanding the circuit and current in Reflect phase were demonstrated their progressions, such as:
I used to think that short circuits are very complicated, but they are not. In addition, I thought that parallel circuits have different current for each bulb. But now, I think that for parallel circuits, the bulbs have the same brightness as the same amount of current is being flowed through it. Only, when the switch is closed, then the electrons can starts flowing. Bulbs in series circuits have lower brightness than bulbs in parallel connection.
Figure 2a. LQM
Figure 2b. MQM
Figure 2c. HQM
Voices of the Teacher and Students

The transcribed interviews were analysed with a view to get feedback on the improvement of the system based on the learning practice. The interview focused on their opinions of the whole system, design features and its implementation in learning activities. In brief, the teacher and students expressed an overall positive attitude towards the WiMVT and their perspectives on the further design and development. Based on the transcript matrixes of the Studiocode data analysis on the interview, we concluded that the teacher provided more suggestions on the functionalities based on her observation of students interaction with the system. We summarized her main comments as follows:
1) She had a better understanding of the lessons which could be designed to leverage on the affordances for the WiMVT cycle: Lesson plans should be adopted for best fit with the WiMVT inquiry cycle; it was proposed that a few appropriate tabs were to be used at each time in normal lessons; individual modelling approach was more suitable for some lessons compared to collaborative modelling approach.
ISLS
215
ICLS2012
2) She suggested some supplementary design of the system: In Pre-model and Model phases, students were allowed to draw models on a more extensive modelling space; to control the learning process, the unused tabs would be locked in the current lesson; the locked tabs would be released when it was unused in the subsequent lessons; lesson folders could be established after students finished their learning activities for a particular topic or chapter; the folders could be printed and saved as documents for recording students learning performance. Meanwhile, students participation could be traced within the WiMVT system.
Most of students provided their comments on the functionalities and the design, as well as the learning activities with the system. The below comments are concluded by their major comments: 1) Most of students thought the learning activities with the system were more interesting and engaging compared what they had used
previously. Students who had the similar prior experience pointed out that the small groups collaboration provided more opportunities to do experiments or other activities in WiMVT system. Students appreciated the synchronized collaborative work, which they thought the work would be finished with a faster space and it benefited their science learning. The synchronized modelling process allowed more than one person to draw the model at one time. It did help them to spot mistakes from each other and learn from that mistake within a student work group. And they thought they enhanced their understanding of electrical circuits bring taught in the lesson through the comparison of premodels and models, as well as a reflection phase to concretize the thinking process. 2) On the aspect of system design, students mentioned the navigation of the system was simple, easy to learn, and user friendly. However, some of them thought judicious use of colour coding in the WiMVT interface may enhance the user-friendliness and presentation of the interface. For the chat function, some of students commented that embedding a voice communication channel would complement the chat function, as students may find it harder to write down their thinking process in some situations. It was hoped that the stability of the system could be improved greatly by the next iteration.
Conclusions and Further Work

In this paper, we propose a revised inquiry cycle as the framework of WiMVT system, describe the components and the core features of the system, as well as interpret the work flow of WiMVT instruction. Intend to establish a benchmark of the type of research and development for the comprehensive learning environment, we then describe the development process of the system. With the aim to evaluate and improve the existing system, a pilot study was conducted to examine the usability and learning practices with WiMVT system. The findings from this pilot study indicated the value of the system for science learning, as well as provided confirmation of further design and development in later versions. In doing so, we hope to narrow the gap between the intended design and its use in a real learning context. At present, the system allows inquiry to be done by using the simplified version. The remaining learning management features that further facilitate collaborative modelbased inquiry are still under development, in particular, the development of structured interfaces and levels of sharing functions. Furthermore, the evaluation of this comprehensive science learning environment is a complex, multifaceted and continuing process, so multiple further researches of the later version need to be done in the pilot school, such as usability test, empirical studies on educational research, and the co-design of curriculum materials. In conclusion, it will be a long journey to pursue research work on the WiMVT implementation. The presentation of the segment of WiMVT development in this paper is a summary of our partial work, which exhibits our stance to cope with challenges for bridging research and practice on the use of a web-based inquiry learning environment.
References
Ainsworth, S. (1999). The functions of multiple representations. Computers and Education, 33(2-3), 131-152. Bell, T., Urhahne, D., Schanze, S., & Ploetzner, R. (2010). Collaborative Inquiry Learning: Models, tools, and challenges. International Journal of Science Education, 32(3), 349 - 377. Buckley, B. C., Gobert, J. D., & Paul Horwitz. (2006). Using Log Files To Track Students Model-based Inquiry. Proceeding ICLS '06 Proceedings of the 7th international conference on Learning sciences. Edelson, D. C. (2002). Design research: What we learn when we engage in design. The Journal of the Learning Sciences, 11(1), 105-121. Grosslight, L., Unger, C., Jay, E., & Smith, C., L. (1991). Understanding Models and Their Use in Science - Conceptions of Middle and High School Students and Experts. Journal of Research in Science Teaching, 28(9), 799-822. Harrison, A. G., & Treagust, D. F. (2000). A typology of school science models. International Journal of Science Education, 22(9), 10111026. Linn, M. C. (2000). Designing the Knowledge Integration Environment. International Journal of Science Education, 22(8), 781-796. Rezba, R. J., Auldridge, T., & Rhea, L. (1999). Teaching & Learning the Basic Science Skills. From www.pen.k12.va.us/VDOE/instruction/TLBSSGuide.doc Schwarz, C.V., & N.Gwekwerer, Y. (2007). Using a Guided Inquiry and Modelling Instructional Framework (EIMA) to Support Preservice K-8 Science Teaching. Published online 17 August 2006 in Wiley InterScience (www.interscience.wiley.com), Inc. Sci Ed 91:158 186. Schwarz, C. V., & White, B. Y. (2005). Metamodelling Knowledge:Developing Students Understanding of Scientific Modelling. Cogintion and Instruction, 23(2), 165-205. Urhahnea, D., Schanzeb, S., Bellc, T., Mansfieldd, A., & Holmese, J. (2010). Role of the Teacher in Computer-supported Collaborative Inquiry Learning. International Journal of Science Education, 32(2), 221-243. White, B., Frederiksen, J., Frederiksen, T., Eslinger, E., Loper, S., & Collins, A. (2002). Inquiry Island: Affordances of a Multi-Agent Environment for Scientific Inquiry and Reflective Learning. In P. Bell, R. Stevens & T. Satwicz (Eds.), Proceedings of the Fifth International Conference of the Learning Sciences (ICLS). Mahwah, NJ: Erlbaum. White, R., & Gunstone, R. (Eds.). (1992). Probing Understanding (43 ed.): London and NewYork: The Falmer Press.
Acknowledgements
This research is supported by the project: Fostering Collaborative Inquiry Modelling and Visualisation Practices in Secondary Science Learning (WiMVT) founded by National research Foundation in Singapore (Project #: NRF2009-IDM001-MOE-019, IDM SST Future School-Science project).We would like to thank WiMVT team members and our collaborators: Gao Shan, Karel Mous, Fu Weikai, Oon Pey Tee, Teo Guai Wei, Tan Sze Ghee, Chan Kin Chuah and their students for working with us on the project.
ISLS
216
ICLS2012
Identity and digital media production in the college classroom

Erica Rosenfeld Halverson, Michelle Bass, University of Wisconsin-Madison, 225 N. Mills St., Madison, WI 53705 Email: erhalverson@education.wisc.edu, michelle.b.bass@gmail.com Abstract: We sought to understand whether digital media production in the freshman college classroom could afford students the opportunity to engage with identity. Specifically, students were tasked with producing autobiographical radio documentaries in the style of This American Life. We used the development of metarepresentational competence (MRC) as our measure for how effectively students engaged with identity in their production process. We found three types of MRC trajectories: Students who demonstrated MRC throughout the process, students who demonstrated MRC late in the process, and students who never demonstrated MRC. Students who demonstrated MRC created pieces that took advantage of the affordances of radio as a mode for self-representation as they communicated their personal stories and reflected on the relationship between tools and ideas in sophisticated ways. Our findings can inform the design of learning environments within fields of artistic production as well as other content areas emphasizing the importance of identity work in learning, a focus in recent design-based learning sciences research.
Identity in the Learning Sciences

Over the past 15 years, identity has emerged as a focal concept in education research (Sfard & Prusak, 2005). Identity has become a bridge between student engagement and learning goals that is, if we want students to succeed (and to be motivated to succeed) in academically valued domains we must attend to students identity development as an integral part of the learning process. In the Learning Sciences, we are particularly interested in how identity influences the design of learning environments. Design researchers have described three mechanisms for engaging identity in learning environments: 1) bringing students out-of-school lives vis--vis their cultural practices formally into the teaching and learning space (Nasir & Hand, 2008); 2) making explicit the adaptive identities that are necessary in order to be successful in a more traditional academic context (Lee, Spencer, & Harpalani, 2003) and; 3) designing experiences explicitly for the purpose of exploring students psychosocial identities (Halverson, 2010; Bers, 2001). Regardless of approach, bringing an identity focus to the design of learning environments has been linked to improved academic outcomes. We focus on the ways in which adolescents develop positive, adaptive identities (Lee et al., 2003) through their engagement with artistic production processes. In prior work, we have found that artistic production supports adolescents in exploring possible selves and in affiliating with traditional stigmatized identities in adaptive ways. Furthermore, artistic production processes support both individualistic and collectivistic conceptions of identity, opening up the possibility that multiple models for development can be accommodated within the same type of instructional setting (Halverson, 2010). This paper explores a college freshman seminar classroom as a design space for explicitly addressing the development of adaptive identities. Specifically, we ask whether students learn to construct and represent adaptive identities through participation in an autobiographical digital art production process.
Digital Literacy, Identity Representation, and the Transition to College

Digital media production has become an essential outlet for many adolescents to think about and represent their changing identity as they progress from adolescence to adulthood; many young people make this transition in a collegiate environment. While the literature chronicling new college students participation in digital media literacy and production activities is sparse, the literature on out-of-school media production organizations who work with adolescents aged 14-20 shows the value of these opportunities for empowering young people through digital media literacy (e.g. Halverson, 2010; Hull & Katz, 2006; Soep, 2006). In this paper we focus on the use of radio as a digital medium for self-representation for first year college students. Research on radio production as literacy and representation has thus far been studied in out-ofschool settings (Chvez & Soep, 2005; Walker & Romero, 2008). Radio production as a medium for expression has empowered young people to represent ideas that matter to them and bring these issues to a public audience (Chvez & Soep, 2005). Through the process, young people learn valuable literacy skills, most notably the ability to monitor the quality of their own work through critique (Soep, 2006). The organizations and programs that work with youth to create digital radio productions target adolescents in middle and high school, despite a growing understanding that college-aged young people are struggling with issues of identity and representation (Arnett, 2000).
ISLS
217
ICLS2012
The use of representational trajectories for design and assessment of learning

One way to research students competence with digital production processes and technologies is to focus on the artifacts that students generate as measures of learning. The ability to construct an external representation of a complex idea is the marker of intelligence across disciplines (Enyedy, 2005). Producing external representations that demonstrate mastery of a topic is often taken as evidence for learning in a constructivist-oriented curriculum. di Sessa and Sherin (2000) call this, metarepresentational competence the ability to select, produce, and productively use representations but also the abilities to critique and modify representations and even to design completely new representations (p. 386). Metarepresentational competence (MRC) explicitly draws on students intuitive understanding of how to create representations of systems and marks learning through their increasingly reflective and sophisticated understanding of the relationship between concept and external representation. The development of MRC as a guide for the design of learning environments has been taken up across several domains. Enyedy (2005) describes progressive symbolization, a pedagogical strategy for developing robust mathematical understanding by beginning with students intuitive understandings and building increasingly more formalized representations. Emerging work in composition indicates that recursive attention to the relationship between ideas to be represented and the tools of the medium that afford appropriate communication is a method for improving the quality of student writing (Magnifico, 2010). In our work, we have found that learning to produce autobiographical art can be taught and documented through a series of representations that begins with a focus on content, moves to a focus on how the tools of the medium afford a representation of the content, and end with a consideration of and a reflection on the relationship between content and tools the two key components of art making (Halverson, in press). We call this path from initial idea to final product the representational trajectory. The content students engage with in the development of representational trajectories ranges from mathematics (as in Enyedys work) to observations of the natural world (as in Magnificos work). In our case, identity was the content of interest; our instructional aim was for students to engage with the tools of the radio medium, specifically the tools of the radio documentary genre in order to represent themselves. We measure students learning in terms of how they use (and reflect on their use of) the radio medium in order to represent their autobiographical identity stories.
Methodological Approach
Our interest in this paper is to assess whether students creating digital media products in the formal learning setting of a freshman seminar classroom were engaged in a representational trajectory that allowed them to explore the relationship between identity and digital media production. To do so, we asked the following research question: How do students display metarepresentational competence as they create their radio pieces?
Data Collection
The data were collected in the context of a course designed by Halverson entitled, Representing self through media: A personal journey through This American Life. The major assignment and primary outcome for the course was the creation of radio documentaries in the style of the popular National Public Radio show This American Life. Each student was required to produce one ten-minute segment of their own and to help create an episode around a theme that married three or four of their segments. To capture the digital media production process and students evolving understanding of identity and representation, the researcher (Bass) engaged in three primary data collection methods: (1) ethnographic observations during each 75-minute course period, twice weekly for 15 weeks; (2) collection of all student-produced artifacts including representations created during class discussions, written reflections on the major content topics in the course, and all representations of students radio segments and episodes, from initial idea to final product; Additionally, (3) semi-structured interviews were conducted throughout the course. The purpose of these interviews was to understand how participants represented their identities independently of and through the creation of their radio pieces.
Data analysis
In order to focus on students representational trajectories we employed what we have termed bi-directional artifact analysis a framework for analyzing young peoples creative production processes through ethnographic observations of participants in situ, the artifacts they create, and interviews with participants as they describe their activities over time. Bi-directional artifact analysis involves: a) Identifying a learner-created artifact; b) documenting relevant data around the artifact and; c) constructing narrative threads across the data types that trace the core ideas and tools present in the final product back through their development. This framework echoes Enyedys (2005) description of bi-directional analysis: go[ing] backwards in time in an attempt to trace the origins of this intervention and forwards in time to examine what subsequent impact it had on the way other students reasoned (p. 437). Since the primary outcome for the course was the creation of autobiographical digital art in the form of This American Life-style radio documentaries, we conducted a bi-directional artifact analysis of the radio
ISLS
218
ICLS2012
production process. We examined the radio show itself, the artifacts students produced along the way, and their oral and written reflections on all of these artifacts. We identified the core ideas of students pieces over time and traced the changes in the representation of their ideas through the segments various instantiations. We also conducted a thematic analysis of students reflections including interviews, written reflections, and class discussions, to trace students awareness of their evolving representations of their personal stories.
Major Findings
In conducting a bi-directional analysis of students radio documentaries over time, we found student work fell into one of three categories: 1) Trajectories that demonstrated early MRC; 2) Trajectories that demonstrated growth in MRC and/or converged on MRC at the end; 3) Trajectories that never approached MRC. Table One describes how many students fell into each of the MRC categories, separated out by the episodes in which their pieces appeared. Table 1: The development of MRC through representational trajectories. Episode Title The American Dream ...And One For All Wisconfidential Sex in Perspective Where the Heart Is TOTAL NUMBER OF STUDENTS Early MRC 1 1 1 0 1 4 Late MRC 1 2 3 2 3 11 No MRC 2 0 0 2 0 4
Multiple representational trajectories Early MRC

Four of our students demonstrated MRC early and then throughout the radio production process. That is, they seemed to understand the relationship between the representational tools afforded by the This American Life genre and their interpretations of their own adaptive identities from the beginning of the course. Chris, for example, was aware of the affordances of radio from the first artifact he created about his radio piece. He knew he wanted his story to be about his parents influence on him and how their following their American Dream to the United States from Jamaica has allowed him to live his own American Dream. During his first episode group meeting, he described several ways to make explicit connections between his story and his parents through the use of voiceovers, script and music. For instance, he planned on, asking both of my parents questions at the same time so that it would feel as if I am sitting around a table having a conversation with them instead of the feeling of just an interview. These choices were reflected in Chris final piece, which used all of these modalities individually and in combination. Chris piece allowed him to explores the relationship between how he sees himself, how his parents see him, and how he fits into the multiple communities to which he claims membership, including the Jamaican American community.
Late MRC
The majority of our students (11 of 19) converged on MRC at the end of their processes. This is the trajectory we expected; We designed the course with explicit checkpoints toward the development of MRC, anticipating that these checkpoints would serve as formative feedback for students along this path. Bryans trajectory is representative of the developmental path for students in the course and the one we had designed for. He began his production process by identifying a compelling narrative without consideration for how that narrative was explicitly connected to identity or how the radio medium would afford him the opportunity to communicate the narrative. After many discussions with classmates and the instructor Bryan settled on, An Improbable Dream, the story of his grandfathers life, specifically the, years trying to come from Mexico and become a citizen of the United States, and his grandfathers trials in the US including losing a finger working in a factory and being shot while delivering pizzas. As the initial idea moved into a more formal story pitch, Bryan accomplished three tasks: 1) he expanded the narrative of his grandfather in greater detail; 2) he linked his piece to the American Dream theme and; 3) he began to identify the ways in which he could use the radio medium to communicate his story. In describing the tools of the radio medium, Bryan said: And then I wanna interview my grandma but have her speak Spanish but have my mom do the English voiceover for her about the whole experience coming
ISLS
219
ICLS2012
to America. The decision to interview his grandmother, to use his mother as a translator, and to include both of their voices in the radio piece begins to demonstrate an understanding of how to transform an oral narrative of personal experience into a multimodal representation. Bryans understanding of the relationship between his story and the tools of the radio medium continued to develop as he put his piece together. During an editing session, Bryan discussed with Bass some representational options for his piece. In one section, he wanted to include his uncles reflections on the day he took his own father to the hospital but also wanted to respect his uncles request not to use his actual voice. Bryan puzzled through how to aurally represent emotion without acting like his uncle. He decided to transcribe and read his uncles words in his own voice, incorporating longer pauses into the responses yet balancing the time constraints imposed on a This American Life segment. Also during this editing session, Bryan discussed whether to include music with his reading of his uncles responses: I was wondering if I should put music to that at all, since its just gonna be my voice? Or should I just have it be like, just the voice? The tone of his question implies that he thinks just his voice might sound boring and that the inclusion of music could be more aesthetically pleasing to the audience. Bass scaffolded him to think more explicitly about the narrative, the characters, and how the radio medium might serve the story by asking him, Is there music you associate with your uncle? After a short pause, Bryan reflected: Every time we talk about my grandpa my uncle really likes to play Viva la vidathats the song we played at his funeral a lot and stuff. For the first full version of his piece, Bryan used his own voice as his uncles narration. There was no music in the background and Bryan did not alter his normal speaking voice while voicing his uncles words. In the final version of his piece - re-edited and shared during the second semester - Bryan had a classmate read his uncles narration. He added melancholy piano to the background of the narration and used meaningful pauses, sighs, and intonations to express feeling. The editing between the narration is crisp and Bryan made the exchanges sound more like an interview; he tells the audience the question he asked his uncle and then uses his classmates narration to provide the answer compared to his first iteration of the piece where it he just reads off his uncles words without a break in the dialogue. Additionally, Bryan is able to more clearly distinguish his thoughts on his grandfather from his uncles by using a different voice. His reflections heartfelt expression is supported through the uplifting piano selection playing as Bryan speaks. It is clear when Bryan is talking about his grandfather and when his uncle is reflecting on the day his father was shot and how his father affected his life.
No MRC
The final group of four students in our study did not display MRC at any time during the course, or in their final radio pieces. The lack of MRC was characterized by few changes from initial project ideas to final pieces. Stacys initial idea, for example, was to focus on the people in her life who inspire her the most: my father, my boyfriend, and Katie Couric. Stacy did not make any significant changes from her initial idea to her pitch; this is marked both by the similarity in discourse features between multiple descriptions of her piece over time and her lack of response to instructor prompts in the online discussion board asking her to reflect on her initial idea in greater depth. In her pitch, she does allude to the need for understanding how the audio medium might afford her telling the story of her personal influences: Mine is more focused on me and Im just going to talk about the three people who influenced me, and I just figured out how Im gonna do it. Her follow-up description, though, is more about the features of the narrative she plans to focus on Im gonna open up with a line from one of the stories my dad used to tell me and not on the features of This American Life or audio more generally. Stacys script outline did represent some narrative changes. Included in the outline is a basic attention to the features of the audio medium for creating multimodal representations of self. She states: I will not be interviewing people, I will set the scene for each of [the] stories through noise and music In each story, I will tell how each person inspired me to be a journalist Here she described opportunities (or rejections of opportunities) to engage with the tools of the radio medium to represent her identity piece. Stacy made the decision a priori not to interview the people in her life who inspire her and later decided not to include anyones voice other than her own. The story of her grandfather has some underscoring, though the decision of when the music fades in and out does not seem connected to the story itself, nor is the song loud enough to be discernible. The other two stories she tells are underscored by two different songs, Beauty and the Beast for her father, and, My Heart Will Go On for her boyfriend. These two songs have symbolic value for Stacy, and during an editing session she refers to her use of them as, oh my god, so cool! However, there is no attention to the relationship between the music and her monologues, as we see in pieces that demonstrate MRC. Rather, the music choices are purposeful but non-reflective and do not seem to contribute to an understanding of the function of the representational medium.
ISLS
220
ICLS2012
Conclusions & Implications

Though identity has become a focal concept in learning sciences research, few studies have documented identity development as integrated in a learning process (Halverson, 2010; Bers, 2001; Nasir & Hand, 2008). While our earlier work provided an existence proof for the relationship between identity development and artistic production (Halverson, in press), in this paper we attended to the scope of representational trajectories students engaged in as they learned to produce digital art about the stories of their lives. We found that the majority of students followed the expected path, from an initial focus on what story they wanted to tell, to a focus on how to use the tools of the digital audio medium for representing their ideas, to an understanding of the relationship between their identity stories and how the digital audio medium affords representation of self (MRC). We also found that some students displayed MRC at the beginning of the process while others never seemed to get there at all. Our work has the potential to inform the design of learning environments within fields of artistic production as well as other content areas emphasizing the importance of identity work in learning, a focus in recent learning sciences research, particularly with respect to science. Understanding how to design for an integrated relationship between identity and learning processes allows us to take seriously the role of identity in learning. Furthermore, the use of identity narratives as content for the development and display of MRC could expand access to a complex cognitive process and potentially interest more students in fields of study that require the development of MRC. Finally, we hope our work can speak to the importance of transitional times such as the first semester of the freshman year of college as design spaces for identity work. We know that identity work in learning is important, especially during times of transition; our work can help interested designers to create experiences that engage, rather than minimize, that transition.
References
Arnett, J. J. (2000). Emerging adulthood: A theory of development from late teens through the twenties. American Psychologist, 55(5), 469-480. Bers, M. U. (2001). Identity construction environments: Developing personal and moral values through the design of a virtual city. Journal of the Learning Sciences, 10(4), 365-415. diSessa, A. A. & Sherin, B. (2000). Meta-representation: An introduction. Journal of Mathematical Behavior, 19, 385-398. Enyedy, N. (2005). Inventing mapping: Creating cultural forms to solve collective problems. Cognition and Instruction, 23(4), 427-466. Halverson, E. R. (2010). Artistic production processes as venues for positive youth development (WCER Working Paper No. 2010-2). Retrieved from University of Wisconsin-Madison, Wisconsin Center for Education Research website: http://www.wcer.wisc.edu/publications/workingPapers/papers.php Halverson, E. R. (in press). Digital art-making as a representational process. The Journal of the Learning Sciences. Hull, G. & Katz, M. (2006). Crafting an agentive self: Case studies of digital storytelling. Research in the Teaching of English. 41(1), 43-81. Lee, C. D., Spencer, M. B., & Harpalani, V. (2003). Every shut eye aint sleep: Studying how people live culturally. Educational Researcher, 32(5), 6-13. Magnifico, A. (2010). Writing for whom? Cognition, motivation, and a writers audience. Educational Psychologist, 45(3), 167-184. Nasir, N. & Hand, V. (2008). From the court to the classroom: Opportunities for engagement, learning, and identity in basketball and classroom mathematics. Journal of the Learning Sciences, 17, 143-179. Sfard, A. & Prusak, A. (2005). Telling identities: In search of an analytic tool for investigating learning as a culturally shaped activity. Educational Researcher, 34(4), 14-22. Soep, E. (2006). Beyond literacy and voice in youth media production. McGill Journal of Education, 41(3), 197-214. Walker, D. & Romero, D. (2008). When literacy is a bennie: researching contested literacies in bilingual youth radio. Ethnography and Education, 3(3), 283-296.
ISLS
221
ICLS2012
Mathematics Learning in a Racial Context: Unpacking Students Reasoning about Asians are Good at Math
Niral Shah University of California, Berkeley niral@berkeley.edu This paper draws on interview data (n = 35) from a larger study that explored how high school students make sense of racial discourse in mathematics learning contexts. All students interviewed reported awareness of the Asians are good at math narrative. Frequently, students linked it to narratives that position other racial groupsparticularly Black, Latina/o, and Polynesian studentsas mathematically inferior. Further, students situated this racialmathematical discourse within broader racial discourses about innate intelligence and cultural traits (e.g., parenting). These findings suggest that racial-mathematical discourse may limit certain students access to productive identities as capable doers of mathematics, thereby impeding learning and participation. The paper concludes with a discussion of how the findings might inform the design of pedagogical environments that disrupt racialmathematical discourse and promote equitable learning opportunities for all students.
Introduction
Recent research in the learning sciences has called into question perceptions of mathematics as a neutral or culture-free discipline. Working from a sociocultural perspective on learning (Lave & Wenger, 1991), researchers have illustrated the centrality of students engagement and domain identities to the learning process (Cobb & Hodge, 2002; Nasir, Hand, & Taylor, 2008). And yet, while the emphasis on culture has led to a more comprehensive conceptualization of learning, less is known about how issues of race mediate the mathematics learning process. This paper builds on sociocultural analyses of learning by reporting findings from a study that explored how high school students in the United States reason about race in the context of their mathematics learning experiences. Historically, research on race in mathematics education has been limited to quantitative studies of racial achievement gaps on standardized tests and disparities in course-taking patterns (Oakes, 2005; Reyes & Stanic, 1988). However, because gap-oriented research conceptualizes race as purely a demographic variable (i.e., as a set of racial categories), some have questioned the incremental utility of this approach (Gutierrez, 2008; Martin, 2009). To better understand the impact of race (and racism) on mathematics learning at the level of everyday experience, the present study operationalized race as a discourse, or the expressions, symbols, and practices that facilitate both individual reasoning and inter-personal dialogue about race (Goldberg, 1993). Racial narratives (i.e., stereotypes) represent one type of expression central to racial discourse. In the United States there exists a racial narrative specific to mathematics education that explicitly connects race with mathematical ability: the widespread belief that Asians are good at math. On its surface, the Asians are good at math narrative seems a benign compliment directed at only a single racial group. But is this how students make sense of it? What implications might it hold for students of other racial backgrounds, as well as for the learning process in general? In exploring these questions, this paper draws on interview data (n = 35) from a larger study conducted at Eastwood High School (a pseudonym), a racially diverse, urban public school located in Northern California. The findings presented here highlight two themes that reveal subtleties in students reasoning about the Asians are good at math narrative: a) students implicitly linked Asians are good at math to narratives that position other racial groupsparticularly Blacks, Latina/os, and Polynesiansas mathematically incapable; and b) students related the Asians are good at math narrative to broader discourses outside mathematics, such as racial discourses about innate intelligence and cultural traits (e.g., parenting). These findings are significant because they reveal an underlying complexity that demands a more nuanced conceptualization of the Asians are good at math narrative. Rather than understanding it to be a harmless compliment, educators and researchers should situate this narrative within a larger array of racial discourses that tends to position students from historically marginalized communities as less capable of succeeding in mathematics. Another way of framing the issue is that racial-mathematical discourse creates inequities by obstructing many students access to productive identities as doers of mathematics, which from a sociocultural standpoint limits students opportunities to learn. The paper concludes with a brief reflection on how educators might leverage these findings to design more equitable learning environments for all children.
Pivoting from Achievement Gaps to Racial Discourse
ISLS
222
ICLS2012
Studies of racial achievement gaps remain the predominant form of research on issues of race in mathematics education (Reyes & Stanic, 1988; Riegle-Crumb, 2006). Departing from this paradigm, recent research on African Americans in mathematics education has instead investigated the impact of race and racism on students everyday learning experiences vis--vis their identities as mathematics learners (Martin, 2006; Stinson, 2008). This body of literature has demonstrated that some learners perceive their mathematics identities as inextricably linked with their racial identities, and that mathematics learning and participation can be considered racialized forms of experience (Martin, 2006, p. 198). While consistent with the basic theoretical commitments of this body of literature, the theoretical approach employed here differs in two ways. First, instead of exploring how issues of race affect students pathways through mathematics, the goal here was to unpack students reasoning about racial discourse in mathematics. Framing the study in this way led to a new analytical focus: racial narratives (e.g., Asians are good at math). A few exceptions notwithstanding (see Nasir, et al., under review; Shah, 2009), little is known about how students engage in and reason about racial-mathematical discourse. Second, whereas prior research has been confined to the traditional Black-White paradigm of race, this study sought to operationalize race in a more expansive way by including perspectives from students of other racial backgrounds. In his analysis of racial discourse, Goldberg (1993) argued that since its inception, the concept of race has been rooted in the notion of hierarchy. This means that race must be conceptualized in relational terms. For example, to understand how issues of race affect a Latina/o student in mathematics, one must consider how racial discourse positions Latina/os relative to students of other racial backgrounds. Based on this theoretical premise, recruiting a racially diverse interview sample was an intentional aspect of the studys design.
Methods
Data collection took place during the 2010-2011 school year at Eastwood High School (a pseudonym), a large, comprehensive urban high school located in Northern California. The racial and ethnic demographics of the school at the time of the study were 48% Latina/o, 25% Black or African American, 14% White, 8% Asian (includes Filipina/o), 3% Polynesian, and 2% other. Although this paper analyzes interview data, the larger study also included participant-observation conducted in four mathematics classrooms. Of the 35 students from those classes that were interviewed, participants ranged from 14 to 17 years in age and self-identified as follows: 29% White, 23% Asian, 17% Latina/o, 14% Black, African, or African American, 14% Polynesian, and 3% mixed race. Interviews relied on a semi-structured protocol and lasted approximately 30 minutes. The first half of the interview focused on students beliefs about mathematics and their self-perceptions as mathematics learners. The second half of the interview probed students reasoning about racial-mathematical discourse. To prompt this part of the conversation, students were asked: Have you heard people say that some groups are better than others at math? That interviewees tended to interpret this question in racial termstypically making reference to the Asians are good at math narrativerequires critical reflection from a methodological standpoint. In an interview such as this, the racial background of the researcher matters at least as much as the racial backgrounds of the students. The author, who conducted all of the interviews, is a brown-skinned male with parents of South Asian origin. Thus, it is possible that rather than student utterances reflecting omnipresent societal discourses, that the authors positionality cued students to think about Asians and frame them in a positive light. However, this seems unlikely. Students elaborated on the Asians are good at math narrative in extensive detail, often citing multiple examples of moments they saw it arise in conversation with peers and in the media. This would suggest that interviewees were tapping into their knowledge and experiences, instead of merely attempting to appease or perform for the researcher. All interviews were transcribed, and common themes were identified across the data corpus and subsequently refined using an iterative process (Glaser & Strauss, 1967). It was not the purpose of this study to determine the extent to which students either endorsed or rejected particular racial narratives. Instead, the aim was to better understand how they made sense of the aspects of racial-mathematical discourse most salient to them. For that reason, questions were designed to gauge their perceptions of what people say about race, rather than their own personal beliefs.
Findings
All 35 students interviewed reported awareness of the Asians are good at math narrative. By itself, this finding illustrates the pervasiveness of this particular narrative. Students spoke at length about other racial narratives as well, but Asians are good at math was usually the first mentioned. Subtleties in students reasoning about the narrative contradict its surface connotation as nothing more than a benign compliment directed only at Asians. The rest of the paper unpacks two themes that emerged from the data: a) students linked Asians are good at math to narratives about the mathematical incapacity of other racial groups; and b) students related racial-mathematical discourse to broader racial discourses beyond mathematics, such as racialized perceptions of innate intelligence.
ISLS
223
ICLS2012
Implicit Linkages to Narratives about Other Racial Groups

If the Asians are good at math narrative pertained only to Asians, students would have discussed it in terms of its implications for only that racial group. The data, however, do not support this perspective. Students consistently linked the Asians are good at math narrative to narratives about the mathematical incapacity of other racial groups (e.g., Latina/os are bad at math). Consider the following comment from Troi, a 12th grader in Precalculus who identified as Polynesian (Samoan): So there's a lot of jokes that say "Where the Asians at?" cause, you know, Asians are supposed to be the good ones at mathor like, you know, the Indians like Sanjay [a classmate of his in Precalculus] are supposed to be good at math, and uh Tariq [another classmate] is supposed to be good at math, but I don't think they'll notice me like, "Oh Troibig Troi, Samoan Troi is good at math." (Laughs). In noting that, Asians are supposed to be the good ones at math, Troi mentions several of his Asian classmates as examples of students that are expected to succeed in mathematics. But he does not stop there. Instead, he connects this narrative to societal beliefs about his own ethnic group (i.e., Samoans are not good at math). Talking about disparate racial narratives in conjunction with each other was typical of students reasoning, but students often connected them in subtle ways. A common rhetorical strategy was for students to challenge the validity of the Asians are good at math narrative by citing examples of mathematically successful non-Asians as exceptions to the narrative. For example, Monet, a 12th grader in Algebra 2 who identified as Black, describes her stance on the Asians are good at math narrative by highlighting herself as a counterexample: I don't I don't agree (with the stereotype) because I feel like it doesn't really make a difference what race you are. Like I'm Black but I'm good at math. So are you not going to ask me for help because I'm not Asian? So I don't really agree with the stereotype. Embedded in Monets reasoning is the idea that only Asians can be good at math (i.e., that other racial groups cannot be good at math). Thus, a narrative about Asians, which superficially seems to have little to do with other racial groups, actually positions other students as less capable of doing mathematics. Students appear to be articulating what Martin (2009) has termed a racial hierarchy of mathematical ability, where some racial groups are expected to excel at the expense of others that languish at the bottom. Again, students do not necessarily endorse this hierarchy; what their comments show is that they believe that others talk as if such a hierarchy exists.
Relationships to Broader Racial Discourses Beyond Mathematics

Besides linking narratives about Asians to narratives about other racial groups, data suggest that students also contextualized their racial-mathematical reasoning within racialized discourses outside mathematics. For instance, in addition to asserting that people tend to believe that Asians good at math, nearly half of the students interviewed also indicated that Asians are thought to excel in school overall. Some students explained these beliefs in terms of perceived differences in intellectual capacity. Silvia, an 11th grader in Precalculus who identified as Mexican, argued that if she had been born Asian instead of Mexican, students would have thought I was smarter because, like, they say: Asians are smart. Silvias comment reflects what Mills (1997) has termed a racial-rational discourse, which frames some racial groups as inherently more intelligent or rational than other racial groups. Given the long history in the United States of using intelligence to index both race and mathematical ability, students engaging in racial-rational discourse in a mathematics context is unsurprising (Shah, 2010). Apart from engaging in racial-rational discourse, most prevalent in students reasoning were explanations rooted in perceptions of the cultural practices of certain racial groups. Twenty-four of thirty-five students mentioned that Asians were thought to have stricter parents, which resulted in an enhanced ability to focus and be disciplined about school compared with other groups. Consider the following comment from James, a 12th grader in Precalculus who identified as African American: I think, like, not their religion, but just they way they're brought up, like, cause, um, I used to have Asian friends--well I still do--but like, just the way it's all put together it's like, it's like real serious, it's very serious in the household, like, likeone of the girls here--she had a 4.5 (grade point average) or she had a 4.3, and she was kind of like down about it. And it's like, woah, that GPA is amazing to me, and I'm like dang, and she was kind of down about it. She about to go far, for sure
ISLS
224
ICLS2012
In touting their very serious households, Jamess comment echoes the oft-voiced notion of Asians as a model minority in the United States that succeeds through hard work in school and strong parental support (Wu, 2003). Of course, as with all racial discourses, the existence of a model minority necessarily implies the existence of a non-model minority. Indeed, students mentioned a variety of narratives along these lines, such as Polynesian parents dont care and Blacks are lazy in school.
Figure 1. Nested Racial Discourses To summarize, as illustrated in Figure 1 students contextualized racial-mathematical discourse within discourses about academics in general, intellectual capacity, and cultural traits. It should be noted that Figure 1 uses Asians only as an example; the data suggest that based on the narratives mentioned by students, similar models of nested racial discourses are applicable to other racial and ethnic groups (e.g., Latina/os are bad at math, Whites are bad at school, etc.).
Conclusion and Implications

Sociocultural theory has done much to broaden our understanding of the behaviors and needs of students from diverse cultural backgrounds, specifically with respect to the dynamics of students engagement and domain identities (Nasir, Hand, & Taylor, 2008). And yet, the emphasis on cultural practices has not yet shed light on how issues of race mediate the learning process. This paper represents one of the first empirical efforts to address this void in the learning sciences. In lieu of focusing on racial achievement gaps, the present study used interview techniques to explore how high school students in the United States engage with racial discourse in mathematics. Much of students racial-mathematical reasoning revolved around the widely known Asians are good at math narrative. On its surface, this narrative seems little more than a benign compliment directed at Asians. However, the findings of this study suggest a more complex picture. Rather than existing in isolation, the Asians are good at math narrative is but one element of a more complex racial-mathematical discourse that holds implications for students of all racial backgrounds. What makes this discourse problematic is that it positions non-Asian students in the United Statesusually Black, Latina/o, and Polynesian studentsin pejorative ways, thereby limiting these students access to productive identities as capable mathematics learners (Nasir & Shah, 2011). This is significant because from the sociocultural perspective, students identities are deeply consequential for learning. Further research is needed to detail the particular ways in which racial discourses affect processes of identity formation in everyday classroom activity. Altogether, the findings from this study would appear to paint a bleak picture for the possibility of designing interventions that might challenge the status quo. Certainly, educators have limited control over certain aspects of societal beliefs that may transcend the local classroom context. But still, there is reason for optimism. Research has shown that certain pedagogical structures, when grounded in malleable theories of intelligence (Dweck, 2000), can in fact disrupt hierarchical discourses by equalizing status among students and challenging views of mathematical ability as genetic (see Boaler & Staples, 2008). Given that students in this study tended to conflate both mathematical ability and race with perceptions of innate intellectual capacity, such approaches may warrant renewed consideration. To be sure, designing learning environments equitable for all students is a challenging proposition. By analyzing the content and structure of racial-mathematical discourse, this paper offers a starting point for identifying potential points of leverage.
ISLS
225
ICLS2012
References
Boaler, J., & Staples, M. (2008). Transforming students' lives through an equitable mathematics approach: The case of Railside School. Teachers College Record, 110(3), 608-645. Cobb, P., & Hodge, L. L. (2002). A Relational Perspective on Issues of Cultural Diversity and Equity as They Play Out in the Mathematics Classroom. Mathematical Thinking and Learning, 4(2 & 3), 249-284. Dweck, C. S. (2000). Self-theories: Their role in motivation, personality, and development. Philadelphia: Taylor & Francis. Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. Chicago: Aldine. Goldberg, D. (1993). Racist culture: Philosophy and the politics of meaning. Oxford: Blackwell Publishers. Gutierrez, R. (2008). A Gap-Gazing Fetish in Mathematics Education? Problematizing Research on the Achievement Gap. Journal for Research in Mathematics Education, 39, 357-364. Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge: Cambridge University Press. Martin, D. B. (2006). Mathematics learning and participation as racialized forms of experience: African American parents speak on the struggle for mathematics literacy. Mathematical Thinking and Learning, 8(3), 197-229. Martin, D. B. (2009). Researching race in mathematics education. The Teachers College Record, 111(2), 295338. Mills, C. (1997). The racial contract. Ithaca: Cornell University Press. Nasir, N. S., Hand, V., & Taylor, E. V. (2008). Culture and mathematics in school: Boundaries between "cultural" and "domain" knowledge in the mathematics classroom and beyond. Review of Research in Education, 32(1), 187. Nasir, N. S., O'Connor, K., & Wischnia, S. (under review). Knowing about racial stereotypes versus believing them. Unpublished manuscript. UC Berkeley. Nasir, N. S., & Shah, N. (2011). On defense: African American males making sense of racialized narratives in mathematics education. Journal of African American Males in Education, 2(1), 24-45. Oakes, J. (2005). Keeping track: How schools structure inequality: Yale University Press. Reyes, L., & Stanic, G. (1988). Race, education, sex, socio-economic status and mathematics. Journal for Research in Mathematics Education, 19(1), 26-43. Riegle-Crumb, C. (2006). The path through math: Course sequences and academic performance at the intersection of race-ethnicity and gender. American Journal of Education, 113(1), 101-122. Shah, N. (2009). A student's causal explanations of the racial achievement gap in mathematics education. Paper presented at the 31st annual meeting of the North America Chapter of the International Group for the Psychology of Mathematics Education, Atlanta, GA. Shah, N. (2010). Race and mathematics learning in discursive alignment: A new theoretical vector. Paper presented at the American Educational Research Association, Denver. Stinson, D. W. (2008). Negotiating sociocultural discourses: The counter-storytelling of academically (and mathematically) successful African American male students. American Educational Research Journal, 45(4), 975. Wu, F. (2003). Yellow: Race in America beyond black and white. New York: Basic Books.
Acknowledgements
The research reported here was supported in part by the Institute of Education Sciences pre-doctoral training grant R305B090026 to the University of California, Berkeley. The opinions expressed are those of the authors and do not represent views of the Institute of Education Sciences or the U.S. Department of Education.
ISLS
226
ICLS2012
Dimensions of scaffolding in technology-mediated discovery learning environments

Mark J. W. Lee and Barney Dalgarno Charles Sturt University, School of Education, Locked Bag 588, Wagga Wagga, NSW 2678, Australia Email: malee@csu.edu.au, bdalgarno@csu.edu.au Abstract: In this paper, we revisit and re-examine the importance of and possibilities for supporting and scaffolding learner activity, with a particular focus on discovery learning tasks mediated by information and communications technologies. We propose a new framework for classifying scaffolding in technology-mediated environments that consists of two dimensions, the first of which pertains to the type of knowledge development being supported and the second of which identifies the pedagogical technique used to provide the scaffolding. Having introduced our framework, we proceed to describe examples that illustrate the ways in which each category of scaffolding within the framework might be realized.
Introduction
The need for a supportive dimension in learning and teaching has long been recognized (Biggs, 1999), but many educators claiming to espouse a discovery learning approach tend to overlook the need to provide students with adequate and appropriate structure and scaffolding. Mayer (2004) has cautioned against such pure discovery learning approaches, pointing to a substantial body of evidence that demonstrates their inefficiencies and shortcomings. Similarly, Kirschner, Sweller, and Clark (2006), drawing on research on human cognitive architecture, expertnovice differences and cognitive load, criticize and highlight the problems inherent in unguided and minimally guided approaches to instruction. They argue that instructional guidance should only be reduced or removed when learners have amassed sufficient prior knowledge to permit internal guidance. This perspective is shared by Roblyer, Edwards, and Havriluk (1997), who maintain that discovery learning is most successful when students have prerequisite knowledge and undergo some structured experiences (p. 68). Many opponents of pure discovery learning advocate instead the use of guided discovery (Leutner, 1993), which brings together aspects of discovery learning and principles from cognitivist instructional design theory. Guided discovery attempts to strike a balance between learner-driven discovery and open-ended exploration on one hand, and instructor-supplied guidance and structure on the other. In doing so, it is believed to increase the likelihood of deep learning occurring when compared to didactic, transmissive modes of teaching, while at the same time mitigating or avoiding the problems with pure discovery learning (de Jong & van Joolingen, 1998). Guided discovery emphasizes the need for scaffolding, a term coined by Wood, Bruner, and Ross (1976) as a metaphor to depict a process in which an expert or other competent individual (e.g., teacher) provides support to a novice (learner) in accomplishing a task or attaining a goal. Scaffolding involves systematically providing supportive aids in the form of tools, strategies, and guides targeted to the learners zone of proximal development (ZPD, the gap between the learners current or actual development level and his/her emerging or potential levelVygotsky, 1978), to assist progression to the next potential developmental level. Since the 1970s, a wealth of theoretical and empirical research conducted in traditional, face-to-face settings has been dedicated to the formulation of design principles, guidelines, and models for scaffolding learning. A key question is how to go about applying these to the design of technology-mediated learning environments, especially those with a discovery or guided discovery learning focus. In the next section, we propose a new framework for understanding the types of scaffolding that are possible in technology-mediated learning environments, after which we expound on the framework through illustrative examples.
Scaffolding in Technology-Mediated Environments: A New Framework

A number of attempts have been made by various researchers to classify and taxonomize the types or applications of scaffolding that may be used in technology-mediated learning environments. According to Winnips and McLoughlin (2000), for example, applications of scaffolding can be categorized according to who regulates the scaffolding (e.g., the teacher, the software, peers, the learner him/herself), the technology used, the pedagogy used, or the intended learning outcome. McLoughlin (2002) defines nine somewhat overlapping categories of learner supports or scaffolds: orientation: communication of expectation, coaching, eliciting articulation, task support, expert regulation, conceptual scaffolding, metacognitive scaffolding, procedural scaffolding, and strategic scaffolding. A shortcoming of this categorization is that some of the categories represent the type of knowledge development that is being scaffolded (e.g., conceptual scaffolding, procedural scaffolding), while others represent the way in which the scaffolding is provided (e.g., coaching, task support). To address this shortcoming, we have devised a two-dimensional framework, with the first dimension being the type of knowledge development intended to be supported, and the second dimension being the way in
ISLS
227
ICLS2012
which the scaffolding is provided, that is, the overarching pedagogical technique used. The knowledge dimension consists of three categoriesprocedural, conceptual, metacognitiveand the pedagogical dimension consists also of three categoriesinstruction, coaching, and the provision of supporting/enabling tools.
The Knowledge Dimension

The categories in our knowledge dimension draw on the definitions of knowledge types in the revised version of Blooms Taxonomy proposed by Anderson and Krathwohl (2001). Each of the three categories in this dimension is described in turn below. Procedural scaffolding. An important element of constructivist learning theory is the idea that learners should be given the opportunity to carry out realistic tasks, with the assistance of scaffolding to enable them to complete the larger task without needing to learn all of the subsidiary or sub-tasks involved (Bruner, 1986). Ideally, as a by-product of such procedural scaffolding, the learner will learn how to complete the sub-tasks so that eventually he/she will be able to carry out the larger task unassisted. This type of scaffolding corresponds to Anderson and Krathwohls (2001) procedural knowledge category, which they consider to be knowledge of how to do something (p. 214). This comprises knowledge of subject-specific skills and algorithms, knowledge of subject-specific techniques and methods, as well as knowledge of the criteria to use to determine when to use the appropriate skills, algorithms, techniques, and methods. Applied to technology-mediated learning environments, procedural scaffolding focuses on suggesting various ways to leverage the available resources and tools within the environment to achieve a desired outcome. Conceptual scaffolding. Conceptual knowledge entails an understanding of how basic factual elements are interlinked to form a larger pattern or structure. Conceptual knowledge includes knowledge of classifications and categories, knowledge of principles and generalizations, and knowledge of theories, models, and structures (Anderson & Krathwohl, 2001). Conceptual scaffolding therefore involves the provision of support to assist the learner in assimilating or constructing internal representations of these knowledge structures within the subject or problem domain at hand. Metacognitive scaffolding. Metacognitive knowledge is knowledge about cognition in general as well as awareness of and knowledge about ones own cognitive abilities and processes (Anderson & Krathwohl, 2001). Metacognition is a key element of self-directed and self-regulated learning (Biggs, 1987; Zimmerman & Schunk, 1989; Boekaerts, Pintrich, & Zeidner, 2000), and hence has especially important implications for the design of learning tasks and environments that seek to afford a high level of learner autonomy and control. Metacognitive scaffolding is support for the learner as he/she manages and monitors his/her own learning, which includes setting learning goals, formulating and selecting cognitive strategies, regulating learning processes, and engaging in reflection on both the processes and the outcomes. In addition to procedural, cognitive, and metacognitive knowledge, Anderson and Krathwohl (2001) identify a fourth knowledge category, factual knowledge, which we consider to be less relevant to the present discussion on scaffolded discovery learning. This is because it is arguable that the learning of factual knowledge through discovery learning methods does not really require scaffolding in the way the learning of procedural, conceptual, and metacognitive knowledge does. Although scaffolds may be implemented that point or direct learners to materials or resources aimed at developing their basic factual knowledge, our contention is that suggesting resources in this way is a means of supporting the learner in regulating his/her own learning strategy, and as such is more appropriately classed as a form of metacognitive scaffolding.
The Pedagogical Dimension

The second dimension of our scaffolding framework, the pedagogical dimension, categorizes the ways in which scaffolding can be provided in technology-mediated learning environments. The three categories in this dimension are explained below. In arriving at these categories we examined the various techniques implied by categories used in earlier frameworks, such as that proposed by McLoughlin (2002), and attempted to identify the conceptually distinct pedagogical techniques encompassed within them. Instruction. This is the provision of support for the learner through lectures, demonstrations, or the modeling of ideal or expert behaviors. It normally occurs prior to the commencement of a learning task or activity (known as front-loaded instruction), although instances of back-loaded instruction exist, for example in the form of summaries or reminders about key points following the completion of an activity. Coaching. This involves the provision of support for the learner during learning tasks. Such support is normally reactive in that it is delivered to address identified misconceptions or gaps in the learners knowledge or understanding. It is also adaptive in that each learner is provided with coaching that is individualized to suit his/her specific needs as distinct from those of other learners and changed as needed over the course of a particular task. Tools. Not all forms of scaffolding entail intervention by an external agent. Scaffolding can also be achieved through making available supporting tools that the learner is able to invoke when necessary, choosing when, where, and how to use these to support his/her learning. (Here, tools refers to both discrete software
ISLS
228
ICLS2012
objects or functions within the application or environment as well as features or attributes of the overall environment.) Such tools enable the learner to undertake given tasks or support the performance of those tasks by making certain aspects of the tasks easier. They include tools that help with procedural tasks, tools that help with understanding concepts, and tools that help the learner to perform metacognitive functions. Metacognitive scaffolding, for example, can be provided by supplying a cognitive tool (Jonassen, 1994) such as an electronic notepad or a concept-mapping tool and encouraging learners to use to it record their thinking and articulate/externalize their mental models.
Examples Illustrating the Framework

As Table 1 shows, the three techniques that comprise the pedagogical dimension of our framework can each be used to target the three types of knowledge development, yielding nine distinct types of scaffolding in total that are possible within a technology-mediated learning environment. General examples of the categories produced by the intersections of the pedagogical techniques and knowledge types are contained within the nine corresponding cells of the table. More specific, detailed examples from various disciplines that are representative of the categories are included in the ensuing subsections. Table 1. Dimensions of scaffolding in technology-mediated learning environments PEDAGOGICAL Dimension Coaching
Example: Advice about how to navigate through a part of the environment (i.e., when the learner has become stuck) Example: Alternative explanation of a concept, delivered in response to the learners unsuccessful attempt to carry out a task Example: Rhetorical or leading questions in the form of prompts asked of the learner to help him/her solve a problem
KNOWLEDGE Dimension Procedural
Instruction
Example: Presentation of a recommended sequence of steps to carry out an overall learning task, as part of an orientation or briefing preceding the task Example: A video lecture on a particular theoretical concept in the learning domain Example: Suggestion of a number of possible learning tasks that the learner could undertake to further develop or reinforce his/her understanding
Tools
Example: Provision of a tool that allows the learner to bookmark a page or location for later viewing
Conceptual
Metacognitive
Example: Provision of a visualization tool to aid the learners understanding of concepts within the learning domain Example: Provision of a conceptmapping tool that helps the learner to articulate and reflect on their current understanding of an aspect of the learning domain
Procedural Instruction
Procedural instruction is instruction about the steps involved in a particular procedure, which may either be an overall task to be undertaken or a subsidiary task (sub-task) that forms part of the larger, overall task. For example, an accounting student could be provided with a list of the steps involved in developing an annual budget for a small business, presented as a set of clickable hyperlinks that take him/her to a more detailed explanation of each step. Moreover, a digital video could be provided in which an explanation of the key aspects of the task are presented. Screencasts could be used to demonstrate how to perform various procedures in the accounting software package that is to be employed to prepare the budget. In an immersive virtual learning environment, procedural instruction may be supplied via a personified agent within the environment whose actions and speech are controlled by a simple script. For example, a virtual teacher appearing as an avatar within the environment might model the correct procedure for operating a piece of equipment or machinery.
Procedural Coaching
Procedural coaching is similar to procedural instruction except that it is provided in response to some action (or lack thereof) performed by the learner, perhaps illustrating deficiencies or gaps in knowledge of a specific procedure. The simplest form of such coaching would be demonstrations of procedures provided as an option when it is detected that that the learner is attempting a particular procedure. An example of this can be seen in the case of a veterinary science student undertaking a problem-based learning activity calling for the diagnosis of the condition suffered by a simulated animal. Here, the student could be provided with suggestions about particular examination procedures or tests to be done on the animal, along with the option to view multimedia content relating to those procedures. Rather than being completely dynamic, procedural coaching could consist of a scripted animation performed by an agent, a pre-recorded audio-visual presentation or a series of text messages displayed to the learner, in a similar way to procedural instruction. The provision of such pre-fabricated or scripted support could be implemented using a set of relatively simple rules or triggers, specifying which animations, videos, recorded audio or text messages to activate in response to which learner actions. For example, in the veterinary science activity, the choice of what suggestions to provide to the learner could be determined through simple
ISLS
229
ICLS2012
conditional or branching logic that detects whether the learner has completed certain identified milestones within the procedure. The suggestions could be delivered proactively by the system or in response to a request from the learner (e.g., by clicking on a button/menu item labeled Help or Hints).
Procedural Tools
Many general-purpose tools commonly found in technology-mediated learning environments, such as calculators, graph generators, and spell checkers, can be considered instances of procedural tools. An example of a more specialized procedural tool is a template given to a teacher education student to simplify the process of designing a lesson; another is an applet made available to a nutrition student within a web-based scenario learning environment that he/she can use to help calculate the fat, sugar, and carbohydrate content of a particular meal when attempting to plan a diet for a patient.
Conceptual Instruction
Conceptual instruction is instruction to help learners understand the concepts involved in a task that they are either about to undertake or have just completed. As with procedural instruction, this type of instruction could be provided as video, audio, or text material displayed on screen to the learner, or as scripted animations carried out by a personified agent. For example, a biology student could be provided with an animation illustrating the concept of photosynthesis with an audio track explaining the key aspects.
Conceptual Coaching
Conceptual coaching differs from conceptual instruction in that it occurs as a result of an action carried out by the learner implying he/she is in need of assistance or guidance in relation to a conceptual aspect of his/her current task. If conceptual coaching is to be provided by the systemas distinct from a real tutor present within a multi-user virtual environment, for examplethen complex inferences about the learners cognitive learning processes are required. That is, unlike procedural coaching, which may be provided by the system in response to an unsuccessful attempt at a procedure by the learner, system-provided conceptual coaching requires the software to make assumptions about the cognitive aspects of the current task and about possible misconceptions indicated by learners behaviors or actions. As an example of conceptual coaching, an oncology student using virtual microscopy to analyze images of cancer cells might be provided with an animation with an instructional audio track explaining the key indicators of certain cell abnormalities. This might occur as a result of the learner specifying an incorrect classification or diagnosis, or it could occur simply as a result of the learner clicking on Help or Hints .
Conceptual Tools
Conceptual tools are tools that help the learner to understand the concepts involved in a task they are undertaking or in an aspect of the environment they are exploring. These tools can, for instance, be visualization tools that allow the learner to observe aspects of the environment not normally visible to the naked eye. They can also be interactive components of the virtual environment that the learner can use to magically carry out a task to help them to understand a particular concept. For example, a chemistry student undertaking an experiment in a virtual laboratory could be provided with a tool that allows him/her to zoom in to the molecular level to observe a simulation of the molecular structures occurring as a result of the reaction. Some tools that make certain procedures quicker or more efficient may also be considered conceptual tools because they actually scaffold the concepts to be encountered through the task. For example, in a simulated chemistry laboratory, a tool that dynamically calculates and displays the ratio of one solution to another during a titration experiment may help the learner understand the chemistry involved more effectively than in the traditional approach, where he/she must calculate the ratios upon completion of the experiment.
Metacognitive Instruction
Metacognitive instruction may consist of instruction about the big picture or high-level tasks that the learner should undertake within the environment in order to optimize or enhance his/her learning. Mayer (2004) cites the absence of this initial orientation as being one of the key reasons for the failure of pure discovery learning. For example, a student of political science could be provided with a series of focus questions to answer as he/she reads a newspaper article, parliamentary report, or other archival document, to help him/her identify some of the less obvious aspects of the written work. An additional element of such introductory instruction could be the suggestion of self-regulatory and reflective strategies.
Metacognitive Coaching
Metacognitive coaching is coaching to encourage and facilitate metacognitive tasks such as goal setting, formulation and selection of cognitive strategies, self-regulation and management of learning processes, and reflection. Providing opportunities for articulation may be beneficial because it not only highlights and draws
ISLS
230
ICLS2012
the learners attention to specific concepts in the context of the task being performed, but also promotes deep learning by encouraging the learner to be aware of and consciously reflect on his/her evolving understanding of those concepts. For example, a social work student trying to identify the appropriate intervention in a domestic situation based on reading of a set of case notes could be provided with a series of questions such as What are the key aspects of the case that need to be considered? or What legal and regulatory requirements are applicable here?. Such questions would encourage the student to focus on the important aspects of the case and important elements of his/her own prior knowledge while undertaking the analysis. Like conceptual coaching, programmatically implementing metacognitive coaching within a technology-mediated learning environment is not straightforward, due to the need to deduce the cognitive needs of the learner at a particular point in time. Again, an alternative is to provide a list of metacognitive support options potentially relevant to the current task, and have the learner choose the support needed and/or desired.
Metacognitive Tools
Metacognitive tools are tools aimed at helping the learner to undertake metacognitive tasks such as goal setting, self-regulation, and reflection. As an example, a clinical psychology student attempting to diagnose the disorder suffered by a hypothetical patient after reading the transcript of a series of sessions with a psychologist could be provided with a tool allowing him/her to embed comments in the text, coded with different colors for patient history, patient relationships, and patient attitudes.
Conclusion
In this paper, we have proposed a new framework for classifying and understanding the types of learning scaffolds possible in technology-mediated environments. The first dimension of the framework relates to the knowledge development intended to be supported, while the second characterizes the pedagogical technique or way in which the scaffolding is provided. This results in a total of nine scaffolding types, each of which has been illustrated and typified in the paper by means of examples. Future work is needed that assesses the appropriateness and completeness of the proposed framework. This can be accomplished through of theoretical analyses of scaffolding examples from the literature together with empirical studies exploring the characteristics of scaffolding provided by teachers within both face-to-face and online learning environments. Beyond that, research is needed to ascertain precisely which scaffolding types are most appropriate in different technologymediated learning situations and scenarios, and to test their efficacy in supporting different types of learning goals and outcomes within various subject disciplines and domains.
References
Anderson, L. W., & Krathwohl, D. R. (Eds.). (2001). A taxonomy for learning, teaching, and assessing: A revision of Blooms Taxonomy of educational objectives. New York: Longman. Biggs, J. (1987). Student approaches to learning and studying. Melbourne, Australia: ACER. Biggs, J. (1999). Teaching for quality learning at university. Oxford, UK: SRHE and Open University Press. Boekaerts, M., Pintrich, P. R., & Zeidner, M. (Eds.). (2000). Handbook of self-regulation. London: Academic. Bruner, J. S. (1986). Actual minds, possible worlds. Cambridge, MA: Harvard University Press. de Jong, T., & van Joolingen, W. R. (1998). Scientific discovery learning with computer simulations of conceptual domains. Review of Educational Research, 68(2), 179-201. Jonassen, D. H. (1994). Technology as cognitive tools: Learners as designers. Paper presented on ITForum. http://itech1.coe.uga.edu/itforum/paper1/paper1.html Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75-86. Leutner, D. (1993). Guided discovery learning with computer-based simulation games: Effects of adaptive and non-adaptive instructional support. Learning and Instruction, 3(2), 113-132. Mayer, R. E. (2004). Should there be a three-strikes rule against pure discovery learning? The case for guided methods of instruction. American Psychologist, 59(1), 14-19. McLoughlin, C. (2002). Learner support in distance and networked learning environments: Ten dimensions for successful design. Distance Education, 23(2), 149-162. Roblyer, M. D., Edwards, J., & Havriluk, M. A. (1997). Integrating educational technology into teaching. Upper Saddle River, NJ: Merrill. Vygotsky, L. S. (1978). Mind and society: The development of higher mental processes. Cambridge, MA: Harvard University Press. Winnips, K., & McLoughlin, C. (2000). Applications and categorization of software-based scaffolding. In J. Bourdeau & R. Heller (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia, and Telecommunications 2000 (pp. 1798-1799). Chesapeake, VA: AACE. Wood, D., Bruner, J. S., & Ross, G. (1976). The role of tutoring in problem solving. Journal of Child Psychology and Psychiatry, 17(2), 89-100. Zimmerman, B. J., & Schunk, D. H. (Eds.). (1989). Self-regulated learning and academic achievement: Theory, research, and practice. New York: Springer-Verlag.
ISLS
231
ICLS2012
How a power game shapes expressing opinions in a chat and in an argument graph during a debate: A case study
Galle Molinari, Distance Learning University Switzerland, Techno-Ple 5, Case Postale 218, 3960 Sierre, Switzerland, gaelle.molinari@unidistance.ch Kristine Lund, ICAR Lab (UMR5191)-CNRS, University of Lyon, ENS, 15 Parvis Ren Descartes, BP 7000, 69 342 Lyon Cedex 07, France, kristine.lund@ens-lyon.fr Abstract: We investigate the dynamics of power and influence in a CSCL debate. We show how the power game between students shapes their use of a chat tool and an argument graph tool, in particular for expressing opinions. The power of one student over another is expressed as better mastering content as well as mastering the specificity of marking opinions on graph elements. The originality of our work is to study the particular relation of uncertainty and control of the use of two CSCL tools and also to show how it may influence opinion change.
Introduction
In this paper, we examine a collaborative argumentative debate with a focus on how socio-relational processes, such as power and influence, shape the use of computer-mediated communication tools (and vice versa) during the debate. Classroom debates can be characterized by a dynamic equilibrium of cooperation and competition. At some points in the debate, students make an effort to cooperatively interact so as to reach an agreement whereas simultaneously a latent competition co-exists between them. Competition in a group can be seen as a relationship of tension, constraints and asymmetry between group members who may be motivated to exert influence over their partners beliefs, opinions and actions. In a debate, a power game can progressively emerge where participants may use the representation of their partners doubts, questions and uncertainty so as to tailor their behavior and impose their knowledge and points of view. This kind of interpersonal relationship may lead to an increase in face-threatening issues and behavior (Muntigl & Turnbull, 1998). Uncertainty and control may be viewed as fundamental factors in the dynamics of power and influence between group members (Ducheneaut, 2002). In a CSCL environment, we argue that collaborative partners may gain power by demonstrating not only their level of expertise in a particular learning topic but also their ability to understand and control the technology used in their workplace. In other words, to achieve power, CSCL participants need to show a certain level of certainty and control in the way they use the technological tools provided by their learning environment. They have to demonstrate that they are able not only to master the technical aspects of tool usage but also to appropriate tools by adapting them to their goal-directed activity. Moreover, in the process of a debate, the distribution of power may depend on the way participants express and structure their opinions. It has been shown that people with higher power in a group are more likely to engage in voice (Islam & Zyphur, 2005). A power game may also occur when participants of the debate challenge their partners positions and make them feel uncertain regarding the correctness of their opinions. Finally, coherence is one criterion that defines the quality of the space of debate (Quignard et al., 2003), and individuals who want to dominate the debate need to appear consistent in their opinion, in particular by providing a balance of pro and against arguments that is consistent with their expressed opinion. Our study investigates the dynamics of power and influence between two students Alice and Sally who debated about authorizing genetically modified organisms (GMOs) using a chat tool and an argument graph tool (DREW). The graph tool was designed so that students could express their opinions for any element in the graph and also visualize divergences in their opinions. Previous to the debate, students showed divergent opinions on the authorization of GMOs (Alice was against, Sally in favor). After the debate, Alice changed her opinion and moved closer to Sallys opinion; in contrast, Sally stayed true to her initial view. Our goal is twofold. First, we show how a power game between both students progressively emerged during the debate and shaped the use of the chat and graph tools, in particular for the argumentative activity of expressing opinions. In other words, we illustrate how uncertainty and control seen as critical elements in the power game influence the way students use both the chat and the graph tools to express and argue their opinions. Second, by examining Alice and Sallys power game, our goal was also to provide some clues for understanding how Alice changed her opinion after the debate. To our knowledge, the literature does not focus on the particular relation of power to CSCL tool use. We therefore argue that it is crucial to consider all modes of interaction within a CSCL system when examining how power games occur and may influence opinion change.
Method
In this section, we present our population, our task sequence, our motivation for choosing the dyad we did for our case study, and the analysis methods used. Our dyads were composed of French secondary school students (16-17 years old) and we chose Alice and Sally for the study presented here (their names have been changed).
ISLS
232
ICLS2012
Task Sequence
The global pedagogical sequence scenario instantiated for this case study can be described as follows: (0) Training. Alice and Sally were introduced to the basic notions and techniques necessary for debating. They were also trained on the CMC tools to be used in phase 2 (debate), that is, the chat and the argument graph tools. (1) Preparation. After reading multiple texts designed to provide (contradictory) opinions of various social actors (e.g., Greenpeace, Limagrain) in the GMO debate, Alice and Sally were asked to individually write an argumentative text (pre-text) in which they had to express their own opinions on the use of GMOs. (2) Debate. Alice and Sally debated the question of authorizing GMOs through parallel use of the chat and the argument graph tools. When using the graph tool, they had the possibility to give their respective opinions for any elements (boxes and links) in the graph (each students opinion appeared in a different color). Moreover, boxes with respect to which opposed opinions have been expressed appeared in a crushed form. (3) Consolidation. Alice and Sally were asked to modify their pre-texts in the light of the debate (post-texts).
Motivations for the Dyad Choice and dyad characteristics in terms of individual texts
Alice and Sally were chosen for this study based on the analysis of differences between their individual argumentative texts before (pre-texts) and after (post-texts) the debate using the criteria that define the Quality of the Space of Debate (QSD, Quignard et al., 2003): richness (number of arguments), degree of elaboration of arguments, balance (a text is well-balanced when it contains approximately as many pro and contra arguments across a variety of topics), coverage (a text has a wide coverage when the arguments cover the different topics and sub-topics of the question), and coherence (a text is coherent when the expressed opinion is consistent with the general balance of pro/against arguments). Alice and Sally were also chosen because there was a change in opinion as a result of the debate. One student Alice explicitly expressed in her post-text that she had changed her opinion on the topic discussed after having debated it with her partner (change in opinion). Alice was initially against the authorization of GMOs; her final opinion was more balanced after the debate. It seems that the main driver for the change in her opinion was the potential benefits of GMOs for human medicine (after the discussion, I changed my point of view about GMOs. It seems to me that GMOs will be of great help for therapeutic purposes). In parallel to this change in opinion, the number of arguments in Alices text increased (increase in richness score, +4). In particular, she added four arguments in favor of GMOs in her post-text (arguments against her initial opinion), and 3 low-elaborated arguments about health issues (all these arguments were introduced by Sally in the debate). As a result, her post-text was more well-balanced in terms of arguments in favor of and against compared to her pre-text (increase in the balance score, +0.45). However, the coherence of Alices text that is, the consistency between the opinion she expressed and the general balance of the pro/against arguments she provided after debate also decreased (decrease in the coherence score, -0.25). Finally, there was an increase in the number of topics treated in Alices text after debate (increase in the coverage score, +0.29). The other student, Sally, initially in favor of GMOs, did not modify her opinion after the debate. She added an equal number of arguments in favor of (3) and against (3) GMOs (increase in the richness score, +6); as a consequence, her pre- and post-texts did not differ with respect to balance and coherence scores. Among the 3 arguments against GMOs she added in her post-text, 2 of them were introduced by Sally herself, during the debate. All the arguments added were low-elaborated. She did not add any new topics to her post-text. It should be noted that the modifications both students made to their pre-texts after debate mainly consisted in the addition of an extra-part relating to what was expressed in the interaction, tacked on the end of the pre-text. They did not reorganize their texts and they almost never erased any of their initial arguments.
Analysis Methods of Alice and Sallys interaction

We used the RAINBOW method to analyze the argumentative (opinions, argumentation, broaden and deepen) and non-argumentative (social relation, interaction management, task management) activities in both the chat and the argument graph (see Baker et al., 2007). We also used ADAM (Argumentation Diagram Analysis Method; see Lund et al., 2007) to examine how the argument graph was structured. Tatiana (Trace Analysis Tool for Interaction Analysts; Dyke, Lund, & Girardot, 2009) was used to synchronize chat and graph interventions from DREW (Dialogue Reasoning Educational Website) and to generate temporal visualizations of where opinion expressions occurred in both the chat and graph and for both Alice and Sally.
Results
A first zoom-in on the dyads interaction (RAINBOW and ADAM analyses)
The RAINBOW analysis shows that Sally preferentially used the chat for task management purposes and the graph for marking opinions (supporting and challenging actions on boxes and links). Alice performed more task management actions (creating, moving or deleting boxes and links) in the graph compared to Sally. Alice also initially preferred using the chat to the graph when expressing her opinion.
ISLS
233
ICLS2012
The interaction between Alice and Sally can be divided into 2 successive sub-debates: health issues and food issues. Chat utterance analysis shows that Alice and Sally contributed equally to the chat for both subdebates. The analysis of graph events shows that in the health debate, Sally was consistently more active than Alice, to the greatest extent in task management and in marking opinions. In contrast, Alice was much more active in task management in the food debate whereas Sally continued to mark more opinions. The ADAM analysis focuses on how the argument graph was constructed and organized. It is Sally who created the first box (GMO) immediately followed by Alice who created the GMO poison in food box; thereafter Sally and Alice alternated creating boxes in the graph. Sally provided arguments that systematically defended the authorization of GMOs whereas Alice provided arguments that systematically attacked this thesis. Alice and Sally diverged in opinion about 4 boxes; among these 4 boxes, 3 were added by Alice to the graph. Here is a brief description of the graph as it appeared at the end of the interaction. It consisted of 2 main branches, each one corresponding to a sub-debate (the health branch and the food branch) and containing a thesis box (the GMO production box for the health branch and the GMO poison in food box for the food branch). There were 2 levels of arguments in both branches: the 1st-level arguments are arguments directly linked to the thesis box, the 2nd-level arguments are arguments on arguments. Most of 1stlevel arguments were created by Sally whereas Alice mainly created 2nd-level arguments. In the health branch, the GMO production box was placed at the top and corresponded to the thesis box of this branch. This box was the first box added to the graph by Sally. Sally initially named it GMO but Alice decided to modify its content at the middle of the interaction (the beginning of food branch): the GMO box became the GMO production box. In the health branch, Sally created two 1st-level arguments in favor of GMOs (creates vaccinations and medicaments, allows treating illness) and positively linked them to the GMO production box (thesis). There were four 2nd-level arguments: one in favor of GMOs (heal cancer) created and positively linked by Sally (it was linked to the creates vaccinations and medicaments box), and three against GMOs created and negatively linked by Alice (the resists against antibiotics box was linked to the allows treating illness box; both the humans = guinea pigs and they go against ethics boxes were linked to the creates vaccinations and medicaments box). For 2 of the arguments against GMOs created by Alice (resists against antibiotics, human = guinea pigs), there was a divergence in opinion. In the food branch, the GMO poison in food box (an argument against GMOs) corresponded to the thesis box. This box was the second box added to the graph by Alice and both students diverged in opinion about this box. Initially, this box was negatively linked by Sally to the GMO box and corresponded to a 1stlevel argument. At the very end of the interaction, Alice deleted this link so that the GMO poison in food box became the second thesis (with GMO production as the first thesis). Note that when an argument branch is disconnected from the rest of the graph, the first box on which arguments are made acquires the status of thesis. In the food branch, Sally created 2 arguments in favor of GMOs (reduction of malnutrition and hunger, better quality products) and negatively linked them to the GMO poison in food box. These 2 arguments in favor of were initially 2nd-level arguments but became 1st-level arguments after the food branch was cut off. For one of these 2 arguments (better quality products), a conflict of opinion between both students occurred. Moreover, 2 arguments against GMOs (product without flavour, lose nutritional value) were created and negatively linked by Alice to the better quality products box. These 2 arguments against were initially 3rdlevel arguments but became 2nd-level arguments after the food branch was cut off.
A second zoom-in on the interaction (three sequences: opening, middle, closing)

Using Tatiana software (Dyke, Lund, & Girardot, 2009), temporal visualizations were generated to show the dyads dynamics of expressing opinions in both the chat and the graph. We chose 3 sequences of the interaction on which to zoom in because of what they illustrated about Alice and Sallys positioning game: the beginning, a middle sequence and the closing sequence. We show hand-drawn visualizations of these sequences below.
Opening sequence
In this sequence, Alice writes HELLO (in French SALUT; see Figure 1). Then, after floundering a bit with how to connect to the graph, she asks Sally if she masters the interface (TU MAINTRISE?; notice that Alice opens with uppercase letters in the chat). Sally seems to be more comfortable with how to connect to the graph. She answers affirmatively to Alices question (oui, a va) and then creates the first box (GMO) in the graph. Alice then provides her negative opinion about GMOs in the chat (moi je suis contre les ogm) which is consistent with her initial position as she expressed it in her pre-text. Alice however forgets to also mark her negative opinion on the GMO box in the graph. Sally reacts to Alices intervention in the chat by marking her opinion on the GMO box she just created: she chooses the opinion in favor of which is consistent with what she wrote in her pre-text. However, in the chat, Sally then writes that she is neither against nor is she in favor of GMOs (moi je suis pas contre et pas pour non plus). Compared to Alice, Sally seems to have the strategy of not confessing her opinion in a clear and coherent way. In addition, she contradicts herself between modalities (chat: not against and not in favor; graph: in favor). This strategy might confuse Alice and make her uncertain
ISLS
234
ICLS2012
about Sallys opinion. It might also create tension between them. Perhaps Sally is testing Alices knowledge about how to mark opinions on the graph, but Alice does not notice the inconsistency in Sallys behavior.
Figure 1. Temporal visualizations for the beginning (left) and middle (right) of Alice and Sallys interaction
Middle sequence
At the beginning of this sequence (Figure 1), Sally uses the chat to ask Alice to express her opinion on a box Sally just created (are you for or against what I just said?, tu es pour ou contre ce que je dis?), and then Sally marks her opinion for on this box. A game of creating a box and marking an opinion on the box was then played between Alice and Sally with the implicit goal of displaying consistency between the argument provided and the opinion marked on this argument: (1) Alice creates the resists against antibiotics box and marks her opinion against on this box; (2) Sally creates the creates vaccinations box and marks her opinion for on this box; (3) Alice creates the humans = guinea pigs box but does not immediately mark her opinion on this box; (4) Sally uses the chat to check whether Alices marked opinion on the resists against antibiotics box really reflects what she claims (do you think that GMOs resist against antibiotics?, tu penses que les OGM resistent aux antibiotiques?). Then Sally marks her opinion against the humans = guinea pigs box; (5) Alice is flustered by Sallys question (no, I mean yes yes, non, si si) thus perhaps showing her incertitude; (6) Sally takes off her opinion from Alices box humans = guinea pigs but then puts it back on. Finally, Sally points out to Alice (in an authoritarian manner) that she did not use the graph in the correct way (you must complete your arrows!). Therefore, the tension between both partners still remains high. Sally tries to destabilize and gain power over Alice by forcing her to systematically express her opinion on each element in the graph, and thus by highlighting Alices difficulties in understanding and mastering the graph as a tool for expressing opinions. Sally also tries to challenge Alices opinion by questioning her capabilities to provide arguments consistent to her own position (Sally directly attacks the coherence of Alices space of debate).
Closing sequence
In the closing sequence, Sally initiates closing (ok, were going to stop now, thank you for your collaboration). But Alice seems to want to pursue a bit more and proposes to conclude (ok, we should conclude): indeed according to the instructions, the partners should summarize their main points of convergence after debating and Alice is reminding Sally of that. Sally does not seem to agree that there is still work to be done and again tries to close the conversation: thanks and see you soon. Alice gives up on the summary task without further argument (is it useless to try and convince Sally of anything?) and seems to admonish her: First of all, goodbye and thank you for your collaboration (in French dabord adieu et merci de ta collaboration). This is a somewhat strange formulation: Adieu is a kind of final goodbye that means you dont expect to see the person again. Sally responds with an its nothing (de rien), which presumably responds to Alices thanking her. After this passage of chat utterances, Alice marks her opinion in favor of the two last boxes she created in the food branch (product without flavour and lose nutritional value), both arguments against better quality products. Then she writes in the chat that she did not change her mind (Im still against), although after the debate, she will privately change her post-text opinion. To save her positive face, Sally replies that she was not affected by the fact that Alice did not change her opinion (it does not bother me at all). Sally also writes that she did not change her mind either. Next, Sally definitely closes the conversation in the chat (bye its over), but a kind of erasing game ensues in the graph where both Alice and Sally erase each others arrows. Sally cleans up the graph by mainly erasing unconnected and therefore useless arrows created earlier by Alice. Alices erasing action has greater consequences on the final form of the graph. As already stated, Alice erases Sallys link between the GMO production box and the GMO poison in food box around one minute before the end of the interaction. This event is noteworthy in that it dramatically changes
ISLS
235
ICLS2012
the way the final argument graph is structured: Sallys GMO production becomes the thesis box of the health branch whereas Alices GMO poison in food becomes the thesis box of the food branch. It is as if Alice tries to repair her face and reclaim the debate by reorganizing the argument graph into two sub-debates and deciding its final form. Since Alice created the GMO poison in food box originally, turning it into a stand-alone debate and thus giving it equal status to Sallys health branch gives Alice more equal power in the debate. Notice that Alice deleted this link after Sally had left and was not there to attack or interrupt her.

Our first goal was to examine the dynamics of Alice and Sallys chat and graph usage in particular in the argumentative activity of expressing opinions, and how it illustrated their power and influence game. To our knowledge, there is little research that has addressed the particular relation of power to tool usage (Ducheneaut, 2002). We argue however that there is a need to consider all modes of CSCL communication when understanding how power games occur and influence participants actions and views. In this study, we particularly focused on how uncertainty and control (Ducheneaut, 2002) played a role in the emergence of the power game between Alice and Sally and influenced the way they used both the chat and graph tools to express and argue their respective opinions. First, it was shown that Sally gained power over Alice not only by dominating the debate Sally had the role of proposing arguments whereas Alice took the role of reacting to them but also by demonstrating a higher level of understanding and mastery of the argument graph. Compared to Alice, Sally was more likely to engage in voice (Islam & Zyphur, 2005) and used the graph as the main tool for expressing and arguing opinions (Sally used the chat mainly for task management purposes). Alice had more trouble using the graph to give her opinions and seemed to find it easier to do that via the chat. Sally also tried to destabilize Alice by constantly pointing out her difficulties in using the graph to express opinions. She tried to attack Alices positive face using strategies such as: (a) testing Alices ability to detect her (true) opinion about GMOs by providing contradictory opinions in the chat and the graph, (b) forcing Alice to systematically express her opinion on each element in the graph, and (c) challenging the consistency of Alices opinion. In other words, Sally mastered the tool, checked up on and goaded Alice, exposing Alices uncertainties and inconsistencies. Alice was somewhat humiliated in the interaction; she therefore did not want to publicly admit that Sally had convinced her, although she was not above re-using Sallys arguments privately, while modifying her post-text. Our second goal in this study was to examine Alice and Sallys power game so as to provide some clues for understanding how Alice changed opinions after the debate. One way to understand this change is to examine Alices strategies used in the second part of the debate to regain her lost face. Alice seemed to react to Sallys provocation and attacks by being more active in the building of the graph, and also more systematic in marking her opinion in the graph. She also tried to appropriate the debate for her own purposes by rearranging it into two sub-debates health and food and therefore deciding its final form. In particular, she tried to gain more equal power in the debate by turning the GMO poison in food box she originally created into a stand-alone food branch thus by taking the lead of the food debate, and by leaving the lead of the health debate to Sally (as Sally was the one who created the thesis box of the health branch). Notice that the arguments from Sally that Alice decided to add to her post-text after the debate were arguments in favor of GMOs that came from the health branch. To conclude, this study shows that the analysis of how a power game occurs and shapes the use of CSCL tools (e.g. chat, graph, etc.) and how restorative strategies are used to repair face that was lost during the power game is a promising perspective to foster the understanding of learners potential belief change.
References
Baker, M., Andriessen, J., Lund, K. van Amelsvoort, M. & Quignard, M. (2007). Rainbow: A framework for analysing computer-mediated pedagogical debates. IJCSCL, 2, 315-357 Ducheneaut, N. (2002). The social impact of electronic mail in organizations. Information, Communication, & Society, 5(2), 153-188. Dyke, G., Lund, K., & Girardot, J.-J. (2009). Tatiana: an environment to support the CSCL analysis process. In C. O'Malley, P. Reimann, D. Suthers & A. Dimitracopoulou (Eds.), CSCL 2009 Conference Proceedings (pp. 58-67). Rhodes, Greece: International Society of the Learning Sciences. Islam, G & Zyphur, M.J. (2005). Power, voice and hierarchy: Exploring the antecedents of speaking up in groups. Group Dynamics: Theory, Research and Practice, 9(2), 93-103. Quignard, M., Baker, M., Lund, K. & Sjourn, A. (2003). Conception d'une situation d'apprentissage mdiatise par ordinateur pour le dveloppement de la comprhension de l'espace du dbat. Actes de la confrence EIAH 2003 (pp. 355-366). INRP-ATIEF. Lund, K., Molinari, G., Sjourn, A. & Baker, M. (2007) How do argumentation diagrams compare when student pairs use them as a means for debate or as a tool for representing debate? IJCSCL, 2, 273-295. Muntigl, P. & Turnbull, W. (1998). Conversational structure and facework in arguing. Journal of Pragmatics, 29(3), 225-256.
ISLS
236
ICLS2012
Using the learner-generated drawing strategy: How much instructional support is useful?
Annett Schmeck (nee Schwamborn), Luisa Friedrich, Maria Opfermann, & Detlev Leutner, Instructional Psychology, Duisburg-Essen University, 45117 Essen E-mail: annett.schmeck@uni-due.de, luisa.friedrich@uni-due.de, maria.opfermann@uni-due.de, detlev.leutner@uni-due.de Learner-generated drawing is a learning strategy which is used to improve students text comprehension. Research has shown that benefits of the learner-generated drawing strategy strongly depend on drawing-accuracy and that students need instructional support to draw accurate drawings. However, less is known about how much support is needed. Thus, in the present study, one hundred and two 9th graders read a science text and were instructed to generate drawings during reading with varying degrees of instructional support. Results show that students who learnt with a toolbar showing all the relevant elements for drawing showed higher drawing-accuracy scores associated with less learning time than students who learnt with pre-drawn drawing backgrounds or those who learnt without any support during drawing.
Objective and Theoretical Framework

Students reading complex and difficult scientific texts do often have problems with the cognitively highly demanding processes of text comprehension. Based on models of self-regulated learning (e.g., Boekaerts, 1999; Weinstein & Mayer, 1986; Zimmerman, 2001) the so-called learner-generated drawing strategy is one possible way to foster students text comprehension (Alesandrini, 1984; Schwamborn, Mayer, Thillmann, Leopold, & Leutner, 2010; van Meter & Garner, 2005). In doing so, students are instructed to read a text and to generate drawings by themselves that reflect the important ideas of the text. According to van Meter and Garner (2005) this generation process causes students to engage in cognitive and metacognitive processes which foster deeplevel understanding of the to-be-learned information. The integration of the internal verbal and pictorial representations into a coherent mental model, for example, is induced (Mayer, 2009). Additionally, metacognitive processes such as self-monitoring and self-regulation are induced. Thus, drawing a picture to a text is a productive learning activity that encourages generative processing and thus fosters deep level understanding (de Jong, 2005; Kirschner, Sweller, & Clark, 2006; Mayer, 2003, 2004; Schwamborn et al., 2010). However, instructing students to generate drawings to a text does not automatically guarantee deep level understanding. Problems concerning the product of drawing can occur, like a low quality of the students self-generated drawings (Leopold, 2009; van Meter, 2001; van Meter, Aleksic, Schwartz, & Garner, 2006). A low quality of the self-generated drawing means that the drawing does not accurately reflect all important ideas and causal relations that are described in the text which are necessary for deep-level understanding. Thus, benefits of the learner-generated drawing strategy depend on the quality respectively the accuracy of students self-generated drawings during reading which is the so-called prognostic drawing effect (Schwamborn et al., 2010). Accordingly, research on the learner-generated drawing strategy showed that the accuracy of students self-generated drawings respectively the benefits of the strategy depend on scaffolding students to use the strategy appropriately. That is, positive effects of the strategy on text comprehension depend on the instructional support mediated by the accuracy of the learner-generated drawings (e.g. Schwamborn et al., 2010). In studies of Leutner, Leopold, and Sumfleth (2009) and Alesandrini (1981), for example, students did not benefit from using the pure, unsupported learner-generated drawing strategy. Lesgold, Levin, Shimron, and Guttman (1975), van Meter et al. (2001, 2006), and Schwamborn et al. (2010), however, instructionally supported the drawing process in their studies and found benefits of the drawing strategy. Van Meter et al. (2001, 2006), for example, supported students with illustrations and prompting questions after the drawing process. Lesgold et al. (1975) supported students during the generation process by giving them cut-out figures. Accordingly, Schwamborn et al. (2010) implemented a drawing prompt to instructionally support students during the drawing process and thus to increase the accuracy of the generated drawings. The drawing prompt included (a) a toolbar showing (and labeling) all the relevant elements for drawing and (b) partly pre-drawn backgrounds for students drawings (see Figure 1). The authors found a significant positive effect of the learner-generated drawing strategy on text comprehension. Thereby high-accuracy drawers outperformed low-accuracy drawers. In sum, research on the learner-generated drawing strategy has shown that benefits of the strategy strongly depend on the accuracy of students drawing during reading the text. That is, the accuracy of learners drawings during learning can predict the quality of their text comprehension. Thus, the accuracy of drawing during learning is a reflection of the actual use of the learner-generated drawing strategy (e.g. Schwamborn et
ISLS
237
ICLS2012
al., 2010; van Meter et al., 2001, 2006). Additonally, results of Schwamborn et al. (2010) give indication that students do need instructional support during the use of the learner-generated drawing strategy to generate highaccuracy drawings. However, we do not know how much instructional support is necessary to enable students to generate high-accuracy drawings. Indeed, Schwamborn et al. (2010) suggested a drawing prompt; however, they did not conduct a treatment-check of their drawing prompt. Thus, it would be useful to vary the amount of drawing support given to the learner ranging from high support, as in the study of Schwamborn et al. (e.g., giving the learner a partially drawn background and a toolbar that presents and labels each element to be drawn) to low support (e.g., giving the learner only a partially drawn background). Thus, in the present study four experimental drawing conditions with varying degrees of instructional support during the use of the learner-generated drawing strategy were implemented: (a) a toolbar showing (and labeling) all the relevant elements for drawing; (b) partly pre-drawn backgrounds; (c) a complete drawing prompt, that is a toolbar and partly pre-drawn backgrounds; (d) a control condition without any support during drawing. The goal of our study was to investigate how much instructional support students actually need to generate good respectively accurate drawings while using the learner-generated drawing strategy. The accuracy of students drawings during learning was used as predictor of the quality of students text comprehension. __________________________________________________________________________________________
__________________________________________________________________________________________ Figure 1. Screen shot of the complete drawing prompt for the second paragraph of the learning booklet (cf. Schwamborn et al., 2010; p. 875).
Data Source
One hundred and two 9th graders in higher track secondary schools participated in this study. The mean age was 14.3 years (SD = 0.55) and 56.9% were female.
Method
There were four treatment groups: 25 students served in the toolbar group, 26 served in the pre-drawnbackground group, 25 served in the complete drawing prompt group, and 26 served in the control group without any instructional support during drawing. The materials used were adapted from Schwamborn et al. (2010). In addition, we also measured students individual learning times. The dependent variables were students drawing accuracy as well as their individual learning times. Participants were tested in classrooms at their schools. All materials were paper-pencil based. Within their classes, students were randomly assigned to one of the four groups. First, students completed a participant questionnaire (e.g., current chemistry-grade). Second, students were tested on spatial ability (paper-folding test; Ekstrom, French, & Harman, 1976). Third, students were given instructional booklets corresponding to their assigned group. They were instructed to read the instruction and were then tested on their current motivation (questionnaire on current motivation; Rheinberg, Vollmeyer, & Burns, 2001). Then they were instructed to learn with the text. The text was about the chemistry of doing laundry, consisted of about 1000 words, and was divided into six paragraphs (cf. Schwamborn et al., 2010). Students did not have prior-knowledge about the topic of the text, as according to the curriculum this topic is introduced at the end of grade nine. Students in all four conditions were instructed to carefully read the text in order to comprehend the material and additionally were instructed to draw pictures for each text paragraph that represent the main ideas of that paragraph. Students in the toolbar condition were instructed to read the text and additionally to draw pictures for each paragraph
ISLS 238
ICLS2012
using the elements from the toolbar. Students in the pre-drawn-background condition were instructed to read the text and additionally to draw the pictures for each paragraph using the partly pre-drawn picture backgrounds. Students in the complete drawing prompt condition were instructed to read the text and then to draw pictures using the drawing prompt, which means to use the elements from the toolbar and draw the pictures into the predrawn background. Students in the control group were instructed to read the text and additionally to draw pictures for each paragraph without any instructional support. Each student was allowed to learn at his/her own pace whereby learning time was measured.
Results and Conclusions

Scoring
We determined the drawing-accuracy score using a coding scheme. This coding scheme was based on expert drawings and a checklist specifying important features of the drawings (cf. Schwamborn et al., 2010). Students could get a maximum of 27 points on the drawing-accuracy score. Two independent raters, student assistants from science education, scored each of the six drawings for each student with an acceptable interrater reliability of Goodman-Kruskal gamma = .78.
Are the groups equivalent on basic characteristics?

Before looking at group differences on the dependent variables, drawing accuracy and learning time, we analyzed whether the four experimental groups differed on basic characteristics. A chi-square analysis indicated that there were no significant differences on gender (p > .05). Separate one-way analyses of variance revealed that the four experimental groups did not differ significantly on age, F(3,98) = 2.02, p = .12; spatial ability, F < 1; chemistry-grade, F(3,98) = 2.15, p = .09; or motivation, F < 1. Overall, we concluded that the groups were equivalent on basic characteristics.
Are the groups equivalent on drawing accuracy and learning time?

First, mean proportion correct and SDs on drawing accuracy during learning for all four conditions are presented in Table 1. There was a significant main effect of group on the accuracy score, F(3,98) = 6.49, p < .001, partial eta2= .17. Tukeys test (with p < .05) showed that students in the control group and the pre-drawn-background group generated significantly less accurate drawings than students in the toolbar group and in the completedrawing-prompt group. There were no additional significant differences between the groups (all p > .05). Second, results revealed that students in the toolbar group (M = 30.64, SD = 4.67) and in the complete drawing prompt group (M = 31.08, SD = 4.27) needed significantly less learning time to work with the science text than students in the pre-drawn background group (M = 34.31, SD = 3.53) and in the control group (M = 33.73, SD = 4.50), F(3,98) = 4.79, p < .01, partial eta2= .13. Again there were no additional significant differences between the groups (all p > .05). Overall, our results show that the most effective instructional support seems to be the toolbar showing (and labeling) all the relevant elements for drawing. Both, students in the complete-drawing-prompt group and students in the toolbar group showed higher drawing-accuracy scores associated with less learning time than students in the pre-drawn-background group or those in the control group. Students in the pre-drawnbackground group, however, did not differ significantly from those in the control group on drawing accuracy and learning time. Apparently, the pre-drawn background was not an adequate instructional support for using the learner-generated drawing strategy. The toolbar, however, was successful in supporting students during the use of the learner-generated drawing strategy. Students in the toolbar groups did not differ significantly from those in the complete-drawing-prompt group that is, toolbar and pre-drawn backgrounds on drawing accuracy and learning time. In short, giving students all the relevant elements they need for drawing increases the accuracy of their self-generated drawings and simultaneously decreases the time they need to work with the text. Table 1: Mean proportion correct on drawing accuracy during learning for all four groups. group control toolbar pre-drawn background complete drawing prompt n 26 25 26 25 drawing accuracy score M .17 .31 .19 .31 SD .14 .15 .15 .14
ISLS
239
ICLS2012
Scientific and Educational Significance

Concerning scientific implications, the results of this study are consistent with previous research on the learnergenerated drawing strategy (cf. Lesgold et al., 1975; Schwamborn et al., 2010; van Meter & Garner, 2005). That is, students do need instructional support during the use of the learner-generated drawing strategy to generate high-accuracy drawings. With regard to the amount of instructional support that is needed to enable students to generate high-accuracy drawings, our results give evidence that the proposed drawing prompt respectively the toolbar by Schwamborn et al (2010) seems to be a better instructional support of the learner-generated drawing strategy (by means of drawing-accuracy and learning time) than the pure, unsupported drawing instruction respectively the pre-drawn backgrounds, only. One reason might be that the toolbar which shows (and labels) all the relevant elements for drawing, presents the students important concepts that they need for drawing a coherent mental model of the chemical processes discussed in the text. That is, constraining and structuring the drawing activity by giving students the main concepts presented in the text fosters additional generative processing while minimizing additional extraneous processing (Schwamborn et al., 2010). Thus, concerning educational implications, this study shows that using the learner-generated drawing strategy as proposed by Schwamborn et al. (2010) can be an effective way to improve students drawing accuracy, which in turn should result in better text comprehension. As several studies have shown positive correlations between drawing accuracy and text comprehension (e.g. Schwamborn et al., 2010; van Meter et al., 2001, 2006), in this study we assessed drawing accuracy only. That is, we used an indirect measure of text comprehension. However, to ensure our conclusions are appropriate, future work is needed to investigate whether the reported effects can also be replicated with direct measures of text comprehension (i.e. learning outcome tests on retention and transfer).
References
Alesandrini, K. L. (1981). Pictorial-verbal and analytic-holistic learning strategies in science learning. Journal of Educational Psychology, 37, 358-368. Alesandrini, K. L. (1984). Pictures and adult learning. Instructional Science, 13, 6377. Boekaerts,M. (1999). Self-regulated learning: where we are today. International Journal of Educational Research, 31, 445-457. De Jong, T. (2005). The guided discovery principle in multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 215228). New York: Cambridge University Press. Ekstrom, R. B., French, J. W., & Harman, H. H. (1976). Manual for kit of factor-referenced cognitive tests. Princeton: Educational Testing Service. Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41, 7586. Lesgold, A. M., Levin, J. R., Shimron, J., & Guttman, J. (1975). Pictures and young chil-drens learning from oral prose. Journal of Educational Psychology, 67, 636-642. Leutner, D., Leopold, C., & Sumfleth, E. (2009). Cognitive load and science text comprehension: Effects of drawing and mentally imagining text content. Computers in Human Behavior, 25, 284-289. Leopold, C. (2009). Lernstrategien und Textverstehen [Learning strategies and text comprehension]. Waxmann: Mnster. Mayer, R. E. (2003). Learning and instruction. Upper Saddle River, NJ: Prentice-Hall. Mayer, R. E. (2004). Should there be a three-strikes rule against pure discovery learning? The case for guided methods of instruction. American Psychologist, 59, 1419 Mayer, R. E. (2009). Multimedia learning (2nd ed). New York: Cambridge University Press. Rheinberg, F., Vollmeyer, R., & Burns, B. D. (2001). FAM: Ein Fragebogen zur Erfassung aktueller Motivation in Lern- und Leistungssituationen [QCM: A questionnaire to assess current motivation in learning situations]. Diagnostica, 47, 57-66. Schwamborn, A., Mayer, R. E., Thillmann, H., Leopold, C., & Leutner, D. (2010). Drawing as a generative activity and drawing as a prognostic activity. Journal of Educational Psychology, 102, 872879. van Meter, P. (2001). Drawing construction as a strategy for learning from text. Journal of Educational Psychology, 69, 129140. van Meter, P., Aleksic, M., Schwartz, A., & Garner, J. (2006). Learner-generated drawing as a strategy for learning from content area text. Contemporary Educational Psychology, 31, 142-166. van Meter, P., & Garner, J. (2005). The promise and practice of learner-generated drawings: literature review and synthesis. Educational Psychology Review, 12, 261-312. Weinstein, C. E., & Mayer, R. E. (1986). The teaching of learning strategies. In M. C. Wittrock (Ed.), Handbook of research on teaching (pp. 315-327). New York: Macmillan. Zimmerman, B.J, & Schunk, D.H. (Eds.). (2001). Self-regulated learning and academic achievement: Theoretical perspectives (2nd ed.). Mahwah, NJ: Erlbaum.
ISLS
240
ICLS2012
Unpacking traces of collaboration from multimodal data of collaborative concept mapping at a tabletop
Roberto Martinez Maldonado1, Judy Kay1, Kalina Yacef1, Beat Schwendimann2 1 School of Information Technologies, 2Faculty of Education and Social Work University of Sydney, Sydney, NSW 2006, Australia {roberto, judy, kalina}@it.usyd.edu.au, beat.schwendimann@sydney.edu.au Abstract: During collaborative student work, teachers aim to support groups effectively and ensure that each group member can contribute at some level. Analysing the final product of group work does not reveal individual contributions or the collaborative process. This research investigates indicators of collaborative learning, captured by a novel tabletop environment, which can provide real-time information about the ongoing collaborative work. This can enable teachers to direct their attention more effectively. This paper reports findings from a case study on collaborative concept mapping. Five triads of university students engaged in a collaborative activity on the topic of human nutrition. The variables, level of participation, symmetry of participation, similarity of previous knowledge, knowledge contribution, transactivity, interaction and knowledge creation, were used to describe the collaborative process. Results offer promise for transforming this collaboration data to give informative feedback to teachers and support collaborative learning.
Introduction and related work

Research suggests that collaborative work can promote active participation of students and improve learning. It can enable students to, consider alternative ideas, negotiate shared knowledge and move more readily from receiving knowledge to constructing understanding and creating new artefacts (Paavola & Hakkarainen, 2005). Advantages of face-to-face collaboration, compared with other ways of learning include that learners can communicate continuously, move between individual and group work, and be more productive in completing their tasks (Olson, et al, 2002). However, students working in groups often do not naturally collaborate to reach common goals. To make collaboration effective, groups often close teacher support. In traditional classrooms, teachers can often only assess the final product, which does not reveal individual contributions. Nor does it give the opportunity for providing feedback at the time it was needed. Teachers need real-time information about which groups and group members need their attention to provide such support. Concept maps can be used as social artefacts through which students communicate (Roth & Roychoudhury, 1993). Their high degree of explicitness makes concept maps an ideal vehicle for exchanging ideas for collaborative constructing knowledge. It has been reported that students who collaboratively generated concept maps achieved higher scores than those who constructed theirs individually (Okebukola, 1992). When concept maps are generated collaboratively they become shared social artefacts that elicit existing and missing connections and spur discussion among students and teachers. Both, concept maps and collaborative learning, have been found to have educational benefits. Combining the two could produce synergistic beneficial effects. Interactive tabletops are promising to improve collaborative learning and there is growing interest in how they can enhance classroom activities. The promise of interactive tabletops is that they offer a tangible shared space that can enrich collaborative work by providing access to digital content. However, existing tabletops do not automatically capture and analyse collaborative interactions between students in real-time to support teachers guidance. The rich content that can be captured by these devices can offer teachers and researchers new opportunities to inspect the collaborative process through tools that help recognising behaviour patterns and analyse how learners collaborate (Medina & Suthers, 2008). A number of researchers have explored the potential of interactive tabletops to support collaboration and learning. Rick et al. (2009) analysed the effects of the tabletop environment on students participation. They found that technical aspects such as software design, students physical arrangement or the hardware features affect groups collaboration and their time to complete tasks. Evans et al. (2010) presented their work on tabletops that support mathematical skill acquisition and suggested that interactive tabletops make it possible to track the students actions and provide real-time feedback. Martinez et al. (2011d) reported initial work on extracting patterns of physical interaction that are linked to the strategies followed by high achieving groups of learners. This research goes beyond previous work by describing findings of a case-study that aimed to a) implement a tabletop as a shared space for group collaboration where students can integrate their individual ideas and create a shared artefact, and b) capture student behaviour during the group activity in a pervasive manner to provide teachers with real-time information about the ongoing collaborative process. The contribution of this paper is the novel approach to automatically exploiting captured audio and application traces of students activity at a tabletop to summarise and provide indicators of the collaborative process.
ISLS
241
ICLS2012
Method
The learning environment captures the interactions of users with the tabletop, their verbal participation and the representation of their knowledge in the form of concept maps. The research question this study addresses is: What measures can be automatically captured from a tabletop learning environment that complements the empirical observations made by a teacher? Tabletop learning environment. The interactive tabletop offers a comfortable space for up to four learners and recognises input from each user through an over-head depth sensor that tracks learners position (Martinez, et al., 2011b). This system allows direct-manipulation of the virtual objects on the tabletop and tracks who is touching what in a pervasive manner. Discussions are recorded through a directive microphone array that can identify when each learner is speaking. For this study, only the frequency of verbal contributions was used. Figure 1 shows the main parts of the learning environment.
Figure 1. Learning environment being used by three learners and the variables of study. Participants and design. Fifteen university students participated in the study. Most were enrolled in engineering courses and were aged between 20 and 26. Participants were assigned to groups of three and knew each other. The experiment had two phases: an individual one at a personal desktop followed by a collaborative one at the tabletop. The duration of the experiment included (a) a 10 minute reading phase, in which all participants read the same text about the learning domain (human nutrition and healthy diet), (b) up to 30 minutes for building the individual maps using the desktop application CmapTools (http://cmap.ihmc.us), and (c) up to 40 minutes for the collaborative phase at the tabletop (see Figure 2). For the individual phase, learners were requested to answer the focus question: What types of food should we eat to have a balanced diet? They received an initial list of suggested concepts extracted from the instructional text to start building their concept map. They were free to add their own concepts, hierarchical arrangement, and linking words. The collaborative phase consisted of building a collaborative concept map at the tabletop, using an application developed by Martinez, et al. (2011a) which gives learners access to their individual concept maps and supports collaboration to create a shared concept map. Learners have access to the list of concepts they used in the individual stage. They share their concepts with others, create new concepts from scratch, and choose linking words to describe the relationship between concepts. All elements on the tabletop are coloured to distinguish which learner created which object. This phase was scripted in four stages (see Figure 2): i) Selecting concepts; participants are asked to add the concepts that they consider are the most general from their individual concept maps. New concepts can also be added later. ii) Adding propositions that the individual learners have in common; the tabletop accesses the individual maps, identifies similar propositions and asks learners whether they want to add these. iii) Linking phase, in which users build relationships between concepts (propositions). iv) Reflection on the resulting map; the application highlights potential errors (e.g. duplicate concepts) and suggests the learners consider propositions present in the individual maps but missing in the group map. Data sources. Between 450 and 1102 meaningful physical actions on the interactive surface were recorded per group. Discussions were captured, distinguishing when each group member was speaking. On average, individual participants spoke between 2 and 15.6 minutes. Qualitative observations of the videos were used to assess whether most of groups speech was on-task (an average of more than the 89% in each group). Each utterance in a sample of each session (more than the half of each session) was coded using a modified coding scheme for Collaborative Decision Making (Kennedy-Clark, et al, 2011). Off-task speech included all the utterances about anything other than the subject material, management of the group processes or the interaction with the application. Data analysis. This study focuses on analysing a set of variables that measure the degree of participation, knowledge distance, and interaction within the groups (Figure 1). These variables were chosen based on previous investigations that have used them to aid empirical and quantitative studies of group learning (Dillenbourg, 1998; Molinari, et al., 2008; Paavola & Hakkarainen, 2005; Taricani & Clariana, 2006) 1) Level of Participation physical and verbal. This dimension of the collaborative group is a very basic indicator of the extent to which one learner participates compared with other learners and groups. The first dimension of participation (physical) includes all the meaningful touches that are performed on the interactive surface (e.g. create concept, add link, and edit linking words). Verbal participation has further impact on the group decisions in face-to-face collaboration. Therefore, it was captured.
ISLS 242
ICLS2012
Figure 2. Approach: 1) individual work, 2) tabletop group activity, and 3) data captured automatically. 2) Symmetry of participation physical and verbal. It has been found that groups in which students participate asymmetrically are frequently related to cases of free-riding or disengagement from the activity (Dillenbourg, 1998). Groups that behave collaboratively tend to allow contributions by all group members. The Gini coefficient was used as an indicator of symmetry of participation in both dimensions: physical and verbal. For this coefficient, a value of zero means total equality and a value of one indicates maximal asymmetry. Previous research on collocated collaboration in small groups found that coefficients close to .5 identify group situations that are already asymmetric (Martinez, et al., 2011c). 3) Distance among individual perspectives. This study aims to measure the extent to which the individual perspectives about the topic are similar to each other. This study uses concept maps as indicators of understanding. Taricani & Clariana (2006) developed a technique for scoring open-ended concept maps by comparing them with a master map. Inspired by this approach, a similar automatic technique was applied with the purpose of comparing concept maps among group members. 4) Individual contribution to the group artefact. Applying the same technique (Taricani & Clariana, 2006) this indicator measures the distance between the individual concept maps and the group map. This distance can be measured throughout the collaborative process to observe how individual contributions vary across time. 5) Transactivity. In the context of collaborative concept mapping, transactivity has been described as the extent to which a group member refers to or builds upon their peers contribution while adding their own ideas to the group map (Molinari, et al., 2008). This indicator was measured in terms of the number of links that each learner creates using concepts that other learners had added to the group map. 6) Interaction with others objects. Another way to measure interactions among a group of learners is to see how often they interact with the same objects (e.g. one learner adds a concept, then a second learner moves and edits the concept word, then a third learner creates a link using this concept). This indicates that the three learners are at least aware of the presence of the concept in the context of the activity. 7) Knowledge Creation. Following the metaphor of knowledge creation of Paavola & Hakkarainen (2005) this indicator measures the created knowledge in the context of concept mapping by distinguishing links that were used in the shared map but not in any individual map. This variable does not describe the correctness of these links; nor does it compare the map with a master map.
Results
Table 1 shows data of four trials, each with distinct behaviours for different aspects of collaborative work. It summarises the results of the variables that were measured. Level of Participation (group) is the number of touches or speaking time of the three participants of each group compared with the mean over all groups (312 touches per group, SD =241; 7.2 minutes of talk per group, SD=3.5). For example, the group where participants worked independently (Group 1) had 817 touches and 12 minutes of talk. Table 1: Measures of group activity process: 1) a group in which group members worked independently; 2) a balanced group; 3) a group with a dominant student and 4) a group with a free-rider.
Groups: Propositions 1-Level of participation (group) a) Participant 1 b) Participant 2 c) Participant 3 2-Symmetry (Gini coefficient) 3-Individual map differences a) P1 and P2 b) P1 and P3 c) P2 and P3 4-Individual contribution a) Participant 1 1-Independent work 2-Balanced work 1- Dominant 4- Free-rider 55 14 27 20 Touch Verbal Touch Verbal Touch Verbal Touch Verbal Med Low Low Med High Med Med Med 817touches 12 min 432touches 22.6min 1102touches 27.4 min 1383touches 23 min Med-30% Med-43% High-45% Med-29% Med-23% Low-14% Med-37% High-47% Med-28% Med-39% Med-29% Med-28% Med-21% Med-28% High-51% Med-37% High-41% Low-17% Med-25% High-42% High-54% High-56% Low-10% Low-15% 0.30 5% 23% 50% 55% 0.29 0.27 14% 8% 17% 28% 0.16 0.35 9% 4% 40% 30% 0.44 0.50 21% 20% 35% 51% 0.34
ISLS
243
ICLS2012
b) Participant 2 c) Participant 3 5-Transactivity (avg p/student) 6-Interaction (avg p/student) 7-Knowledge building (max) 59% 54% 1 links (SD =2) 7 actions (SD =4) 45% 55% 8% 7 links (SD =1) 36 actions (SD =27) 71% 23% 54% 10 links (SD =4) 58 actions (SD =48) 11%

53% 24% 8 links (SD =6) 28 actions (SD=22) 20%
The tags Low, Med and High were calculated for each group (Row 1) and each student (rows 1.a, b and c) from the relative number of touches and amount of speech compared to the mean participation among groups and within each group respectively. Low means < mean 1 Standard Deviation (SD), High > mean + 1 SD and Med around the mean. The indicator of symmetry for the physical and verbal participation was calculated using the average of the Gini coefficient sampled every two minutes. For example, the minimum Gini coefficient corresponds to the verbal participation of Group 2 (0.16) indicating that the speaking time of group members was balanced. On the other hand, a Gini coefficient of 0.5 for verbal participation of Group 4 reflects the lack of participation by one of the group members compared with the other two. Row 3 represents the distance between the individual maps of the members of the same group. P1, P2, and P3 correspond to Participants 1, 2 and 3. A score of 5% indicates that the individual concept maps shared 5% of the propositions, indicating how close their initial conceptions about the topic were before the collaborative phase. Individual contribution (Row 4) was measured as the minimum distance reached between each individual map and the final group map, or in other words, the number of propositions included in the group map from the individual maps. For example, the maximum distance was found in Group 3 between Participants 1 and 3 (4%), and the minimum distance in Group 1, where Participants 2 and 3 shared the 50% of their propositions with their group map. However, if the proposition was not added by the participant, the contribution of that specific proposition was halved. Row 5 and 6 correspond to the indicators of transactivity and interaction with other students objects. Row 7 was measured as the maximum percentage of propositions of the group map that were not present in any individual map (e.g. a maximum created knowledge of 71% for Group 2, and a minimum of 11% for Group 3).
Analysis and discussion

This study aimed to link quantitative indicators that can be automatically captured from a face-to-face collaborative learning environment to the observed group behaviour. Results indicate that groups presented diverse quantitative indicators according to their interactions style: Independent work / balanced work. Two groups showed opposite behaviour in terms of degree of collaboration. In Group 1, students worked mostly separately without showing awareness of others actions. They worked independently, giving some comments to the others about their artefact but mostly being engaged in their individual work. Interestingly, the measures of symmetry indicated rather balanced group work with the participation of each group member close to the overall average (272 touches and speaking time of 4 minutes per student). However, the indicator of transactivity averaged just one link. This means that during the activity each student created just one link using a concept that other student added. Each student also only averaged 7 actions performed on objects created by others. As a result, they ended up building three distinct concept maps that were barely connected. Additionally, a high degree of contribution (54%) and knowledge building (45%) were observed. However, given the fact that they worked almost separately, facilitators may argue that this created knowledge cannot be attributed to the group work. By contrast, the second group performed better in terms of collaboration. Table 1 shows that the participation was more balanced, especially for the verbal participation (symmetry close to 0, for touch=0.27 and speaking time=0.16). Even when their individual concept maps were not very similar (rows 3.a, b and c, <17% for the 3 students) their indicators of transactivity and interaction with elements created by others were higher than group 1 (7 links and 36 actions per student). The level of verbal participation as a group was close to the total average across all groups. However, their effective work was relatively small (only 14 propositions in the final map). A teacher could have encouraged them to include more of their individual propositions in the shared concept map. Nevertheless, this group scored the highest rank in knowledge creation (0.71) because they created most propositions collaboratively. Dominant student / Free-rider. In Group 3, Participant 3 performed most of the changes to the group map, and controlled most of the decisions (more than 600 actions while the two other students had less than 250 each) and he spoke during 16.6 minutes (around 60% of the total group speaking time). This asymmetry of participation was captured by the Gini coefficients (touch=0.35 and verbal participation=0.44). The transactivity and interaction measures were higher than in Group 2. However, these averages are skewed due to the dominance of Participant 3 who used others concepts to create links and built the shared concept map almost by himself (Participant 3 performed 113 actions on others elements, the others only 41 and 21 each, mean=58 actions, SD=48). As a result, the final group map showed a low score for knowledge building because the final solution was very close to the dominant students map (knowledge building = 0.11 and individual contribution of Participant 3 = 54%). Another important aspect is the connection between individual map distances and the contribution to the final map. It could be expected that if two learners have similar individual maps (in this case Participants 2 and 3 shared 40% of their propositions) the contribution of both participants would be higher if
ISLS 244
ICLS2012
one added most of his links. However, the metrics are not affected by this effect. In this case even when Participant 3 added more than 54% of their individual propositions, Participant 2 added just 23% of their map. By contrast, the fourth group showed two learners who collaborated well to merge their ideas (Participants 1 and 2). However, the third participant did not contribute to the group effort and had significantly lower levels of participation compared with the other students (free-riding effect) contributing only to 10% and 15% of the physical and verbal participation respectively (Participant 3 had only 62 touches and 3 minutes of speech). Due to the lack of participation of this student, the symmetry of participation of the group was high (physical=0.50 and talking time=0.34). The indicators, transactivity and interaction with others objects, were similar to all but Group 1 because the two collaborative participants worked together on the same task, building their propositions on the others concepts and links (transactivity of 8 links and interaction of 28 actions per student). Participant 3 only interacted three times with others objects and created few propositions as indicated in their low individual contribution score (24% against 51% and 53%).
Conclusions and future work

The learning process that takes place during the creation of collaborative artefacts is not normally visible in the final product. This paper presents indicators that provide summary information about the collaborative knowledge construction process that is not otherwise available to teachers from the final product. Data from case studies was automatically and unobtrusively captured from the tabletop learning environment and used to identify differences between groups. The case studies illustrate distinctive group behaviours: independent work, balanced work, a dominant student and a free-rider. Results showed the feasibility and utility of using the captured data to generate indicators that can effectively describe particular group behaviour, when compared with analysis of the video and observations. These indicators can serve as a foundation for a new form of support for teachers in classrooms where students work in small groups. Future work will focus on turning tabletop systems into adaptive learning tools that can support students. Teacher dashboards will display patterns of interactions during collaborative work using visualisation, statistical and data mining techniques. This study will extend to consider the quality of the collaboration of the group, learning gains and its implementation in school classrooms.
References
Dillenbourg, P. (1998). What do you mean by 'collaborative learning'? Collaborative Learning: Cognitive and Computational Approaches. Advances in Learning and Instruction Series. (pp. 1-19): Elsevier Science. Evans, M. A., Drechsel, E., Woods, E., & Cui, G. (2010). Multi-Touch Tabletop Computing for Early Childhood Mathematics: 3D Interaction with Tangible User Interfaces. ICLS 2010. (pp. 274-275) Kennedy-Clark, S., Thompson, K., & Richards, D. (2011). Collaborative Problem Solving Processes in a Scenario-Based Multi-User Environment. CSCL2011. (pp. 706-710) Martinez, R., Kay, J., & Yacef, K. (2011a). Visualisations for longitudinal participation, contribution and progress of a collaborative task at the tabletop. CSCL2011. (pp. 25-32) Martinez, R., Collins, A., Kay, J., & Yacef, K. (2011b). Who did what? who said that? Collaid: an environment for capturing traces of collaborative learning at the tabletop. Interactive Tabletops and Surfaces, ITS 2011, (pp. 172-181). Martinez, R., Wallace, J., Kay, J., & Yacef, K. (2011c). Modelling and identifying collaborative situations in a collocated multi-display groupware setting. AIED 2011. (pp. 196-204) Martinez, R., Yacef, K., Kay, J., Kharrufa, A., & Al-Qaraghuli, A. (2011d). Analysing frequent sequential patterns of collaborative learning activity around an interactive tabletop. EDM 2011. (pp. 111-120) Medina, R., & Suthers, D.D. (2008). Bringing representational practice from log to light. ICLS 2008. (pp.59-66) Molinari, G., Sangin, M., Nssli, M.-A., & Dillenbourg, P. (2008). Effects of knowledge interdependence with the partner on visual and action transactivity in collaborative concept mapping. ICLS 2008. (pp. 91-98) Okebukola, P. A. (1992). Concept mapping with a cooperative learning flavor. The American Biology Teacher, 218-221. Olson, J. S., Teasley, S., Covi, L., & Olson, G. (2002). The (currently) unique advantages of collocated work. Distributed work, 113-135. Paavola, S., & Hakkarainen, K. (2005). The Knowledge Creation Metaphor An Emergent Epistemological Approach to Learning. Science Education, 14(6), 535-557. Rick, J., Harris, A., Marshall, P., Fleck, R., Yuill, N., & Rogers, Y. (2009). Children designing together on a multi-touch tabletop: an analysis of spatial orientation and user interactions. Paper presented at the 8th International Conference on Interaction Design and Children, Como, Italy. (pp. 189-196) Roth, W.-M., & Roychoudhury, A. (1993). The concept map as a tool for the collaborative construction of knowledge: A microanalysis of high school physics students. Journal of Research in Science Teaching, 30(5), 503-534. Taricani, E., & Clariana, R. (2006). A Technique for Automatically Scoring Open-Ended Concept Maps. Educational Technology Research and Development, 54(1), 65-82.
ISLS 245
ICLS2012
Motivation in the Learning Sciences: Connecting to Practice by Design

Phillip Herman, Peter S. Wardrip, Jolene Zywica. Learning Sciences and Policy, University of Pittsburgh pherman@pitt.edu, peterwardrip@gmail.com, jolenezywica@gmail.com Abstract: Learner motivation, as a research and design priority, remains relatively marginalized in the Learning Sciences (LS) compared to the disciplines focus on designing for and researching learning. Yet, the learning environments we develop require high levels of motivation. In this conceptual paper, we provide evidence that motivation is marginalized, explain why that might be the case, and provide a way forward that may help ensure our designs better instantiate what we know about motivation in practice. Design-based research (DBR) offers the possibility of connecting motivation research to practice in ways that have not occurred in other disciplines, such as educational psychology. A clearer focus on applying and translating motivation research in design settings will inform our own designs and test motivation theory in ways that are necessary to make motivation more relevant for practice. Designed learning environments, like those propagated by the Learning Sciences (LS), often require relatively high levels of learner motivation and cognitive engagement compared to more traditional learning settings (Blumenfeld, Kempler, & Krajcik, 2006). In project-based learning, for example, learners need to be active, strategic participants who metacognitively monitor their understanding during extended group inquiry activities that often address real-world problems. Learners in these settings need to be cognitively engaged, not just behaviorally active, in their own learning. Ensuring that our designs support the requisite levels of learner motivation and cognitive engagement is an unmet challenge in LS. Simply put, if we fail to design learning environments that effectively tap, support, develop, and sustain high levels of learner motivation and cognitive engagement, the learning that is the ultimate goal of LS designs will likely remain unrealized, sporadic, or only routine for certain subgroups of learners. After 20 years as a discipline, we still know too little about how to specifically design learning environments that ensure motivated learning for most or all learners. There are general recommendations in the literature about how one might apply motivation theory and research to practice: learners should be provided with moderate levels of challenge; learners should be offered choices in learning; and learning should be relevant to real-life contexts, just to name a few (Urdan & Turner, 2005). However, the empirical evidence connecting these design principles to learner motivation remains relatively weak (Herman & Gomez, 2006). Furthermore, we have avoided some of the most critical questions that arise when a designer wants to incorporate motivation research in practice. For example, we do not know how to determine if and when a learner perceives a task as being relevant for their lives. We understand the general premise that relevance is important, but we do not know, and have not asked as a field, how to design to ensure that the potential motivating power of working on real-world problems is actualized and sustained for all learners. Too many of the basic questions designers face remain unanswered. In this paper, we will argue that even though motivation has not been an explicit priority in design, LS, with its focus on collaborative, design-based research (DBR) methods is in a strong position to help test and build motivation theory in learning settings in ways that are not common in existing motivation research paradigms. As a discipline, we need a renewed commitment to ensure LS attends more deliberately to issues of motivation and cognitive engagement in the design of, and research on, learning environments. Doing so should not only improve our designs, but should inform motivation theory in ways that have not typically happened in other disciplines, such as educational psychology, the home of much motivation research. We begin with evidence that motivation is not a common topic in LS research. We then describe challenges associated with applying extant motivation theory and research to traditional classroom practice, and to LS design contexts. Finally, we recommend ways forward that will help to improve our designs, test motivation theory in practice, and develop new, practice-focused theories of motivation.
The Prevalence of Motivation Research in the Learning Sciences

It is difficult to objectively measure the relative importance of a topic like motivation in LS scholarship. We believe, based both on extensive experience as researchers in LS and on our reading of the research, that motivation has to a substantial degree not been a priority of LS compared to its focus on, for example, learning, conceptual change, technology, or other priorities. To better gauge how much LS has explicitly focused on researching motivation and motivation-related topics, we reviewed published articles in the Journal of the Learning Sciences (JLS) from its inception in 1991 through 2011, and papers in the published proceedings from the last three conferences of the International Society of the Learning Sciences (ICLS 2006, 2008, 2010). This
ISLS
246
ICLS2012
review searched all titles that included the keywords motivation, identity, interest, self-efficacy, selfregulation, intrinsic, extrinsic, goals, and self-determination. Though these keywords in titles are not meant to be comprehensive in terms of identifying all articles related to motivation, they should reasonably return an estimate of how much motivation scholarship is represented in these LS venues. For a comparison with the frequency of other keywords in these same venues, we also searched for terms such as online, inquiry, and design. The motivation-related keywords are much less common. In JLS, motivation-related keywords (excluding identity, which is arguably motivation-related) were present in the titles of 10 journal articles throughout nearly 20 years of publication. Learning was in over 400 titles and design in over 150. Engagement and Motivation were present in fewer than 25 titles each. The relative marginalization of motivation research has been noted by other scholars in LS, including Blumfeld, Keppler, and Krajcik (2006) in a handbook chapter and in a recent piece on the impact of designbased research, which found that 13% of titles and abstracts related to motivation themes (Anderson & Shattuck, 2012). Next, we identify reasons why motivation research has not had a greater impact on practice to date.
The Limits of Motivation Research in Impacting Practice

There appears to be a growing awareness that motivation research continues to have a relatively limited impact on educational practice, despite all the progress over the last decades in both theory development and empirical studies of motivation. Recent scholarship (Turner, 2010; Kaplan, Katz, & Flum, 2012) describes challenges that have limited the impact of motivation research on practice. Perhaps the most significant challenge is that motivation theories tend to be underspecified for practice. This under specification, in part, is likely attributable to the fact that motivation theories were typically developed in the disciplines of educational or developmental psychology, and reflect the priorities and methods of inquiry of those disciplines. As such, motivation theories are typically broad explanatory frameworks that describe and account for individual behavior, cognition, and beliefs in various contexts (Herman & Gomez, 2006). Motivation theories were not primarily developed in practice contexts and thus do not prioritize practice-based challenges and questions that arise for practitioners. For example, one prominent motivation theory, self-determination theory (Deci & Ryan, 2000), has been influential in accounting for the ways individuals behave in contexts in which they are intrinsically, versus extrinsically, interested in a domain. The theory is an explanatory framework that can account for the ways individuals make sense of, act, and particularly, persist, in various contexts, whether those contexts are traditional classrooms, careers, or videogames. There is a strong evidentiary base that when individuals are intrinsically interested in a task or activity, they tend to voluntarily persist in that task longer than others who are extrinsically interested (Deci & Ryan, 2000). But the theory was not developed per se to help teachers (or designers) develop and structure learning environments. Self-determination theory, which is relatively better specified than most motivation theories, still remains silent on many of the issues a practitioner would face when applying these insights and theories to a teaching or design context. Some of the questions might include: How can I know what a learners goals really are?; What are learners interests?; Is there an important difference between being interested in a way of learning (e.g., group vs. individual) versus the content of learning? The preceding questions seem related to the theory in fairly direct ways, even if the theory has not been translated well or researched enough to address these questions in practice. A second set of questions also arises, that are critical in practice, but are further afield from the theory, and ignored by motivation researchers. These questions may be critical for practitioners: How will a particular students focus on extrinsic goals hinder or help a teacher manage instruction, including the behavior management of her class?; How does a teacher think about, and act on instructionally, one persons interest while also being responsible for the management and learning of the whole class?; How can a teacher cover required standards when her students may not be intrinsically interested in particular content? In summary, a motivation lens on practice often raises more questions than it answers. This can lead to frustration for practitioners and marginalization of motivation theory in practice. In fact, it is rare that a motivation theory has the kinds of answers that teachers need (Turner, 2010). The Learning Sciences has essentially inherited motivation theory, constructs, and measures from other disciplines that have not successfully connected motivation research to practice. As a field, we have tended to include these inherited ideas about motivation in our design contexts with varying degrees of success (Herman & Gomez, 2006). But, it is worth wondering to what extent the theories currently work in the practice-focused world of LS research and design and how we might benefit by re-examining this inheritance. There are important cultural differences between motivation researchers and practitioners that can work to limit the applicability of motivation research in practice (Turner, 2010; Kaplan, Katz, & Flum, 2012). The goals, assumptions, epistemologies, values, and purpose of the two enterprises are fundamentally different. In some cases, motivation researchers and practitioners can converge, at least nominally, but in other ways they are strikingly divergent. Kaplan, Katz, and Flum (2012) go so far as to question the enterprise of applying motivation research to classroom settings and raise issues that need to be addressed before motivation research is likely to impact learning practice broadly. Most motivation theories have been developed in research carried
ISLS
247
ICLS2012
out with individuals, not classrooms. In some classroom research, studies aggregate the responses of individuals on a survey to better understand something about classroom motivation. But this methodological approach is somewhat troubling when the theory guiding the research was developed to account for individual-level phenomena. Although teachers sometimes work one-on-one with students, much of their instructional time, and thinking, is taken up in small group or whole group instruction. This aggregation of individual experience to group dynamics is a major, largely unaddressed problem of practice. There is, however, a growing number of researchers who adopt sociocultural views of motivation (e.g., Jrvel, Jrvenoja, & Veermans, 2008; Hickey, 2003) that focus less on individual processes and more on participatory patterns in collaborative settings. This is a very important new focus of some emerging designs in LS. The last three decades have witnessed the development of an ever-growing number of motivation related theories and associated constructs. A partial list includes: Goal theory, attribution theory, value theory, social cognitive theory, socio-cultural theory, and self-determination theory. Associated constructs include: Efficacy beliefs, identity, mastery goals, and performance goals, to name a few. Though this proliferation no doubt helps to account for more phenomena in more finely conceptualized theoretical frameworks, the proliferation of theory and constructs and soft borders between them also work to increase confusion and, again, may limit application to practice. Because each construct is generally researched separately, we know little about the interactions of constructs in practice. The sheer number of constructs may work against practitioners making concerted efforts to incorporate them into practice. Hearing about these seemingly similar theories often frustrates practitioners, who want to know how they can use these ideas in practice. A lack of clarity about the malleability of motivation-related constructs also represents a significant challenge. Many theories are unclear or ignore questions related to whether motivation constructs are malleable in learning contexts and whether learning designs need to account for the malleability of constructs. For example, goal theory, with its distinction between Mastery and Performance goals, is unclear about whether goal orientations change or even should be expected to change over the course of, say, a time-limited learning experience. The theory is not clear about whether goal orientations are better thought of as stable, trait-like characteristics of learners or whether goal orientations are more variable, state-like characteristics. How one conceptualizes goal orientation is then critical to whether it even makes sense to focus on goal orientation in learning settings. Many studies and designs assume that goals are malleable, others do not. In some studies, it is hard to tell because the researchers or designers have simply ignored this important issue in goal theory. What a teacher or other practitioner believes about motivation is a powerful predictor of whether he or she takes up specific motivation theory in practice (Turner, 2010). If a teacher believes that efficacy beliefs are critical, she is more likely to wonder about ways to build and support high efficacy. In some cases, teachers may possess micro theories about their students motivation, or about how students learn, that contradict motivation research and make the application of that research to practice, at least in that teachers classroom, unlikely. Some in LS also have beliefs about motivation that may work to marginalize its application to practice. Since the connection between motivation and learning is underspecified, designers may feel justified in not attending to motivation with the assumption that, What matters is learning and motivation must have been there if learning occurred. This can be a pervasive mindset that may severely limiting progress in developing practicebased motivation theories. Next, we describe how a focus on design in the context of DBR can begin to address some of the challenges listed above to both improve our learning designs and inform motivation theory.
The Potential of Design to Connect Motivation Research to Learning Practice

There is often an unclear relationship between motivation and learning that creates challenges when designers work to connect motivation theory to practice. We argue that a design focus may move the field of motivation research forward by connecting motivation to practice in ways that have not yet been realized. Design can help test extant motivation theory, better specify existing theory to address practice-based considerations, and develop new practice-focused theories that better account for motivation in learning contexts. In short, we believe a DBR focus on motivation in learning contexts is what is needed to both better ensure the requisite levels of learner motivation and cognitive engagement that will, in part, determine the success of our designs. Also, a DBR focus can inform motivation research in exactly the ways that motivation researchers like Turner argue are necessary. A DBR focus forces designers to be explicit in their choices, and to work with teachers or other practitioners in real world, messy contexts that challenge and stretch existing theories. The designer, who straddles academic and practice settings may be more likely to think deeply about motivation theory, recognizing what needs to be answered in practice, and researching ways to translate motivation into practice in reified designs. Acting as a bridge between academic research and practice has been a role taken on by those in LS in the context of learning theory. It needs to become more of a priority in the context of motivation theory. Designers can also bring data and insights back from situated, contextualized learning contexts to test, reformulate, or generate theory. As diSessa and Cobb (2004) have argued, the purpose of design studies is to test and inform theory. To date, that testing has largely been focused on theories of learning, but should rightfully also focus on theories of motivation and cognitive engagement. Design contexts offer the possibility
ISLS
248
ICLS2012
of iteratively informing motivation research about how well theories account for motivated behavior, and can provide rich sources of data about motivation constructs like interest, effort, persistence, and attributions. In short, design has the potential to move forward motivation theory and the impact that motivation theory can have on improving learning environments. In order to integrate motivation theory with design, we as a field need to take the next step. Thus, in the following section we provide recommendations for moving the field of LS in a direction in which theories of motivation more prominently drive research and design.
Making Motivation Research more Useful in Design: Recommendations

Motivation has slipped to the sidelines in some of the debates about where we should focus our efforts. Although increased learner motivation and interest in learning was one of the original driving forces for developing innovative LS designs, like project-based science in the 1990s (e.g., Blumenfeld et al., 1991), there appears to be relatively little explicit focus on motivation in contemporary LS scholarship. There is good reason for some skepticism because designing for cognitive engagement in learning is obviously difficult. Motivation researchers have not solved this challenge for those of us who want to simply apply motivation principles to our learning designs. There are still too many unanswered questions about how to leverage motivation research to design more effective learning environments. But, we believe that there has been some progress. We begin with several recommendations that we think are feasible and would ultimately help develop the requisite knowledge base to better connect motivation to practice. Some of the recommendations are certainly easier to implement than others, but we hope they serve to open a discussion across a broad range of participants in LS. We end with a brief description of two studies that incorporate some or all of these recommendations.
Designers need to explicitly address and problematize research questions that are specifically related to testing, translating, or generating motivation theory in practice. Too
often, studies in LS are focused on learning and mention motivation in passing, usually as a desired outcome. So, a study of problem-based science might focuses on learners understanding of the nature of scientific inquiry but also adds interest in science as a possible student outcome that their design might support (Herman & Gomez, 2006). In this framing, we may learn very little about ways that teachers can actually build interest in science in practice because those discussions are limited in articles and there is often not data collected to support these discussions. In brief, we argue that if motivation is a driving theory it should appear in research questions, abstracts, and paper titles.
Designers need to be more explicit about their stance on motivation and their view of the relationship between motivation and learning. They need to more explicitly identify how and why they are working to incorporate motivation theory(ies) in their designs. The explicitness
needs to be connected to their design rationales, specific design decisions, measurement, and iteration of design. They also need to identify the questions of practice that their work is intending to solve. Designers need to clearly justify why a motivation theory is applicable for a specific design context. For example, designers need to explain why they choose constructs, what they believe about the malleability of constructs over the timeframe and implementation strength of a design, measurement of constructs, and hypotheses.
Designers need to better understand and document learners Opportunity to be Motivated

(Herman & Gomez, 2006). By this we mean that we need to better understand learners actual, not idealized, experience of implementations to understand how motivation is, or is not, supported in practice. To do so, designers need to develop better context-specific indicators of both the quality and comprehensiveness of an implementation. If we are trying to determine if inquiry increases learners interest in science, it is important to determine if learners experienced inquiry, for how long, in what ways, and if the motivational principles that inquiry was supposed to support were present in the implementation. For instance, did learners experience meaningful choice over learning, or was choice limited by teacher expertise?
Designers need to better leverage technology to collect information on learners motivation and explicitly measure changes in motivation. To date, most information about motivation in LS
contexts has been through anecdotal reports from teachers (my kids are really interested in using the technology), or from surveys or interviews with learners. Less has been done to use the extensive data that some learning environments provide about learner motivation. The kind of data that DBR can generate needs to be better used in understanding moment by moment and across time motivation and cognitive engagement.
Designers need to better understand motivation theory so as to build models of motivation and engagement that include information about influences that are endogenous or exogenous to the designed learning environment. It matters what happens at home and it matters
what has happened for the whole life of a learner before participating in the designed learning experience. We cannot account for all of the sources of motivation that are relevant to designed learning environments, but our designs might improve, and we might develop better motivation theory, if we work on the challenge of connecting learners experiences in the learning environment to the rest of their lives.
ISLS
249
ICLS2012
Fortunately, there are useful examples of studies that instantiate some of these recommendations and work to build the kind of knowledge base that needs to be constructed to improve design and theory. Habgood and Ainsworth (2011), in a paper on videogame learning, clearly focus on researching a motivation theory (selfdetermination theory) in a design context. They are explicit about the theory, what they are testing, how they measure and conceive of motivation, the explicit design decisions that influenced their effort to instantiate and test the notion of intrinsic integration in practice and, their understanding of the relationship between motivation and learning. Allen (2004), in a museum setting, describes design efforts to increase visitors cognitive engagement with exhibits. Visitors had complete volition over which exhibits to visit, for how long, and for which they could choose their level of involvement. Allen specifies the motivation theory guiding her work and identifies ways the theory did not work to support the kind of engagement desired. A result of this research is the beginning of a knowledge base that other designers could use to understand how to better leverage visitor (or learner) perceptions of physical spaces to increase cognitive engagement.
Discussion
Design-based research as a paradigm has many affordances that could support an effort to make motivation research more useful for learning practice. The Learning Sciences may be somewhat reluctant to take on this challenge. To some extent, we have passed that challenge off to practitioners and have largely assumed that teachers will figure out how to get the learners to play the game. We need to do more of that ourselves, both because it is necessary and by doing so, we will develop a knowledge base that will improve our designs. We are not productively leveraging a methodology that might help us get there. Design-based research may be the way to move this agenda forward. Some of what we need to do is to prioritize an explicit focus on motivation and its connection to learning. It is hard, the theories may not hold up, and there are significant challenges in practice, but, as learning becomes more diffuse in form and content, more voluntary and life-long, we believe learner motivation and cognitive engagement will become relatively more important in the design of successful learning environments. When the focus is no longer on students, who must attend school, but learners who can simply click away from an elaborate, online learning environment, we need to ensure that we are learning, sharing, and theorizing about how to sustain motivation and cognitive engagement in practice.
References
Allen, S. (2004). Designs for learning: Studying science museum exhibits that do more than entertain. Science Education, 88, 17-34. Anderson, T., & Shattuck, J. (2012). Design-based research: A decade of progress in educational research? Educational Researcher, 41, 16-25. Blumenfeld, P., Soloway, E., Marx, R., Krajcik, J., Guzdial, M., & Palincsar, A. (1991). Motivating projectbased learning: Sustaining the doing, supporting the learning. Educational Psychologist, 26, 369-398. Blumenfeld, P., Kempler, T., & Krajcik, J. (2006). Motivation and cognitive engagement in learning environments. In R. Sawyer (Ed.), The Cambridge handbook of the learning sciences (pp. 475-488). New York: Cambridge University Press. Deci, E. L., & Ryan, R. M. (2000). The what and why of goal pursuits: Human needs and the self determination of behavior. Psychological Inquiry, 11, 227268. diSessa, A. A., & Cobb, P. (2004). Ontological innovation and the role of theory in design experiments. Journal of the Learning Sciences, 13(1), 77-103. Habgood, M. & Ainsworth, S. (2011). Motivating children to learn effectively: Exploring the value of intrinsic integration in educational games. Journal of the Learning Sciences, 20, 169-206. Herman, P., & Gomez, L. (2006). Motivation in Project Based Science Classrooms: New measures better coupled to students experiences. Proceedings of the Seventh International Conference of the Learning Sciences (pp. 250-256). Mahwah, NJ: Erlbaum. Hickey, D. T. (2003). Engaged participation vs. marginal non-participation: A stridently sociocultural model of achievement motivation. Elementary School Journal, 103 (4), 401-429. Jrvel, S., Jrvenoja, H., & Veermans, M. (2008). Understanding the dynamics of motivation in socially shared learning. International Journal of Educational Research, 47, 122-135. Kaplan, A., Katz, I. & Flum, H. (2012). Motivation theory in educational practice: Knowledge claims, challenges, & future directions. In T. Urdan (Ed.), Educational Psychology Handbook (pp. 165-194), American Psychological Association: Washington, DC. Turner, J. (2010). Unfinished business: Putting motivation theory to the classroom test, in T. Urdan & S, Karabenick (Eds.), The decade ahead: Applications and contexts of motivation and achievement (pp. 109-138), Emerald Group Publishing: Bingley, UK. Urdan, T., & Turner, J.C. (2005). Competence motivation in the classroom. In A.E. Elliot & C. Dweck (Eds.), Handbook of competence motivation (pp. 297-317). New York: Guilford
ISLS 250
ICLS2012
Supporting Student Learning using Conversational Agents in a Teachable Agent Environment

James R. Segedy, John S. Kinnebrew, Gautam Biswas, Institute for Software Integrated Systems, Vanderbilt University, Nashville, TN, USA {james.segedy, john.kinnebrew, gautam.biswas}@vanderbilt.edu Abstract: This paper presents a design experiment and case study of a four-fold approach to agent feedback in Bettys Brain, an open-ended Teachable Agent learning environment that requires students to gain familiarity with and employ a number of cognitive and metacognitive skills to achieve success in their learning and teaching tasks. Our approach is based on designing feedback according to the principles of goal-alignment, context-relevancy, integrated cognitive and metacognitive support, and a conversational approach to feedback delivery. We tested this feedback approach on 43 8th-grade students, and our findings illustrate that: (1) students conversational choices provide valuable insight into their current mental state, (2) despite the help, some students struggled to learn using the system. These results prompt us to make new recommendations for continued development of feedback in complex, open-ended learning environments.
Introduction
For a number of years, our group has used the learning-by-teaching paradigm to design an open-ended learning environment called Bettys Brain (Leelawong and Biswas, 2008). Students teach a virtual agent, named Betty, about science topics by reading source material and constructing a causal map (Figure 1) that represents relevant science phenomena as a set of entities and causal relations (links) among them. Once taught, Betty (the Teachable Agent) can use her map to answer causal questions (e.g., if cold detection increases, what effect does it have on an animals blood vessel constriction?) and explain those answers by reasoning through chains of links (Leelawong and Biswas, 2008). The goal for (human) students using Bettys Brain is to teach Betty a causal map that matches a hidden, expert model of the domain. To gauge their progress towards this goal, students can make Betty take quizzes, which are sets of questions created and graded by a virtual mentor agent named Mr. Davis. Mr. Davis compares Bettys answers with those generated by the expert model and assigns grades accordingly. Thus, when Betty is unable to answer quiz questions correctly, the (human) students can use that information to discover Bettys (and their own) misunderstandings and correct them by modifying the causal map.
Figure 1. An example causal map on body temperature regulation More generally, Bettys Brain is an open-ended and choice-rich learning environment that presents students with the complex task of carefully reading about and modeling science phenomena. To achieve success in the environment, students must employ a number of cognitive and metacognitive skills (see Figure 2). At the cognitive level, they need to identify important information from the reading materials, represent that information in the causal map format, and use questions and quizzes to check how Betty performs. At the metacognitive level, they need to decide when and how to acquire information, build and modify the causal map they are creating to teach Betty, check Bettys progress, reflect on their own understanding of the science knowledge, and seek help or feedback from Mr. Davis. One important goal of working with learners in such an open-ended
ISLS
251
ICLS2012
environment is developing their abilities to independently regulate their learning process in preparation for future learning (Bransford and Schwartz, 1999). Such preparation, through the development of metacognitive and self-regulatory skills, is important for developing effective learners in the classroom and beyond (e.g., Zimmerman, 2001). However, previous studies have shown that middle school students do not have well-developed independent learning strategies; and as novices they often adopt suboptimal trial-and-error methods when they encounter difficult, open-ended, and exploratory learning tasks (Azevedo, 2005).
Figure 2. Cognitive skills and metacognitive strategies important for success in Bettys Brain In our system, both Betty and Mr. Davis provide dialogue-based feedback to promote effective metacognitive strategies for knowledge construction, monitoring, and help-seeking. However, the agents intentionally avoid providing students with direct answers (e.g., a correct causal link), and they never require users to make a specific, correct next step. Instead, Bettys interactions illustrate her metacognitive awareness. On the other hand, Mr. Davis proposes metacognitive strategies or prompts for reflection as feedback in response to student difficulties and unproductive behaviors. Our hypothesis is that using this form of agent dialogue and feedback to help students adopt productive metacognitive strategies will be more effective in preparing them for future learning than providing domain-specific answers and hints. Testing the effectiveness of the agents strategy feedback in previous classroom studies, however, has produced limited success. For example, an analysis of video-taped sessions of students using Bettys Brain during a classroom study in 2009 revealed that 77% of the feedback statements delivered by agents appeared to be ignored by students, and students rarely pursued help from Mr. Davis. Additional analysis demonstrated that many students lacked understanding of the cognitive skills required for teaching Betty, and this promoted unproductive learning behaviors. These analyses lead us to hypothesize that students need more explicit dialoguebased support. Thus, we re-designed our agent dialogue around four principles: goal-alignment, contextrelevancy, integrating cognitive and metacognitive support, and a conversational approach to feedback delivery. We report a design experiment and case study (Brown, 1992) in an 8th grade science classroom, where we investigate the effectiveness of our feedback approach by analyzing students performance, learning behaviors, and conversations with Mr. Davis. Like other design experiments, our goals are two-fold: (1) analyze experimental results to study the effectiveness of the current approach, and (2) use the results to develop and refine our theory of feedback for complex, open-ended learning environments.
A Four-Fold Approach to Feedback Design

In order to address the various student difficulties mentioned previously, we modified our dialogue design to comply with four design principles: goal-alignment, context-relevancy, integrated cognitive and metacognitive support, and a conversational approach to feedback delivery. The goal-alignment principle dictates that feedback needs to explicitly align with student goals. In other words, the agents should not only suggest strategies, but should also explain how students can employ them. The context-relevancy principle advocates that feedback be specific to the learners current situation, so that the learner can more easily apply the recommended strategy. The third principle states that the feedback should provide support for all of the skills and strategies that students need to master (e.g., by providing worked examples and explanations). Finally, the fourth principle relates to the mechanism by which feedback is delivered, and it prescribes back-and-forth conversations with multiple exchanges that engage learners in social interactions with the agent. We use conversation trees (Adams, 2009) to represent dialogue and feedback in Bettys Brain. In a conversation tree, nodes represent a computer characters dialogue and the branches represent conversational choices available to the user after a response. This structure avoids the complexity of natural language pro-
ISLS
252
ICLS2012
cessing, and the alternation of agent dialogue and student responses allows a joint negotiation of both the conversations direction and follow-up activities.
Experimental Study and Results

To explore the effectiveness of our new feedback framework, we implemented several conversational structures in Bettys Brain. We then ran a study on 43 8th-grade students, who were asked to learn about thermoregulation using the system. In analyzing the data from this study, we divide students into high- and low-performing groups (with 18 students in each group), as determined by scoring the causal maps they created to teach Betty 1, dropping a small set of students in the middle. In this study, the maximum possible causal map score was 15. The mean and standard deviation of the high groups map score was 14.6 and 0.77, respectively. Conversely, students in the low group achieved map scores with a mean and standard deviation of 2.8 and 1.59, respectively. Considering the big difference in the success of these two groups, we conducted a series of analyses to understand the differences in students learning behaviors and dialogue choices. As a first step, we analyze aggregate student behavior by employing hidden Markov models (HMMs) with previously developed methods to provide a concise representation of student learning strategies and behaviors (Biswas et al., 2010). HMMs include a set of states and probabilistic transitions (i.e., more likely transitions are assigned higher probabilities) between those states. Like a students mental processes, the states of an HMM are hidden, meaning that they cannot be directly observed, but they produce observable output corresponding to student actions in the learning environment.
Figure 3. Learning behavior models for high- and low-performance groups We derive aggregate HMMs from a group of students learning activity traces. We first characterize each student activity as one of five, qualitatively distinct actions: reading the science materials, editing the causal map, querying Betty, probing Bettys understanding by asking her to explain her answers, and making Betty take a quiz. However, this abstraction of the raw activity trace logs removes useful contextual information (e.g., was the students query related to a recent map edit?). To maintain a balance between simplicity and interpretability, we use context information to label each action by its relevance to immediately preceding actions (Biswas et al., 2010). Using this relevance metric, we split each categorized action (other than quizzes, for which this characterization does not apply) into two distinct actions: high relevance, meaning the action is related to at least two of the previous three actions, and low relevance otherwise. Figure 3 illustrates the two HMMs derived from the high- and low-performing student activity traces. The percent value listed within each state is the frequency of that state occurring relative to the other states in the model. Using the interpretation methods detailed in (Biswas et al., 2010), we interpreted these states as representing: (1) Systematic Reading students are reading material that has high relevance to previous actions (2) Unfocused Reading students are reading material that has low relevance to previous actions; (3) Editing and Monitoring students are monitoring the correctness of their causal maps using queries and quizzes and are making both high and low relevance edits to their causal map; and (4) Uninformed Editing students are making low-relevance edits to their map, possibly indicating the use of guessing strategies (e.g., trial-and-error). In comparing the derived HMMs for the high- and low-performance groups, the largest difference is that the uninformed editing activities are split into a separate state (Uninformed Editing) in the high group rather than being combined with the informed editing activities in the Editing & Monitoring state as in the low group. This suggests a more strategic use of edits in the high group when in the editing and monitoring state, i.e., their
1
Causal map scores provide a measure of task performance and were calculated by subtracting the number of incorrect map links from the number of correct map links.
ISLS
253
ICLS2012
edit actions are more relevant to their recent monitoring activities. In contrast, the low groups editing activities seem to be less focused, suggesting greater reliance on trial-and-error approaches. Another important difference in focus and systematic strategy use between the two groups is the additional (Extended) Unfocused Reading state in the low group HMM. The extremely high self-loop probability (99%) in this state indicates that upon entering the state, a student would generally continue performing unfocused reading activities for an extended period of time. Further, the low group HMM indicates a greater overall reliance on unfocused reading (59% expected state occurrence, combining the two Unfocused Reading states) compared to the high group (45%).
Figure 4. A flowchart showing how the low group moved through the help conversations. Overall, these results suggest that students in the high group adopted more effective learning strategies that combined directed reading, checking, and focused map building. Together, these effective strategies could explain their success in map building and teaching efforts. On the other hand, low-performing students were more likely to apply what we characterize as suboptimal learning strategies, such as unfocused reading and uninformed editing. However, these results do not directly elucidate how the cognitive and metacognitive dialogue with agents affected student learning behaviors. It could be that the high-performing group was better able to understand and use the dialogue than the low group. Another possibility is that the high group was more willing to engage in dialogue with the agents or knew when to ask for help. To investigate these issues, we analyze how students used the on-demand help feature provided by Mr. Davis.
Figure 5. A flowchart showing how the high group moved through the help conversations. At any time, students could ask Mr. Davis for help. He responded by asking learners where they needed help, providing five possible options, illustrated in Figures 4 and 5. To understand students help-seeking be-
ISLS
254
ICLS2012
havior, we investigated students choices of help topic and the level of detail to which they pursued that topic using clarifying questions. Figures 4 and 5 illustrate the help dialogue differences between the low and high groups. All conversations started in the Mentor Help state when the student requested help from Mr. Davis, and the arrows show the percentage of times students chose each subsequent help option. For example, in the low group, 14.3% of the times that a student asked Mr. Davis for help, that student wanted to know how to determine if they were done teaching Betty. The dashed line from the mentor help state represents the proportion of times students requested help but chose to end the conversation instead of asking a specific question. Examining the flowcharts in Figures 4 and 5 reveals some interesting differences in how the two groups used the help system. First, students in the low group were much more likely to end the help conversation immediately after the mentor asked them what kind of help they needed (14.3% versus 7.9% for the high group). Second, students in the low group were also much more likely to tell Mr. Davis that that they didnt know what to do next (26.2% vs. 13.2% for the high group). These observations suggest that students in the low group were struggling to determine relevant learning activities while teaching Betty.
Discussion and Conclusions

In this paper, we presented the results of a design experiment to explore the effectiveness of a four-fold approach to designing agent dialogue that supports independent learning in a complex, open-ended science learning environment. This approach calls for feedback based on the principles of goal-alignment, context-relevancy, integrated cognitive and metacognitive support, and a conversational approach to feedback delivery. In this paper, we mainly report on the effects of the conversational approach to feedback. The data analysis and case studies provided insight into the feedbacks impact on the students ability to achieve success in teaching Betty. We found that students conversational choices contain information about their understanding of the learning task, as students from the low group were more likely to indicate that they were unsure of how to proceed. By analyzing differences between the stronger and weaker performers, we also concluded that low performers were not provided sufficient support to succeed at the task. They were less able to utilize the help system, and they spent much of their time performing unfocused reading and uninformed editing. While the feedback designed for this study was much more explicit than in previous studies, many students were still unable to achieve success. This suggests that the four-fold approach to feedback may not be enough to help all learners. Based on our analysis of videos from a previous study, we have learned that students dismissal of agent dialogue often implies that they are unable or unwilling to understand the feedbacks meaning and how it applies to their task. An important direction for future work is to continue to refine our feedback principles to better support students as they face these and similar difficulties. One of our goals is to extend our HMM approach of modeling student learning behavior in order to detect more detailed patterns of weaker student behavior during learning and provide appropriate feedback. Currently, we are refining our conceptual framework for feedback and incorporating these improvements in preparation for our next experiment.
References
Adams, E. (2009). Fundamentals of game design (2nd ed.). New Riders Press. Azevedo, R. (2005). Using hypermedia as a metacognitive tool for enhancing student learning? The role of selfregulated learning. Educational Psychologist, 40(4), 11. Lawrence Erlbaum Associates, Inc., Journal Subscription Department, 10 Industrial Avenue, Mahwah, NJ 07430-2262. Biswas, G., Jeong, H., Kinnebrew, J.S., Sulcer, B., and Roscoe, R. (2010). Measuring self-regulated learning skills through social interactions in a teachable agent environment. Research and Practice in Technology-Enhanced Learning (RPTEL), 5(2): 123-152. Bransford, J. D., & Schwartz, D. L. (1999). Rethinking transfer: a simple proposal with multiple implications. Review of Research in Education, 24(1999), 61. Brown, A. (1992). Design experiments: theoretical and methodological challenges in creating complex interventions in classroom settings. The Journal of Learning Sciences, 2(2): 141-178. Leelawong, K., & Biswas, Gautam. (2008). Designing learning by teaching agents: the Bettys Brain system. International Journal of Artificial Intelligence in Education, 18(3), 181208. IOS Press. Schwartz, D. L., Chase, C., Wagster, J., Okita, S., Roscoe, R., Chin, D., & Biswas, G. 2009. Interactive metacognition: monitoring and regulating a teachable agent. In D. J. Hacker, J. Dunlosky, and A. C. Graesser (Eds.), Handbook of Metacognition in Education (pp. 340-359). Mahwah, NJ: Erlbaum. Zimmerman, Barry J. (2001). Theories of self-regulated learning and academic achievement: An overview and analysis. In Barry J. Zimmerman & D. H. Schunk (Eds.), Self-regulated Learning and Academic Achievement: Theoretical Perspectives (pp. 1-37). Mahwah, NJ: Erlbaum.
Acknowledgments
This work is supported by the National Science Foundations IIS Award #0904387.
ISLS
255
ICLS2012
Epistemic Mentoring in Virtual and Face-to-Face Environments

Elizabeth Bagley, University of Illinois at Urbana-Champaign, ebagley@illinois.edu David Williamson Shaffer, University of Wisconsin-Madison, dws@education.wisc.edu Abstract: This study examines whether having mentors communicate with players through a virtual chat program rather than face-to-face changes anything about the players experience. Specifically, this study compares the virtual chat versus face-to-face conditions during the epistemic game, Urban Science, and examines the mentor and player discourse during reflection meetings. The results of this study suggest that mentoring via chat is a viable method for mentoring in the context of epistemic games.
Major issue(s) addressed

Virtual environments have the potential to allow young people to explore complex concepts in simulated form. In simulations, complex social and environmental problems that are too expensive, dangerous, or difficult for players to solve in the real world can be appropriately scaffolded in a dynamic model, giving young people opportunities to more easily interpret the interconnections in the model and develop professional thinking. Virtual environments can make the experience more widely available by simplifying logistics through using non-player characters (NPCs) to support players learning. This paper examines a virtual environment called Urban Science, an epistemic game designed to simulate an urban planning practicum experience (Bagley, 2010). In Urban Science novices, called players, take action in a supervised setting and reflect on their results with peers and mentors. The mentors facilitate the novices work in the practicum by scaffolding tasks the novices are not yet ready to undertake and help them reflect on that work. Following this practicum model, mentors and players interact through planned reflection meetings where the mentor helps players discuss previously completed activities and plan next steps in the project. This paper examines mentors and players reflection meetings in a face-to-face and a virtual condition and asks whether there were differences in the mentors and players discourse between the conditions.
Theory
There is a growing body of research that suggests that video games (used in both formal and informal environments) can support learning (Bagley & Shaffer, 2009; Barab & Dede, 2007) because games allow young people to explore complex concepts in simulated form. Yet, even with a complex model available, logistical concerns, such as identifying experts with adequate experience to mentor players make virtual environments difficult to implement in educational settings. Virtual environments have the potential to simplify logistics by providing students with interactions with a variety of virtual NPCs that can play the role of scripted mentors, community members, or colleagues in a fictitious firm. The NPCs performance can be automated, lowering the logistical overhead and making the virtual environment more widely available. One type of virtual environment designed specifically for young people to explore complex concepts in simulated form with lower logistical overhead is the epistemic game. In epistemic games, novices inhabit a virtual environment in which they learn the epistemic framethe combination of linked and interrelated skills, knowledge, identity, values, and epistemologyof a particular profession by simulating professional training (Shaffer, 2007). Novices are supported by peers and mentors and by simulations that scaffold some of the skills and knowledge necessary for young people to build professional epistemic frames. Mentors in epistemic games facilitate cycles of real-world learning through frequent and strategically-placed reflective conversations with the novices (called players in epistemic games) about their authentic tasks. Mentors model a professional epistemic frame by asking players to reflect on what worked, what did not work, and why and scaffolding a way of seeing and solving problems that the players can adopt. In epistemic games, mentoring has traditionally been conducted face-to-face with players. However, face-to-face mentoring has high logistical overhead and limits the availability of the game. One way to simplify logistics might be to have mentors communicate with players virtually. However, some are skeptical of virtual mentoring and argue that when communication goes electronic, the richness associated with face-to-face conversation diminishes and a considerable amount of information is lost (Bierema & Merriam, 2002). Brennan and Lockridge (2006) argue that in chat-based interactions, mentors have no access to the players body language, tone of voice, or the variety of other signals that can only be detected in a shared physical environment, and as a result, miscommunication can occur. It is unclear whether the constraints of virtual mentoring, namely the possibilities for lost information and miscommunication, outweigh the affordances. Therefore, since there are practical and theoretical reasons to explore virtual mentoring, this study explores whether having mentors communicate with players through a
ISLS
256
ICLS2012
virtual chat program rather than face-to-face changes anything about the players experience. To do so, we measured the quality of the reflection meeting discourse content in two conditions of the epistemic game, Urban Science, a virtual environment in which players work as interns at a fictional, virtual urban planning firm (Bagley & Shaffer 2009).
Methods
This section describes the setting and the activities for both conditions of Urban Science and then discusses the collected, segmented, coded, and analyzed discourse data from the reflection meetings.
Participants
21 high school aged players (11 females, 10 males) recruited by educators at the Massachusetts Audubon Societys (MAS) Drumlin Farm Wildlife Sanctuary played a 10-hour version of Urban Science as part of a week-long Conservation Leadership Program in August 2010. The education specialists recruited young people who had previously participated in at least one MAS program and had no prior experience with urban planning. The two mentors (called planning consultants) in the game were an education researcher (the primary author) and a Drumlin Farm educator. Both mentors completed a training that covered the urban planning profession, the games activities, and preferred mentoring strategies. The mentors were given a script to follow, and they were instructed to keep the conditions as similar as possible.
Intervention
The data for this study were collected from two conditions of Urban Science. In the game, players were assigned to one of two conditions, face-to-face or virtual chat (typed, not video) and one of three teams representing a specific set of stakeholder concerns. Each stakeholder team worked with one of the two mentors. Aside from two adult chaperones being present in the chat condition room, everything about the two games was the same (or as close to the same as possible). During Urban Science, received instructional emails from NPCs controlled by their mentor and were asked to produce land use plans for a community. Throughout the game, mentors were available to help the players if they struggled and to guide players reflection on their work. Mentors also held synchronous reflection meetings to ask players what they finished doing, what they learned during the last activity, what they thought should happen next, and what additional information would be helpful. The mentors were instructed to listen to the responses before interjecting, and after players responded, the mentor revoiced (Cazden, 2001) and extended the players responses to include specific epistemic frame elements pre-determined to be important for that specific point in the game.
Data collection, analysis, and coding

This study examines the mentors and players reflection meeting discourse collected in both the face-to-face and chat conditions of Urban Science. In the chat condition, all of the players and mentors reflection meetings were recorded by the online portal. In the face-to-face condition, the reflection meetings were audio recorded and transcribed. All records were de-identified by replacing references to usernames (names used to sign into the online portal) with pseudonyms. Mentor and player discourse from four reflection meetings were analyzed to determine whether the discourse was different between conditions. The reflection meetings were segmented by conversational turn and coded using a set of 21 codes developed using (1) the American Planning Associations (2011) description of what professional planners know, do, and care about and (2) epistemic frame theory as a guide for categorizing professional thinking. A set of representative excerpts were chosen from the full data set, and grounded theory (Strauss & Corbin, 1998) was used to develop a more specific set of qualitative codes representing aspects of urban planning expertise. While coding the data, the coder read each excerpt separately and applied one code (presence = 1, absence = 0) at a time. The validity of the coding process was checked through an inter-rater reliability analysis in which an educational psychology researcher working in a non-planning domain was trained on the coding scheme and independently coded 150 randomly selected excerpts of the data. All codes had a Cohens Kappa greater than 0.6. The qualitative analyses uncovered several themes in the reflection meeting discourse data, and an emerging quantitative technique called Epistemic Network Analysis (ENA) was used to triangulate the qualitative data and examine how the themes unfolded and whether or not they unfolded similarly or differently across conditions. ENA measures relationships among epistemic frame elements within an epistemic network (Shaffer et al., 2009) and was applied to the mentor and player reflection meeting discourse data. A classical multi-dimensional scaling (MDS) algorithm was applied to the reflection meeting adjacency matrices in order to identify the dimensions that captured the most variance in the data. In this paper, the first and second dimensions (the dimensions that captured the most variance in the data) are plotted. The plots with the reflection
ISLS
257
ICLS2012
meeting data were created from adjacency vectors that included both the mentor and player reflection meeting data.
Results
The results described below examine both the face-to-face and chat conditions of Urban Science. The first section explores the mentors reflection meeting discourse using qualitative and quantitative techniques, and the second section applies the same techniques to the players reflection meeting discourse.
Mentors
During Reflection Meeting (RM) 1 in both conditions, Elise, the People for Greenspace (PGS) team mentor, talked about stakeholders desires by asking players if they trusted the stakeholders to know what is best for the site. For example, in the chat condition Elise asked, Do we trust the stakeholders to know what's good for the site? and in the face-to-face condition she similarly asked, do you guys trust that the stakeholders know whats best for the site? Do you trust the stakeholders, like do you think they know whats best? Elises discourse in the face-to-face condition contained similar content to her discourse in the chat condition, however her face-to-face discourse contained additional filler (Tannen, 1982) words (like) and repetition. During RM 2 in both conditions, Elise talked about generating hypotheses with data or an interactive model by informing the players that iPlan measures the projected social and environmental impacts of zoning changes (Table 1). In both conditions, she discussed iPlans ability to test ways of making the site work for the stakeholders without bringing in actual bulldozers and ended that portion of the meeting by reminding players in both conditions that the site is a complex system, which means that changing one parcel impacts more than one indicator and that there may be trade-offs with every change. As in RM 1, Elises discourse in the face-to-face condition contained similar content to her discourse in the chat condition, however, her faceto-face discourse contained additional filler (Tannen, 1982) words (well, so) and verbal acknowledgements of what the players already said or knew: but what all of you were saying is. Table 1: Excerpt from PGS RM 2 showing (in color) that Elise covered similar content in both conditions. Chat Face-to-face iPlan measures the projected social and iPlan can measure the projected social and environmental impacts of zoning changes, it allows environmental probability changes, it makes you test you to test ways of making the site work for the ways of making the site work for the stakeholders stakeholders without bringing in actual bulldozers. without actually bringing in bulldozers. Well you You discovered that one characteristic of the site is discovered one characteristic of the site, especially, that it is a complex system, which means that but what all of you were saying is that its a complex changing one parcel impacts more than one systemThat means that changing one parcel indicator. There may be trade-offs with every impacts more than one indicatorSo there may be change. tradeoffs with every single change. Since Elise discussed similar content regardless of condition, her discourse showed similar patterns of epistemic frame co-occurrence between conditions. The similar patterns of epistemic frame co-occurrence in both conditions are illustrated by the locations of the mentor points (means) for each condition in Figure 1 where points closer together have more similar patterns of co-occurrence than points farther apart. For example, Elises discourse during RM 1 is located on the far right of the x-axis and her discourse during RM 2 is located high on the y-axis (Figure 1).
Figure 1. Mentors RM discourse (with means) showing that regardless of the communication mode, the mentors covered similar content during the RMs.
ISLS 258
ICLS2012
The mentors patterns of co-occurrence of epistemic frame elements were similar during RMs 1 and 2, and those similarities can be seen qualitatively (Table 2) in mentor excerpts and quantitatively in the ENA analysis (Figure 1). Meeting-by-meeting, t-tests on ENA-generated discourse means for both chat and face-toface conditions showed no significant differences. In other words, the variance between the RMs was larger than the variance between the conditions, and regardless of the communication mode, the mentors covered similar content during the RMs.
Players
During RM 3 in both conditions, players in the PGS team discussed addressing stakeholders desires using a model (Table 2). For example, in both conditions, players talked about their experiences using iPlan to address the stakeholders desires and specifically mentioned Mavens (one of the stakeholders) disapproval of their plans. In the chat condition, one player asked Why Maven was mad? and in the face-to-face condition a player also struggled with Mavens feedback and told the team that Maven called their plans completely unacceptable. Table 2: Excerpts from PGS RM 3 showing (in color) that players discussed similar content in both conditions. Chat Face-to-face I just sent my 2nd map in also I see why Krista still doesnt like traffic but why is Maven mad we cannot get it lower than 2I found that CCl4 needs to be lower and thus we must lessen M1 and M2 areas. if you heighten up one thing and another thing goes downI managed to even out most things, but for some reason, they [stakeholders] werent really like happyMaven called our plans completely unacceptable
During RM 4 in both conditions, players talked about the practice of reporting data. For example, in both conditions, players announced that, I found that or We got a more exact idea indicating that they had data to report to the team. In the chat condition, one player reported that she learned that the stakeholders have high demands. Similarly, in the face-to-face condition, one player reported that the team had a better idea of what they [stakeholders] wanted. Though the players discourse contained different filler (Tannen, 1982) words (like, so), regardless of condition, the players discourse showed similar patterns of epistemic frame co-occurrence between conditions. The similar patterns of epistemic frame co-occurrence in both conditions are illustrated by the locations of the player points (means) for each condition in Figure 2. For example, the players discourse during RM 3 is located to the left of the origin on the x-axis and their discourse during RM 4 is located below the origin on the y-axis (Figure 2).
Figure 2. Players discourse from reflection data (with means) showing that regardless of the communication mode, the players discussed similar content during the RMs. The players patterns of co-occurrence of epistemic frame elements were similar during RMs 3 and 4, and those similarities can be seen qualitatively in player excerpts (Table 4) and quantitatively in the ENA analysis (Figure 2). Meeting-by-meeting, t-tests on ENA-generated discourse means for both chat and face-toface conditions showed no significant differences, with one exception. The t-test comparing the first dimension of each condition in RM 1 did show a significant difference (p < 0.05). However, the difference between the conditions may be attributed to players in the chat condition becoming familiar with the online meeting style.
ISLS
259
ICLS2012
However, overall, the variance between the meetings was larger than the variance between the conditions, and regardless of the communication mode, the players covered similar content during the RMs.
Conclusions, limitations and implications

This study extends the research on virtual education by showing that regardless of the mentoring condition, players showed no significant differences in their RM discourse. Specifically, this study identified that mentors, whether face-to-face or virtual, used similar professional discourse to guide players through the game. Their mentoring led players in both conditions to likewise use similar professional discourse and develop similar epistemic frames. In other words, the co-occurrence of epistemic frame elements within the discourse for both the mentors and the players in each reflection meeting followed similar patterns respectively. These results suggest that the key function of the mentors, to communicate professional ways of thinking, was not diminished in the chat condition. Bierema and Merriam (2002) suggest that the richness associated with face-to-face conversation often diminishes when communication goes electronic. However, in this study, the mentors were instructed to follow a script while leading reflection meetings. Therefore, the similar efficacy of the mentoring in each condition may seem less surprising. The players discourse, however, was similar between conditions without the benefit of scripts to follow, which suggests that if the mentors follow a script while leading reflection meetings, virtual mentoring can be just as effective as face-to-face mentoring at getting players to discuss specific topics, and thus build professional ways of thinking. Further, it is possible that what is lost in the limited communication medium of chat is either peripheral to the professional substance of the conversation or is provided somewhere else in the game. Nevertheless, that virtual mentoring can succeed equally as well as face-to-face mentoring with the same supports suggests that even the human interactions in a mentoring relationship can work virtually. The study presented is, of course, limited. First, the small sample size means that any conclusions are limited to what the sample population did in the context of the epistemic game. Thus, the purpose of the significance tests presented in this paper were to show that additional observations made under the same conditions would show similar results. Second, while this study showed that the mode of communication used by the mentors did not affect the players reflection meeting content, this evidence does not support claims about why mentoring is important in epistemic games. Future studies will examine the important role of mentors in epistemic games. Third, the goal of this particular study was not to not judge the specific merits of the particular epistemic frame developed: a question which has been examined before (Bagley & Shaffer, 2009; Bagley, 2010). Since the mentoring condition did not affect the players reflection meeting discourse, the results of this study can influence the design, implementation, and assessment of virtual environments. Thus, learning in a virtual environment like Urban Science is viable and desirable because virtual environments can expand the range of what players can realistically do and thus also the problems they can address, the possible collaborations they can participate in, and the communities they can inhabit.
References
Bagley, E. (2010). Epistemography of an urban and regional planning practicum: Appropriation in the face of resistance. Madison: University of Wisconsin-Madison, Wisconsin Center for Education Research. Bagley, E., & Shaffer, D. W. (2009). When people get in the way: Promoting civic thinking through epistemic game play. International Journal of Gaming and Computer-Mediated Simulations, 1(1), 36-52. Barab, S., & Dede, C. (2007). Games and immersive participatory simulations for science education: An emerging type of curricula. Journal of Science Education and Technology, 16(1), 1-3. Bierema, L. L., & Merriam, S. B. (2002). Virtual Mentoring: Using technology to enhance the mentoring process. Innovative Higher Education, 26(3), 211-227. Brennan, S. E., & Lockridge, C. B. (2006). Computer-mediated communication: A cognitive science approach. In K. Brown (Ed.), ELL2, Encyclopedia of Language and Linguistics. Oxford, UK: Elsevier Ltd. Cazden, C. (2001). Classroom Discourse: The Language of Teaching and Learning. Portsmouth, NH: Heinemann. Kirk, J., & Olinger, J. (2003). From Traditional to Virtual Mentoring: United States Department of Education. Shaffer, D. W., Hatfield, D., Svarovsky, G., Nash, P., Nulty, A., Bagley, E., Franke, K., Rupp, A.A., Mislevy, R. (2009). Epistemic Network Analysis: A prototype for 21st century assessement of learning. The International Journal of Learning and Media, 1(1), 1-21. Shaffer, D. W. (2007). How Computer Games Help Children Learn. New York: Palgrave. Strauss, A., & Corbin, J. (1998). Basics of qualitative research (3 ed.). Thousand Oaks: Sage Publications, Inc. Tannen, D. (1982). Oral and Literate Strategies in Spoken and Written Narratives. Language, 58(1), 1-21.
ISLS
260
ICLS2012
Two Models of Authenticity: Signature Pedagogy, Problem Based Learning, and Cultural Context
Moshe Krakowski, Yeshiva University, 500 W. 185th NY, NY, moshe.krakowski@yu.edu Abstract: This paper argues that a common paradigm of authenticity, best exemplified by problem-based learning, is not appropriate for foundational learning at the early stages of a domain. A novel second model is introduced, drawing on the notion of signature pedagogy, which contains powerful enculturating features that are highly authentic, but which, surprisingly, do not resemble real world activities.
Introduction: The Problem of Authenticity and Enculturation

In the last twenty years significant attention has been given to the concept of authenticity in school settings. In their seminal article Brown Collins and Duguid (John S. Brown, Allan Collins, & Paul Duguid, 1989) highlighted the tension that often exists in schools between school culture on the one hand and the authentic culture of the learning domain on the other. Adopting the perspective that all learning involves a process of enculturation, they argued that the conceptual understanding of any domain cannot be divorced from the context of conceptual use, and that school activity structures need to be refashioned along more authentic lines. Citing apprenticeship models, such as Jean Laves (1988) study of tailors (from which later emerged the notion of legitimate peripheral participation (J. Lave, & Wenger, E., 1991)) they promoted the notion of cognitive apprenticeship (A Collins, Brown, & Holum, 1991; Allan Collins, Brown, & Newman, 1989), which took the apprenticeship model and adapted it to cognitive strategies in the classroom. Over time, a number of different but related bodies of literature have operated out of a similar situative perspective, investigating ways in which school structures may be authentic, how they acculturate students into modes of reasoning, communicate norms, and significantly, how they can be made more authentic. These approaches have understood all learning to involve a process of enculturation into the discourses of a community (J. Lave, & Wenger, E., 1991), and an increasing socialization into particular norms of behavior and dispositions (or, ways of thinking) associated with a particular community (Putnam & Borko, 2000). Activity structures in schools are not and cannot be neutral packages of learning. Whether the focus is on classrooms as cultural settings (Gutierrez, 2002, p. 315; Westby, 1997), student identity (Sfard & Prusak, 2005), or norms and dispositions (M Gresalfi, Martin, Hand, & Greeno, 2009), the recognition that enculturation constitutes a central aspect of learning complicates the implicitly cultural question of authenticity in learning. Different conceptions of authenticity imply different pedagogical structures, which will impact the social and cultural elements of learning in different ways. Perhaps the most notable example of instantiating authenticity in clearly defined activity structures is the constellation of project- and problem-based approaches (e.g., Barron et al., 1998; Edelson, 2001; Schank, Fano, Bell, & Jona, 1993/1994) which tie individual and social theoretical accounts of learning and motivation to particular classroom activity structures. The numerous attempts to pinpoint the defining features of PBL (e.g. Savery, 2006; Thomas, 2000) have identified problem centrality, student-driven learning, autonomy and relevance. The most frequent descriptor, however, is real-world authenticity (Merrill, 2002; Savery & Duffy, 1996; Walker & Leary, 2009), an essential component of the theoretical justification of problem-based learning (Blumenfeld et al., 1991; Stepien & Gallagher, 1993). And yet, there is a substantial amount of ambiguity in the use of the term authenticity in these contexts. Brown, Collins and Duguid (1989) originally conceptualized authenticity in terms of replicating the ordinary practices of a culture in the classroom. Ann Brown et al. (1993), however, noting the impracticality of apprenticing very young children into the practices of mathematicians, for example, understood authenticity as practices that prepared students for lifelong learning and future participation in the culture of the disciplinean approach that does not require replication of actual disciplinary practices. Others have argued for authenticity as a personal, or emotional, investment on the part of the student (Renzulli, Gentry, & Reis, 2004). In a metaanalysis of research on authentic learning, Rule (2006) identified four components of authenticity: real world problems, open ended inquiry, engagement with a community of learners, and student directed learning. Any of these understandings of authenticity can be instantiated in a variety of ways in the classroom: with reference to the content of real-world research problems (e.g., pollution in a river), to the methods or tools used to assess them (e.g., using authentic graphical interfaces to represent data), or simply to the general types of problems engaged by a discipline, even if the specific question and methodology only approximate those used by professionals (e.g., testing variants of model rockets for launch height). Authenticity can also be found in the engagement of the participants in the classroom; when students interact in real ways with one another (e.g. distributed expertise), rather than the individual, and artificial, standard model of classroom learning. There is a second, related, area of the literature that has remained unclear. Proponents of authentic activity in the classroom have not defined, in a theoretically precise manner, when and where different types of
ISLS
261
ICLS2012
authentic activity should be used. For example, there has been a great deal of work on problem based learning in middle and high school science (referenced above), some work on history and social studies (e.g. Brush & Saye, 2002), on classroom norms and dispositions in math (e.g. MS Gresalfi, 2009, referenced above), and on cognitive apprenticeship in language arts (e.g. cultural modeling, Lee, 1995, 2001). Yet, there has not been a broader articulation of the range of appropriate learning domains, the appropriate ages of students, and their point in a learning trajectory. One illustration of this problem is the relative dearth of research on early childhood learning and on learning foundational skills in this literature. For example, while comprehension strategies have been addressed in contextually sensitive ways (e.g. Palinscar & Brown, 1984), the basics of decoding text have not. Similarly, heavily constrained, highly structured problems at all levels have not been seen as a good fit for project and problem based learning (Walker & Leary, 2009).
Two Authenticity Paradigms: Comparative vs. Constituent

This paper will use the context of Jewish religious education to more precisely explore the use of authenticity as a construct. Specifically, an example drawn from an in-depth case study (consisting of videotaped classes, interviews, survey, and test-score data) of three ultra-Orthodox elementary schools will be used to explore issues of authenticity, and to determine the appropriate circumstances in which different types of authenticity may be desirable. This context, Jewish education, connected as it is to a religious community of practice, provides fertile ground for exploring issues of socialization and authenticity. Indeed, socialization has been widely noted as a goal of religious schooling, and it has been argued that ultra-Orthodox schools, in particular, have as their central purpose the inculcation of the ultra-Orthodox worldview and culture (Krakowski, 2008, p. 17). To address questions of authenticity, this paper will present a case study of new paradigm of authenticity, contrasting it to an existing paradigm (which I call comparative authenticity) that is implicit in problem-based learning. I argue that each serves different pedagogic and cultural goals, and are therefore appropriate under different circumstances. PBL, I argue, is beneficial in later stages of learning trajectories, as it models real world activities, bringing the school and real-world contexts closer together. In so doing, however, the culture created remains a school culture, albeit one that has more personal relevance to the students. The class activities are coherent regardless of whether the students participate in or are expected to participate in an outside culture in the future. The novel authenticity paradigm that I will introduce (termed constituent authenticity) is, however, more useful in the early stages of a learning domain. This view of authenticity draws on the concept of signature pedagogy developed by Lee Shulman (Shulman, 2005b) in the context of professional education. Using a 1st grade Chumash (Bible) class in an ultra-Orthodox Jewish elementary school, I will argue that A. chumash learning is a prototypical example of signature pedagogy, and B. while its practices do not resemble real world practices, they are an authentic component of real world practices, and are particularly well suited for the acquisition of basic knowledge and skill acquisition early in the learning trajectory.
Signature Pedagogy in Elementary School Constituent Authenticity

Rather than conform to uniquely classroom- or school-based structures, the PBL curriculum replicates the activity structures of the real world. This model of comparative authenticity is not applicable in all contexts, however. In the remainder of the paper Id like to introduce a model of authenticity derived from literature on professional education, which I argue complements comparative authenticity in early stages of learning. Lee Shulman has used the term signature pedagogy to describe key similarities in professional instruction across a wide range of fields (Shulman, 2005b). Certain instructional formats, such as the Socratic dialogue in law school or new doctors medical rounds, are clearly and unambiguously identified with particular fields. Though this distinctiveness is what makes them signature, these pedagogic approaches nevertheless all share certain characteristic features (Shulman, 2008); they are pervasive and uniform in their fields, involve fixed routines and rituals, are visible and interactive, and each inculcates particular habits in students. Such habits differ between professions: legal training develops habits of mind (thinking like a lawyer), while medical training develops habits of practice (acting like a doctor), while clergy training creates habits of the heart. Shulman argues that signature pedagogies are found in professional schools precisely because professional schools require more than just understanding; they demand education for practice (Shulman, 2005a). In such contexts, the pedagogy must include mechanisms for socialization into a professions culture and epistemology (Shulman, 2005b, p. 54).
Elementary Chumash Study

Shulmans key insight is that the development of norms and dispositions in professional education through pervasive and uniform interactive routines is essential for socialization into the culture of a community of practice. Id like to suggest that signature pedagogies may also be found at the K-12 level, constitute a type of authenticity (constituent), and are particularly appropriate for the earliest stages of a learning domain. To illustrate, I will describe introductory chumash instruction in ultra-Orthodox elementary schools. Boys in this
ISLS
262
ICLS2012
community are expected to engage in religious study throughout the entirety of their lives, giving school religious material such as chumash a clearly articulated context of use that is central to the entire endeavor of boys ultra-Orthodox education (much like the clearly expected use of professional education). As with signature pedagogy, chumash pedagogy is pervasive (some variant is present in nearly all ultra-Orthodox chumash classrooms); uniformly executed, with fixed routines and rituals; public and interactive; and inculcates habits of heart, practice, and mind (or, dispositions) that are central to participation in ultra-Orthodox life. Unlike PBL, signature pedagogies do not directly resemble real-world activities, because the structure of signature pedagogy does not reflect authentic practices in the real world, but expands the culture of the real world to include the practices of school. In most domains, professional training bears little resemblance to the real world activity in the profession itself (and law school pedagogy has been criticized on these grounds (e.g. Hyland & Kilcommins, 2009)). Similarly, the differences between chumash learning in the elementary school classroom and adult learning practices are quite substantial, most notably visible in the absence, in adult practices, of choral reading, which is central to 1st grade practice. Chumash study in 1st grade is profoundly enculturating, and genuinely authentic, but does not resemble practices outside of school. To demonstrate, I will describe one short example from a corpus of videos of chumash class drawn from the case study, in which the students review a verse from the chumash story of Noah and the flood that deals with how the wickedness of man prompted Gods decision to destroy the world. The verse in question (one of four reviewed in this class) describes the wickedness of the sons of the elohim, who took wives inappropriately (Gen. 6:2). Throughout the entirety of class, the rebbe (religious instructor) circulates throughout the room, holding flashcards with two or three Hebrew words from the text, color-coded to indicate the different parts of the Hebrew word that translate into separate English words, or to indicate the words root, tense, or gender. For this specific verse, the rebbe begins by asking an individual student to identify the root of the Hebrew word and they saw (va-yiru, from the three Hebrew letters reish, aleph, and hei), by pointing to a previous verse that contains a word with the same root. He then points out the function of three additional Hebrew letters contained in the word va-yiru: the prefix vov, which means and, and the yud-prefix/vov-suffix combination that renders the verb third person plural (they). The students translate the root together, and then, at the rebbes instruction, the class engages in choral reading. They chant/sing the verse together in Hebrew, interspersing every few words with the English translation they have previously learned. The rebbe then points out (slowly, and with much repetition and questioning of students) that another Hebrew word from the verse, they chose (bacharu), also has a vov-suffix denoting the third person plural. The rebbe continues in this vein, first asking one and then the other half of the class to chant the verse together again, and concludes with a summary of the content of the chumash story to that point. He explains that God was unhappy with the wickedness of man (specifying kidnapping wives, killing husbands, and worshipping idols) and that God gave the people 120 years to repent before he would destroy the world. In reviewing this verse, the rebbe focused on the grammar and meaning of two particular words in the verse, asked the class to chant the verse and its translation several times, and briefly discussed its meaning in the context of the larger biblical story. This complex structure allows the students to gain basic fluency in reading and translating the text through frequent practice and repetition, and importantly, reflects two distinct enculturating elements: students development of dispositions regarding the nature of the chumash text and student identity development in relation to that text. They acquire the ultra-Orthodox epistemology of chumash study in that the text is assumed to be essentially literal, and textual interpretations (such as elohim translating as ruler, rather than God) are clearly marked as fixed by tradition, rather than open to careful new readings. The structure of chumash learning also leads students to understand the nature of the practice in their lives; for example, vocabulary, grammar, and reading practice are integrated with the study of the text itself, conveying the sense that the practice of chumash study (which they will all engage in as adults) is itself valuable, not any one component skill. They also learn that the text has active moral and religious implications, such as the importance of repentance or punishment and rewardbasic principles that shape the ultra-Orthodox worldview. The norms, dispositions, beliefs, and assumptions that students developabout the text and its role in their lives, about religion and the nature of realitydemonstrate that 1st grade chumash activities must be understood as part of the same larger practice of chumash study that adults engage in; the separate context of study (school) is simply the first stage of the practice. The authentic practice of chumash study must be understood quite broadly; it extends to childhood school learning as well as every other authentic context in which chumash study is engaged, yet the school activity does not necessarily resemble the adult activities. In the next section I argue that this is appropriate to the aims of early foundational instruction in any domain.
Two models of Authenticity, Two Models of Cultural Connection

The implicit assumption that authentic activity is activity that models the real world has rarely been made explicit, and the benefits and limitations of this paradigm have consequently not been well articulated. In particular, authenticity has not been explored in detail at the earliest stages of learning domains, which may stem from the difficulty of modeling authentic practices that already take the skills being developed in the classroom
ISLS
263
ICLS2012
for granted; authentic modeling cannot realistically be used to acquire these basic skills themselves. Yet a larger enculturating context providing a clearly defined, specific cultural context of use (which informs essential conceptions of what activities mean to students) is desirable even in early learning; without such a context, the practice and repetition necessary to develop fluency in basic skills can seem boring and meaningless to students. In this paradigm of authenticity classroom activities act as a distinct first stage of an authentic communal practice that spans school, culture, and community. Chumash learning socializes students into an actual community of learners that extends beyond the classroom itself, in which communal understandings of the role the material will play in students lives impacts the particular narrative that students develop about what they are doing when they study chumash. This type of authenticity is a particularly good fit for foundational learning in any domain, as the activity structures need not closely resemble authentic practices (which is in any event impossible at such an early stage of learning), but might be better reconceptualized as the first stage of enculturation into a larger cultural context of knowledge use. By fostering cultural dispositions and communal narratives in the context of foundational skill acquisition, school activity structures are themselves considered an essential part of the communal practice. In contrast, the type of real-world modeling found in PBL might be more profitably employed much later in the learning trajectory. As with professional education, the context of religious education is highly specific, and reflects an unusually well-structured set of cultural assumptions that drive the learning and socialization process. To apply this paradigm more broadly we need to first articulate a clear vision of what mature authentic practices looks like. In which authentic contexts will these skills used? What mechanisms will tie classroom activities to future contexts in ways that facilitate basic skill acquisition? We need to develop theoretical accounts of how practice, repetition, and skill development can be given an authentic enculturating context that socializes students into future mature learning practices. What are the essential dispositions, beliefs, or worldviews that motivate this practice in the real world, and how can we incorporate them into current instructional structures? Drawing on the common features of signature pedagogy identified by Shulman, I would begin by suggesting that whatever the mechanism, it should, as Shulman suggests, first entail some element of public performance, be uniform across contexts, and should involve fixed routines and rituals. Second, it should include a clear account of the epistemology of the learning practice and a clear articulation to the students of the expected contexts of use. In reading, for example, the nature of the texts used to learn how to read can implicitly shape student conceptions of both the nature of reading itself, and the role that reading will play in their own future lives. This will create a coherent group culture around the practice that is explicitly connected with future engagement in the practice, shaping student understandings of the nature of the material and its role in their lives.
References
Barron, B. J. S., Schwartz, D. L., Vye, N. J., Moore, A., Petrosino, A., Zech, L., . . . Vanderbilt, T. C. a. T. G. a. (1998). Doing with understanding: Lessons from research on problem- and project-based learning. Journal of the Learning Sciences, 7(3&4), 271-311. Blumenfeld, P., Soloway, E., Marx, R., Krajcik, J., Guzdial, M., & Palincsar, A. (1991). Motivating projectbased learning: Sustaining the doing, supporting the learning. Educational Psychologist, 26(3), 369398. Brown, A., Ash, D., Rutherford, M., Nakagawa, K., Gordon, A., & Campione, J. (1993). Distributed expertise in the classroom. Distributed cognitions: Psychological and educational considerations, 188-228. Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. 32-42. Brown, J. S., Collins, A., & Duguid, P. (1989). Situated Cognition and the Culture of Learning. Educational Researcher, 18(1), 32-42. Brush, T., & Saye, J. (2002). A summary of research exploring hard and soft scaffolding for teachers and students using a multimedia supported learning environment. The Journal of Interactive Online Learning, 1(2), 1-12. Collins, A., Brown, J., & Holum, A. (1991). Cognitive apprenticeship: Making thinking visible. American Educator, 15(3), 6-11. Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the craft of reading, writing, and mathematics. In L. B. Resnick (Ed.), Knowing, learning, and instruction: Essays in honor of Robert Glaser. Hillsdale, NJ: Lawrence Erlbaum. Edelson, D. C. (2001). Learning-for-Use: A framework for the design of technology-supported inquiry activities. Journal of Research in Science Teaching, 38(3), 355-385. Gresalfi, M. (2009). Taking up opportunities to learn: Constructing dispositions in mathematics classrooms. Journal of the Learning Sciences, 18(3), 327-369. Gresalfi, M., Martin, T., Hand, V., & Greeno, J. (2009). Constructing competence: An analysis of student participation in the activity systems of mathematics classrooms. Educational Studies in Mathematics, 70(1), 49-70.
ISLS
264
ICLS2012
Gutierrez, K. D. (2002). Studying Cultural Practices in Urban Learning Communities. Human Development (0018716X), 45(4), 312-321. Hyland, A., & Kilcommins, S. (2009). Signature Pedagogies and Legal Education in Universities: Epistemological and Pedagogical Concerns with Langdellian Case Method. Teaching in Higher Education, 14(1), 14. Krakowski, M. (2008). Isolation and Integration: Education and Worldview Formation in Ultra-Orthodox Jewish Schools. (PhD), Northwestern University, Evanston, IL. Lave, j. (1988). The culture of acquisition and the practice of understanding: Institute for Research on Learning. Lave, J., & Wenger, E. (1991). Situated Learning: Legitimate peripheral participation. Cambridge: Cambridge University Press. Lee, C. (1995). A culturally based cognitive apprenticeship: Teaching African American high school students skills in literary interpretation. Reading Research Quarterly, 30(4), 608-630. Lee, C. (2001). Is October Brown Chinese? A cultural modeling activity system for underachieving students. American Educational Research Journal, 38(1), 97. Merrill, M. (2002). First principles of instruction. Educational Technology Research and Development, 50(3), 43-59. Palinscar, A., & Brown, A. (1984). Reciprocal teaching of comprehension-fostering and comprehensionmonitoring activities. Cognition and instruction, 1(2), 117-175. Putnam, R. T., & Borko, H. (2000). What Do New Views of Knowledge and Thinking Have To Say about Research on Teacher Learning? Educational Researcher, 29(1), 4-15. Renzulli, J., Gentry, M., & Reis, S. (2004). A time and a place for authentic learning. Educational Leadership, 62, 73-77. Rule, A. (2006). The Components of Authentic Learning. Journal of Authentic Learning, 3. Savery, J. (2006). Overview of problem-based learning: Definitions and distinctions. The Interdisciplinary Journal of Problem-based Learning, 1(1), 9-20. Savery, J., & Duffy, T. (1996). Problem based learning: An instructional model and its constructivist framework. Constructivist learning environments: Case studies in instructional design, 135-148. Schank, R. C., Fano, A., Bell, B., & Jona, M. (1993/1994). The design of goal-based scenarios. The Journal of the Learning Sciences, 3(4), 305-346. Sfard, A., & Prusak, A. (2005). Telling identities: In search of an analytic tool for investigating learning as a culturally shaped activity. Educational Researcher, 34(4), 14-22. Shulman, L. (2005a). Pedagogies of Uncertainty. Liberal Education, 91(2), 18-26. Shulman, L. (2005b). Signature pedagogies in the professions. Daedalus, 134(3), 52-59. Shulman, L. (2008). Pedagogies of Interpretation, Argumentation, and Formation: From Understanding to Identity in Jewish Education. Journal of Jewish Education, 74, 5-15. Stepien, W., & Gallagher, S. (1993). Problem-based learning: As authentic as it gets. Educational Leadership, 50, 25-25. Thomas, J. (2000). A review of research on project-based learning. San Rafael, CA: Autodesk Foundation. Walker, A., & Leary, H. (2009). A problem based learning meta analysis: Differences across problem types, implementation types, disciplines, and assessment levels. Journal of Problem-based Learning, 3(1), 628. Westby, C. (1997). There's More to Passing Than Knowing the Answers. Language, Speech, & Hearing Services in Schools, 28(3), 274-287.
ISLS
265
ICLS2012
New Evidence on Productive Failure - Building on Students Prior Knowledge is Key!

Katharina Westermann, Nikol Rummel, Ruhr-Universitt Bochum, Institute of Educational Research, Universittsstr. 150, D-44780 Bochum, Germany, Email: katharina.westermann@rub.de, nikol.rummel@rub.de Abstract: The work on Productive Failure (Kapur, 2009) suggests that students can benefit from generating representations during collaborative problem-solving followed by instruction that builds on the student-generated representations. However, Productive Failure has usually been compared to Direct Instruction in studies with several confounds. These evaluations, therefore, do not allow conclusions regarding the underlying learning mechanisms. To address this criticism, we compared four conditions in a quasi-experimental study with 154 10th graders: We varied the form of instruction in two Direct Instruction conditions (a regular DI condition and a DI-S condition where instruction built on typical student-generated representations) and the type of support students received in two Productive Failure conditions (a regular PF condition and a PF+ condition with additional cognitive support). Our results indicate that students benefit from introducing new concepts based on student-generated representations. Moreover, during generation motivational support seems sufficient, that is, students do not need cognitive support.
Collaborative Learning with a Diversity of Representations in Mathematics

The mathematics education literature points at the importance of supporting students in developing an understanding of different representational formats, such as tables, graphs, and algebraic notations (e.g. Cuoco & Curcio, 2001). Teaching students to understand and connect different representational formats not only improves their representational competency, but also fosters their understanding of the underlying mathematical concepts and problem-solving procedures. These assumptions are in line with findings from educational psychology research concerning the added value of multiple representations for learning (e.g. Ainsworth, 2006). However, several studies also demonstrated that students do not automatically benefit from multiple representations. One problem that has been identified is that students often tend to treat the representations in isolation instead of making connections and integrating the information provided by the different representations (e.g. Ainsworth, 1999, 2006). One possible approach to promoting learning with multiple representations, and particularly to promoting connection-making across representations, might be to include collaborative learning opportunities (Bodemer, Kapur, Molinari, Rummel, & Weinberger, 2011; Rummel & Braun, 2009): Within a small group, members may have different perspectives on the representations. By discussing and elaborating the different perspectives, a deeper processing of the mathematical concepts underlying the representations may be fostered as students co-construct a shared understanding that goes beyond the understanding of each individual (Moschkovich, 1996). Indeed, research on collaborative learning shows that students discuss connections between information provided by different representations (e.g. Suthers & Hundhausen, 2003). Similar to the learning environment used in the study by Suthers and Hundhausen, many collaborative learning environments (especially computer-based environments) include different external representations; however, research has usually focused on how to support collaboration by introducing representations, and rarely the other way around. Hence, it remains to be investigated how to best design collaborative settings in order to support learning with a diversity of representations. Based on the way mathematics textbooks and courses are most commonly designed, collaborative learning with different representational formats would likely be implemented in the following way: The textbook or the teacher introduces a new mathematical concept by using different representational formats, for example, symbolic representations (i.e. formulas) and graphical representations (e.g. diagrams). Afterwards students practice problem-solving using these representational formats in small groups. An alternative approach would be to begin by having small groups of students interact with different representations based on their formal and informal prior knowledge, and have the teacher provide instruction thereafter. In other words, students first generate and/or learn with different representations in order to develop an intuitive understanding of the targeted mathematical concept and only afterwards the teacher provides instruction building on these representations. Such a delay of instruction lies at the heart of the Invention Paradigm (e.g. Kapur, 2009; Roll, Aleven, & Koedinger, 2011; Schwartz & Martin, 2004). One approach within the Invention paradigm is Productive Failure (e.g. Kapur, 2009). In a series of experiments conducted by Kapur in Singaporean classrooms, students in a Productive Failure condition (PF) engaged in problem-solving activities in small groups without receiving instructional support. Process data showed that students collaboratively generated a diversity of representations during this phase. Thereafter, typical student-generated representations were
ISLS
266
ICLS2012
contrasted in a teacher-led discussion which finally led to the presentation of the canonical solution by the teacher. Although students in the PF condition were rarely able to find the canonical solution during collaborative problem-solving, they significantly outperformed students who first received Direct Instruction (DI) in the posttest (e.g. Kapur, 2009). Upon closer inspection, however, students in the DI condition in fact received a different form of instruction than students in the PF condition: The teacher directly presented the canonical solution, rather than building on typical student-generated representations and misconceptions. Thus, when comparing the two conditions, the timing of the instruction and the form of instruction were confounded. It could be hypothesized that the DI setting would also benefit from a classroom discussion about typical student-generated representations and misconceptions (Hammann, 2003). Furthermore, in the PF condition it is so far unclear whether a complete lack of support during collaborative problem-solving is indeed optimal for learning. Advocates of the Invention Paradigm seem to argue for a delay of instruction and support. In contrast, the broader literature on collaborative learning suggests supporting students collaborative activities at least to some extent as fruitful collaboration does not occur automatically (ODonnell, 1999). It can be assumed that in the context of the Invention Paradigm, the benefits of collaborative activities would also unfold best if some support was given (Westermann & Rummel, 2012). It therefore seems to be a logical next step to investigate what type of support students need when they collaboratively learn with a diversity of representations. Indeed, in the Productive Failure studies students received at least some support during problem-solving: In earlier studies on Productive Failure (Kapur, 2009), students received motivational support to persist in generating different representations. In later studies by Kapur (personal communication, March 2010), students additionally received cognitive prompts in form of contrasting cases to their generated solution approaches. These prompts should support students to improve their solution approaches from one representation to the next. However, the differences between the support features in the former and the latter studies have not been investigated empirically so far.
Methods
In our study, we aimed to shed light on the impact of building instruction on typical student-generated representations and misconceptions, and on support features that might make the problem-solving phase of Productive Failure even more effective for learning. In order to address these questions, we compared four conditions in a quasi-experimental study as shown in Table 1: We varied the form of instruction in two Direct Instruction conditions (a regular DI condition and a DI-S condition where instruction built on typical studentgenerated representations) and the type of support students received when collaboratively generating representations in two Productive Failure conditions (a regular PF condition and a PF+ condition with additional cognitive support). Table 1: Conditions of the study. Learning phase 1 DI Instruction with canonical representations DI-S PF PF+ Instruction based on typical student-generated representations Problem-solving in small groups Problem-solving in small groups with cognitive prompts
Learning phase 2 Problem-solving in small groups Problem-solving in small groups Instruction based on typical student-generated representations Instruction based on typical student-generated representations
The regular Direct Instruction condition (DI) served as a control condition. In the DI condition the teacher explained a yet unknown concept (the concept of variance and deviation) and introduced the canonical solution by using different representational formats. In order to address the confound between timing of the instruction and form of the instruction, we implemented a second Direct Instruction condition (DI-S) where the teacher introduced the canonical solution by discussing and building on typical student-generated representations (i.e. representations that students in the Productive Failure conditions typically generate during problem-solving). In both Direct Instruction conditions the instructional phase was followed by problem-solving in small groups of three students. Both Productive Failure conditions started with problem-solving in groups of three students to the yet unknown concept. Students were instructed to try different solution approaches by generating different representations, such as tables, graphs and formulas. They used tablet PCs to generate representations. During this phase students in the standard Productive Failure condition (PF) only received motivational prompts encouraging them to persist in solving the task (e.g. you are doing a good job together, keep going). In the augmented Productive Failure condition (PF+), students additionally received cognitive prompts, that is, students were supported in their critical evaluation of the representations they were generating by providing
ISLS
267
ICLS2012
them with contrasting cases to their solution approaches (e.g. maybe there are situations where your solution does not work, have a look at this (counter-)example). In both Productive Failure conditions the problemsolving phase was followed by an instruction phase where the teacher introduced the canonical solution by building on typical student-generated representations as in the instruction phase of the DI-S condition. Participants were 154 10th graders recruited from two secondary schools in Germany. Learning outcomes were assessed by a posttest after the second learning phase. The posttest included items testing for procedural skills and for conceptual knowledge. We further recorded process data (video and screen recordings) of students interaction during the collaborative problem-solving phase. This data enables us to investigate the possible impact of the cognitive prompts in the PF+ condition on the collaborative learning process. The analysis focuses on the number and quality of the generated representations as well as on how students collaboratively generate and learn with multiple representations.
Results
Only the 144 students who were present during both learning phases were included in the analyses. A factor analysis of the posttest items revealed two factors with an eigenvalue of 1 or more (2.47, 1.26), that accounted for 53.53% of the total variance. The rotated factor loadings were in line with our intended differentiation between procedural skills and conceptual knowledge. The items are shown in Table 2. Table 2: Rotated factor loadings. Factor 1 (Conceptual knowledge) .67 -.03 .64 .22 .63 .60 .66 Factor 2 (Procedural skills) .21 .84 .03 .77 .27 .41 -.25
Item 1: Remembering the main features of the canonical solution Item 2: Calculation Item 3: Explaining graphical representation Item 4: Calculation Item 6: Explaining error Item 7: Explaining error Item 8: Sorting graphical representation
To assess differences between the experimental conditions, we calculated a MANCOVA with the factor condition and the covariate prior knowledge (i.e. previous grade in mathematics) that revealed significant differences between the conditions for both, procedural skills and conceptual knowledge (Fprocedural[3, 139] = 3.45, p = .02; Fconceptual[3, 139] = 11.50, p =.00). Mean scores and standard deviations are displayed in Table 3. We calculated three a priori contrasts: First we compared both Productive Failure conditions to both Direct Instruction conditions to analyze the effect of the sequence of the problem-solving phase and the instruction phase. Second, we compared the Productive Failure conditions with each other to assess the effect of the cognitive prompts in the PF+ condition. Third, we compared the two Direct Instruction conditions, that is, the different forms of instruction with each other. Concerning procedural skills the a priori contrasts revealed only one effect with a small effect size: The Direct Instruction conditions outperformed the Productive Failure conditions (F[1, 139] = 7.02, p = .01, 2 = .05); differences between PF+ and PF (F[1, 139] = 3.63, p = .06) and between DI-S and DI (F[1, 139] = 0.14, p = .71) were not significant. For conceptual knowledge the a priori contrasts revealed large and medium effects: The Productive Failure conditions outperformed the Direct Instruction conditions (F[1, 139] = 26.67, p = .00, 2 = .15) and DI-S outperformed DI (F[1, 139] = 19.0, p = .00, 2 = .12). However, there was no significant difference between PF+ and PF (F[1, 139] = 0.07, p = .79). For conceptual knowledge an a posteriori Scheff test indicated significant differences in the pair-wise comparisons between DI and all other conditions (p = .00 for each comparison with DI), but no significant differences between DI-S, PF, and PF+. The Scheff test determined two separate homogeneous subsets: One homogenous subset comprises DI-S, PF, and PF+ (p = .36), that is, the means of these three conditions do not differ significantly. The second homogeneous subset comprises DI (p = 1). Table 3: Posttest results N 19 40 39 46 Procedural skills Mean SD 3.68 0.95 3.56 0.62 2.90 1.32 3.45 0.93 Conceptual knowledge Mean SD 3.37 1.47 5.59 2.19 5.79 2.53 6.54 1.86
DI DI-S PF PF+
ISLS
268
ICLS2012
Discussion
The posttests revealed three major findings: First, we found that the Direct Instruction conditions outperformed the Productive Failure conditions on items testing for procedural skills, yet with a small effect size. This finding is not surprising: Students in the Direct Instruction conditions who received instruction first solved eight practice problems in the following collaborative problem-solving phase. In comparison, students in the Productive Failure conditions solved only one problem during the problem-solving phase prior to instruction. Against this background, the small effect size (as compared to the effect sizes for the other results) seems to indicate that intensive practice may not be the most important learning mechanism. Secondly, our results show that regarding conceptual knowledge students in the DI condition were outperformed by students of all other conditions, while we did not find any significant differences between DIS, PF, and PF+. This finding is especially interesting considering that the most common form of mathematics education is Direct Instruction most likely resembling the DI condition. DI-S, PF, and PF+ have in common the form of instruction: In all three conditions the instruction was based on students prior knowledge and misconceptions as they are externalized in student-generated representations. Thus, our results suggest that building on students prior knowledge and misconceptions during instruction fosters conceptual knowledge. But this instructional approach requires identifying students prior knowledge and misconceptions first. Productive Failure seems to be one fruitful approach for triggering students to externalize their prior knowledge and misconceptions as they generate representations in the process of collaboratively solving problems prior to instruction. The subsequent instruction can then build on these student-generated representations. It should be noted, however, that the representations used in the instruction phase of DI-S, PF, and PF+ were not the very representations produced by our participants in the two Productive Failure conditions. Rather, the representations were typical student-generated representations (taken from pilot studies) that matched the representations most often generated in the Productive Failure conditions. It seems that it is not necessary to use students own self-generated representations in order to successfully build on their (typical) prior knowledge and misconceptions. Yet, up to date, it has not been systematically investigated whether using typical studentgenerated representations during instruction is sufficient. So far research that compared learning with own representations to learning with typical representations has focused on the collaborative learning phase prior to instruction: It seems that during the collaborative learning phase students learn more from generating own representations than studying typical representations (Kapur & Bielaczyc, 2011; Roll, 2011). It remains to be investigated whether this difference between learning with own representations and learning with typical student-generated representations also holds true for the instruction phase. Our third finding was that students collaborative problem-solving prior to instruction enhanced conceptual knowledge equally well in both Productive Failure conditions. This finding suggests that the additional cognitive support that was provided in the PF+ condition is not necessary. This finding matches the results of two studies by Weinberger, Ertl, Fischer, and Mandl (2005). In their studies, providing epistemic support that (similar to our cognitive prompts) pointed learners towards specific aspects of the task did not promote learning, while interaction support in form of a role script did. In a previous study (Westermann & Rummel, 2012) we could fruitfully support students interaction with a role script in a delayed instruction setting that was similar to the procedure of our Productive Failure conditions presented here. It therefore seems promising to further investigate the differential effects of different dimensions of support for collaborative learning as proposed in the framework by Diziol and Rummel (2010). Even though the posttest results of the two Productive Failure conditions did not differ significantly, we are interested in finding out whether the problem-solving processes in the two Productive Failure conditions differed, that is, whether the cognitive support had any impact on the learning process. The currently ongoing analysis of our process data will help us to shed light on this question. One aspect is the impact of the cognitive prompts in the PF+ conditions on the number and quality of student-generated representations. A recent study by Wiedmann, Leach, Rummel, and Wiley (2012) indicates that the learning outcome may depend more on the total number of generated representations than on their quality (see also Kapur & Bielaczyc, 2012). In other words, it seems that students should externalize their prior knowledge by generating a large number of representations, including those representations that are based on misconceptions. If this finding was replicated by our data, it could at least partly explain why the cognitive support had no added benefits: The cognitive support probably helped students to improve the quality of their representations, but most likely did not influence the number of generated representations. Another aspect is to investigate how students collaboratively generate and learn with different representations. Specifically, we focus on the following questions: Do those students who introduce ideas for generating representations during collaborative problem-solving benefit more from this phase than the other group members? Literature points at the importance of active processing of learning relevant representations for learning (Atkinson & Renkl, 2007), but active processing may not require to generate ones own ideas. Indeed, this assumption is supported by our posttest data: Students in the DI-S condition benefited equally well from learning with typical student-generated representations as students in the Productive Failure conditions who first
ISLS
269
ICLS2012
generated own representations. To investigate the assumption that collaborative learning can foster elaborative discussion and co-construction (Moschkovich, 1996), we analyze whether students co-construct representations and how this might be related to their learning outcome. The literature on learning with multiple representations (e.g. Ainsworth, 2006) indicates that making connections across representations might be especially fruitful for learning. At the same time, the literature points at students difficulties to make connections across representations. We therefore analyze whether and how students make connections across the representations that they have generated themselves. In summary, the findings from our posttest data support the notion that it is central to build instruction upon students prior knowledge and misconceptions. The results of the ongoing process analysis will give further insights into the potential impact of the prompts in the PF+ condition on the collaborative learning process, and shed light on how students collaboratively learn with self-generated representations, that is, representations that exhibit their prior knowledge. At the conference, results of the process analysis will be presented and discussed along with the quantitative results.
References
Ainsworth, S. (1999). Designing effective multi-representational learning environments (No. 58). Nottingham: ESRC Centre for Research in Development, Instruction & Training Department of Psychology. Ainsworth, S. (2006). DeFT: A conceptual framework for learning with multiple representations. Learning and Instruction, 16(3), 183-198. Atkinson, R. K., & Renkl, A. (2007). Interactive example-based learning environments: Using interactive elements to encourage effective processing of worked examples. Educational Psychology Review, 19, 375-386. Bodemer, D., Kapur, M., Molinari, G., Rummel, N., & Weinberger, A. (2011). MUPEMURE: Towards a Model of Computer-Supported Collaborative Learning with Multiple Representations. Computer Supported Collaborative Learning CSCL2011 Proceedings, Vol. 3 (pp. 1065-1072). International Society of the Learning Sciences. Cuoco, A. A., & Curcio, F. R. (2001). The roles of representation in school mathematics. Reston, Virginia: National Council of Teachers of Mathematics. Diziol, D., & Rummel, N. (2010). How to design support for collaborative e-learning: A framework of relevant dimensions. In B. Ertl (Ed.), E-Collaborative knowledge construction: Learning from computersupported and virtual environments (pp. 162-179). Hershey, PA: IGI Global. Hammann, M. (2003). Aus Fehlern lernen (Learning from errors). Unterricht Biologie, 27(288), 31-35. Kapur, M. (2009). Productive Failure in mathematical problem solving. Instructional Science, 38(6), 523-550. Kapur, M., & Bielaczyc, K. (2011). Classroom-based Experiments in Productive Failure. In L. Carlson, C. Hlscher, & T. Shipley (Eds.), Proceedings of the 33rd annual conference of the cognitive science society (pp. 2812-2817). Austin, TX: Cognitive Science Society. Kapur, M., & Bielaczyc, K. (2012). Designing for productive failure. The Journal of the Learning Sciences, 21(1), 4583. Moschkovich, J. N. (1996). Moving up and getting steeper: Negotiating shared descriptions of linear graphs. Journal of the Learning Sciences, 5(3), 239-277. ODonnell, A. M. (1999). Structuring dyadic interaction through scripted cooperation. In A. M. ODonnell & A. King (Eds.), Cognitive perspectives on peer learning (pp. 179-196). Lawrence Erlbaum Associates, Publishers. Roll, I., Aleven, V., & Koedinger, K. R. (2011). Outcomes and Mechanisms of Transfer in Invention Activities. In L. Carlson, C. Hlscher, & T. Shipley (Eds.), Proceedings of the 33rd Annual Conference of the Cognitive Science Society (pp. 2824-2829). Austin, TX: Cognitive Science Society. Rummel, N., & Braun, I. (2009). Kooperatives Lernen mit digitalen Medienverbnden [Collaborative Learning with Multimedia]. In R. Pltzner, T. Leuders & A. Wichert (Eds.), Lernchance Computer - Strategien fr das Lernen mit digitalen Medienverbnden [Computer as a Chance for Learning Strategies for Learning with Multimedia] (pp. 223-240). Mnster: Waxmann. Schwartz, D. L., & Martin, T. (2004). Inventing to prepare for future learning: The hidden efficiency of encouraging original student production in statistics instruction. Cognition and Instruction, 22(2), 129184. Weinberger, A., Ertl, B., Fischer, F., & Mandl, H. (2005). Epistemic and social scripts in computer-supported collaborative learning. Instructional Science, 33(1), 1-30. Westermann, K., & Rummel, N. (2012). Delaying Instruction Evidence from a Study in a University Relearning Setting. Instructional Science. doi: 0.1007/s11251-012-9207-8 Wiedmann, M., Leach, R. C., Rummel, N., & Wiley, J. (2012). Does group composition affect learning by invention? Instructional Science. doi: 10.1007/s11251-012-9204-y
ISLS
270
ICLS2012
Explanandum and visibility condition change childrens gesture profiles during explanation: implications for learning?
Audrey Mazur-Palandre, Kristine Lund, ICAR Lab (UMR5191); CNRS; University of Lyon; cole Normale Suprieure de Lyon, 15 Parvis Ren Descartes, BP 7000, 69 342 Lyon Cedex 07 Email: Audrey.Mazur_Palandre@ens-lyon.fr, Kristine.Lund@ens-lyon.fr Abstract: We show how 6-year old first-graders gesture profiles evolve with both the type of explanandum and whether or not their interlocutor is visible. Sixty participants played two online educational games: one on numbers and the other involving manual dexterity. Our analyses focused on how 30 child-instructors explained both games to 30 child-learners in either one of two conditions: 1) face-to-face or 2) separated by a curtain. Our results give evidence for a multimodal view of language production that is closely tied to context. We show how some gestures remain stable across explanatory tasks, irrespective of interactional situations but how other gestures remain stable across interactional situations. We suggest implications for teaching-learning situations regarding gestured explanations.
Introduction and Theoretical Framework

We set out to see how the elaboration of gestures evolves with explanation task and interactional condition for 6-years olds, a population whose language and gesture skills are developing. We chose the context of how explanation production because researchers have mostly focused on why explanation production and we built an experimental situation where two different on-line game types were explained (the variable of context is tested) in two different interactional conditions (the variable of visibility is tested). We argue for a model of multimodal language production where both verbal productions and gestures are influenced by the context in which they are produced and are co-elaborated as human interaction proceeds. We present a brief typology of both explanations and gestures and present our theoretical hypotheses concerning gesture production for how explanations depending on both task content and interactional situation. Both gestures (Alibali, Kita & Young, 2000; Cook, Mitchell & Goldin-Meadow, 2008) and explanations (e.g. de Vries, Lund & Baker, 2002) are well known for their positive role in aiding learners conceptualization and in this paper, we hypothesize how their context of production can influence that role.
The Role of Gestures in Models of Language Production and Comprehension

Gestures can help to understand the production and the comprehension of language. The message that commands movement corresponds to an early step in the production process (Colletta, 2007, p. 21) and in this view, gestures would be elaborated when the verbal message is conceptualized (McNeill, 1992; Kita, 2000). According to this interface hypothesis, gestures as well as language help in the development of ones thinking in order to produce language, but also help others to understand ones language (Kita, 2000). Gestures are thus generated in relation to speech within a spatio-motor representation organized with the goal of speaking (Kita, 2000; Kita & zyrek, 2003); this model is based on the Growth Point theory of McNeill (1992;200).
Explanation and Gesture Typology

Explanation appears very early during child development and evolves during childhood (Colletta & Pellenq, 2009; Veneziano & Hudelot, 2002). Children are capable of coherent and cohesive discourse at ten or eleven years old (e.g. Mazur-Palandre, 2009). Explanation is an interesting theme as it is difficult to define linguistically. If the researchers that work on explanation agree that explanation is organized around an explanandum (phenomenon or behavior to explain and an explanans (cause, reason, motive that which explains) (Hempel & Oppenheim, 1948; Lund, 2003), they do not deny that the verb to explain is polysemiotic. Grize (1990) gives six definitions: (1) communicate, (2) develop (3) teach, (4) interpret, (5) motivate and (6) give an account for. According to Grize, explanation can respond to three questions: what, how, and why? One of the originalities of our study is that we concentrate on how explanations, corresponding to the type of explanation to teach in Grizes typology, very little studied for a child population. Gestures can be classified into two large categories: autonomous gestures (or quasi-linguistic gestures, Cosnier (2008)) and co-verbal gestures (amongst others, Colletta, 2007; Cosnier 2008; Kendon, 2004; McNeill, 1992). We focus in particular on co-verbal gestures for this study and focus on the following gesture typology: deictic gesture (pointing gesture), representational gesture (represents a concrete object), framing gesture (expresses an emotional or mental state of the speaker), discursive gesture (structures the discourse), interactive gesture (verifies the attention of ones interlocutor), enunciative gesture (expresses word searching) and
ISLS
271
ICLS2012
performative gesture (non-verbal language act that modifies the illocutionary force (i.e. speakers intentions; Austin, 1962) of the spoken word it accompanies). The aforementioned multimodal model of language production and the descriptions we have given of both gestures and explanation types allow us to present our theoretical and operational hypotheses.
Theoretical and Operational Hypotheses

First, Colletta & Pellenq (2009) showed that depending on the type of context narration, description or argumentation, childrens posture-mimo-gestuality specializes. For instance, during narrative and descriptive production, children produce more concrete gestures, whereas during argument production, they produce more interactive gestures. So, although both games are explained with how explanations, will the explanandum have an effect on gesture production, in particular since the spatial game seems to call for gesturing? Second, research illustrates the importance of the interlocutors visibility on adults gesture production (among others, Alibali, Heath & Myers, 2001; Bavelas, 1994; Bavelas, Chovil, Lawrie & Wade, 1992). Such results confirm Kendons theory (1987) that the conditions of transmission have an important impact on both linguistic units and gestures. However, to our knowledge only one study concerning children confirms an effect of visibility on gestures (Doherty-Sneddon & Kent, 1996). And it states that only older children are sensitive to this factor. So will our 6-year old children gesture differently according to the visibility of their interlocutor? Third, in previous studies, authors underline that gesture does not always have the same function. Gesture could both help the elaboration of the message by the speaker and the comprehension of the message by the interlocutor (Kita, 2000; Kita & zyrek, 2003; Kita, et al., 2007; Lozano & Tversky, 2006). Is it thus possible to distinguish gestures performed in order to help the elaboration of the message from gestures performed in order to facilitate the comprehension of the message by the interlocutor? These theoretical hypotheses are captured in three operational hypotheses: OH1, OH2 and OH3. First, we expect that depending on the game played (explanandum), gesture profiles will be different (OH1). Secondly, depending on the visibility condition, gesture profiles will be different (OH2). Third, if OH2 is validated, we may see which gestures are geared more for production and which for communication (OH3).
Methodology
Here, we first describe our population and our protocol. Second, we illustrate how we performed our analyses by describing how we coded speaker gestures during explanation. All of the first-graders in three schools in France (130 students) filled out a questionnaire from which we selected the participants of our study. All of the children were French native speakers, non-bilingual and did not have any behavioral or learning problems. 60 students participated: 30 girls and 30 boys (mean age: 6.6 years; standard deviation: 4.7 months. The children had parental authorization and participated voluntarily. We filmed the students during regular classroom hours inside their school buildings. Children worked in pairs; each one played the role of either child-instructor or child-learner. In phase 1, the child-instructor was first asked to play a game (either the number or spatial game). After having played, the child-instructor explained the game to the child-learner (phase 2, the focus of our analyses for this article). Next, the childlearner played the game under the watch of the child-instructor (phase 3). The two games played are part of CogniKizz Net, a French on-line educational game site from www.cognik.net. All children played both games: race to the numbers (a player is asked to recognize numbers or count objects by clicking on the correct image) and break the bricks (the player must hit back a ball with the help of a kind of racket). The experiment was carried out under two conditions: children were either face-to-face or were separated by a curtain. Transcription and coding of gestures was done in ELAN TM. A movement is identified as a gesture when it is sufficiently noticeable (high amplitude and visible by the interlocutor). Only hand and head gestures were taken into consideration (no leg and trunk movements, nor self-focus gestures were coded). Once the movements were tagged as gestures, one coder categorized them according to whether they were Deictic, Representational, Framing, Discursive, Interactive, Word-searching or Performative gestures (as described in the section on Gesture typology). Inter coder reliability calculations are in progress.
Results
Four sets of aggregated gesture profiles are shown: Figure 1 reflects explanations given by the childinstructor to the child-learner for the spatial game and Figure 2 does so for the numbers game. In both figures, the face-to-face condition is on the left and the separated condition is on the right. In all the graphs, the x-axis shows gesture types and the y-axis shows the percentage scale. The error bars represent the standard deviation, often quite large in studies on childrens language acquisition. Proportions are regrouped in Tables 1 and 2; bold type shows the highest values. The amount of discursive and word searching gestures produced while explaining the spatial game is dependent on the visibility condition (OH2 is confirmed and elements are given for OH3; cf. Figure 1). The amount of interactive, representational and word searching gestures produced while explaining the numbers
ISLS
272
ICLS2012
game is dependent on the visibility condition (OH2 is confirmed and elements are given for OH3; cf. Figure 2). Also, OH1 is confirmed: depending on the explanandum (game explained), gesture profiles are different. Gestures: spatial game face-to-face condition Gestures: spatial game - separated condition
Figure 1. The proportion in percentages of gesture types in the spatial game in the two conditions Table 1. Proportions of gestures while explaining the spatial game according to the visibility condition Gesture type Face-to-face Separated Representational 38.62 % 44.86% Discursive 35.46% 22.13% Interactive 12.01% 10.7% Word searching 9.07% 22.31% Deictic 4.1% / Performative 0.74% /
Gestures: numbers game face-to-face condition
Gestures: numbers game - separated condition
Figure 2. The proportion in percentage of gesture types in the number game in the two conditions Table 2. Proportions of gestures while explaining the number game according to the visibility condition Gesture type Face-toface Separated Representational 22.14% 7.2% Discursive 32.47% 40.42% Interactive 28.69% 19.63% Word searching 8.55% 28.62% Deictic 6.67% / Performative 1.48% / Framing / 3.70%
Discussion, Conclusion and Perspectives

When child-instructors explain the spatial game to child-learners, the majority gesture type is representational irrespective of visibility condition (cf. Table 3, results a and b). The fact that explaining a spatial task leads to representational gestures validates both OH1 and aligns with Kita & zyreks (2003) multimodal model of language production; if a spoken production contains notions of spatiality, then more representational gestures will be produced, as both verbal and gestural aspects are co-elaborated (c.f. Bergmann & Kopp, 2008). Table 3: Syntheses of results concerning the 60 child explanations, taken up in order in the discussion Explaining the spatial game a. Majority of representational gestures c. Less discursive gestures b. Majority of representational gestures d. More discursive gestures Explaining the numbers game e. Majority of discursive gestures f. Majority of discursive gestures
Absence of visibility Face-to-face condition
ISLS
273
ICLS2012
Absence of visibility Face-to-face condition
g. Word-searching gestures 2nd most produced
h. Word-searching gestures 2nd most produced l. Interactive gestures diminish strongly in the curtain condition across tasks i. Word-searching gestures 4th most produced j. Word-searching gestures 4th most produced k. Deictic gestures only occur in the no curtain condition across tasks
But Kita and zyreks model also states that language production depends on context and communicative goal and our results show that representational gestures remain stable even though visibility changes. Either representational gestures are important for both producing and comprehending messages or the explanandum type overrides the visibility condition. If visibility doesnt influence representational gesture production, this argues against a generalized OH2 whereas our other results report that the visibility condition does give different gesture profiles. Other research, but concerning adults also shows that visibility can indeed have an impact on the proportion of certain gestures. For example, Bavelas et al. (1992, 1994) have identical results to ours: adult speakers perform more interactive gestures in the face-to-face condition and they perform as many topic gestures (considered as equivalent to our representational gestures) in the face-to-face condition as in the separated condition. On the other hand, Alibali et al. (2001) shows contrary results to ours (but again, for adults): representational gestures depend on the visibility condition whereas beats (gestures that are synchronized with speech rhythm and emphasize words or phrases) have the same proportion regardless of the visibility condition. However, the explanandum was a cartoon to be narrated and not a spatial game to be explained, so gesture impetus differed. Another explanation for representational gestures in the spatial game remaining the same regardless of visibility condition (other than the force of representational gestures) could be that children of our study are too young to be sensitive to visibility. Indeed, Doherty-Sneddon and Kent (1996) show that six-year olds perform the same proportion of gestures in the two conditions (face-to-face and separated) whereas eleven-year olds make more communicative gestures in the face-to-face condition. But these results call for prudence insofar as no distinction is made between different communicative gestures and the gestures counted in the separated condition include those where children stand up in order to be seen, thus eliminating the visibility condition. Now consider the following two results: there are more discursive gestures when explaining the spatial game in the face-to-face condition as opposed to the separated condition (cf. Table 3, results c and d; OH2 confirmed) whereas when our child-instructors explain the numbers game to child-learners, the type of gesture they produce the most is discursive, but there is no distinction between visibility conditions (cf. Table 3, results e and f; argues against OH2). This difference in discursive gesture production according to visibility condition between the two games could perhaps again be explained by the explanandum nature: it makes sense to emphasize linguistic units having to do with breaking bricks when face-to-face because the action involved is visually shared. In the numbers task, there is an action of selecting a picture, but not of carrying out movement. The relative prevalence of discursive gestures (unless overridden by a spatial explanation task) is confirmed by Colletta & Pellenq (2009) who found that during why explanations on social and family topics, children from a similar age group as ours produced a majority of discursive gestures, when speaking to an experimenter. But then, must we choose in an either-or situation? Is it that either children are sensitive to the visibility condition (more discursive gestures in the face-to-face condition for the spatial task) or they are not (amount of representational gestures are highest for the spatial task and amount of discursive gestures are highest for the numbers game, both regardless of condition). Alternatively, perhaps the nature of the explanatory task that children accomplish determines their level of sensitivity to the visibility condition (a crossing of OH1 & OH2). If we continue to compare the interactional conditions, we first observe that word-searching gestures are the 2nd most produced for explaining both the spatial game and the numbers game in the separated condition whereas they are the 4th most produced for both games, face-to-face (cf. Table 3, results g, h, i and j). Is it that being blocked from seeing ones interlocutor hinders how explanation production and forces speakers to perform lexical or syntactic searches (OH2)? Second, deictic gestures only occur in the face-to-face condition (OH3); they seem to be specifically produced to facilitate the interlocutors comprehension of the message (cf. Table 3, result k). Third, interactive gestures diminish strongly in the separated condition, but still exist (cf. Table 3, result l). Are they mostly for facilitating message comprehension, yet pertinent for production (OH3)? We have shown that the context of young childrens explanation production (explanandum type + visibility condition) changes their gesture profiles; how can this result influence the positive role both explanation and gestures have on learners conceptualization? In this short paper, we can only invoke our perspectives for further research. A study from Cook, Mitchell & Goldin-Meadow (2008) shows how gesturing during problem solving makes for longer lasting learning gains, but perhaps the explanandum, the visibility condition and the type of gesture performed makes a difference for such learning gains. In summary, the multimodal model of language production argues that gestures are done both for production and comprehension (Lozano & Tversky, 2006) but we show that this is true to different degrees, depending on both explanandum
ISLS
274
ICLS2012
and visibility condition. Understanding the factors that influence speakers gesture production while knowing that gestures are important for conceptualization (Alibali, Kita & Young, 2000), should help us to design more effective teaching-learning tasks that are built around explanation.
References
Alibali, M.W., Kita, S. & Young, A.J. (2000). Gesture and the process of speech production: We think, therefore we gesture. Language and Cognitive Processes, 2000, 15 (6). 593-613. Alibali, M. W., Heath, D. C., & Myers, H. J. (2001). Effects of visibility between speaker and listener on gesture production: Some gestures are meant to be seen. Jrnl. of Memory and Language, 44, 169-188. Austin, J.L. (1962). How To Do Things With Words. Oxford: Clarendon Press. Bergmann, K. & Kopp, S. (2008). Multimodal content representation for speech and gesture production. In M. Theune, I. van der Sluis, Y. Bachvarova and E. Andr (Eds.), Symposium at the AISB Annual Convention: Multimodal Output Generation (pp. 6168). Aberdeen, UK Bavelas, J. B. (1994). Gesture as part of speech: Methodological implications. Research on Language and Social Interaction, 27(3), 201-221. Bavelas, J. B., Chovil, N., Lawrie, D.A.&Wade, A.(1992). Interactive gestures. Discourse Processes, 15, 469489. Colletta, J.-M. (2007). Signaux corporels et acquisition du langage: des relations constantes et troites. Langage et pratiques, 39, 20-33. Colletta, J.-M. & Pellenq, C. (2009). The development of multimodal explanations in French children. In M.A. Nippold & C.M. Scott (Eds.), Expository Discourse in Children, Adolescents, and Adults. Development and Disorders, (pp. 63-100), Taylor & Francis. Cook, S., Mitchell, Z., & Goldin-Meadow, S. (2008). Gesturing makes learning last Cognition, 106 (2), 10471058 Cosnier, J. (2008). Les gestes du dialogue, La communication, tat des savoirs, Editions Sciences Humaines, (pp. 119-128). Doherty-Sneddon, G.& Kent, G. (1996).Visual signals and communication abilities of children. British Journal of Developmental Psychology, 18, 595-608. Grize, J.-B. (1990). Logique et langage. Gap/Paris, Ophrys. Hempel, C. G., & Oppenheim, P. (1948). Studies in the Logic of Explanation. Philosophy of Science, 15, 567579. Kendon, A. (1987). On Gesture: Its complementary relationship with speech. In A. Seigman and S. Feldstein, (Eds.), Nonverbal Communication, (pp. 65 - 97.) Hillsdale, New Jersey: Lawrence Erlbaum Associates. Kendon, A. (2004). Gesture: visible action as utterance. Cambridge: Cambridge University Press. Kita, S. (2000). How representational gestures help speaking. In David McNeil (Eds.), Language and gesture, (pp. 162-185), Cambridge, Cambridge University Press. Kita, S. & zyrek, A. (2003). What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language, 48, 1632. Mazur Palandre, A. (2009). Le flux de linformation, aspects syntaxiques et discursifs, Une tude fonctionnaliste et dveloppementale. Thse de doctorat, Universit de Lyon. Lozano, S.C. & Tversky, B (2006). Communicative gestures facilitate problem solving for both communicators and recipients. Journal of Memory and Language 55, 4763 Lund, K. (2003). Analyse de l'activit explicative en interaction : tude de dialogues d'enseignants de physique en formation interprtant les interactions entre lves. Thse de doctorat, Universit de Lyon. McNeill, D. (1992). Hand and Mind: What gestures reveal about thought. Chicago: Chicago University Press. McNeill, D. (2000). Language and gesture. Cambridge, Cambridge University Press. Veneziano, E. & Hudelot, C. (2002). Dveloppement des comptences pragmatiques et thories de l'esprit chez l'enfant : le cas de l'explication. In J. Bernicot, A. Trognon, M. Guidetti & M. Musiol (Eds.), Pragmatique et psychologie (pp. 215-236). Nancy : Presses Universitaires de Nancy. De Vries, E., Lund, K. & Baker, M.J. (2002). Computer-mediated epistemic dialogue: Explanation and argumentation as vehicles for understanding scientific notions. The Journal of the Learning Sciences, 11(1), 63103.
Acknowledgments
This study was funded by the Rhne-Alpes Region and Europe as a FEDER project. We thank the students, parents, principal and teachers at Tignieu-Jameyzieu and Saint Romain de Jalionas primary schools for participating. We thank Gregory Pescini and Jean-Marc Colletta for help in obtaining authorizations and JeanMarc for discussions on gesture analysis. We are grateful to Nathalie Besacier, Clment Moret and Pascale Pauly for their help with collecting data and to Cline Faure and Marilyne Goutagny for their statistics advice.
ISLS
275
ICLS2012
Bidirectional artifact analysis: A method for analyzing creative processes

Alecia Marie Magnifico, University of Illinois at UrbanaChampaign, amagnif@illinois.edu Erica Rosenfeld Halverson, University of WisconsinMadison, erhalverson@wisc.edu Abstract: In order to understand and design learning environments that support new literacies, we must develop methods to describe the creative production of literacy artifacts. In this paper, we describe bidirectional artifact analysis, a framework that employs ethnographic observations of participants in situ, interviews over time, and the artifacts they create to trace young peoples creative production practices. While typical descriptive analyses move forwards, we move bidirectionallyfrom final product backwards and from initial idea forwardsto better understand participants learning processes and the role of social, collaborative audiences in that learning.
The Need for a New Methodology

In recent years, research on new media and new literacies has re-focused the meaning of literacy (e.g. Moje, Overby, Tysvaer, & Morris, 2008; New London Group, 1996). Beyond comprehending and producing texts, being literate has come to describe the use of medium- and context-appropriate tools, practices, modes, and epistemologies to produce textual, visual, and multimodal artifacts. These understandings have particular purchase in digital environments and creative arts education, where social interactions are often shaped by the production and critique of artifacts for and with a responsive audience (Magnifico, 2010). In order to understand and design learning environments to support these complex literacy practices, we must develop methods that describe creative production and literacy artifacts. We must build analytical lenses that take creative worksvideo, writing, multimedia collage, etc.as units of analysis and trace them through complex processes of creation and revision. This paper is thus guided by two methodological questions: (1) How can we study the relationship between processes and products in creative new literacies spaces?, and (2) How can we document and describe the ways in which participants become literate in these environments? In this paper, we describe bidirectional artifact analysis, an analysis framework for creative production processes that employs ethnographic observations of participants in situ, the artifacts they create, and interviews over time. Marrying narrative analysis (Halverson, 2008; Labov & Waletzky, 1967/1997), discourse analysis (Wood & Kroger, 2000), and artifact analysis, this framework echoes and extends Enyedys (2005) description of bidirectional analysis: go[ing] backwards in time in an attempt to trace the origins of this intervention and forwards in time to examine what subsequent impact it had on the way other students reasoned (p. 437). While typical descriptive analyses move forwards, we, like Enyedy, have found that moving bidirectionally from final product backwards and from initial idea forwardshelps us to understand participants learning, as well as the role of social and collaborative audiences in that learning.
Theoretical Perspectives
In order to produce creative work successfullyor to teach creative processes to othersartists of all kinds must consider not only the quality of their original ideas, but also how these concepts will be translated to consumers. In our attempts to trace the learning of creative processes, we have found it useful to attend to several factors: (1) the representations that artists use to build ideas and describe them to others, (2) the audiences for creative processes, and (3) the metrics of quality for evaluation of creative products.
The Role of Representation in Creative Processes

diSessa (2004) argues that representation design is a creative endeavor, a venue in which creative and artistic skills are at a higher than normal premium (p. 300). Thus, observing creative production helps us to understand the role of external representation in creativity and learning (e.g. Eisner, 2002). Getting smart in the context of producing art means coming to know the potential of the materials in relation to the aims of a project or problem (Eisner, 2002, p. 72). The capacity to build representations from different materials for multiple situations is a marker of creative expertise (Hayes, 1989), and has been called essential in investigations of creativity (p. 262) (Hasirci & Demirkan, 2007). Since artistic production requires creative representation, learning art means learning how to construct and evaluate these representations. Many methods for studying educational settings focus on examining one kind of data to evaluate learning. Educational researchers have developed deep frameworks for understanding classroom participants conversations, written texts, interviews, visual art, and multimodal online activities. Creative production settings, however, often encourage novices to engage with multiple media to produce a poem, script, video, or piece of visual art. While representations may shift, and ideas may transform through drafting, feedback, and
ISLS
276
ICLS2012
revision, understanding how all of these artifacts contribute to the final textthe final draft of the creative productionis necessary to understand the processes that artists employ.
How Producers Engage with Their Audiences

Our understanding of audience and collaboration have broadened as new literacies and new media have pushed writers and artists into a world where they may create, read, and view, as Lunsford & Ede put it, among the audience (2009). This shift can transform writers into speakers, readers into critics, and both into collaborators. Online artists can request audience feedback by soliciting email or comments and respond with another draft if they choose. Writers and creators in the same physical space can work together to solve problems and clarify ideas. Within such artistic communities, it is increasingly plausible to think of readers and viewers in terms of Ongs (1975) collectivity, acting on one another and on the speaker [or writer, or artist] as members of an audience do (p. 11). Considering how feedback guides artistic process suggests that audiences for creative work may play both a cognitive role, in that sharing work forces creators to consider readers, and a social one, in that creators define themselves as community members by the work that they share (Magnifico, 2010).
Method: Bidirectional Analysis

Most education research moves forward in time. Teachers and researchers create intervention curricula; they teach the curricula; they observe the effects by comparing pre-intervention and post-intervention data. Learning builds up from prior knowledge, and therefore, teachers and researchers teach and analyze in this direction, too. Priors (2004) multi-textual tracing methodology, developed to examine students learning in higher education, deepens this notion of learning over time by capturing multiple contextual threads of teaching and learning practice. Analyzing assignments and other initiating texts, classroom discourse, writing drafts, teacher and peer feedback, and students reflections, Prior traces how scholarly arguments and ideas are written into being, often as hybridizations of student interest and teacher guidance. Similar multi-layered evolutions might be traced over time for any student, either in schools or creative production environments. In his studies of elementary classrooms, Enyedy (2005) turns this forward-development timeline metaphor on its head. Students worked progressively to inven[t] representations and iteratively refin[e] solutions to problems that the class ha[d] identified (p. 428). Instead of looking at the students successive representations over time, however, Enyedy (2005) analyzes these interactions by reaching backwards and forwards, examining how each new representation was a revision of prior ideas. By turning the analytical lens, this study shows how students reflections became a base from which the classs inquiry progressed. In our work, we extend Priors tracing and Enyedys bidirectionality to creative production. Learning models in creative production settings often resemble the common instructional models of writers workshops (e.g. Atwell, 1998) or design crits (e.g. Soep 2006). Young creators present drafts to a group of mentors and peers, assess their current successes and difficulties, and receive revision suggestions. Final drafts are built from past representations, and reflect not only the artists changing ideas, but also those offered by readers in peer and mentor feedback. While we focus on creative and artistic production here, the cross-content implications are clear: When we analyze drafts as representations that grow and change as a result of individual cognitions, mentor responses, and social reflections, we can build models for developmental learning progressions (e.g. Duncan & Hmelo-Silver, 2009) that account for collaboration in many different subject areas.
Examples of Bidirectional Analysis in Action

To demonstrate this method, we draw on two studies of new literacies learning. We present a brief example from each data set to demonstrate how bidirectional artifact analysis allows researchers to see how students engage with their compositions and audiences, as well as their growing mastery of appropriate representational tools. We demonstrate how the method works and how it helps us to understand young peoples emerging access to and use of the new literacy practices that researchers describe as crucial for 21st century success.
Example 1: Frank as Border Crosser

This example is drawn from Halversons case studies of United States youth media arts organizations. Data collection traced students digital art from initial program applications to final public presentations, and included ethnographic observations, artifact collection, semi-structured interviews, and post-mortem reflections. Franks piece (Figure 1), is an autobiographical graphic design piece created through Street Level Youth Medias 2007 program, Represent!: exploring your identity through history and culture. This piece includes three elements: 1) a photoshopped image of himself in front of two flags; 2) a black and white line drawing of a person split down the middle; 3) below the images, a series of names/nicknames printed on individual placards. The meaning and genesis of these elements can be traced by examining Franks journal, interviews, and final piece. Franks artist statement describes his work as: a graphic design piece dealing with him being a Mexican American, crossing borders each day going to different places with his art work and
ISLS
277
ICLS2012
gathering all of his names. Each of the elements is analogous to the components described in the statement; they are all connected with this vision for his art and how it represents his identity. In other work, we use bidirectional artifact analysis to trace the evolution of Franks whole piece as a method for documenting learning (Halverson, in press). For illustrative purposes here, we trace one of the three core components of Franks piece. The largest image in Franks final piece is a screen print, a color image of him and two simplified flags, American and Mexican (see Figure 1). The initial idea for the dual flags appears in his journal as a drawing of a black figure standing in front of a box that is half white and half grey. This image is accompanied by the text, border crossing from Mexico to USA. Frank attributes this idea to his father. In an post-mortem interview, he described struggling to represent himself in his final piece of art. His father asked him, Why dont you do something about you being Mexican American? Not only did Frank embrace the idea, he felt motivated by this contribution: Thats like, part of the reason I did this. Because of my dad. He helped me too (09/15/08). Interviews with Frank reveal how he got from border crossing to the two flags present in the piece. He describes beginning by, tak[ing] a regular flag from here and a regular Mexican flag and put[ting] them in the background, behind his photo. The simplified flags emerged from his artist-mentors suggestion[he] gave me the concept of make your own flag and an image from an album cover. I saw a flag that Trent Reznor Figure 1: Frank - Represent! did Its basically black. Its all red. Its like blood dripping down from the flag. I wanted to do that, but it kind of, it strayed from who I am (Interview, 09/15/08). In describing the flag from a Nine Inch Nails album, Frank identifies what he likes about a self-made flag. The reason that he does not import this image, however, was that, it strayed from who I am. He wanted to keep himself in the center of his art, so he chose an aesthetic and created a Mexican-American border-crossing version of the image. The black and white image to the right replicates the sketch from Franks journal. He describes this image as, basically the same thing, have to dealing with borders. Only it's...it's the same thing as this only it's stripped down of everything. He sees the two images conveying similar meaning: Frank as border crosser. However, where the large image refers to Franks Mexican-American identity, this stripped down version is different: It can mean anything. Im basically walking into anywhere. With the same concept of walking into different borders or different places. (Interview, 09/15/08). The final image included both border crossing representations, the full color image of Frank in front of two flags and this initial sketch. But Frank transforms the meaning of the initial sketch to represent himself as a more general border crosser. He understands and represents himself as someone who can go anywhere, a broader scope than his dads original idea. The above analysis traces one of the three key components of Franks piece from initial conception to final piece. However, this division is more analytic than temporal; Frank did not conceptualize each of these elements independently. Bidirectional analysis allows us to understand Franks artistic process and the evolving relationship between his concept (identity) and his representational choices (graphic design). Frank struggles initially, but through conversations and journal entries, he determines that he wants to create a visual representation of himself that portrays his identity as a Mexican-American, a border-crosser more generally, and a friend and family member with many roles and names. He marries his narrative perspective to the affordances of the tools, which results in his final piece and an artist statement that explains this relationship to an audience.
Example 2: Noahs Poetry Collaborations

The second study, is drawn from Magnificos (under review) case studies that examined creative writing practice and young writers work with different audiences. Data collection included ethnographic observation, artifact collection, peers and mentors feedback, final products, assessments, and semi-structured interviews. This second example details the effects of reader feedback on one student, Noah (all names are pseudonyms), who used poetry to describe everyday events as epic occurrences. In his 11th grade classroom, Noahs teacher, Mr. Caswell, designed an open workshop format for students to write creative pieces and respond to each others work. Tracing the artifacts of Noahs poem drafts reveals different revision patterns as he worked with and received feedback from different peers. This bidirectional analysis shows the genesis of Noahs revisions on Pitter Patter, his poem about driving in a rainstorm, and suggests pedagogical structures to encourage generative critique in classroom workshops. Noahs experience exposes a problematic feature of this pedagogy: The continued development of students writings seemed to depend on their workshop critique sessions, but the content of these conversations was quite diverse in practice.
ISLS
278
ICLS2012
Abstract Critique
Mr. Caswell often asked his class to complete a written close reading exercise before they launched into verbal critique. On a close reading day, Henry writes comments on Noahs poem Pitter Patter when Mr. Caswell asks the class to exchange papers, choose a literary element to analyze, and write comments on each others work. Henry suggests changes, noting, for instance, this sentence doesnt make sense grammatically (see below). The car slows and stops where Soaked grass begins over there Rubber tires brush the damp fuzz With pitter-patters the grass is abuzz (from Draft 4 of Pitter Patter). He does not identify specific errors, however. Next the to the last two lines of this stanza, Henry writes what made you pick this?! The students never have a chance to discuss these edits, though, because the hour ends. Henrys comments are emphatically punctuated?! but do not offer clear suggestions or justifications for why Noah should consider making changes. Perhaps as a result, Noahs revisions never address these ideas.
Concrete Critique
Noah and Kira work together for two class periods to discuss Pitter Patter. Their conversations, however, feature close readings and textual antecedents. While they challenge each other, they often resolve arguments and questions with textual evidence. Here, Kira and Noah discuss Pitter Patters setting in detail, considering the scene that inspired the poem. Kira notes the unclear setting, invoking herself and people as readers: Kira: You were THAT off on the side of the road? Noah: Nooo, no, here, let me sketch the scene... This is mine, this is the road, this is the church... This is the soccer field behind the church... I drove my car up, and it was pouring. Kira: So you're in a parking lot. So you stop. So you drive into a parking lot and stop You need to tell me that somewhere in this poem, because I didnt get that at all I mean, do you want the reader to know youre stopped in a parking lot? Noah: It doesnt matter it doesnt matter... I dunno. [] Kira: You just have to remember that the people reading this have never been here. Like when I read it, I imagined [our schools] road, right there And your car brushing the grass would be on the side... Noah: Are you saying Im a bad driver? But yeah Maybe that would be it. Clarify Kira: Like, you don't HAVE to, but if you wanna create that image then well, obviously. (Classroom transcript, 4/1/2009) While Noah argues that Kiras failure to understand the setting doesnt matter, Kira persists, explaining the uncertainty that might confuse other readers. She asks questions, pointing out that if the poem doesnt clarify where [she is] then it is difficult to create that image. She justifies her suggestion by implying that poetry creates images for readers and noting that people cannot understand the poem without the setting. Kira aligns her critiques with Noahs text, pointing out textual elements and the consequences of those choices. In the next draft, Noah modifies two stanzas, making changes to clarify the events of the poem. During their meetings, Kira makes many explicit written suggestions, including word substitutions, image clarifications, and perspective suggestions. To this, she adds oral suggestions that refer to Pitter Patters text rather than general criticisms. As a result of these exchanges, Noah makes 12 revisions to his poem, including two new or expanded stanzas, several clarified images, and a strengthened first-person perspective. In contrast to his collaboration with Henry, Noahs critique conversations with Kira led to many more suggestions and revisions. Examining Noahs work in a bidirectional way, tracing his poems through drafts and critique conversations, shows us that while most students made comments that were relevant, few general suggestions led to textual changes in future drafts. Rather, Noahs responses to critiques and revisions present a clear pattern. He revises when a reader shows him that his language is obscuring rather than conveying his ideasfor example, Kiras notes about Pitter Patters perspectiveand suggests how these lines might be improved. In this way, audience response seems critical to Noahs creative and representational processes.
Significance and Implications

While these two examples explore different forms of creative production, bidirectional analysis reveals similar processes. Frank designs his autobiographical screenprint with several media. He consults with others and captures his evolving thoughts about self-representation in his journal. These data sources show why he made representational decisions and how he used media tools to represent himself. Noah revises his poetry through
ISLS 279
ICLS2012
critique conversation with his peers. While his poetic medium does not change, he uses in-depth feedback to improve his poems and his understanding of readers reactions. In both cases, audiences make valuable contributions to the creative work. With its focus on drafts, artifacts, and young artists interactions around those artifacts, bidirectional analysis shows us how representations shift through creative processes and how feedback helps young artists to refine their ideas and knowledge of representational materials (Eisner, 2002). For researchers seeking to design environments that encourage the development of new literacies, taking up production-oriented definitions of literacy requires a complementary shift in analysis. Methodological tools such as case study (e.g. Stake 1995), ethnographic observation (e.g. Geertz, 1973), thematic coding (e.g. Saldaa, 2009), and discourse analysis (e.g. Wood & Kroger, 2000) represent vital steps in documenting literacy practices of a learning environment in context. No single analytic tool, however, provides enough information to understand how production processes lead to literacy learning in complex environmentsthose focused on creative production and on developmental content learning (Duncan & Hmelo-Silver, 2009). These complex tasks and questions require broader methods that help researchers to parse data drawn from observation, conversational discourse, and artifacts in careful combination. Bidirectional artifact analysis offers a way forward by articulating an analysis framework that combines methodologies for conducting inquiry with complex qualitative datasets.
References
Atwell, N. (1998). In the middle: New understandings about writing, reading, and learning. Portsmouth, NH: Boynton/Cook Publishers, Inc. diSessa, A. (2004). Metarepresentational competence: Native competence and targets for instruction. Cognition & Instruction, 22(3), 293-331. Duncan, R.G. & Hmelo-Silver, C. (2009). Learning progressions: Aligning curriculum, instruction, and assessment. Journal of Research in Science Teaching, 46(6), 606-609. Eisner, E. (2002). The arts and the creation of mind. New Haven, CT: Yale University Press. Enyedy, N. (2005). Inventing mapping: Creating cultural forms to solve collective problems. Cognition and Instruction, 23(4), 427-466. Geertz, C. (1973). The interpretation of cultures. New York: Basic Books. Halverson, E.R. (2008). From one woman to everyman: The reportability paradox in publicly performed narratives. Narrative Inquiry, 18(1), 29-52. Halverson, E.R. (in press). Digital art-making as a representational process. Journal of the Learning Sciences, 21(2). Hasirci, D. & Demirkan, H. (2007). Understanding the effects of cognition in creative decision making: A creativity model for enhancing the design studio process. Creativity Research Journal, 19(2-3), 259-271. Hayes, J. R. (1989). Cognitive processes in creativity. In J. A. Glover, R. R. Ronning, & C. R. Reynolds (Eds.), Handbook of creativity. New York: Plenum. Labov, W., & Waletzky, J. (1967/1997). Narrative analysis: Oral versions of personal experience. Journal of Narrative and Life History, 7(14), 338. Lunsford, A.A. & Ede, L. (2009). Among the audience: On audience in an age of new literacies. In M.E. Weiser, B.M. Fehler, & A.M. Gonzales (Eds.) Engaging audience: Writing in an age of new literacies. (pp. 42-69). Magnifico, A.M. (under review). Authenticity as a design element for writing pedagogy. Under review by Research in the Teaching of English. Magnifico, A.M. (2010). Writing for whom: Cognition, motivation, and a writers audience. Educational Psychologist, 45(3), 167-184. Moje, E. B., Overby, M., Tysvaer, N. & Morris, K. (2008). The complex world of adolescent literacy: Myths, motivations, and mysteries. Harvard Educational Review 78(1), 107-154. New London Group. (1996). A pedagogy of multiliteracies: Designing social futures. Harvard Educational Review, 66(1), 60-92. Ong, W. (1975). The writer's audience is always a fiction. PMLA, 90(1), 9-21. Prior, P. (2004). Tracing processes: How texts come into being. In Bazerman, C. & Prior, P. (Eds.), What writing does and how it does it: An introduction to analyzing texts and textual practices (pp. 167-200). Mahwah, NJ: Lawrence Erlbaum Associates. Saldaa, J. (2009). The coding manual for qualitative researchers. Thousand Oaks, CA: Sage Publications. Soep, E. (2006). Critique: Assessment and the production of learning. Teachers College Record, 108, 748-777. Stake, R. (1995). The art of case study research. Thousand Oaks, CA: Sage Publications. Wood, L.A., and Kroger, R.O. (2000). Doing discourse analysis: Methods for studying action in talk and text. Thousand Oaks, CA: Sage Publications.
ISLS
280
ICLS2012
Learning from the folly of others: Learning to self-correct by monitoring the reasoning of projective pedagogical agents
Sandra Y. Okita, Teachers College, 525 West 120th Street, New York, NY 10027, okita@tc.columbia.edu Azadeh Jamalian, Teachers College, 525 West 120th Street, New York, NY 10027, aj2334@columbia.edu Abstract: Students find it relatively natural to catch other peoples errors, even if they are not inclined to catch their own. A total of sixty-two nine- to eleven-year-old students participated in two studies that tested the hypothesis that monitoring the reasoning of a pedagogical agent solving math problems, can help students learn the skill of monitoring, and eventually selfcorrect when solving math problems. A projective pedagogical agent ProJo was designed to openly displays its reasoning when solving math problems allowing children to look for mistakes. Two testing environments Doodle Math, and Puzzle Math were implemented to directly compare self-monitoring and self-other monitoring treatment. The results showed initial evidence that self-other monitoring may be an effective way to help students develop metacognitive skills to self-correct and accurately solve problems.
Introduction
Careless mistakes can easily be avoided if children learn to check their answers. One way to minimize mistakes is for people to monitor their thinking and activity. This way, they can catch potential mistakes or confusions before they lead to problematic errors. There are various viewpoints on how the ability to self-monitor develops in childhood and how to promote it (Markman, 1977). One viewpoint is that children develop abilities to selfmonitor as they work to resolve their own confusions and internal conflicts (Karmiloff-Smith, 1974; KarmiloffSmith 1979; Brown 1987). From this perspective, one way to help children learn to self-monitor is to put them in situations that will lead to cognitive conflict. Often times people turn to others to understand themselves. In this viewpoint children learn to self-monitor by observing other people who self-monitor (Palincsar & Brown, 1984; Okita & Schwartz, 2006). In this case, presenting children models of others who overtly self-monitor should lead to improvements. This research examines a third alternative: Children learn to self-monitor by monitoring other people, and with practice, they turn the external monitoring inwards. Children may find it difficult to be attentive to their own mistakes when they are concentrating on a problem or task. However, they may find it relatively easy to catch other peoples mistakes. Gelman and Meck (1983) found that children, who had difficulty checking their own work, found it easier to point out mistakes of others (e.g., puppets). By focusing children on catching anothers mistakes, children may learn to the skill of monitoring, and eventually self-monitor.
Related Works
Studies have shown that learning among peers and comparing ourselves to peers can be quite effective (Chi, 2001; Graesser, Person, & Magliano, 1995). For example, comparing test scores helps people learn where they stand academically. Another way is observing others to better understand the self. For example observation can be better than doing when students cannot solve a math problem, but observing others solve a math problem may help you learn how to solve problems. This is because the person they are observing can provide a model of competent performance. Research has also shown that observing somebody who knows less (or knows about the same as you) can be beneficial. Other studies in the area have shown (e.g., reciprocal teaching) that students may spontaneously compare their understanding to what they observe in another person, and any discrepancies can alert them to think more deeply about who is right (Seifert & Hutchins, 1992). This implies that observing an imperfect peer, under the right circumstances, might be a great trigger for learning and reflection. It is expected that the effect of self-monitoring may not be immediate while the task is still challenging and unfamiliar, and may take some time for the behavior to turn inward. Norman (1983) suggests that at first, students who self-other monitor may do worse than students who do not, but over time they may surpass those students without the self-other monitoring treatment. An alterative approach to a human peer is the use of a pedagogical agent. Studies also demonstrates that students can learn by observing, an agent that knows less than they do (Schwartz, Chase, Chin, Oppezzo, Kwong, Okita, Roscoe, Jeong, Wagster, & Biswas, 2009). An ideal learning environment would be an interactive pedagogical agent designed to solve math problems that would engage the student in catching mistakes and monitoring agent behavior. Students with practice can turn this external monitoring inwards and learn to self-correct. The Projective Pedagogical Agent, ProJo, was developed so that students could learn to monitor and check for potential mistakes. ProJo displays the reasoning behind the problem solving so that the student can follow ProJos lead and check the answer. The paper introduces two studies that test the hypothesis of whether modeling a pedagogical computer agent solving math problems may help students learn to self-monitor and solve math problems with better accuracy.
ISLS
281
ICLS2012
Two testing environments Doodle Math (study1) and Puzzle Math (study 2) were created to allow a direct comparison between self-other monitoring to self-monitoring.
Figure 1. ProJo agent in Doodle Math (Left), Puzzle Math & Experimental Setting (Right)
Study 1: Self vs. Self-Other Monitoring

Study Design and Procedure
In the first study, forty elementary school students between the ages 9- to 11- years participated in a one-to-one forty-five minute session. The students were randomly assigned to the self-monitoring treatment or self-other monitoring treatment. Self-other treatment was compared directly to self-treatment as a between subjects design. Students were taught a math trick on divisibility rules for multiplication. In the self-other treatment, ProJo appears in the learning environment (See Figure 1), whereas in self-treatment, ProJo is not present. The treatment session has ten trials (5 pair trials). Children in the self-treatment solve all ten trials on their own. In the self-other treatment, the student and ProJo, each take turns solving five trials each. In other words the student in the self-other treatment condition will be solving 5 trials on their own, and 5 trials monitoring the agent ProJo solve problems. The students were responsible for monitoring and catching any mistakes ProJo made. ProJo solved problems that were sometimes correct and other times incorrect. When it was ProJos turn, he would say, I think I learned a lot by watching you play. Let me try. I dont like mistakes, so if you can catch me when I make a mistake that would be great. We are a team, and I wouldnt want you to lose a point because of my mistake. ProJo would make a few intentional slow motion mistakes so the student could monitor and catch (or not catch) the error. The relative effects of self-other monitoring were evaluated using calculation time and accuracy measures during the treatment sessions, and pre- post-tests were administered at the beginning and end of the study. To see the effect of each treatment (self vs. self-other) the analysis looked only at the 1st of the pair for the self and self-other treatments. Therefore the first analysis included 1st of the pair (five problems) from the self-other treatment, and the 1st of the pair (five problems) from the self-treatment condition.
Materials and Measures

The testing environment Doodle Math (See Figure 1 Left) was created to allow students to practice monitoring. Students used a pen and tablet device to write out their calculations on the screen. Doodle Math was developed to run in a web-based environment. The key measures from this study include a pretest and posttest, math problems used in the 10 trials during treatment session, screen shots during treatment, and a play back log file of students behavior. A pretest was used to examine the current math level of the students and to identify the calculation errors students made. The errors were later implemented as ProJos calculation mistakes, and displayed during the self-other monitoring treatment sessions. The relative effects of self-other monitoring were evaluated using time and accuracy measures during the treatment sessions, as well as on pre- and post-tests. The posttest showed if self-other treatment had an overall effect on math accuracy. The screen shots showed students reasoning and method when problem solving (e.g., did they follow the math trick, where did they make a mistake, did they accurately catch ProJos mistakes). The log file recorded the hand written calculation for each student by keeping a log of the data points (e.g., x, y points of the white drawing area) from the pen and tablet. The log file allowed ProJo to play back the reasoning process and calculation of the student in the students original handwriting and timing. This playback log file was an important assessment tool for teachers because they could playback the recording and see where the student hesitated during problem solving.
Results
The initial thought was that students in the self-other treatment would slow down when solving problems on their own. During the treatment session, students in the self-other treatment (Figure 3 dotted line) showed a more gradual decrease in time than the self-treatment (See Figure 2 dotted line). The students in the selftreatment increased calculation speed immediately after the first trial. Students in both treatments spent most time on the first problem, possibly getting acquainted. The effect of accuracy and time was short lived where the
ISLS
282
ICLS2012
two treatments showed little difference by the posttest. Comparing calculation time and accuracy across the trials, results showed that students in the self-treatment immediately picked up speed and did worse over time.
Figure 2. Accuracy and time for self-treatment.
Figure 3. Accuracy and time for self-other treatment.
Students in the self-other monitoring treatment slowed down after the first exposure to the agent, then gradually picked up speed and improved in accuracy over time (Figure 3). One possible interpretation is that students are working hard to internalize the monitoring, which cashed out on later problems. The results were not sufficiently strong to guarantee this interpretation, but showed a promising trend. As mentioned earlier, the pretest identified some calculation errors students made. The students errors were later implemented as ProJos calculation mistakes, and displayed during the self-other monitoring treatment. During monitoring, the agent ProJo would randomly make three mistakes and correctly solve two problems, so the student did not automatically expect ProJo to always make an error. The three mistakes included two similar calculation mistakes that the student made on the pretest (mix-up procedure and losing track in procedure), and one general calculation mistake (different from the students mistake). The results showed that similar mistakes (as their own) were difficult to catch, while students had no trouble catching general calculation mistakes, or acknowledging when the ProJo solved the problems correctly. One possible explanation is that general calculation errors are matters of fact types of problems, and students can easily remember their math facts and compare them to ProJos computation. In contrast, for the mix-up and lose track problems, the errors are procedural, and students need to follow the procedure and check it against their own answer. This may be more difficult because they are new to the procedures, whereas math facts are familiar. An alternative explanation is that the calculation errors are different from the mistakes the students would make, so it is easy for them to catch. The critical limitation was that ProJo was successful at replaying errors, but unsuccessful in making the students notice their own mistakes. The study provided initial evidence that self-other treatment using ProJo may be an effective way to help students learn to self-correct. The studies were short in duration and designed to see if there were any short-term effects. A longer intervention may determine if more sustained practice at monitoring would lead to stronger effects. Another limitation is that that current study falls short in determining whether the students were monitoring or just copying behavior. The second study extended the session to an hour and continued treatment for two days to see if this would sustain monitoring practice and improve accuracy. A new testing environment Puzzle Math (See Figure 1 Right) was created for the second study so that it included a detection features to measure student monitoring.
Study 2: Extended Treatment and Detecting Monitoring and Behavior

Study Design and Procedure
Twenty-two elementary school students from Northern Manhattan participated in a one-hour treatment session. The students were between the ages 9- to 11-years and participated for one session a day for two days. The students were randomly assigned to one of two conditions, self-monitoring treatment or self-other monitoring treatment. Self-other treatment was compared directly to self-treatment as a between subjects design. The treatment session had seven puzzles. Children in the self-treatment solved all seven puzzles on their own. In the self-other treatment, the student took turns with ProJo, the student solving five puzzles, and monitored ProJo solving two puzzles. The relative effects of self-other monitoring were evaluated using accuracy measures, a monitor log that detects if students were monitoring (i.e., self correcting) their own work, and a pre- and posttests administered at the beginning and end of the study.
Materials and Measures

The second test environment Puzzle Math incorporated many of the lessons learned from the first study. Puzzle Math placed emphasis on, 1) extended treatment over two days, 2) a puzzle that helps identify students
ISLS
283
ICLS2012
procedural calculation mistakes, and 3) the addition of a log monitor that detection when students are self correcting. In the first study, students made a lot of procedural mistakes (i.e., mixing up numbers, losing track and calculating the same number twice), which were not calculation mistakes, but procedural mistakes. In the first study, we found that the calculation steps could differ by the students math level. A student who may not be able to use the math efficiently, may look at 25 x 240 and write out 5 x 5 x 2 x 2 x 2 x 2 x 3 x 5 and start calculating each number one by one increasing their chances of losing track and mixing up numbers and getting the answer wrong. A student, who is more efficient, may look at 25 x 240; see that 240 can be broken down to 4 x 60 to make the calculation easier because 25 x 4=100, and 100 can be easily calculated by 60 = 6000. To address this issue, Puzzle World has a set of six to eight problems where the divisible numbers are small (1~2 digits), requiring fewer factoring (e.g., 4 = 2 x 2, 12 =3 x 4), and fewer chances of losing track. More details on the problem sequence are described in the next section. The pretest determined the students math level, and any prior knowledge to similar math puzzles. The new math game involved divisibility rules for multiplication (Figure 1 Right). Taking the form of a puzzle, the math game provided opportunities to monitor the agents reasoning when problem solving. Each puzzle was created from a small set of six to eight math problems on divisibility rules. As the game progresses, restrictions were imposed on the puzzle (e.g. Type in any number between 1-9 where the product equals the bold numbers). The restrictions got more challenging as the game progressed, where the easiest rule was Calculation Only: use any number between one to nine, medium level rule was Calculation & Rule: use any number between one to four, and the hardest rule consistency across Entire Puzzle: use any number between 1 to 4 whose products equal the blue bold number, and only have one of each number (1,2,3,4) in each row and column.
Figure 4. Pre-post-test Accuracy.
Figure 5. Accuracy over two days.
Figure 6. Self correcting activity.
Results
The accuracy data on student performance was categorized into three different scores. The math problems in study 2 involved a puzzle that consisted of a set of problems. The pretest showed that there was no difference in the groups prior to the study (Figure 4). Figure 5 showed the three accuracy scores during treatment session over the two days. Overall, students in the self-other mode had a higher accuracy score compared to the selfmode. The students in the self-other mode seemed to improve at a higher rate from treatment day 1 to day 2. The difference between the two groups on performance (during the treatment session) started to appear by the end of the second day. For accuracy on calculation between the two groups was approaching significance at (Self monitoring M=68%, Self-other monitoring M=87%) t(20)= -1.825, p=.083. For accuracy on following rules & calculation the difference between the two groups were approaching significance at (Self monitoring M=59%, Self-other monitoring M=78%) t(20)= -1.833, p=.082. By the time the students took the posttest, there was a significant difference between the two groups, where the Self-other monitoring performed better than self-monitoring. The posttest accuracy score for the calculation-only problems were significantly different between the two treatments (Self monitoring M=55%, Self-other monitoring M=96%) t(20)= -3.28, p<.05. Even with the rule restrictions, there was a significant difference between the two groups where Self-other monitoring scored higher than the self-monitoring at (Self monitoring M=43%, Self-other monitoring M=88%) t(20)= -3.60, p<.05. The monitor detection log kept of the number of incidents where the student self corrected during problem solving, and changed their answer before finishing the puzzle (Figure 6). In addition to the number of times the student self corrected, of those corrections, we looked at how many were actually correct. The results showed that self-other monitoring had significantly more cases of self correction compared to the self monitoring (Self monitoring M=9.6, Self-other monitoring M=32.6) t(20)= -2.25, p<.05. There was also a significant difference between the two groups on the accuracy of the self correction (Self monitoring M=8.6, Self-other monitoring M=18.8) t(20)= -3.03, p<.05. Although the number of self-corrections slightly decreased from day 1 to 2, the accuracy in self-correction increased from day 1 to 2 for the Self-other monitoring. The accuracy score on the posttest indicated that students in self-other monitoring performed better compared to the students from self-monitoring (See Figure 4). This was also seen during the treatment session
ISLS
284
ICLS2012
where self-other monitoring had a higher performance rate as the treatment progressed from day 1 to day 2 (See Figure 5). Possibly due to the late-gain effect the difference between the two treatments were not significant until the posttest. As an indicator of monitoring, the number of self-corrections was significantly higher in the self-other monitoring treatment (See Figure 6). Although the number of incidents slightly decreased from day 1 to day 2, the quality of self-correction seemed to increase for self-other monitoring where accuracy increased from day 1 to 2, while in self-monitoring the number of incidents and accuracy decreased.
General Discussion
One challenge that remains is how ProJo is still unsuccessful in making the students notice their own mistakes. However, seeing ProJos errors still had a modest benefit for student learning. This was seen in the post-test where students solved problems on their own, away from ProJo. If students made an error on their own problem, and then monitored ProJo make a similar error, the students were less likely to make the error on the post-test. If the students had not made an error, but saw ProJo make a mistake on a similar problem, this tended to hurt the students post-test performance. Students in the self-monitoring treatment who made an error, but did not get to monitor ProJo make a similar error, were likely to make a similar error on the post-test. These tentative results suggest a practical hypothesis that if a student gets a problem right, then the student should solve another problem on their own. If the student gets a problem wrong, have them monitor an agent. These findings have some implication where a more sophisticated version of ProJo could be designed where the system makes realtime decision to have the student monitor an incorrect ProJo, a correct ProJo, or simply continue working on their own. There may be a need to sequence the agents mistakes to further improve student learning. By identifying unfavorable problem sequencing, ProJo can have important implications on the learner.
Summary
The two studies provided some evidence that self-other monitoring may be an effective way to help students develop metacognitive skills. The present measures showed a late gain effect and some evidence that students were monitoring (self-correcting) rather than copying behavior. The studies also provided two testing environments that isolated and adjusted factors (e.g. kinds of errors, what was monitored by the student, how agent would respond, and the data retrieved), and tracked student progress and achievement in various data forms (screen shot of worked out problems, time and accuracy scores).
References
Brown , A. (1987). Metacognition, executive control, self-regulation and other more mysterious mechanisms. In Weinert, F.E, & Kluwq, R.H. (Eds.), Metacognition, Motivation and Understanding, NJ, LEA. Chi, M.T.H., Silver, S.A., Jeong, H., Yamauchi, T., & Hausmann, R.G. (2001). Learning from human tutoring. Cognitive Science, 25, 471-533. Gelman, R., & Meck, E. (1983) Preschoolers counting: Principles before skill, Cognition, 13, 343-359. Graesser, A.C., Person, N., & Magliano, J. (1995). Collaborative dialog patterns in naturalistic one-on-one tutoring. Applied Cognitive Psychologist, 9, 359-387. Karmiloff-Smith, A., & Inhelder, B. (1974-1975). If you want to get ahead, get a theory. Cognition, 3, 195-212. Karmiloff-Smith, A. (1979). Micro- and macro- developmental changes in language acquisition and other representational systems. Cognitive Science, 3, 91-118. Markman, E. (1977). Realizing that you don't understand: Elementary school children's awareness of inconsistencies. Child Development, 48, 986-992. Norman, D.A. (1983). Design Rules Based on Analyses of Human Error, Comm. of the ACM, 26(4)254-258. Okita, S. Y., & Schwartz, D. L. (2006). When observation beats doing: Learning by teaching. In S. Barab, K. Hay, & D. Hickey (Eds.), Proceedings of the 7th International Conference of the Learning Sciences, Vol. 1 (pp. 509-515). New Jersey: Lawrence Erlbaum Associates. Bloomington, USA. Palincsar, A. S., & Brown, A. L. (1984). Reciprocal teaching of comprehension fostering and comprehensionmonitoring activities. Cognition and Instruction, 2(2), 117-175. Schwartz, D. L., Chase, C., Chin, D. B., Oppezzo, M., Kwong, H. Y., Okita, S. Y., Biswas, G., Roscoe, R., Jeong, H., & Wagster, J. (2009). Interactive metacognition: Monitoring and regulating a teachable agent. In D. J., Hacker, J. Dunlosky, & A. C. Graesser (Eds.). Handbook of metacognition in education (pp. 340-358). New York, Routledge. Seifert, C.M., & Hutchins, E.L. (1992). Error as Opportunity: Learning in a cooperative task, Human-Computer Interaction, 7, 409-435.
Acknowledgments
We thank all the student participants in the study, the Sociable Technology and Learning (STL) lab members, Delci Wright, Selen Turkay, Mihwa Kim, and Yanjin Long (now at CMU).
ISLS
285
ICLS2012
Understanding influence in collaborative group work: The importance of artifacts

Lesley Dookie, University of Toronto, 252 Bloor St. W., Toronto, Canada, lesley.dookie@utoronto.ca Indigo Esmonde, University of Toronto, 252 Bloor St. W., Toronto, Canada, indigo.esmonde@utoronto.ca Abstract: When students work together on collaborative tasks, they typically have unequal influence over group decisions. This paper builds on the Differential Influence framework developed by Engle, Langer-Osuna and McKinney de Royston to consider the powerful role of artifacts in group work. Specially, we argue that if a student has greater influence over a shared artifact developed by the group, then they may have greater Perceived Authority as well as Access to the Conversational Floor and Interactional Space. They may also be able to influence the groups collaborative decisions despite challenges to the Perceived Merit of their ideas. Based on the impact of the shared learning artifact on these interdependent dimensions, we discuss how a students power and influence is perpetuated in collaborative group work.
The construction of influence in mathematical group work

When collaborative group work is used in school contexts, the work is rarely equally distributed. Some students do the bulk of the work while others are left to more peripheral forms of participation. Recognizing the importance of investigating equity issues associated with classroom peer collaboration, Engle, Langer-Osuna and McKinney de Royston (in press) have developed an analytic framework that tries to account for the differential influence that group members have on the groups ongoing task. Their work is valuable in that it considers the process by which a group makes ongoing decisions related to the task at hand, decisions that may marginalize some group members while supporting the learning of others. Engle et al. elaborate a model for understanding Differential Influence (DI), in which they consider how students Access to the Conversational Floor, Access to Interactional Space, Perceived Merit of Ideas, and Perceived Authority, are interdependently implicated in determining a students Influence with the rest of their group. (These constructs will be defined in more detail in the body of the paper.) In this paper, we apply the DI framework to understand an interesting case of group interaction and extend the framework to account for the role of artifacts in constructing influence. Cultural-historical activity theory (CHAT) uses human activity as a unit of analysis (Engestrm, 1987; Cole, 1996) and recognizes that when engaged in object-driven activity, people use artifacts to accomplish their emerging goals (Saxe, 2004). As such, artifacts play a central role in mediating activity. In the case we focus on here, three Grade 7 students were working together on an open-ended mathematics task and shared a worksheet, which we came to call the learning artifact. Throughout this interaction, one of the students was clearly the groups influential leader, while another student was just as clearly ignored and silenced. We use this case to highlight the role of a shared artifact in contributing to the construction of differential influence and marginalization.
Methods
As part of a broader study in which we were interested in understanding processes of marginalization in group work, we reviewed some classroom video data to find a compelling example in which a student made efforts to be involved in the group task, and was relatively unsuccessful. We selected a video from a corpus of classroom video data. The teacher in this particular classroom video (pseudonym Emma) cared deeply about equitable teaching and wanted all of her students to become independent mathematical thinkers. The video we chose was 33 minutes long, and involved a group of three students: Kyla, Ariel and Monica. Other participants included Emma and a researcher, Indigo. The students were given a worksheet to share that depicted two maps of different neighborhoods in their city. The maps identified the locations of all grocery stores in those neighborhoods, and students were tasked with identifying food deserts on the maps, using any mathematical concepts and tools. The school was a girls-only independent school in a Canadian city, with tuition of approximately $14,000 per year. The students were drawn from all over the city, and although there were some bursaries and scholarships, most families paid the full tuition rate. The school was founded on a feminist philosophy and the teachers were committed to aligning their teaching with their understanding of girls learning needs.
Analysis
To begin analysis, we created a basic transcript that included talk, gestures and body positioning of each of the participants. The transcript included a column for analytic notes and coding, and a column for sketches of the placement of the three students and any other participants in the camera frame. As we watched the video repeatedly to understand the dynamics of group interaction (Erickson, 2007), we became increasingly interested
ISLS
286
ICLS2012
in the central importance of the groups shared worksheet. Our observations of the video led us to believe that this learning artifact mediated the groups collaboration in important ways. Rather than implementing the DI framework as a coding scheme, we searched for examples from the video to highlight how access to and influence on the groups learning artifact had an impact on the four components of the DI framework. These examples, with commentary, are presented as the results of this paper, which expands on the DI framework.
Results
In the sections that follow, we will demonstrate the ways in which students access to and influence over the learning artifact was related to each of the four dimensions of the DI framework: (1) Perceived Merit, (2) Perceived Authority, (3) Access to the Conversational Floor and, (4) Access to Interactional Space, and ultimately, on influence. Ariel held on to the groups worksheet during most of the activity, and we use vignettes to illustrate how her access to the worksheet facilitated each of the dimensions of the DI model and, in turn, helped her to become influential in the group work activity.
Perceived Merit
In the DI framework, Perceived Merit captures the groups assessment of the value of one anothers ideas. It is not an attempt at an objective assessment of the merit of an idea; rather, this code captures the groups talk and actions that indicate whether they find an idea to be meritorious. Perceived Merit can be seen when participants explicitly evaluate the validity of an argument, when they use gestures (such as eye rolling, head nodding or shaking) to convey an evaluation, or when they verbally align themselves with or against an argument. In the DI framework, the Perceived Merit of ones idea directly impacts ones influence on the groups actions. That is, a person who presents an argument with Perceived Merit gains a corresponding increase in influence. Likewise, when a persons ideas are perceived to be of low merit, there is a corresponding decrease in their overall influence. Access to and influence on the learning artifact allowed Ariel to gain influence on the groups final product without having to convince others of the Perceived Merit of her ideas. This can be seen from the following example. The vignette begins early on in the groups work, just after Ariel had read aloud the instructions from the learning artifact and the group members had inspected the map. Vignette 1. While Kyla stood up and leaned over Ariel to be able to see, the group engaged in a lively discussion about the boundary of the food desert on the map. Ariel traced a line around a region, implicitly identifying the inside of this region as a food desert. As she drew the boundary, she explained what she was doing, and told the group if you dont agree with me for any of this tell me. Kyla and Monica both watched as she continued to draw her line. Very quickly, Monica began to voice disagreement. The specific point of disagreement was that Ariels outline crossed what appeared to be a ravine. But wait, theres no roads there so youd have to...theres probably like a ravine there so you cant access there, said Monica. There followed a discussion of the ravine, involving Kyla, Monica, the teacher, and another student. Everyone except Ariel agreed that people would not want to walk across a ravine to get their groceries, but Ariel continued to draw her boundary. Finally, Monica reminded Ariel of her earlier words: Yeah! You told us if we didnt agree with you we could say it and youd change it. Ariel still continued to draw her boundary on the map. In the vignette, despite the fact that her classmates explicitly disagreed with her multiple times, Ariel was able to mark her boundary on the map. In the language of the DI framework, Monica and Kyla positioned Ariels idea as being without merit, and yet Ariel was still able to use her influence over the learning artifact to carry it out. Even though her contribution was without merit, this example illustrates how Ariels influence on the learning artifact allowed her to increase her overall influence on the group work.
Perceived Authority
In the DI framework, Perceived Authority is the extent to which a participant is seen as a credible source of information. According to the model, Perceived Authority is negotiated through interactions between students and has a direct impact on ones influence in the group. Examples of Perceived Authority include when someones credibility is accepted without hesitation, when they are oriented to as an expert (for example, if a group member asks them questions or for help), and their suggestions are trusted and taken up by others. When a person gains perceived authority, they have a corresponding increase in influence as well as Access to the Conversational Floor and Interactional Space. As can be seen in vignette 1, before seeking her group members approval, Ariel began marking the worksheet with her ideas. The fact that her ideas were initially accepted without hesitation (since she wrote them down before having checked with her group) indicates that she was taking up a position of authority. This
ISLS
287
ICLS2012
was a common occurrence during the group work activity and will be illustrated again in vignette 2. Ariels access to and influence on the learning artifact helped to increase her Perceived Authority, allowing her to write ideas without the permission or approval from the group.
Access to the Conversational Floor

Engle and colleagues define the conversational floor as a socially negotiated space wherein people make verbal contributions to a discussion. The DI framework considers how individuals gain initial Access to the Conversational Floor, as well as whether they are able to maintain this access to present a point in its entirety. For example, an individual who is able to begin presenting an utterance by simply opening their mouth and starting to speak has greater Access to the Conversational Floor than an individual who needs to wait to be called on. Similarly, an individual who is able to share their point until they signal that they are finished has greater Access to the Conversational Floor than someone who is continuously interrupted. An individual with strong Access to the Floor can also affect who else gets access to it and greater Access to the Conversational Floor can lead to greater overall Influence in a discussion. The learning artifact was a shared and central component of the activity and we observed that when a group member had influence over the worksheet, they were granted easier Access to the Conversational Floor. Ariel maintained her influence over the worksheet throughout most of the activity and we illustrate how this influence granted her Access to the Conversational Floor in the following vignette. This vignette occurred relatively late in the groups work session, after several instances in which Kyla and Monica had asked Ariel if they could write on the worksheet. Vignette 2. The worksheet was in front of Monica and there was some back and forth talk between the girls about the map and what general assumptions they had to make when identifying food deserts. Ariel interrupted Kyla and said to Monica, You should write, we assume that they cant walk over that [meaning, cant walk more than 500 m.], so maybe they have physical problems. As she spoke, Ariel took the worksheet from Monica and began to write. As she wrote, she also spoke the words out loud, thus maintaining her position in the conversational floor. While this was happening, the previous back and forth conversation involving Monica and Kyla ceased. Both Monica and Kyla oriented their bodies towards Ariel and the worksheet. After about 40 seconds, Monica indicated that she had an idea, but refrained from actually sharing this idea. Ariel continued to write and to dictate aloud, and a moment later Kyla asked, how long is your sentence going to be, miss run-on? but also refrained from speaking further. After another 50 seconds of dictation, Ariel finished and signified this by saying, period! Throughout this time, both Monica and Kylas bodies remained oriented towards Ariel and the worksheet. Later in this exchange, Ariel handed the worksheet back to Monica and said, Your turn. Write. When Monica demonstrated some uncertainty as to what she should write, Ariel probed her with questions after which Monica suggested, Maybemaybe the other person doesnt want to go to the grocery store because theyve already got a corner store near their house?. Ariel then replied, Sure. Write that. The vignette demonstrates how Ariels influence on the learning artifact provided her with Access to the Conversational Floor as she not only initiated talk by interrupting Kyla, but was also able to maintain the Floor until she completed her idea. As Ariel spoke she was attended to by both Monica and Kyla based on their bodily orientations and limited talk. When Ariel held the worksheet, she had easy Access to the Conversational Floor because the worksheet was central to the activity and if Kyla and Monica wanted to know what was being written, they had to listen to Ariels dictation. The vignette also demonstrates how Ariel provided Monica with Access to the Conversational Floor by giving her access to the learning artifact, probing her with questions, and giving her the opportunity to offer up an idea. Ariel maintained control over both the learning artifact and the Floor but created specific entry points for Monica. Taken together, influence over the central learning artifact mediated Ariels Access to the Conversational Floor, which, in turn, allowed her to become more influential in the group work activity.
Access to Interactional Space

According to the DI model, access to Interactional Space occurs as a result of an individuals bodily orientations in physical space. These orientations are socially negotiated through interaction. Engle et al. define Access to Interactional Space as, the degree to which the student is visually attended to and physically oriented to by others when speaking or listening, and is able to affect the spatial access of others (p. 12). Access to Interactional Space can be shaped by the physical organization of an activity including how people are seated, their relative proximity to the center of activity, and whether they are blocked by other objects or people. Access to Interactional Space is said to have an indirect effect on influence in that greater Access to Interactional Space
ISLS
288
ICLS2012
can lead to greater Access to the Conversational Floor and thus greater influence. For example, Engle et al. demonstrate how a student who is more often physically oriented toward became influential in a debate. In our analysis of the group work activity, we observed that proximity to the worksheet facilitated Access to Interactional Space. Specifically, Ariel maintained influence over the worksheet for the majority of the activity and the physical organization of this activity was, for the most part, centered around her and the learning artifact. The learning artifact was a focal point of the activity and when it did switch hands, there was a corresponding, noticeable shift in the physical organization of the group as individuals re-centered themselves around it. For example, referring back to vignette 2, after Ariel handed the learning artifact over to Monica, she turned her body towards Monica and the learning artifact. Kyla stood up, walked over, and also oriented herself towards Monica and the learning artifact. Figure 1 demonstrates the groups body positions while Ariel held the worksheet, and then when Monica did.
Figure 1. Body positioning with the worksheet in front of Ariel (left image) and Monica (right image). In the DI model, increased Access to Interactional Space also facilitates Access to the Conversational Floor and so influence over the learning artifact facilitated both of these processes. We can therefore argue that by facilitating access to both the Conversational Floor and Interactional Space, influence over the learning artifact helped Ariel to become, overall, more influential in the cooperative group work activity.
Access to the Learning Artifact Versus Influence Over the Learning Artifact
An important distinction that came to light during our analysis was that access to the learning artifact was very different from influence over the learning artifact. While Monica had some access and Kyla had minimal access to the worksheet, this access alone did not facilitate Access to the Conversational Floor or Interactional space and nor did it help them to become more influential. As was depicted in vignette 2, simply having the worksheet in front of her did not grant Monica control over the Floor and Ariel still had clout over what ideas ended up on the worksheet. Furthermore, Kylas attempts to write on the groups worksheet are instructive as to the distinction between simple access to, and influence over, the worksheet. Vignette 3. Throughout the group activity, Kyla made several comments on the groups strategy for locating food deserts. She was vocal during the debate about the ravine and disapproved of Ariels placement of food desert boundaries. She was enthusiastic about an early strategy for locating food deserts by measuring the distance between grocery stores. When she was not given access to the worksheet to carry out this strategy, she took out her own sheet of paper and started to measure and write down these distances. The group never oriented towards her, looked at what she was writing, or talked with her about it. At several other points, she asked to be able to write a contribution on the worksheet but was dismissed. Finally, Ariel allowed Monica to write down two answers on the worksheet. The group continued to discuss the problem of food deserts, while the worksheet lay on the desk in front of where Kyla stood. Ariel read aloud the groups last question (about a method for locating food deserts), suggested use google maps, and then turned to Kyla to ask her to write that. Kyla leaned forward and wrote this sentence on the worksheet: use google maps to identify. When the teacher came over, Kyla told her that Theyre making me write the shortest answers and I want to write. This vignette illustrates how Kyla continued to try to access and influence the learning artifact, without success. She even created her own learning artifact, but the group did not attend to her work and they positioned her written notes as peripheral to the task. At the end of the activity, when the worksheet was sitting on the desk in front of Kyla, she did not write until instructed to do so by Ariel, even though she had complained earlier that
ISLS
289
ICLS2012
she wanted to write more. Thus, Kyla had access to the worksheet through physical proximity, but she seemed to require approval from her peers before writing.
Implications
This paper has expanded on the DI framework presented by Engle et al (in press). We found that the artifact that was the central focus of the groups work mediated their participation in important ways. The student who had the most access to and influence on the groups worksheet was also able to gain easier Access to Interactional Space and the Conversational Floor, was able to assert her Authority without waiting for approval from peers, and was able to influence the groups final product despite repeated challenges to the Perceived Merit of her ideas. It is important to note that our analysis was not causal. Ariels Access to the learning artifact may have positively influenced the other dimensions of the DI framework, or vice versa. Most probably, there was a reciprocal relationship so that increases to Ariels overall influence in the group increased her access to the learning artifact, and increased access to the learning artifact also increased her overall influence. How this underlying influence over the learning artifact came to be is another important question that is beyond the scope of this paper, but requires, to begin with, further examination of previous classroom activities and student interactions. Our aim with this paper was to build on previous research on influence in group work, in light of the cultural historical frameworks that guide our understanding of learning. In keeping with CHAT, we found that the artifact that was central to the group activity played an important role in the circulation of influence within the group. CHAT also underlines the importance of using the activity system in this case, the teaching and learning practices of that particular classroom as our unit of analysis (Engestrm, 1987). We would expect different activity systems to support the construction of influence in different ways. Specifically, other kinds of learning artifacts would certainly mediate the groups interaction in different ways. If all three students had separate worksheets, then the arrangement of body positioning and eye gaze would have shifted, thus changing the nature of the interactional space. Many other arrangements of learning artifacts within activity systems could be imagined, each with unique impacts on the four components of the DI framework. These findings call on educators to consider all aspects of the context and physical arrangement of group work, as they work towards more equitable teaching practices. The physical arrangement of desks, students bodies in relation to one another, their physical connection or visual access to the worksheet, were all critical in determining how the group worked together, and who ultimately influenced their final product. While an analysis of student talk would certainly reveal some aspects of Ariels influence and Kylas marginalization, our analysis of the groups embodied interaction highlights the very physical ways in which influence and marginalization were constructed. This is a direct challenge to more cognitively based analyses in which students physical embodiment is ignored in favor of a focus only on talk.
References
Cole, M. (1996). Cultural psychology: A once and future discipline. Cambridge, MA: Belknap Press of Harvard University Press. Engestrm, Y. (1987). Learning by expanding: An activity-theoretical approach to developmental research. Helsinki: Orienta-Konsultit. Engle, R. A., Langer-Osuna, J. M., & McKinney de Royston, M. (under review). Toward a model of differential influence in persuasive discussions: Negotiating quality, authority, and access within student-led arguments. Cognition and Instruction, Erickson, F. (2007). Ways of seeing video: Toward a phenomenology of viewing minimally edited footage. In Ricki Goldman, Roy Pea, Brigid Barron & Sharon J. Derry (Eds.), Video research in the learning sciences (pp. 145-155). Mahwah, N.J.: Lawrence Erlbaum Associates. Saxe, G. B. (2004). Practices of quantification from a sociocultural perspective. In A. Demetriou, & A. Raftopoulos (Eds.), Cognitive developmental change: Theories, models and measurement (241-263). Cambridge, UK: Cambridge University Press.
Acknowledgments
The research reported here was supported by a research fellowship from the Knowles Science Teaching Foundation (KSTF). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of KSTF. We would like to thank the school, teacher, and students, for generously participating in the study.
ISLS
290
ICLS2012
Theoretical Issues: Indicators of Decentralized and Centralized Causality as a Gauge for Students Understanding of Complex Systems
Lauren A. Barth-Cohen, University of California, Berkeley, Berkeley CA, 94720, lbarth@berkeley.edu Abstract: One approach to conceptualizing students reasoning about complex systems focuses on what I refer to as a series of indicators which function as a gauge for students understanding. Some indicators are associated with centralized causality, a common student difficulty, and some indicators are associated with decentralized causality, a more normative understanding. In this paper, I summarize this approach, and then I discuss a series of embedded theoretical issues such as problematic dichtomous pairing of indicators, ambiguity about kinds of cognitive entities of indicators, context dependency of indicators, and the possibility of certain indicators being bootstrapped for change. I conclude with a discussion of how these results influence the coding of data of students reasoning about complex systems and the modeling of students knowledge about complex systems.
Introduction
Research on student reasoning and learning about complex systems is a growing area within the Learning Sciences. Complex systems, which are also sometimes called decentralized systems or self-organizational systems, contain a characteristic behavior in which simple patterns or simple rules emerge from a larger phenomena, such as a traffic jam arising from localized interactions between many individual cars that speed up or slow down depending on the behaviors of surrounding cars. There are several analysis methods for capturing students reasoning and learning about complex systems, but I address one common method that focuses on centralized and decentralized causality. This analysis method began with Mitch Resnicks work on probing peoples thinking about decentralized systems and self-organizational systems (Resnick, 1996). Using a programmable modeling environment Resnick had students develop ways to model phenomena that exhibit these behaviors, such as the formation of traffic jams, termites constructing mound nests, and ants collecting their dead into piles. One finding of Resnick (1996) is that many students had an inclination towards the centralized mindset. This is a way of thinking in which a centralized cause, such as a leader or a seed, is assumed to be the reason for the particular phenomena. For example, in traffic jams students assumed that a radar trap or broken bridge would be the cause of the traffic jam. Resnick (1996) argues that this inclination towards the centralized mindset is not only a misconception, but it is also a bias that can seen through the history of science and it is not surprising that people have this predisposition as many everyday phenomena are due to centralized causes. Resnick (1996) also found that after students had worked with the modeling environments they began to develop decentralized ways of thinking. Through working with students, he developed a list of guiding heuristics that are a loose collection of ideas associated with decentralized reasoning. He viewed them pedagogically as discussion points and possible rough gauges or yardsticks for conceptual change. For example, one heuristic is that a traffic jam is not only a collection of cars, which means that objects at one level, such as individual cars moving forwards, can behave very differently from objects at another level, such as the entire traffic jam moving backwards. Building from the literature, including Resnick (1996), Jacobsen (2001) developed a coding scheme that was used in an exploratory study of expert and novice problem solving within complex systems. Recently his coding scheme has been employed by others, including: Charles and dApollonia (2004), Penner (2000), and Levy and Wilensky (2008). Jacobsen (2001) refers to this coding scheme as a complex systems mental model framework, as it contains two mental models, one associated with centralized causalitya common misconceptionsand the other associated with decentralized causalitya normative understanding. Although Jacobsen (2001) viewed this as loosely associated with the mental model framework discussed in Vosniadou and Brewer (1994) other researchers have used it as more of a coding scheme than a mental model framework. For example, Levy and Wilensky (2008) investigated students reasoning about everyday complex phenomena. Using this coding scheme they documented a strategy in which students formed small intermediate groups of individuals, and they found that this strategy may be related to students understanding of complex systems. I interpret this coding scheme as based on heuristics, or indicators, of centralized and decentralized causality that are associated with either specific difficulties or with a normative understanding of complex systems. In this way these indicators can be thought of as gauges of students thinking. In Table 1, I present a selected set of these indicators. The white rows contain indicators of decentralized causality and the grey rows contain indicators of centralized causality. This distinction between centralized and decentralized causality follows from the literature (e.g. Jacobsen, 2001; Resnick, 1996). For each set of indicators, I include a
ISLS
291
ICLS2012
description and idealized example from traffic jams. There is some variety in terms of which indicators or codes are mentioned by different researchers. For example, levels is mentioned in Wilensky & Resnick (1999), Resnick (1994), Charles & dApollonia (2004), and Penner (2000), and local perturbations is mentioned in Jacobsen (2001), Levy & Wilesnky (2008), and Penner (2000). Also, the list of indicators has shifted over time, for example role of the environment was included in Resnick (1996) and was not part of Jacobsens (2001) mental model framework. A practical reason for the indicators approach to coding data is because there is no good definition of complex systems. As a case in point, a recent NSF panel on complex systems published a report (Guckenheimer et. al. 2008) that contains no definition of complex systems. Instead this report characterizes complex systems by several properties: internal structure, adaptation and evolution, uncertainty, and behaviors such as emergence that are not characteristic of simple systems. Similarly, Chi (2005) uses a series of attributes to describe emergent processes as a precise definition is impossible. This general method of avoiding a definition and instead focusing on a characterization is also common within the complex systems scientific literature (e.g. Goldstein, 1999). Table 1: Selection of indicators of centralized causality (grey rows) and decentralized causality (white rows). Indicator Description Idealized Examples from Traffic Jam Environment as acted upon Role of the environment Environment as interacted with A leader coordinating the behavior of the members of the group No leader coordinating the actions of the group. Focus on the macroscopic level Similar behavior at different levels Different behaviors at different levels Predictability of an individual No predictability of individuals Removal of components will not influence the system Removal of components will influence the system Small actions leading to small effects Small actions can lead to big effects Two roads merge so a traffic jam forms. A big pot-hole in the left lane of a multi-lane highway causes all the cars from the left lane merge to the right resulting in a slow down. All the cars come to a stop because there was an accident and the police slow everyone down. The average speed of a car may decrease without external mechanism coordination the actions. The entire jam is moving backwards at once. Jams and cars moving forward. Jams moving backwards, cars moving forwards. Following one car you can predict what it will do based on the other cars around it. Following one car, you have no idea what will occur. A few less cars on the road will not change anything because there is still such a large number of cars. Removing a couple cars from the road might be the critical thing between a jam forming or not forming. Adding or removing a few cars cannot change the jam. Adding or removing a few of cars might be enough to create or destroy a jam.
Leader
Levels
Predictability Removal of components influencing the system Local perturbations
Methods
This paper is intended to be a theoretical discussion, however, I draw on data to illustrate the theoretical points. The data comes from a corpus of exploratory clinical interviews conducted with 8-12th graders, undergraduates, masters, and Ph.D. students around a series of problem contexts that exhibit behaviors associated with complex systems including the formation of traffic jams and the movement of sand dunes. The interviews were designed to capture students reasoning processes, as they were asked to reason about approachable situation which are not apart of traditional curriculums but for which I expected them to be able to access prior resources and experiences. To carry out the analysis for this paper, I reviewed the literature and asked a series of open-ended questions, such as: What are the indicators? How do they function? What variables influence them? How do they relate to learning and instruction? The style of these questions is meant to be general although detailed answers to these questions pertain to specific perspectives. The goal in asking these questions is gain a more rigorous understanding of the indicators. From the analysis it became clear that these indicators are not only a coding scheme, and my goal during the analysis has been to clarify them.
Theoretical Issues
In this section I discuss a series of theoretical issues and make a case for new directions that views the indicators as an empirical focus. Overall I argue that some indicators are better or worse gauges of student understanding. More specifically, I argue four things, that currently there is a problematic dichtomous pairing of indicators, that
ISLS
292
ICLS2012
there is ambiguity about the different kinds of cognitive entities embedded within the indicators, that the indicators vary across problem contexts, and that some indicators are central for helping students bootstrapping for change.
Dichotomous Pairing of Indicators

As should be apparent from the Table 1, many indicators occur in dichotomous pairs. Often, but not always, there is one indicator associated with centralized causality and one that is associated with decentralized causality. This approach is used in a variety of papers including Jacobsen (2001) and Levy & Wilensky (2008). For example, lack of a leader is associated with decentralized causality, leader as coordinator is associated with centralized causality. This can be advantageous, because it clearly distinguishes centralized and decentralized causality. I question this dichotomous pairing and present an alternative in which the indicators occur along a spectrum. While not explicitly stated, this pairing may imply mutual exclusivity in which the paired indicators never occur simultaneously. I question this implicit assumption. A more realistic approach would be to view this pairing as occurring along a spectrum with each indicator being some degrees left or right of center. Moving beyond this pairing would have the advantage of opening up more space for addition and subtraction of indicators at different locations along this spectrum; thereby taking into account more possibilities for families of related indicators that capture more variability in students understanding. To illustrate issues around the dichotomous pairing, I focus on a family of indicators based on periodic and limiting behaviors. There are several kinds of periodic and limiting behaviors: they can occur at either a micro or macro level, they can be static or dynamic, and they can be periodic or not. In table 2, I outlined some of these related indicators: convergence to a limit, steady state behavior characterized by a lack of change, periodic motion, and changing composition. Also, the opposite behavior such as far from equilibrium can be key for emergence (Goldstein, 1999). From this list changing composition is associated with decentralized causality, but others are not specified as to being associated with centralized or decentralized causality. Table 2: Selection of indicators associated with periodic and limiting behaviors. Indicators Idealized Examples from Traffic Jam Convergence to a Limit Traffic jams can get bigger or smaller, but over time they stabilize. Steady State The highway is closed due to an accident, nobody is allowed on or off. Periodic Motion Over time the jam getting bigger, small, and the bigger again. Cars join and leave at equal rates while the jam size is constant. Changing Composition In my data there are many cases of students explanations being based on periodic and limiting behaviors and they can be classified according to a spectrum. Often students explanations incorporate changing composition, which is a key part of decentralized causality. For example, Damien a Ph.D student said the following about sand dunes changing size, the dune getting bigger or smaller is just likedoes it gain more sand than it loosesthen it gets biggerIts a question of, do you gain ten per a minute and then loose five per a minute, or the other way around and thats where it grows or shrinks. This explanation might be decentralized, because it connects the micro and macro levels thereby accounting for more dynamics of the system. Another example of a student mentioning a periodic or limiting behavior comes from Keaton a Ph.D. student whose traffic jam explanation accounted for the speeds of cars converging to a limit when there are a certain amount of cars on the road. If you put a lot of cars on the road, eventually if they are too close together they can't safely keep going at that speed, so if there is a car like really close in front of you, you tend to slow down. This explanation is not clearly associated with centralized or decentralized causality, as it accounts for both a diffuse behavior of cars slowing down due to the number of cars on the road and a direct causal behavior of one car slowing down because there is a car close to it. Perhaps it is appropriate to view this explanation as falling somewhere along a spectrum between centralized and decentralized causality or incorporating aspects of both. Finally, a third case of a student mentioning a periodic or limiting behavior comes from Antonio, an undergraduate who upon being asked about dunes appearing to have the same size at two points in time explained that nothing happened, its not a very windy area. cause [the sand] always goes with the wind and nothing touched it, so nothing changed. This explanation accounted for the steady state characterized by a lack of changing behavior and is associated with centralized causality. These examples show that some explanations based on periodic and limiting behaviors do not fall into the centralized/decentralized dichotomy.
Cognitive Entities
As was mentioned previously there have been different approaches as to how these indicators are used: a coding scheme, a loose collection of ideas used to make sense of complex systems, or evidence of a particular mental model. Given these different literature perspectives (e.g. Jacobsen, 2001; Resnick, 1996), it is natural to investigate if all indicators are the same kinds of cognitive entities.
ISLS 293
ICLS2012
In my data I see the indicators acting as different kinds of cognitive entities, some are concept-like with many sub components and some are similar to declarative facts. Below I present two different examples. In the first case Stephen, a Ph.D. student, is explaining the movement of traffic jams. He applies the indicator, lag effects to cars speeding up and slowing down, and uses it as justification for why cars cannot perfectly synchronize. Just because there is a free route in front of you, it takes time for that propagateEven if they could slot together perfectly, they won'tSomeone turns and slows you down on your way there, and you end up slowing down someone else, so the slow down always propagates. But the speed up isn't necessarily, I'm mean, youre not always in a position to take care of that, so I suppose that more people you put together, the slower its going to get as a result. If we could all, somehow synchronize and work perfectly, we could all drive down a residential street at one-hundred miles an hour because no one would ever be in the way, but we can't. In this case lag effects contain a complicated collection of several sub-parts, which is evidence of a robust understanding. Stephen mentions details such as the time for speeding up, the cars slowing down because they cannot slot together perfectly, and the number of cars influencing the speed. While this transcript excerpt illustrate that at a minimum Stephen has an awareness of lag effects such that he uses it in a clear and robust manner. Stephen has a strong academic background in science; for him lag effects maybe functioning as some type of conceptual cognitive entity rather than a collection of intuitions or nave knowledge. Comparably, sometimes certain indicators appear to be like declarative facts, such as in this second case when a student announces that a traffic jam is caused by a centralized leader as coordinator, such as an accident: sometimes there are traffic jams because, because there is an accident, or, like your rubbernecking an accident in the road. (Drew, Ph.D student). In this case he simply declares that traffic jams are caused by accidents. This is not to suggest that all of the indicators are the same type of cognitive entity; other factors influence the indicators. Paying attention to the variety of factors influencing what kinds of cognitive entities are entailed supports asking questions about instruction and learning. For example, how do the indicators appear different when used by students with different levels of background knowledge? A better understanding of the cognitive entities entailed by these indicators will also support investigating instruction and development.
Context Dependence
Do some indicators occur more often in certain problem contexts or scientific domains than others? This issue is important because evidence that not all indicators occur equally in all problem contexts or domains could have implications when comparing students understandings across problem contexts or domains. For example, one indicator used in Jacobsen (2001) is teleological beliefs, which are associated with the clockwork mental model and non-teleological beliefs, which are associated with the complex systems mental model. However, human based problem contexts, such as traffic jams, may cue teleological beliefs at a different rate than, say, biology contexts that are viewed as farther from humans (Inagaki & Hatano, 1987). This could have implications for which indicators are chosen to capture students understanding in particular problem context. Beyond context dependent differences across scientific domains, there could also be context dependent differences across instructional modes. Do indicators occur with similar or different likelihoods across different instructional modes, such as role-playing games, computational modeling environments, and interview settings? One pair of indicators, removal of components influencing the system, although mentioned in the literature have not been seen in my data set in which students were asked about complex systems in clinical interview settings. Potentially this specific indicator may be easier to see in computational modeling environments where one can easily manipulate the number of agents and see how the behavior changes. Comparably, in studies where there is no computational component, explanations that are based on this indicator would involve a complicated thought experimenthow might this system change if there were a few less agents? While this issue of how contexts influence the various indicators is not solved here it is an important dimension of variability.
Bootstrapping for Change

Beyond the centralized mindset being generally associated with students difficulties, there are particular indicators that are associated with specific kinds of difficulties. Wilensky and Resnick (1999) argue that some difficulties with emergent systems are due to confusion or slippage across the microscopic and macroscopic levels. Similarly, Sengupta and Wilensky (2009) argue that common misconceptions about electricity can be reinterpreted as a slippage between levels. What other indicators are central to students difficulties? What about the inverse questionare certain indicators central to helping students bootstrap themselves towards a normative conception? In my data occasionally indicators are mentioned in ways that influence students changing explanations towards decentralized causality. More specifically, I have a case of a student rejecting explanations that are
ISLS
294
ICLS2012
based on certain indicators associated with centralized causality resulting in the student relinquishing those particular explanations and then generating new explanations that are more closely associated with decentralized causality. In the case of Laurel (masters student), she rejects several sand dune explanations that are associated with direct causality in which the wind moves and entire dune at once or the wind moves sand particles in a sequential fashion. This occurs because those explanations do not take into account her prior knowledge about wind having variable direction and speed; her recognition of these holes in her prior explanations contributes to subsequent explanations that are more decentralized. In future work it would be productive to identify some indicators such that if they are incorporated in students reasoning, it would be a strong gauge of bootstrapping towards the target conception.
Contributions and Next Directions

This paper highlights theoretical challenges revolving around a common approach used to code data in which students are reasoning about complex systems. If ones goal is to measure students progress within centralized and decentralized causality, then my results suggest that existing coding schemes should be modified to account for families of related indicators beyond dichotomy and to account for contextual differences in the relevance of the indicators. If ones goal is to model students knowledge within centralized and decentralized causality, then my results suggest that this current approach is problematic because these indicators are not all the same types of cognitive entitiesthey vary across contextsand when certain indicators are incorporated into students reasoning, they may support bootstrapping for change. Future work modeling students knowledge should take these considerations into account by viewing these indicators as not only a coding scheme but also as a continuum of understandings and an empirical focus.
References
Charles, E. S., & d'Apollonia, S. T. (2004). Understanding what's hard in learning about complex systems. Proceedings of the 6th International Conference on Learning Sciences, 590-590. Chi, M. T. H. (2005). Commonsense conceptions of emergent processes: Why some misconceptions are robust. The Journal of the Learning Sciences, 14(2), 161-199. Goldstein, J. (1999). Emergence as a Construct: History and Issues. Emergence, 11, 49-72. Guckenheimer, J. et. al. (2008). Foundations for complex systems research in the physical sciences and engineering. Report from an NSF Workshop. Retrieved from http://www.northwestern.edu/nico/in_the_news/nsf_complex_systems.pdf Inagaki, K., & Hatano, G. (1987). Young children's spontaneous personification as analogy. Child Development, 58(4), 1013-1020. Jacobson, M. J. (2001). Problem solving, cognition, and complex systems: Differences between experts and novices. Complexity, 6(3), 41-49. Levy, S. T., & Wilensky, U. (2008). Inventing a "mid level" to make ends meet: Reasoning between the levels of complexity. Cognition and Instruction, 26(1), 1-47. Penner, D. E. (2000). Explaining systems: Investigating middle school students' understanding of emergent phenomena. Journal of Research in Science Teaching, 37(8), 784-806. Resnick, M. (1996). Beyond the centralized mindset. Journal of the Learning Sciences, 5(1), 1-22. Sengupta, P., & Wilensky, U. (2009). Learning electricity with NIELS: Thinking with electrons and thinking in levels. International Journal of Computers for Mathematical Learning, 14(1), 21-50. Vosniadou, S., & Brewer, W. F. (1994). Mental models of the day/night cycle. Cognitive Science, 18(1), 123183. Wilensky, U., & Resnick, M. (1999). Thinking in levels: A dynamic systems approach to making sense of the world. Journal of Science Education and Technology, 8(1), 3-19.
Acknowledgements
Thanks to Andrea A. diSessa, Randi A. Engle, Tania Lombrozo, and Barbara Y. White for insightful discussions.
ISLS
295
ICLS2012
Engaging in Design: Reflections of Young Paper Engineers

Rob Rouse, Richard Lehrer, Vanderbilt University, Department of Teaching and Learning 230 Appleton Place, Nashville, TN 37203-5721 Email: rob.rouse@vanderbilt.edu Abstract: Despite a recent push to expand the scope of P-12 engineering education, research on how young children understand and engage in the core practice of engineering design remains scarce. In order to investigate how young children engaged in engineering design, we presented 31 children (ages 8-9) from two third-grade classrooms with the task of constructing a pop-up book, (also called paper engineering). Here we report the results of a qualitative analysis of 29 childrens semi-structured interviews in which they answered questions about their pop-up books post-construction. We found that children used troubleshooting and satisficing (among other techniques) to solve problems they experienced while constructing pop-up books. This research has implications for elementary grade engineering design instruction. In particular, it highlights the need to promote activity structures that foster a normative practice of troubleshooting.
Introduction
Recently, there has been an effort to expand P-12 engineering education in the U.S. (Brophy, Klein, Portsmore & Rogers, 2008). Proponents of the expansion claim that in addition to improving childrens learning and achievement in science, technology, and math, increased emphasis on engineering education has the potential to improve childrens (a) awareness of engineering and the work of engineers, (b) understanding of and ability to engage in engineering design, (c) interest in pursuing engineering as a career, and (d) technological literacy (Katehi, Pearson, & Feder, 2009, pp. 49-50). Because empirical evidence supporting these claims is sparse, there have been calls for increased research into P-12 engineering education, and in particular, engineering design. Engineering design is the purposeful, systematic, iterative, and social process that engineers use to solve problems (Katehi et al., 2009). The fundamental aspect of engineering setting it apart from other subjects is its focus on design (Dym, Agogino, Eris, Frey, & Leifer, 2005). In this paper, we report on a study that examined how children solved problems while participating in an engineering design activity. In doing so, we aim to add to the existing literature about how children understand and engage in engineering design.
Paper Engineering
We chose to examine the way elementary age children engaged in engineering design by piloting an instructional unit where they created a pop-up bookan activity known as paper engineering. Paper engineering aims to transition young learners from craft-based activities, where design goals tend to focus on aesthetics and trial-and-error construction, to engineering design, where design goals are subject to more analytic decisions (Hendrix & Eisenberg, 2005). Paper engineering approximates engineering activity in several ways. First, paper engineering presents an ill-structured environment, so that children must make decisions in light of uncertainty about how to construct working pop-up books. Each of these decisions can generate situations where children are required to decompose systems, generate and test solutions, analyze and evaluate results, and optimize their books. Second, design of pop-up books often demand trade-offs among elements of design, such as the amount of pop-up motion, the height of the pop-up off the page, the location of the pop-up inside the book, and material constraints, such as the rigidity of the paper. Third, children produce sketches and prototypes (much like professional engineers do) to describe structural and functional relationships within their pop-up books. Finally, children work in groups to complete cycles of design and redesign where their results are subject to public scrutiny and where sharing information spurs revision and innovation. There are many different types of pop-ups that children can and do make. Here, we give background on only the type (parallel-fold) that will allow readers to follow the distinctions made in the findings section of this paper. To create a parallel-fold pop-up, the paper engineer tapes an unfolded strip of cardstock (cardstock is more durable than regular paper) into an open blank book (Figure 1, left). He or she tapes the strip so that both attachment points (called page positions) are parallel to the center of the book (called the gutter) and aligned with each other. After attaching the strip, the paper engineer closes the book. Because the strip is attached to both sides of the book, it pushes outwards and folds as the book is closed. The right side of Figure 1 shows how the strip looks when the book is reopened. Parallel-folds can be symmetric or asymmetric. Symmetric parallelfolds are attached at page positions equidistant from the gutter and fold in the middle of the strip. The parallelfold in figure one is an example of an asymmetric parallel-fold because its page positions are not equidistant from the gutter. This causes the strip to fold to the right of the gutter.
ISLS
296
ICLS2012
Figure 1. Creating a parallel-fold pop-up (adapted from Benenson & Neujahr, 2009)
Research Questions
We set out to answer two questions about how children understand and engage in engineering design while constructing pop-up books: (a) How do children report solving problems? and (b) What do childrens problemsolving approaches indicate about how they engage in the process of engineering design?
Method
Participants in our study were 31 children (12 boys and 19 girls) from two third-grade classes in a suburban elementary school in the Midwestern United States. The children were all either eight or nine years old. Data collection consisted of video recording lessons where children worked on paper engineering activities. Seven of the nine lessons were recorded, yielding roughly six-and-a-half hours of classroom video. Additionally, approximately one month after instruction, 29 of the 31 children were interviewed individually. The interviews were semi-structured and designed to elicit information about how children constructed their pop-up books. For example, children were asked questions like, Tell me about your pop-up, How does your pop-up work? or How did you make that pop-up? On average, each interview lasted about 7 minutes. We worked with the classroom teachers to plan the paper engineering lessons. The lessons were based on a 12-part paper-engineering curriculum jointly developed by engineers and educators at City College that included investigation of parallel-folds and their composition (Benenson & Neujahr, 2009). The curriculum emphasized engineering design and had been used previously in several New York City classrooms but had not been used by the teachers in this school. Almost all instruction happened while both classes were co-present in the same classroom. In these instances, one of the two teachers tended to lead instruction. Several times, the classes mixed then split in half. In these cases, each teacher led instruction over the same content independently and in different classrooms. The teachers modified the curriculum greatly when they enacted it. In order to connect it to their required curriculum, they combined it with two other units, researching animals of the world, and writing poems. This resulted in hybrid instruction where children first researched animals of the world (e.g., lions, tigers, anacondas, snowy owls), then learned about, and wrote different types of poems (e.g., acrostic, diamante, haiku), and finally performed paper-engineering activities. The enacted instruction led to children creating their own animals of the world pop-up poetry books. On average, each childs book had five pages. Upon completion, the children added their books to the school library (with great pride) so that they, and future generations of students, could check them out and read them. Our analysis focuses on the individual interviews. These interviews were rich with children talking about the problems they faced during construction. In order to find instances where children reported being confronted by a problem while constructing their book, we followed Jordan and Henderson (1995) and Stevens and Halls (1998) technique of looking for disruptions or trouble. We began by transcribing each childs interview. We then sought to identify instances of trouble within each interview. We defined trouble as any place where children reported (a) changing their thinking, (b) copying something, (c) being unsure of what to do, (d) asking for help, (e) being constrained by time, or (e) explicitly stating that something was difficult or troublesome. After isolating these instances (in total there were 43), we moved to the second step of our analysis. Because we were only interested in how children solved problems specific to their pop-ups, our next step was to take a second pass through the instances of trouble to excluded any that did not deal with pop-up construction. Out of 43 total instances, we excluded 27. Many of the excluded instances had to do with aesthetic issues (e.g., how to draw a perfect tiger or how to accurately trace a komodo dragon) that, although they were interesting, fall outside the scope of this analysis. For the remaining 16 instances, we looked for trends in the responses children reported making when they experienced trouble. To uncover trends, we performed an inductive analysis (Thomas, 2006) where emerging themes were developed by both studying transcripts of childrens responses in detail and by comparing across transcripts to identify commonalities in childrens responses.
ISLS
297
ICLS2012
Findings
We classified childrens responses to the trouble they reported experiencing while constructing their pop-up books into three categories. We named these categories troubleshooting, scaling back, and emerging ideas. Children reported troubleshooting to solve problems they experienced with their pop-ups in 5 of 16 instances. When children troubleshot, they tended to focus on retaining their original design goal. Childrens troubleshooting consisted of several different problem-solving approaches. For example, some children reported making changes to previous successful pop-ups. That is, when confronted by a problem, children modeled their solution on a pop-up they had built previously, but made slight changes to it. Other children reported taking an iterative approach to problem solving (e.g., some children systematically moved parts of their pop-up to different places in the book after recognizing a problem). Finally, one child reported soliciting help from the teacher to accomplish her design goal. Children also reported solving problems they experienced while constructing their pop-ups by scaling back. Children reported scaling back in 7 of 16 instances. In contrast to troubleshooting (where children worked to accomplish their original design goal), when children scaled back, they tended to alter their original design goal from something more complicated to something less complicatedthat is, they satisficed (Simon, 1996). Several children reported that their original plans were difficult or hard and that the new approach (the scaled back approach) was easier. For example, one child who had an idea for multiple pop-up components on the same page (both head and claws popping up) altered her pop-up so that only one component (the head) would pop-up. Another child reported scaling back from an animals entire body popping up to just the animals head popping up. In addition to troubleshooting and scaling back, children reported adopting several other approaches to solving problems they experienced while constructing their pop-ups. Although these approaches may represent responses that could eventually be categories themselves, the small number of instances we isolated in our analysis prevented us from considering them categories at present (thus we combined them to make up the emerging ideas category). The four responses in this category included (a) one child explaining a trial-and-error approach, where she attached a pop-up inside her book without knowing what would happen to itand upon closing and reopening the bookwas pleased with the result; (b) one child taking steps to obviate anticipated problems early in the construction of her pop-up; and (c) two children who reported changing their pop-up to address aesthetic issues at the same time as functional issues.
The Case of Suzies Kookaburra

This troubleshooting example highlights the steps one child, Suzie, took to make her pop-up work in a specific way. In the example, Suzie responded to the interviewers question about how she made her kookaburra pop-up. Because the 3-dimensionallity of the pop-up is obscured in the picture below, the asymmetric parallel-fold to which Suzie attached her kookaburras head has been highlighted with a dashed line. This fold resembles the fold in Figure 1. S: Well, it took me a while to figure this one out cause at first, I had itlike this [A, the page position], was way over here [further left] and umthe bend [B] was in the middle [over the gutter] and it [the pop-up] wouldnt work because I wanted it [B] on the side like that [the current position]. But it was popping out in the middle [over the gutter] and the tail kept coming likecoming out of the book [when the book was closed]. So Mrs. D helped me figure out that if you do an asymmetric fold here [B] then it getslike if you do it this way [the current position], it gets it to pop out on this side [right side of the book] and if you do it on the opposite and make it bend here [C], then it pops out on this side [left side of the book].
C A
Suzies account provided a detailed report about the decisions she made to ensure her pop-up worked the way she intended it to work. Suzie originally positioned the parallel-fold symmetrically. Positioning the fold (or the bendas she says) symmetrically caused two problems. First, it caused her kookaburra to pop-up in the middle of the book (possibly occluding part of the poem that Suzie had already pasted to the left side of the book), and second, it caused the kookaburras tail to protrude from the book when she closed it. With her teachers help, Suzie switched her fold from a symmetric parallel-fold to an asymmetric parallel-fold. Because Suzie wanted her pop-up to function in a specific way, it required her to make specific changes to her fold. These changes, made to retain her original design goal for how the pop-up would function, seemed to enhance
ISLS
298
ICLS2012
Suzies understandings about how her pop-up functioned. By switching from a symmetric parallel-fold to an asymmetric parallel-fold positioned on the right side of the book, Suzie solved both of the problems that she encountered.
The Case of Janes Giraffe

In this scaling back example, the interviewer asked Jane about how she made her giraffe pop-up. Here, Jane reported that she managed to create a giraffe but scaled back other components of her pop-up. Janes response is below. J: So II was trying to get the tree and the bush to pop-up but itit wasnt as easy to get those to pop-up as it was for this [the giraffe]. So I have some bushes that I cut out and then taped on. And then I have a tree that I taped onand then I colored in the sky. After reporting that she experienced difficulty trying to get the tree and the bush to pop-up, Jane told how she decided against having the tree and bush pop-up because it wasnt as easy to get those to pop-up. Instead, Jane taped the tree and bushes to the page rather than making them pop-up. In this instance, Jane solved the problem by altering her original design goal (to have more than one component pop-up) by scaling back some of the functional components of the pop-up and instead adding some aesthetic components to the background.
The Case of Marys Koala

This emerging ideas example highlights how one child, Mary, reported anticipating and planning to eliminate measuring parts of her pop-up. She was the only child who reported obviating the need to measure parts of her pop-up. Marys response to the interviewer about how she made her koala pop-up is below. M: When I first tried to do it, I just tried to stick it [the koala] in right here [in the gutter] and see if it would pop-up. But then umI had to put it against this [the parallel-fold]and um All my pop-ups usually are asymmetric because its hard to get a symmetricum fold. I: You found it was easier to do the asymmetric-fold? M: Cause it doesnt matterlike it [the right page position] could be way over here [further right] but when its symmetric, you have to get it alllike you have to measure it and things like that. In her answer, Mary reported that symmetric pop-ups are hard compared to asymmetric pop-ups because they involve measuring; therefore, she usually makes asymmetric pop-ups. When she explains, when its symmetric you have to measure it, we take it to mean she understands that in order to make a symmetric pop-up, the distance between the page position and the gutter must be the same on either side of the fold. In contrast to symmetric pop-ups, when children constructed an asymmetric pop-up, they did not necessarily need to measure because they could let the book do the fold (as shown in Figure 1). By consciously favoring asymmetric folds and thereby obviating measure, Mary eliminated functional considerations that come along with planning for symmetry or asymmetry in her pop-ups. Essentially, by taking advantage of the book doing the fold for her, Mary saved herself considering issues that stem from measuring.
Discussion
In this pilot study, we observed children using diverse problem-solving approaches when constructing pop-ups. Furthermore, we noticed childrens design goals were related to their problem-solving approaches. That is, children sometimes solved problems to reach their goal, and sometimes changed their goal to solve the problem. Although we do not consider one childs problem-solving approach to be more or less valid than another, we do recognize that some problem-solving approaches are more emblematic of engineering disciplinary practices and engineering habits of mind (i.e., concepts or skills that engineering curricula strive to emphasize) than others. For example, in the troubleshooting example, Suzie demonstrated an approach to problem solving that aligned closely with established engineering disciplinary practices. Her approach was purposeful (she had a design goal), systematic (she did not leave her pop-ups motion to chance), and social (she drew on the teachers knowledge of pop-ups by asking for help). Additionally, she sought to optimize her popup by generating a solution through repeated cycles of analysis and evaluation. These problem-solving approaches are most akin to the skills and concepts that accompany engineering design in typical P-12 engineering curricula. Although Jane and Mary came up with resourceful approaches for solving the problems they faced while constructing their pop-ups, their approaches were less like the approaches engineering curricula espouse. For example, Jane switched her design goal. By switching her design goal, she eliminated the constraints she had originally placed on herself (i.e., that the tree and bushes needed to pop-up). Once she eliminated these
ISLS
299
ICLS2012
constraints, the problem did not exist to solve anymore. Similarly, Mary formulated a goal that was based on eliminating the need to think carefully about a specific aspect of her pop-up (measure). Mary removed measure from the components she had to plan for. As a result, Jane and Mary minimally participated in sustained troubleshootingand in a way, avoided it. Essentially, both children identified a problem and specified a solution but eliminated the need to analyze or evaluate alternative solutions. This study has several limitations. First, the interviews were post-design, and were therefore rerepresentations of what children actually did and thought. Also, because we relied on childrens self-reports, we identified a small number of instances of trouble related to pop-up function that represented a cross-section of the entire class (about one third of the children reported functional trouble). Therefore, our sample was not completely representative of the class. In future iterations, we intend to perform analyses on discourse and actions situated within the paper engineering instruction to see the range of childrens problem solving approaches in context. Despite these limitations, our findings nonetheless provide insight into the problem-solving approaches children used while constructing their pop-up books and therefore have implications for elementary grade engineering design instruction. Originally we did not foresee the need to establish a framework for guiding children toward a practice of troubleshooting while they participated in paper engineering activities. In future iterations of working with children paper engineers, we hope to engender an ensemble of recurrent activities that promote children adopting a practice of troubleshooting. Our goal is to design instruction so that there is a normative expectation that children will value and engage in specific activities (e.g., investigation and experimentation, solicitation of others perspectives, modeling, and mathematizing) in the service of troubleshooting. We conjecture that installing a practice of troubleshooting could aid children in finding solutions to emerging or anticipated challenges in their designs. The instructional challenge will be to develop forms of support for the practice of troubleshooting of sufficient generality yet firmly grounded in design difficulties that children identify. We expect that repeated opportunities to participate in troubleshooting practices in contexts of paper engineering could benefit children in other engineering design contexts.
References
Benenson, G. & Neujahr, J. L. (2009). Force and motion. City Technology. Retrieved October 19, 2011 from http://citytechnology.org/unit/pop-ups Brophy, S., Klein, S., Portsmore, M., & Rogers, C. (2008). Advancing engineering education in P-12 classrooms. Journal of Engineering Education, 97(3), 369-387. Dym, C. L., Agogino, A. M., Eris, O., Frey, D. D., Leifer, L. J. (2005). Engineering design thinking, teaching, and learning. Journal of Engineering Education, 94(1), 103-120. Hendrix, S. L., & Eisenberg, M. A. (2005). Computer-assisted pop-up design for children: computationally enriched paper engineering. International Journal on Advanced Technology for Learning, 3(2), 119127 Jordan, B., & Henderson, A. (1995). Interaction Analysis: Foundations and practice. Journal of the Learning Sciences, 4(1) 39-103. Katehi, L., Pearson, G., & Feder, M. A. (Eds.). (2009). Engineering in K-12 education: Understanding the status and improving the prospects. Washington, DC: National Academies Press. Simon, H. A. (1996). The sciences of the artificial (third edition). Cambridge, MA: The MIT Press. Stevens, R., & Hall, R. (1998). Disciplined perception: Learning to see in technoscience. In M. Lampert and M. L. Blunk (Eds.) Talking mathematics in school: Studies of teaching and learning. University Press, Cambridge, England, pp. 107-149. Thomas, D. R. (2006). A general inductive approach for qualitative data analysis. American Journal of Evaluation, 27(2), 237-246.
Acknowledgments
This material is based upon work supported by the National Science Foundation under Grant Number 0733209. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
ISLS
300
ICLS2012
Using Critical Reading Tools to Facilitate the Learning of Argumentation Skills

Jingyan Lu, Nancy Law, Faculty of Education, Pokfulam Road, The University of Hong Kong, Hong Kong jingyan@hku.hk, nlaw@hku.hk Abstract: This study investigates the effectiveness of OASIS (Online Annotation and Argumentation Support for Inquiry System) in facilitating students argumentation skills. Students in two secondary four classes used OASIS as a tagging tool to read online texts and to evaluate the arguments of peers while studying a unit called Gene Screening in the Liberal Studies curriculum. Students end-of-unit essays on were analyzed to assess the quality of their reasoning and the complexity of their argument structures. Students scores on an examination question testing their reasoning skills on a related topic were also collected. Multiple regressions show that students use of WHAT-tags was significantly related to quality of their argument structures and exam scores on reasoning. No relationship was found between their use of tags and the quality of their reasoning on their essays. Implications for further refinements in tool and task design for OASIS to facilitate argumentation skills are discussed.
Introduction
Given that argumentation pervades both daily life and academic life in the form of philosophical reasoning and scientific inquiry (Kuhn, 2005), the development of argumentation skills should be an important goal of education (Mercer, 2009). However, research indicates that students at all academic levels have difficulty with argumentation; even graduating high-school students have trouble producing, understanding and evaluating arguments (National Science Board, 2006). Given the recommendations that students be taught argumentation skills in school, a natural question is what to teach and how to teach it? What are considered to be good arguments? How can we integrate the teaching of argument skills into the regular curriculum? We sought in this study to foster the development of argumentation skills by embedding them in two basic learning activities: reading critically and evaluating arguments. The study investigated the effectiveness of OASISan argumentation tool designed to foster argumentation skills in reading and evaluation in Liberal Studies (LS), a new core subject in Hong Kong secondary schools.
Theoretical framework
Argumentation skills
Argumentation skills involve both thinking skills and discourse skills (Kuhn, 2005) and can be evaluated in terms of discourse structure and quality of reasoning (Erduran, 2007; Kuhn, 1991; Toulmin, 1958). Individuals exercise these skills with differing levels of expertise in formulating and evaluating arguments (Goldstein, Crowell, & Kuhn, 2009). Educational researchers have widely accepted Toulmins (1958) model of arguments as including claims, evidence, justification, and conclusions and have used it to investigate many areas of education (Chang & Chiu, 2008). A number of researchers have used Toulmins model to investigate argument structure as an indicator of argumentation skills. For example, Means and Voss (1996) characterized argument structures as skeletal, enhanced, and elaborated. Skeletal structures, the simplest, only have a conclusion and a reason. Enhanced structures have a conclusion, a reason, and a qualifier, and elaborated structures have two conclusions with multiple reasons and qualifiers. In Chinn et als study (2000), argument structures are graded from low which have simple reasons to high which are composed of complex networks involving multiple arguments and rebuttals. Understanding the different parts of arguments provides a basis for evaluating the structures of arguments and can be linked to the age, academic performance, informal reasoning skills, and learning contexts of students (Chinn, O'Donnell, & Jinks, 2000; Means & Voss, 1996). Argumentation also involves the exercise of informal reasoning skills which arguers use to weigh different opinions in order to make decisions about issues in their daily lives (Voss, Perkins, & Segal, 1991). Quality of reasoning can be based on the soundness of arguments which can be determined by the degree of realism, and relevance of reasons used to support conclusions (Schwarz, Neuman, Gil, & Ilya, 2003; Voss, et al., 1991). Quality of reasoning can also be based on the provision of grounding and arguments can be hierarchically coded based on the claim it has: simple claims, grounded claims, qualified claims, and both grounded and qualified claims (Weinberger & Fischer, 2006).
ISLS
301
ICLS2012
The ability to evaluate the arguments of others is an important argumentation skill. Although there are many occasions in daily life in which people must evaluate the written or spoken arguments of others, people in general tend to evaluate arguments poorly (Goldstein, et al., 2009). For instance, Kuhn (2005) found that students cannot evaluate the epistemological characteristics of arguments effectively. She suggested that students tended to focus on the content of arguments as opposed to their form. She also found that students tended to evaluate the arguments of others based on their personal preferences as opposed to the epistemic strengths or weaknesses of the arguments themselves. While most studies of argumentation have focused on artificial tasks in laboratory settings with significantly fewer focusing on authentic tasks in classroom settings, recent advances in computer technology have made it possible to support and record student argumentation in the classroom.
Argumentation tools
Argumentation tools, such as Belvedere, Digalo, Convince Me, and Sense Maker provide external argumentation frames to scaffold students in their efforts to construct their own arguments (Scheuer, Loll, Pinkwart, & McLaren, 2010). By supporting exploration and negotiation, these tools can scaffold the efforts of students to construct effective arguments. They provide different types of argumentation frames to scaffold different argumentation skills and in so doing support and shape student arguments in different ways. For example, non-linear frames such as graphs and diagrams can support the construction of arguments base on abstract ideas (Suthers, 2003), while linear frames such as lists and threaded discussions can support the construction of arguments base on face-to-face and online discussions. Thus far, most argumentation tools have been designed to support scientific argumentation in the classroom as opposed to supporting the kinds of argumentation that is involved in addressing issues in the humanities. Thus, in developing argumentation tools to support meaningful inquiry learning in LS classrooms, we designed tools to support the development of argumentation skills involved in reading and evaluating arguments. This study investigated whether OASIS (Online Annotation and Argumentation Support for Inquiry System) can foster the development of students argumentation skills by supporting their efforts to read critically and to evaluate arguments. We emphasized two argumentation skills: argument structure and quality of reasoning. This study focused on three questions: 1. Does reading with OASIS support the development of argumentation skills? 2. Does evaluating with OASIS support the development of argumentation skills? 3. Do high performance class (HPC) and ordinary performance class (OPC) students differ in argumentation skills?
Methodology
Participants
Students of two secondary four classes, one HPC and the other OCP, were chosen as convenience samples from schools participating in a university-school partnership project on the effectiveness of OASIS in LS classrooms. Both classes followed the same LS curriculum and used OASIS to support reading, writing, and peer evaluation. A total of 39 students in the OPC and 44 students in the HPC participated in this study.
Design of study
LS is a core course that was introduced into schools in Hong Kong in 2008. It covers six themes: Personal Development and Interpersonal Relationship, Hong Kong Today, Modern China, Globalization, Public Health, and Energy Technology and the Environment. It provides students with opportunities to explore issues relevant to the human condition in a wide range of contexts and enables them to understand the contemporary world and its pluralistic nature (Education and Manpower Bureau, 2007). The module unit examined in this study dealt with Gene Screening which belongs to the theme of Public Health. The unit was divided into 4-sessions and OASIS was designed to scaffold learning processes by helping students: 1) highlight and tag passages in teacher-selected and self-selected articles, 2) organize and manage highlighted and tagged passages and integrate them into written arguments, and 3) evaluate (highlight and tag) the written arguments of peers. Three OASIS supported tasks were selected for further analysis. Task 1 required students to use OASIS in reading and analyzing teacher-assigned articles; task 2 required them to use OASIS to search for, select, read, and analyze more articles; task 3 required them to use OASIS to evaluate each others essays. The highlighting and tagging features of OASIS facilitate the process of reading-to-argue. OASIS first prompts students to highlight passages in their readings and then prompts them to attach tags to these highlighted passages. Students can choose tags from a list of teacher defined (Task tags) tags or from selfdefined (My tags) tags (See Figure 1). The complex process of selecting, highlighting and tagging passages helps students construct their own arguments. OASIS also has a website that includes a series of features such as my collection, my tags, my group and my task that allow students to view, organize, manage, and synthesize their tagged passages.
ISLS 302
ICLS2012
Finally, my report enables students to view and export their tagged passages. Students can determine what to include in reports by choosing task, category, and even tags. They can export reports as Excel files and save them for future reference.
Figure 1: Screenshot of reading with argument tags in OASIS. OASIS also supports evaluation tasks. The evaluation task here was designed to help students identify the key parts of arguments and to judge the quality of arguments. Students can use two sets of pre-defined tags in OASIS to evaluate each others work: DESCRIPTIVE-tags to identify the key parts of arguments and EVALUATIVE-tags to evaluate the quality of arguments.
Data sources and analysis

Two types of data were collected: 1. annotation data from both teacher-selected and self-selected readings, and peer evaluations and 2. Students argumentation skills demonstrated in their essays about Gene screening and in their answers to a related question on the mid-term exam. Reading and evaluation data consisted of passages that had been tagged and highlighted. Total numbers of such passages were recorded for the three tasks. Same sets of tags were designed for task 1 and 2. They included WHAT, WHY, CONTROVERSIES, PROS, and CONS. Two sets of tags were designed for evaluation task: DESCRIPTIVE-tags which included CLAIM, EVIDENCE, PROS, CONS and POSITION and EVALUATIVE-tags which INSUFFICIENT EXPLANATION, UNREASONABLE EXAMPLE/EXPLANATION, REASONABLE included EXAMPLE/EXPLANATION and ILLOGICAL. To reduce the number of variables used in the regression analyses of tasks 1 and 2, similar tags such as WHY, PRO and CON were collapsed as JUSTIFICATION and in task 3, 9 tags were collapsed into DESCRIPTIVE-tags and EVALUATIVE-tags. Students essays were examined with respect to level of reasoning and complexity of argument structure. These perspectives originated in Toulmins (1958) well accepted argumentation framework and have been further developed both bottom up and top down. Analysis of reasoning skills was adapted from the work of Yeh (1998) and Reznitskaya and colleagues (Reznitskaya, Anderson, McNurlen, Nguyen-Jahiel, Archodidou, & Kim, 2001). Six levels of reasoning were coded ranging from 0 to 5 where 0 means no reasoning is provided that relates to the main topic, and 5 means Claim(s) is/are related to main topic with either supportive evidence/example or supportive justification/explanation, or both; in a coherent style. Coding for quality of argument structure was adapted from the work of Clark et al (2007) where structure was coded into 5 levels with 1 referring to Multiple claims with either evidence/example or justification/explanation and 5 referring to claim(s) and alternative theory(s) which include both evidence/example or justification/explanation and rebuttal. Two research assistants coded the essays with an inter-rater reliability of .08. A question dealing with the issue of Liver transfer in the general area of public health was selected from the Semester 2 mid-term exam in order to explore whether argumentation skills could be transferred to similar tasks. The question contained three parts and measured both application and reasoning skills. The first two questions called for the application of information from source readings and the third called for reasoning. Descriptive analysis was used to characterize online reading and evaluation. Simple t-tests were run to determine whether the HPC and the OPC differed with respect to these measures. Finally, regression analysis was run to look for associations between online tagging and argumentation skills. The two control variables were class (HPC, OPC) and Semester 1 exam scores. The independent variables were types of tagging in different tasks, and the dependent variables were reasoning skills, argumentation structure, applying information performance and reasoning performance on the exam.
ISLS
303
ICLS2012
Results
Descriptive analysis
The total number of argument tags on tasks 1 and 2 were about the same (M = 6.66 and 6.20 for tasks 1 and 2 respectively). More CONTROVERSIES and WHAT-tags occurred in task 1 than in task 2. Students created significantly fewer EVALUATIVE-tags (M = .89), in task 3, as compared to DESCRIPTIVE-tags (M = 2.33).
Does reading and evaluating with OASIS support the development of argumentation skills?
Hierarchical multiple regression was run to test the relationship between online reading behavior and argumentation skills. Prior knowledge, was a significant predictor of reasoning in written essays and performance on exams (Essay reasoning: R2 = 0.120, p<0.01; Exam reasoning: R2 = 0.115, p< 0.01; Exam applying: R2 = 0.089, p<0.05). However, prior knowledge did not predict argument structure significantly (R2 = 0.037, p> 0.05). Another control variable, class, was an insignificant predictor of all dependent variables. Regression results indicated that online reading behavior accounted for about 8.4% of the variance of quality of argument structure, 14.8% of the variance of exam reasoning skills, and 14.5% of the variance of exam applying information skills. Among the different types of online reading behaviors, the number of WHATtags in Task 2 contributed significantly to argument structure quality (t = 2.501, p< 0.05) and exam reasoning skills (t = 2.175, p< 0.05); while the number of JUSTIFICATION-tags in Task 1 was significantly related to exam applying information skills (t = 2.545, p< 0.05). The tagging data in task 3 was not related to reasoning and applying information skills in mid-term exam.
Do HPC and OPC students differ in argumentation skills?

Although, we ran a series of t-tests to test whether the argumentation skills of HPC and OPC students differed with respect on written essays and mid-term exams, these measures did not reveal any significant differences between the two classes.
Discussion
This study examined the relationship between online reading and evaluation and argumentation skills demonstrated in written essays and exams. Student effort at understanding Gene screening, particularly with respect to self-selected articles, significantly contributed to their argumentation skills in both written essays and content-related exam questions. Gene screening was a difficult topic as most students had no prior knowledge about it. Thus, simply trying to understand Gene screening was not only a prerequisite but the most important factor affecting argumentation. In fact, most students attached WHAT-tags to highlighted passages that dealt with information and knowledge related to Gene screening or that dealt with such related concepts as test-tube babies or embryo implantation. Thus, students needed to master these concepts to understand the purpose and consequences of Gene screening in order to write their essays. Identifying relevant information in selfselected articles was also important. Students who couldnt find relevant papers had trouble carrying out further investigation. Similarly, this tagging behavior was also related to reasoning skills on the three-part health question from the Semester 2 mid-term exam. This implied that students might be able to transfer this skill to similar tasks. Besides, assigning more JUSTIFICATION-tags (WHY-PRO-CON) in task 1, students were better at applying skills on the mid-term exam. Applying skills involves being able to recognize, organize, and summarize arguments. The required skills resembled those required in task 1 in which students had to identify claim, pro, and con. The ability to write effective arguments is an essential feature of many subject domains. Childrens difficulty in writing arguments may be due to a number of reasons. They may lack basic literacy skills; they may find it difficult to examine and compare ideas within the same text; they may be unable to consider diverse opinions and select the most relevant one; they may be unable to anticipate objections or disagreements and they may be unable to rebut these objections (Muller Mirza, Perret-Clermont, Tartas, & Iannaccone, 2009). Developing the ability to construct argumentation structures while reading could help students overcome these problems. Writing arguments requires students to be able to read-to-argue. That is they must be able to interpret source materials by extracting and analyzing concepts and ideas, organize and compare multiple perspectives and opinions, and analyze and select relevant explanations for use in written arguments. At this stage, it is encouraging to see that reading with OASIS can help students construct better argument structures in their writing. The fact that reading with OASIS is not significantly related to reasoning skills may indicate that developing such skills might need lengthy practice and more extensive modeling. Finally, evaluation is an important argumentation skill. However, evaluating the arguments of peers, either by identifying the relevant parts of their arguments or by critically evaluating them was not significant to
ISLS
304
ICLS2012
reasoning skills on the exam. The students limited use of EVALUATIVE-tags could be one of the reasons. Other possible reason could be that students were reluctant to criticize each other or that they were not given sufficient guidance on how to use EVALUATIVE-tags. Larson, Britt, and Kurby (2009) found that students improved their ability to judge the quality of arguments when they received a little training on evaluating arguments and immediate feedback. Thus, more research is needed to determine how to design tasks and tools that more effectively support such processes. For instance, evaluating arguments and comparing them with those of teachers should first be modeled for students. The tools students use should support the visualization of such comparisons and should facilitate discussions of argumentation.
Acknowledgement
This study was substantially supported by a grant to the first author from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. HKU747909/2009/Social Science and Humanities).
References
Chang, S.-N., & Chiu, M.-H. (2008). Lakatos' scientific research programmes as a framework for analysing informal argumentation about socio-scientific issues. International Journal of Science Education, 30(13), 1753-1773. Chinn, C. A., O'Donnell, A. M., & Jinks, T. S. (2000). The structure of discourse in collaborative leraning. The Journal of Experimental Education, 69(1), 77-97. Clark, D., Sampson, V., Weinberger, A., & Erkens, G. (2007). Analytic frameworks for assessing dialogic argumentation in online learning Environments. Educational Psychology Review, 19(3), 343-374. Education and Manpower Bureau. (2007). Background. Liberal Studies: Currlculum and Assessment Guide (Secondary 4-6). Hong Kong. Erduran, S. (2007). Methodological foundations in the study of argumentation in science classrooms. In S. Erduran & M. P. Jimnez-Aleixandre (Eds.), Argumentation in Science Education (Vol. 35, pp. 47-69): Springer Netherlands. Goldstein, M., Crowell, A., & Kuhn, D. (2009). What constitutes skilled argumentation and how does it develop? [argument; argumentation; development; metacognition; causal reasoning]. Informal Logic, 29(4), 379395. Kuhn, D. (1991). The skills of argument. Cambridge: Cambridge University Press. Kuhn, D. (2005). Education for thinking. Cambridge, Massachusetts: Harvard University Press. Larson, A. A., Britt, M. A., & Kurby, C. A. (2009). Improving students' evaluation of informal arguments. [Article]. Journal of Experimental Education, 77(4), 339-366. Means, M. L., & Voss, J. F. (1996). Who reasons well? Two studies of informal reasoning among children of different grade, ability, and knowledge levels. Cognition and Instruction, 14(2), 139-178. doi: 10.1207/s1532690xci1402_1 Mercer, N. (2009). Developing argumentation: Lessons learned in the primary school. In N. Muller Mirza & A.N. Perret-Clermont (Eds.), Argumentation and Education (pp. 177-194). New York, NY: Springer. Muller Mirza, N., Perret-Clermont, A. N., Tartas, V., & Iannaccone, A. (2009). Psychosocial processes in argumentation. In N. Muller Mirza & A.-N. Perret-Clermont (Eds.), Argumentation and Education (pp. 67-90). New York, NY: Springer National Science Board. (2006). Americas pressing challenge-building a stronger foundation. Retrieved Sept. 20, 2010 http://www.nsf.gov/statistics/nsb0602/nsb0602.pdf Reznitskaya, A., Anderson, R. C., McNurlen, B., Nguyen-Jahiel, K., Archodidou, A., & Kim, S.-y. (2001). Influence of oral discussion on written argument. Discourse Processes, 32(2/3), 155-175. Scheuer, O., Loll, F., Pinkwart, N., & McLaren, B. (2010). Computer-supported argumentation: A review of the state of the art. International Journal of Computer-Supported Collaborative Learning, 5(1), 43-102. Schwarz, B. B., Neuman, Y., Gil, J., & Ilya, M. (2003). Construction of collective and individual knowledge in argumentative activity. Journal of the Learning Sciences, 12(2), 219-256. Suthers, D. (2003). Representational Guidance for Collaborative Learning. In H. U. Hoppe, F. Verdejo & J. Kay (Eds.), 11th international Conference of Artificial Intelligence in Education (pp. 3-10). Amsterdam: IOS Press. Toulmin, S. E. (1958). The use of argument. Cambridge: Cambridge University Press. Voss, J. F., Perkins, D. N., & Segal, J. (1991). Informal reasoning and education. Hillsdale, New Jersey: Erlbaum Associates. Weinberger, A., & Fischer, F. (2006). A framework to analyze argumentative knowledge construction in computer-supported collaborative learning. Computers & Education, 46(1), 71. Yeh, S. S. (1998). Validation of a scheme for assessing argumentative writing of middle school students. Assessing Writing, 5(1), 123-150. doi: 10.1016/s1075-2935(99)80009-9
ISLS
305
ICLS2012
Changing Explanations about Sand Dune Movement

Lauren A. Barth-Cohen, University of California, Berkeley, Berkeley CA, 94720, lbarth@berkeley.edu Abstract: Knowing that students tend to have difficulties understanding complex systems, this study investigates one case of an adult students reasoning about how sand dunes move in the desert. The movement of sand dunes is a phenomenon that exhibits characteristics associated with complex systems. During the course of the interview, the student generates a set of consecutive explanations and over time her explanations shift from being gradually less prototypically centralized to more decentralized. Using the Knowledge-in-Pieces epistemological framework, I document a series of knowledge configurations that are used within her explanations that together support this change. These results have implications for future work focusing on identifying factors that support similar kinds of changing explanations.
Introduction
Students tend to have difficulties when learning about complex systems (Chi, 2005; Jacobson, 2001; Resnick & Wilensky, 1998; Resnick, 1996). Complex systems are composed of individual elements, each of which has a particular pattern of behavior that results in a macro-level behavior that is different from the micro-level behaviors. As a simple example, consider a wave traveling around an arena, each individual follows a simple micro-level rule in which they raise their hands immediately after the person next to them raises their hands. This rule contributes to a macro-level pattern in which the entire wave travels around the arena. Some other examples of complex systems include the formation of traffic jams, the movement of flocks of birds, and the symmetrical patterns in snowflakes. Across students difficulties when learning about these systems is a common theme of applying centralized or direct causality in situations where decentralized or emergent causality is more appropriate. For example, when learning about diffusion, Chi (2005) finds that students misconceptions about emergence are due to a misinterpretation of these processes as a kind of direct process. An example of this difficulty comes from Resnick (1996) who focuses on the so-called centralized mindset, which is a way of thinking, or attitude, in which centralized causality is presumed to account for the systems behavior even in cases where decentralized causality is appropriate. In the context of building computational models of various complex systems, including traffic jams and the formation of termite mounds, Resnick (1996) documents both the pervasiveness of the centralized mindset and ways that students can begin moving towards decentralized causality. This difficulty can be illustrated through the case of the wave; one could use centralized causality by assuming that there is an announcer telling each person when to raise his or her hands or by assuming the existence of a norm in which one seating section of the arena begins the wave every time the favorite team scores. Oppositely one could employ decentralized causality by assuming that the wave has a distributed cause in which each person individually decides when to raise his or her hands. Knowing that students have these difficulties, there is a focus on instructional approaches to support students towards thinking about complex systems using decentralized causality instead of centralized causality. Sengupta and Wilensky (2009) implement a computational modeling curriculum about electricity, and they hypothesize that it supports students by bootstrapping their existing intuitive knowledge. Similarly trying to support students thinking about emergence, Levy and Wilensky (2008) discuss the utility of a strategy in which an intermediate level is used to scaffold students understanding towards decentralized causality. Using this strategy students reason about a pack of deer or clique of students as examples of small groups within larger populations. I build off this prior work, in that I also investigate ways to make progress towards reasoning about complex systems using decentralized causality. However, I investigate a slightly different issue. As mentioned above, prior work generally focuses on whether students can make progress understanding complex systems, what strategies are effective, and how might computational modeling curriculums be effective. Comparably I focus on how progress occurs moment-by-moment in terms of changing knowledge. Penner (2000) also looks at changing explanations, although he focuses on content and not on forms of knowledge or knowledge evolution. This paper falls within a growing body of work in the learning sciences that focuses on the process of learning in terms of developing theories of knowledge and of knowledge evolution using fine-grained momentby-moment analyses (e.g. Schoenfeld, Smith & Arcavi, 1993; Sherin, 2001). While I build off this prior work using a similar methodological approach and emphasizing changing knowledge, this study focuses on a new domain, complex systems, and focuses on pools of knowledge that are activated for systematic use in a context.
ISLS
306
ICLS2012
Theoretical Background: Process Analysis of Local Changes

This paper uses the epistemological framework of Knowledge-in-Pieces (diSessa, 1993) to capture the process of local changes across a students explanations in terms of knowledge. By local change I refer to small changes that occur in a moment-to-moment time scale during a 13-minute discussion. I focus on explanations that address how and why the phenomena occur and on the knowledge activated within the explanations. Specifically, I focus on the pool of knowledge that has been activated for systematic use in a context. I refer to the pool of knowledge as local knowledge configurations, as they depend on and are activated within a context. Knowledge configuration is not a highly technical term, they are not homogenous, and some bear close relationships to existing constructs. Knowledge configurations may be locally coherent (Hammer, et. al. 2005) as they are dynamic, emerge from a pool of knowledge, and hold together for a short time period. Some knowledge configurations bear a resemblance to Dynamic Mental Constructs (Sherin Krakowski & Lee, 2012) as they are dynamic mental states that can shift during an interview and that depend on many factors including interview questions. However, some knowledge configurations are more like resources that influence explanations rather than explanations themselves. A goal in this analysis is to address cases of knowledge facilitating change within and across explanations. While I focus on the process of local changes during short time periods in terms of the knowledge within a students explanations, there is another perspective that focuses on the general causal patterns within students reasoning. For example, Perkins and Grotzer (2005) present a taxonomy of increasingly complex causal models. Andersson (1986) identifies a common causal core to explanations across different scientific topics. Overall these perspectives focus on general causal patterns within explanations. Comparably, my analysis focuses on how complex arrangements of knowledge support shifts within causal explanations. The goal of this paper is to describe the changing explanations in terms of the knowledge configurations about a particular phenomenon that exhibits a behavior associated with complex systemsthe movement and formation of sand dunes. I highlight the interactions of knowledge configurations that support a shift from gradually less prototypically centralized towards decentralized. This paper resides within complex systems, a field which generally does not investigate the process of changing explanations in terms of knowledge, and within an area that often focuses on sequential patterns of causal reasoning rather than a complex arrangement of knowledge.
How do sand dunes move in the desert?

This study focuses on students reasoning about the movement of sand dunes in the desert, an understudied, but useful, example of a complex system because of students relevant intuitions and experiences. There is little discussion about this particular problem, but I hypothesize that it is accessible for the purposes of investigating learning about complex systems. Students have a great deal of underlying intuitive knowledge about the wind and sand from everyday activities such as playing in sand boxes or gardening. Additionally, since students likely have not thought extensively about sand dunes prior to the interview, we can see them activating intuitive knowledge while reasoning about new phenomena. From the scientific perspective there are several different ways to view the movement of sand dunes. Fundamentally, the movement of sand dunes is a complex process, because the movement of the entire dune (collection of many particles) is determined by the movement of individual particles which act as independent agents with no external cause orchestrating their movement. From the geological perspective, one main mechanism for how wind moves sand is known as saltation. Saltation is the process by which the wind moves sand particles through successive small jumps. When a particle lands it may be ejected or it may collide with another particle resulting in that new particle being ejected. Using Newtonian mechanics each jump can be modeled as a projectile. This process is thought to account for the majority of sand moving up the less steep side of the dune and being deposited on the steeper side. Hence dunes often, but not always, move leeward, in the direction of the wind (Pye & Tsoar, 2009). However, there are many types of sand dunes with different shapes and different behaviors. From the perspective of complex systems, the movement of sand dunes occurs through the local interactions of individual sand particles. Each particle has a particular behavior, for example it can join a dune, leave a dune, or not move. The macro-level behavior of the entire dune, moving, staying still, or staying in the same location but changing size, is due to the collective behavior of all individual sand particles. Using the centralized/decentralized distinction, we can hypothesize how students might explain the movement of sand dunes. If students apply centralized causality when explaining the movement of sand dunes, they might focus on the wind controlling the movement of sand particles such that a gust of wind blows all the sand particles simultaneously thus moving the sand dune at once. While this idealized explanation might sound implausible, a version that I have seen in other interviews is that there must be a boulder in the desert, acting as a seed or leader, such that sand hits the boulder and a dune forms at that location. In contrast, one could use decentralized causality by assuming that wind moves each sand particle a distinct and random distance independent from surrounding sand particles. In this more normative model, the desert is composed of many
ISLS
307
ICLS2012
sand dunes and innumerable sand particles, all acting independently. In this model, it becomes irrelevant whether a new dune is composed of the same sand as in the old dune, since all dunes are indistinguishable from one another.
Methods
The data in this paper comes from a short excerpt within a longer open-ended interview with Laurel, an adult student who articulately reasons about the movement of sand dunes. This interview comes from a corpus of open-ended clinical interview data in which 8th-12th graders, undergraduates, masters, and Ph.D. students explain how sand dunes move. This study uses open-ended individual clinical interviews in the style described in diSessa (2007) to focus on moment-to-moment changes, allowing the author who serves as the interviewer, to probe the students knowledge. In the Laurel interview the sand dune discussion began 38 minutes into an hourlong interview and lasts for 13-minutes. The rest of the interview, before and after the sand dune discussion, is spent discussing unrelated physics topics that do not involve sand dunes or emergence. At the beginning of the sand dune excerpt I propose an initial situation of one sand dune moving in the desert. Additional probing questions focus on clarifying her understanding of the underlying mechanisms (e.g. You mentioned at one point that it would, like, the sand, would stick to it. How would that work?). Laurel is a masters-credential student in mathematics education. She has an undergraduate degree in mathematics and has taken introductory physics. She does not have any formal training in geology, sand dunes, or complex systems; therefore, this is a case of an adult reasoning about something that is new to her. I chose the case of Laurel because she is very articulate and clear in her explanations. We can see local changes in her thinking over the short time period. Data consists of the student drawings and video records of the interviews, which I then used to create transcripts. My analytical goal is to capture and describe the ways that her explanations change with an emphasis on knowledge. Through an iterative coding process, it became apparent that certain explanations and knowledge elements are central to her reasoning shifts. This general method falls within a family of related theoretical and methodological approaches which focus on describing the nature and development of knowledge systems known as Knowledge Analysis (Sherin, 2001). There are some interesting methodological differences between the current study and much of the prior work focusing on students thinking about complex systems using decentralized causality. The current literature (e.g. Sengupta & Wilensky, 2009) tends to use computational modeling environments and a pre-/post-test approach, sometimes with interviews, to understand general trends in students learning. Comparably, the current study only uses interviewing techniques in which students engage in a thought experiment after being presented with a hypothetical situation, sand dunes moving in the desert. Although students have the option of drawing, there are no computational or physical manipulatables.
Analysis: Description of the Knowledge Configurations that Facilitated Change

In this analysis, I describe a series of knowledge configurations that are used within her shifting explanations. At the beginning of the discussion, she generates a series of explanations that are based on knowledge about the implausibility of direct causality. As the discussion progresses, her explanations shift as she generates and subsequently rejects explanations based on sand moving in sequential layers. Another shift in her explanations occurs when she generates explanations based on the sand stickiness mechanism for how sand joins and leaves dunes. Simultaneously during these shifts she also occasionally focuses on the origin of dunes, asking herself questions about how dunes begin forming and what resides in a particular location after a dune moves. This list of knowledge configurations that are aligned with shifts in her explanations is not exhaustive, other factors include interviewer questions and her shifts in her interest or focus of attention, but this list is meant to illustrate some central knowledge configurations. Her final explanations account for a configuration of knowledge that I refer to as changing composition, in which the size of the dune may be stagnant while sand particles are constantly joining and leaving at equal rates. By the end of the discussion, she relinquishes explanations that are prototypically centralized and generates the changing composition explanation, which is more decentralized.
Implausibility of Direct Causality

One key way in which Laurel's explanations change is by her recognition and subsequent rejection of the plausibility of direct causality. Throughout the interview, Laurel several times rejects explanations that incorporate knowledge that is based on direct causality. At first Laurel rejects an explanation based on a single dune shifting positions in its entirety: I dont think the wind would blow the whole sand dune all at once. A second time this occurs is when she rejects the idea that all the sand particles that comprise the old dune will be moved to the new dune: I dont know how they [sand particles] would all end up in the same placeI don't know why they [sand particles] would all concentrate in a form, a big pile. A third time this occurs is in the context of discussing the dune shifting locations in sequential layers. Laurel rejects this sequential layers explanation because it relies on the wind being consistent which conflicts with her prior experiences of wind
ISLS
308
ICLS2012
being variable. This top layer of sand would kind of get blown firstoutside of the sand dune [hand motion] would kind of get blown over here and then it would kind of end up somewhere and then now that layer gone so then if the wind kept blowing, then it would kind of blow, whatever, this, some of the outside again.its not like the wind is constant, it's always blowing in exactly this direction at exactly the same, exactly the same strength. Because if it were that would make sense right. The theme in all these examples is that she is considering the plausibility of direct causality and then rejecting the implications of this approach based on some underlying knowledge about the motion of the wind and sand. Rejecting the plausibility of direct causality contributes to her changing explanations, because it opens doors for considering other options. For example, rejecting the idea that all the sand particles that comprise the old dune will be contained within the new dune opens the door for her to consider the role of other sand in the desert, that might join or leave a dune coming from or returning to the surrounding desert, which is associated with decentralized causality. A similar example occurs when rejecting explanations based on the dunes moving in sequential layers. She has an expectation that for this process to hold, there needs to be a consistent wind. However, her prior knowledge suggests that the wind is not consistent, thus, allowing her to reject the sequential layers explanation and opening the door to consider a variable wind.
Sequential Movement
A key point during the interview, when her explanations shift is when she generates and subsequently rejects explanations based on sand moving in sequential layers. She generates two versions of this explanation, in one case the layers of sand move sequentially with the outermost layer moving first, followed by the next layer until the entire dune moves. In the other version of this explanation, individual grains of sand move sequentially. She mentions an imaginary situation in which a bird drops grains of sand sequentially on the same location until a dune forms: If a bird comes and takes one grain of sand and then dumps it, cause if that happened, then obviously, if you keep dumping them on top of each other it would be like, when you dump whole bunch of sand and it makes like a cone shape. For both versions of this explanation, she rejects it because the wind does not work this linearly, she expects there to be a more variable wind. These explanations are an important intermediate step because they direct her to focus on the microscopic level of individual sand grains, and because they highlight the variableness of the wind. As mentioned in the prior section, recognizing the implausibility of direct causality, in this sequential movement, is important because it opens the door for her to consider more variable wind. Also, this explanation is important, as it connects the micro-level movements of individual grains of sand with the macro-level dune changing size. Connecting levels is an important part of decentralized explanations (Sengupta & Wilesnky, 2009).
Sand Stickiness
One factor that influences Laurels shifting explanations is a focus on how sand joins and leaves the dunes, based on a mechanism I refer to as sand stickiness. Sand stickiness is that idea that sand particles suspended in the wind join the dune because they stick to the dune resulting in it getting bigger and sand particles leave a dune because they are no longer stuck resulting in the dune getting smaller. Laurel incorporates this mechanism for how dunes change size in several her explanations. For example, when explaining how dunes get smaller Laurel says: I guess, since the sand is like kind of loose [gesture implies outwards], its not like it's glued together or compacted or something [holds hands together to imply compact]. Then when the wind comes, the sand will freely, the outside layer of sand will start, can just fly other places [gesture implies away] and then maybe that keeps happening as the wind keeps coming then next, other layers of sand keep disappearing or not disappearing but like, being taken off of that and then it's like dissipated around [gesture implies inwards and outwards]. This is one of several explanations Laurel generates based on this mechanism, which is important, because, similar to the sequential movement explanation, it connects the micro and macro levels.
Origin of Dunes
Throughout the discussion, she occasionally focuses on the origin of dunes asking herself questions about how a sand dune began forming and what is left over after a sand dune moves. The first time she mentions the issue of dune origins is in the context of what happens after a dune moves. Laurel asks herself, originally where there was a sand dune, would it be flat or something? Or? Just shorter? At the end of the interview, the possibility of
ISLS
309
ICLS2012
the location of the original sand dune not being flat becomes relevant. Instead of focusing on what is left over after a dune moves, Laurel considers an alternative, but related question, of why a dune doesnt disappear and how it gets bigger after having gotten smaller. But it wouldn't get flat probably, there would still be a little cone there and then maybe that would start collecting more [gestures implies collecting] and growing [gestures implies growing] and then the wind could take it away get smaller [gestures implies getting smaller]. Laurel rejects the idea of a dune disappearing into the flat desert and focuses on dunes getting bigger and smaller cyclically as sand joins and leaves. The question of the origin of dunes is associated with a shift from explanations that are gradually less prototypically centralized to decentralized. Laurels initial approach to this question focuses on the desert being flat after a dune moves, which is aligned with centralized causality while at the end of the interview, she rejects the flat desert idea and mentions a small cone of sand left over which then allows the dune to begin increasing in size.

In the analysis, I describe some candidate knowledge configurations that facilitate change in Laurels explanations from being gradually less prototypically centralized to more decentralized. For example, in this data, recognition of the implausibility of direct causality and focus on the origin of dunes is important to her changing explanations. The ways in which Laurels explanations change may have implications for future work focusing on identifying factors that support students reasoning about complex systems.
References
Andersson, B. (1986). The experiential gestalt of causation: A common core to pupils preconceptions in science. European Journal of Science Education, 8(2), 155-171. Chi, M. T. H. (2005). Commonsense conceptions of emergent processes: Why some misconceptions are robust. Journal of the Learning Sciences, 14(2), 161-199. diSessa, A. A. (1993). Toward an epistemology of physics. Cognition and Instruction, 10(2-3), 105-225. diSessa, A. A. (2007). An interactional analysis of clinical interviewing. Cognition and Instruction, 25(4), 523565. Hammer, D., Elby, A., Scherr, R. E., & Redish, E. F. (2005). Resources, framing, and transfer. In J. P. Mestre (Ed.), Transfer of learning from a modern multidisciplinary perspective (pp. 89120). Greenwich, CT: IAP. Jacobson, M. J. (2001). Problem solving, cognition, and complex systems: Differences between experts and novices. Complexity, 6(3), 41-49. Levy, S. T., & Wilensky, U. (2008). Inventing a "mid level" to make ends meet: Reasoning between the levels of complexity. Cognition and Instruction, 26(1), 1-47. Penner, D. E. (2000). Explaining systems: Investigating middle school students' understanding of emergent phenomena. Journal of Research in Science Teaching, 37(8), 784-806. Perkins, D., & Grotzer, T. (2005). Dimensions of causal understanding: The role of complex causal models in students' understanding of science. Studies in Science Education, 41, 117-165. Pye, K., & Tsoar, H. (2009). Aeolian sand and sand dunes. Springer-Verlag. Resnick, M. (1996). Beyond the centralized mindset. Journal of the Learning Sciences, 5(1), 1-22 Resnick, M., & Wilensky, U. (1998). Diving into complexity: Developing probabilistic decentralized thinking through role-playing activities. Journal of the Learning Sciences, 7(2), 153-172. Schoenfeld, A. H., Smith, J. P., & Arcavi, A. (1993). Learning: The microgenetic analysis of one student's evolving understanding of a complex subject matter domain. Advances in Instructional Psychology, 4, 55-175. Sengupta, P., & Wilensky, U. (2009). Learning electricity with NIELS: Thinking with electrons and thinking in levels. International Journal of Computers for Mathematical Learning, 14(1), 21-50. Sherin, B. L. (2001). How students understand physics equations. Cognition and Instruction, 19(4), 479-541. Sherin, B., Krakowski, M., & Lee, V. R. (2012). Some assembly required: How scientific explanations are constructed in clinical interviews. Journal of Research in Science Teaching, 49(2), 166-198.
Acknowledgements
The author thanks Andrea diSessa and the Patterns Group at UC Berkeley and Ayush Gupta and Ido Roll for insightful comments.
ISLS
310
ICLS2012
High school students epistemic engagement in producing documentaries about public science concerns
David J. DeLiema, Jarod N. Kawasaki, William A. Sandoval, U. of California at Los Angeles, 2027 Moore Hall, Box 951521, Los Angeles, CA 90095 USA Email: ddeliema@gmail.com, jarodkawasaki@gmail.com, sandoval@gseis.ucla.edu Abstract: Recently, situated theories of cognition and learning have attracted personal epistemology researchers, motivating efforts to revise traditional PE theories of developmental stages and independent beliefs. In the situated framework, researchers view knowledge practices as tentative and dependent on particular contexts. In the present study, we qualitatively examine the epistemic aims (i.e. defending a personal conviction about a knowledge claim) and the epistemic actions (i.e. questioning empirical data) that high school students bring to bear in multiple contexts within a classroom where each student creates a documentary film about a public science topic. We describe several socially-situated variables that drive the formulation of epistemic aims and the consequent selection of epistemic actions: dimensions of interpersonal relationships compatible or conflicting with epistemic goals, constraints that arise from completing classroom tasks, and personal affiliations with target topics.
Introduction
Pedagogical interest in epistemic cognition stems in part from empirical evidence suggesting the beliefs students have about knowledge influence the way they learn. For example, students who consider physics to be complex might investigate the complex interactions between variables in physics problems, whereas students who view physics knowledge as simple might memorize formulas (Hammer, Elby, Scherr, & Redish, 2005). We refer to the ideas students have about knowledge as epistemic beliefs (e.g. Schommer, 1990) and the actual knowledge practices they deploy in classroom exercises as epistemic cognition (e.g. Chinn, Buckland, & Samarapungavan, 2011). The promise of correlating beliefs about knowledge with practices around knowledge rests on a core empirical question: Do students beliefs and practices around a knowledge domain persist across contexts within a classroom, and if not, what dimensions of context motivate different expressions of epistemic cognition? We examine the construct of personal epistemology through a situated framework to throw light on the context question. Recently, situated theories of cognition and learning have attracted personal epistemology (PE) researchers, motivating efforts to revise traditional PE developmental stages theories (Perry, 1970) and independent beliefs systems models (Schommer, 1990). The traditional developmental and belief systems frameworks hypothesize that students have robust ideas about knowledge that are stable and correspond to (or surface as) self-assessed beliefs phrased in words during interviews or on surveys outside classroom situations. These professed beliefs are tacitly assumed to transfer into enacted practices around knowledge in classroom activities. Some studies, however, have found that variation in contextwhether across questionnaires, enacted curriculum, or reasoning on science problemsleads to differences in epistemic cognition for the same participants (see Sandoval, 2005). In the situated framework, researchers view knowledge practices as dependent on particular contexts. As such, we examine not only students beliefs about the nature of knowledge and knowing, but also how epistemic cognition varies situationally, what Bromme (2005) alluded to in question form as, why and with what aim epistemological beliefs emerge and for what are they needed? (p. 199). The epistemological resources framework (Hammer & Elby, 2002) in fact preceded Brommes call and launched the situated PE approach, positing that epistemic cognition involves fine-grained cognitive resources such as accumulation, formation, checking that frame actions. Even though the framework spells out a variety of knowledge resources (although openly incomplete) students may activate, the framework remains silent on the contextual variables that matter for their activation. In providing a partial answer to the context lacuna, Chinn et al. (2011) have formulated an ambitious theory that offers some structure to the construct of context as it involves epistemology. Standard PE models consider two epistemic dimensions: the nature of knowledge, including its source and form, and the nature of knowing processes, including ideas about certainty and justification (Hofer & Pintrich, 1997). Chinn and colleagues expand this model to include a variety of other aspects of epistemology, most importantly for our purposes here, the new dimension of epistemic aims. Epistemic aimssuch as gaining knowledge, understanding, or having true beliefare the goals that motivate knowledge practices. Chinn et al. argue that if we fail to model students epistemic aims, then we remain essentially blind to whether students will actually deploy a set of epistemic practices in a particular situation.
ISLS
311
ICLS2012
This paper answers Brommes (2005) call by adopting a mixed version of the epistemological resources model (Hammer & Elby, 2002) and Chinn et al.s (2011) recent framework to account for how social dimensions of activities present constraints and affordances on epistemic aims and resources. The activity system of interest in this study takes place in a high school summer course focused on students creating documentary movies around public science issues. We hypothesize that within this activity system, students epistemic aims arise out of interactions between the goals they are pursuing (namely those resulting from personal affiliations with a topic), features of the task structure (namely time constraints), and social relationships (including, centrally, power relationships). We expected that having students manage a long-term documentary project where they had to find and evaluate diverse source materials, including claims from news reports, family, and friends, would involve some of students everyday experiences with science and generate a variety of constraints on epistemic aims. We created a learning space where public and familial experiences were integrated with classroom activities in order to get a glimpse of how foregrounded social context impacts epistemic cognition.
Method
The study took place during a five-week high school summer course on scientific media literacy and documentary movie making at a local state university. The class met for two and a half hours, five days a week for five weeks. There were 19 students enrolled in the course, of which 11 consented to participate (N=11, 2 girls, 14-18 years old) and seven (1 girl) focused on public science topics. All the participants have been given pseudonyms in the analysis section. Participants came from various ethnic and socioeconomic backgrounds and attended various high schools, both private and public. In a blend of design-based and action research, the second author taught the course and maintained a daily reflective journal while the lead author observed the class four times, recording field notes about specific interactions for three students. Through an iterative process accounting for class experiences and curriculum design, we designed activities around the elements and structure of television news narratives, highlighting rhetorical (e.g. representation, persuasion, power, relevance) and narrative (e.g. perspective/camera angles, symbolism, empathic anchors, verbal-visual-discontinuities) elements. Students engaged with these elements around a self-selected science topic covered in the media, evaluating various sources of information on their topic, including experts in the media, local experts, peers, and family, critiquing popular news media, and creating a documentary film. Students documented their class experiences through online journal entries, responding to the days lesson and additional reflection questions. The primary student artifact was a documentary film about their selected science topic, examined through multiple sources such as the news media, personal interviews with family and friends, a reflective video journal, and selected video clips from the internet, to be shared in a public Q&A screening with over 100 peers enrolled in other classes at the school. Students also maintained a daily media consumption log, where they reflected on various media sources they encountered. In our analysis, we focused on multiple events in the classroom that suggest factors relevant to students epistemic aims and resources. Specifically, we tracked the experiences of three students observed in detail in our field notes, and through a grounded approach, iteratively examined all data sources for those students and identified salient themes that emerged. The first and second authors independently coded written artifacts and student documentaries along the dimensions of epistemic aims, epistemic resources, and social/task constraints. We adopted an approach of writing memos and building conceptual flow charts as a way to develop the conceptual categories in a constant back and forth with the data. Through our analysis, we have identified exemplary events that demonstrate how potential contextual factors motivated different epistemic aims and resources. For data, we triangulate between students verbal discourse, nonverbal behaviors, and constructed artifacts to model the epistemic resources recruited in specific settings and to infer the epistemic aim(s) adopted. In so doing, we take into account the full activity systemtools, rules, divisions of labor, and communities (Cole & Engestrm, 1993)to begin to explore how social dimensions of the setting drive certain epistemic aims. Methodologically, we deviate from Hammer & Elbys (2002) definition of epistemic resources by focusing only on the observable epistemic actions students take rather than accounting also for the cognitive resources that frame those actions. As such, we use the term epistemic action for the remainer of the paper. In summary, we describe the context that forms around epistemic cognition along three dimensions: (1) students observable knowledge practices such as checking a document or asking a question (epistemic actions), (2) what we can infer about students knowledge aims such as attempting to cast doubt on a claim or qualify a claim (epistemic aims), and (3) situational factors that impact the former dimensions. The third dimension identifies non-epistemic variables in the activity system that may have motivated the epistemic aims adopted and the resources deployed.
Results
ISLS
312
ICLS2012
We found that students practices around knowledge in classroom activities varied as a function of multiple factors. Namely, social interactions between the student and the teacher, researcher, family, peers, and the public, in addition to constraints in the medium/activity engaging the student and goals springing from personal affiliations with the topic, seemed to frame epistemic aims and actions. We organize our findings along four different interaction spaces: (1) student and task, (2) student and other authority figures, (3) student and teacher, and (4) student and peers. Vignettes and student artifacts are used to demonstrate how features of settings motivate epistemic aims and actions.
Student and task

First, we examine the interaction between the student and the dominant task in one instance that exemplifies how the full activity system constrains the formation of epistemic aims. While the course attempted to encourage students to independently pursue a chosen topic, the teacher confirmed that students were under conventional constraints, such as needing to complete activities on time. On many occasions, a student named Saahil expresses that he is uncertain of whether global warming is true. A few days before his final project presentation, Saahil is working in a classroom adjacent to his usual room (due to space constraints) when the teacher from the other class overhears Saahil talking about his uncertainty. The teacher introduces himself as an expert on global warming (with a college degree in environmental science) and the following interaction occurs (as described in the researchers field notes): You could put lots of things, the teacher explains, including how CO 2 warms the atmosphere, CO2 dissolves and turns the oceans more acidic, and how ice melts. Saahil perks up: Im doing ice melt! The teacher responds, Are you trying to prove or disprove? Saahil: Giving both sides. The teacher again, Its so complicated, you might want to narrow down your focus. As the teacher walks away, I ask Saahil how much of the discussion made sense. Saahil says, Some of it. Five minutes later, the teacher projects on the white board a list of about ten specific questions Saahil could address. The teacher happily offers his help. Saahil gives the teacher a silent nod and looks back at his computer. (7/22/11) We might suppose that with the list of questions and the offering of expert help, Saahil would pursue an aim to evaluate the questions he had been mulling over all summer, narrow his focus, and take up a more concrete position. After all, in conversations with the researcher, Saahil states that opinions on global warming could change, and on the Schommer Epistemological Questionnaire, he provides multiple answers suggesting that he is very certain that scientists can find truthful answers to almost anything if they try hard enough. But at this point in the activity, Saahil has two days remaining to work on a documentary that is still two minutes short. Saahil, in his role as student needing to finish the assignment on time, prioritizes getting the film to 8 minutes (the assigned length repeatedly enforced by the teacher in whole class discourse), which Saahil manages with a documentary exactly 8 minutes and 1 second in length. The final documentary does not incorporate any of the feedback from the neighboring teacher. Saahil eschews the experts concern about his imprecise question and uses the time instead to add an entertaining video on recycling to fulfill the 8-minute requirement. At the public screening, when several students and the teacher question the jarring ending, Saahil, somewhat exasperated, exclaims, I didnt know how to end it! In conclusion, Saahil reasonably adopts the epistemic aim of remain tied to past knowledge claims and deploys the epistemic action of pretend to understand but actually ignore. His decision may also have reflected that the neighboring teacher belongs to a different community, that the iMovie tool requires large time investments to modify past edits, or that the concepts were beyond his expertise. In summary, Saahils role as student satisfying task requirements, the separate communities encountered, and the inefficient iMovie tool all non-epistemic features of the activity systemmay have constrained his opportunity to take up an evaluative epistemic aim (e.g. thoroughly investigate information sources about global warming), even when an opportunity falls into his lap.
Student and authority

Second, we examine one case in which the involvement of authority figures impacts the epistemic aim of critiquing knowledge claims. The example involves Jake, who discovers to [his] surprise that his parents disagree over whether violent video games lead to violent real world behavior. Jake himself wavers between multiple approaches to evaluating his peers, his parents, and experts views on the causal link. At one point, he argues that everyones views are just beliefs There are no right answers. Yet he also expresses a personal commitment to the position that the two variables are correlated but not causally linked, citing his personal experience and explaining, A friend said pretty much the same thing, a friend who, is a very intelligent person. He also concedes that the dominant media activist who supports the opposing causal position, Jack Thompson, is credible, trying his hardest and credible, as long as [he has] the research. In his assignment on
ISLS
313
ICLS2012
discussing oppositional viewpoints, Jake explains that, Jack Thompson does not have the elbow room to lie about anything. Finally, Jake suggests that he feels open critiquing [his] moms [causal] view even though he says she is an expert because she works with children. These multiple excerpts, which took place over two weeks in interactions with the researcher and in written assignments, suggest that Jake can adopt a variety of approaches to dealing with the causal claim: knowledge as coming partly from personal experience (in his own and his Moms case), knowledge as resulting partly from hard work (in Jack Thompsons case), and knowledge as coming partly from intelligent others (in the peers case). What might we conclude is Jakes epistemic aim in these interactions? Though he appears committed to the position that only an association exists between violent video games and real-world violence, and in fact bases his documentary on promoting this argument, he also hedges the strength of this claim in conversations with the researcher and in his written assignments (e.g. There are no right answers). Jake has entered a complex epistemic and social space. He is reluctant to make the strong claim that Jack Thompson is wrong in some settings but not in others. For example, he critiques Thompson in his final documentary, where he includes multiple cartoon strips criticizing Thompson and even includes the phrase, in small print appearing briefly at the bottom of the screen, Former Attorney NEEDS TO DIE I WILL GIVE ANYBODY WHO KILLS HIM MY SOUL. Even though the latter comment is likely a joke targeting his peers or even an absurdist commentary on the link between video games and real world violence, we see a tendency in Jake to exhibit his disagreement with Thompson. The variation in his criticism of Thompson suggests an epistemic aim that teeters between understanding, criticizing, or wishing away anothers opinion depending on the target discourse participants (peers or teacher/researcher).
Teacher and student

The third context we examine is the student-teacher interaction. In many instances in the present data set, the teacher directs the whole-class discourse from an authoritative position, while the activities the teacher asks students to engage in are largely geared toward critiquing and evaluating sources of knowledge. Accordingly, the teacher demands from an authority position e.g. through phrases such as Look for, Dont just, I want you to, Back to work, Dont push me today that students become evaluators, critiquing the sources and methods of science claims in the media e.g. through phrases such as Show either similarities or differences between media coverage and What makes a source of information good, reliable, trustworthy? These opposing spaces in the classroom discourse essentially divide the student into at least two roles. One role requires the student to defer the selection of many goals and time investments to an authority, while in the next moment, another role demands the student stand up to authority in the media, to parents, and peers. Some testy interactions occurred in the classroom as students grew accustomed to having voice and independence in some roles but not in others. One exemplary moment is noted in the teachers reflective journal. The teacher had taken control of the students computers and locked them to keep students focused on the whole class activity. Multiple students criticized the teachers move: This is like how some governments shut off the Internet at certain times to watch only government programming, It is really ironic, Youre being hypocritical. The teacher, in turn, wrote in his journal that, I knew I was asking these students to insert themselves into a really unusual space, encouraging them to exercise their freedom and voice, while still adhering to my authority as the teacher asking them to complete assignments. The event sheds light on how what we typically consider non-epistemic features of social relationships the power of the teacher to control top-down the students access to resources may spill over into the power, voice, and emotions students tap into in their roles as evaluators in the science activity. The whole-class discourse context, in which the teacher adopts the role of an authority, essentially clashes with the epistemic dimensions of the activity itself, which encourages students to adopt evaluator roles, and the students clearly pick up on that contradiction.
Student and peers

In the final interaction space, we examine discourse around knowledge that occurs in relationships between students and peers. In the example, a student named Manish is arguing with his peers and we find that a personal connection to the content impacts Manishs epistemic aim. In the interaction, Manish takes up an argument with his peers about the popularity of a soccer video game that he loves to play. His peers argue that a first-person shooter game is the most popular game and they turn to the Internet to find statistics about which game has sold more copies. It turns out that the first-person shooter game is the highest selling game at that time. Manish challenges the source of the statistic and adamantly states that it is not enough proof. Whereas Manish trusts statistics about smoking (his topic for his documentary film), he rejects the quantitative evidence in the video game context. Manish seems to have adopted the epistemic aim of defending a personal conviction (love for soccer, which he expresses by wearing soccer jerseys to class and watching soccer games online during class time) and is willing to deploy the necessary actions to achieve that aim, questioning the source of empirical data, even if that action may not be called for elsewhere. Other non-epistemic variables could in principle have influenced the adoption of Manishs epistemic aim. For example, Manish may have wanted to win an argument
ISLS
314
ICLS2012
with a peer, antagonize another student, or avoid working on his own documentary. If none of these factors were relevant, Manish could have had the pure epistemic aim of discovering the truth of which game sold more copies.
Discussion and significance

This study attempted to document how some variables mediate the adoption of epistemic aims and actions in multiple contexts within the classroom, taking into account non-epistemic factors such as social role, task constraints, personal affiliations with topics, and discourse participants. Our analysis shows that the full activity system in and beyond classrooms interacts meaningfully with the core epistemic goals of curricular tasks themselves. The present methodology, which involves examining discourse between students, peers, and teachers and multiple artifacts produced along the way, provides a cross-context angle on how some nonepistemic variables mediate epistemic cognition. We described several contextual variables that appear to drive the formulation of epistemic aims and the consequent selection of epistemic actions: constraints that arise from adopting a student role to work through the activity itself, discourse participants that enable or constrain opportunities to critique authority, dimensions of teacher authority compatible or conflicting with the epistemic aims of class tasks, and personal affiliations with a topic. We see that epistemic aims, in part, reflect values established in students roles and relationships in the social world, such that an epistemic aim (e.g. trying to understand or trying to critique) forms because it serves as an avenue to accomplish a much broader goal potentially unrelated to knowledge. To what extent does repeated acquiescence to the demands of teachers when it comes to prioritizing certain class goals and activities carry over into students willingness to confront authority figures such as science experts interviewed in the media when students evaluate public science topics? How do students balance competing goals between maintaining an evaluator role and dealing with the constraints both longterm factors and minutia that arise in the task environment itself? In future research, we should investigate how students negotiate the overarching epistemic goals of an activity with the pragmatic constraints and affordances of the full activity system. We would expect epistemic cognition around a particular domain in a particular context is as much a product of the design of the learning space including how members of the space treat each other and how the teacher negotiates norms and task goals with students as a product of the knowledge itself. In this way, the factors that lead to the deployment of epistemic actions include non-epistemic variables. As these variables change, students enacted epistemic practices tap into different parts of the their state space for engaging with knowledge. By making the opportunity available to observe whether epistemic cognition is situated, we found that the challenge of working with students to become evaluators of science in the media may involve a much larger initiative: teaching students how to respond to situations in which they have reduced power and voice, constraints on time, and social relationships in which participants have arrived at different conclusions.
References
Bromme, R. (2005). Thinking and Knowing About Knowledge Activity and Sign. In M. H. G. Hoffmann, J. Lenhard & F. Seeger (Eds.), (pp. 191-201): Springer US. Chinn, C. A., Buckland, L. A., & Samarapungavan, A. L. A. (2011). Expanding the Dimensions of Epistemic Cognition: Arguments From Philosophy and Psychology. Educational Psychologist, 46(3), 141-167. Cole, M. & Engestrm, Y. (1993). A cultural-historical approach to distributed cognition. In G. Salomon, ed. Distributed cognitions. New York: Cambridge University Press. Hammer, D., & Elby, A. (2002). On the form of a personal epistemology. In B. K. Hofer & P. R. Pintrich (Eds.), Personal epistemology: The psychology of beliefs about knowledge and knowing. (pp. 169-190): Mahwah, NJ, US: Lawrence Erlbaum Associates Publishers. Hammer, D., Elby, A., Scherr, R. E., & Redish, E. (2005). Resources, Framing, and Transfer. In J. P. Mestre (Ed.), Transfer of learning from a modern multidisciplinary perspective: Information Age Publishing Inc. Hofer, B. K., & Pintrich, P. R. (1997). The Development of Epistemological Theories: Beliefs About Knowledge and Knowing and Their Relation to Learning. Review of Educational Research, 67(1), 88 Perry, W. G., Jr. (1970). Forms of intellectual and ethical development in the college years: Oxford, England: Holt, Rinehart & Winston. Sandoval, W. A. (2005). Understanding students' practical epistemologies and their influence on learning through inquiry. Science Education, 89, 634-656. Schommer, M. (1990). Effects of beliefs about the nature of knowledge on comprehension. Journal of Educational Psychology, 82(3), 498-504.
ISLS
315
ICLS2012
Improving Middle School Students Understanding of Core Science Ideas Using Coherent Curriculum
Joseph Krajcik, Sung-Youn Choi, Michigan State University, 237 Erickson Hall, East Lansing, MI Krajcik@msu.edu, choisung@msu.edu Namsoo Shin, LeeAnn M. Sutherland, University of Michigan, 610 E. University Ave., Ann Arbor, MI namsoo@umich.edu, lsutherl@umich.edu Abstract: This study explores the influence of coherent curriculum on student understanding of core chemistry ideas across the middle grades. Participants in urban and suburban schools used materials designed based on what is known about promoting student learning of science, and that link ideas within and across grade levels in a coherent manner. Using a crosssectional approach, student learning was assessed using an independent measure aligned with a learning progression on the transformations of matter. To develop the learning progression and corresponding assessment, we used construct-center design. Using IRT and analysis of variance, we demonstrate that students latent ability changes across grade level, illustrating that curriculum coherence can, indeed, support students in building understanding of core chemistry ideas. We also found that boys and girls did not perform differently, in contrast with other findings, particularly in urban U.S. middle schools, in which girls outperform boys.
Introduction and Rationale

Research shows that students: 1) demonstrate little understanding of the core ideas of chemistry, 2) lack interdisciplinary connections among key ideas, 3) fail to apply core ideas to understand chemical phenomena, and 4) do not use scientific reasoning and practices (Johnson 2000; Nakhleh, & Samarapungavan 1999).Because research illustrates that many students learn ideas superficially and that such limited understanding is not likely to prepare students for the world in which they live now and will live as adults, deeper understanding of the core ideas of science becomes instrumental for problem-solving and decision making in a competitive global economy, as well as for the personal well being of each citizen. Derived from research, our premise is that students need to develop integrated understanding of core ideas to actively contribute to and to use 21st century knowledge, practices, and skillsto problem-solve and to make decisions. To help students build integrated understanding, one promising approach is developing and using coherent curriculum materials, designed such that ideas build upon one another over time and across contexts such that richer understanding and greater connectedness among those ideas result (Margel, Eylon & Scherz, 2008; Roseman, Linn & Koppal, 2008; Shwartz, Weizman, Fortus, Krajcik, & Reiser, 2008). Curriculum coherencepresenting a complete set of interrelated ideas and making connections among them explicit (Roseman, Linn, & Koppal, 2008)has been identified as a critical predictor of student achievement(Schmidt, Wang, & McKnight, 2005). Unfortunately, most traditional science textbooks and instruction in the U.S. do not support deep, integrated learning due to a lack of coherence (Kesidou & Roseman, 2002). Using learning-goals driven design (Krajcik, McNeill, &Reiser, 2008), we developed a middle school science curriculum for grades 6-8 with coherence as a central design tenet. Investigating and Questioning our World through Science and Technology (IQWST) (Krajcik, Reiser, Sutherland & Fortus, 2011) is a projectbased curriculum comprised of biology, chemistry, physics, and Earth science units. The effects of using coherent curriculum on learning remain largely untested in the U.S. due to a dearth of coherent curricula and of appropriate measures to assess learning. We have researched use of the materials in classroom contexts to examine how the linking of ideas in a coherent manner supports student learning of core science content.
Research questions
Few studies report the result of using a coherent curriculum in the classroom. This situation arises because most curricula do not link content through and across years. The purpose of this study was to examine whether the three units in the chemistry strand, which link content, build understanding of targeted chemistry ideas, and to examine students performance by gender, school, and teacher. As such, the primary research question was: When students experience coherent curriculum materials, does understanding develop across time? We predicted that continuous growth inunderstanding would result as studentsexperienced learning tasks and theirideas became more integrated using coherent materialsacross the middle school years. We also asked if students performance differ by gender, school of teacher.
Methodology
IQWSTs coherence was accomplished by using research literature to identify clusters of science ideas that interrelate and build[ing] an instructional sequence to foster more complex understandings over time
ISLS
316
ICLS2012
(Shwartz, et. al, 2008).This approach matches closely the ideas presented in the Framework for K12 Science Education (NRC, 2011), in that both stress development of core ideas blended with scientific practices across time. To measure the coherence of the curriculum materials, we developed a learning progression and corresponding assessment items, independent of the curriculum materials, for twocore ideas: Structure and Interactions.Learning Progressions may be thought of as a sequence of ways of reasoning about a set of ideas, a sequence successively more complex as learners move from less understanding to more sophisticated understanding as they engage in new learning experiences (NRC, 2008; Smith, et al., 2006). Learning progressions (LPs) provide a developmental perspective on helping students learn. This study focuses on the chemistry sequence across 6th, 7th and 8th grades. Each unit takes approximately 10 weeks to complete. We designed each unit iteratively, following cycles of choosing standards to address, creating standards-based learning performances, developing materials, and collecting and analyzing student data, as well as incorporating feedback from teachers and students during and following classroom enactment (Shwartz, et. al, 2008). We blended content standards with inquiry standards to develop learning performances that specify what we expected students to know and be able to do. We then created learning tasks to support students in meeting those learning performances. During the feedback stage, we revised materials and sequenced them carefully to increase the curricular coherence among the three units. One essential feature of IQWST is that the learning of science content andscientific practices are coordinated across units within each grade level, and across 6th, 7th, and 8th grades to build coherence (Shwartz, et. al, 2008).Learning core ideas and practices across content areas and time provides students with opportunities to develop, reinforce, and use their understanding on an ongoing basis throughout their three-year experience with IQWST. This enables students to develop understanding that they can call on and apply in newly encountered contexts. Scientific practices include designing investigations; data gathering, organization, and analysis; modeling; and constructing evidence-based explanations. Another important feature is that it is project based (Shwartz et. al, 2008) in that students perform investigations and seek information to find solutions to important and meaningful questions. Each unit uses a driving question (DQ) that anchors content in a context that is meaningful to students. DQs are rich, open-ended questions that use everyday language to connect science learning with authentic interests and curiosities students have about the world. By definition, a driving question must be meaningful, worthwhile, feasible, and real world. As such, the DQ is used to link the various learning tasks and investigations, central to coherence of the materials. In the process of finding solutions to the driving questions, students learn important science content. IQWST also provides students with opportunities to develop literacy in science through extensive reading and writing opportunities.
Student Materials
Instrument Development
Explaining what happens during transformation phenomena (chemical reaction, phase change, expansion and contraction) involves incorporating ideas from topic areas including: the structure of matter, conservation, forces and energy. We iteratively built an instrument based onthe learning progression (LP) by considering the sets of ideas that students need in order to explain a range of transformations of matter. To develop the learning progression and corresponding assessment, we followed aconstruct-center design approach (CCD) (Shin, Stevens, & Krajcik, 2011). We carefully unpacked the ideas related to the structure and interaction of matter and examined the learning literature on students development of these ideas. Content experts reviewed this work to determine the validity of the learning progression and assessment items. Their feedback guided revision. As we designed items, we required that students incorporate ideas from multiple content areas, when possible, within a single item, regardless of the level of the LP. We developed assessment items to measure how students applied ideas within and across topics to explain phenomena involving transformations of matter. Items were validated through rounds of internal and external review, followed by pilot studies. To ensure that students interpreted the questions and representations as intended, each item was accompanied by a set of questions that probed student understanding of the item (DeBoer, et al., 2008). Each item was piloted with students from grades 6-14. Semistructured interviews with a subset of students supplemented the pilot test data. To measure the entire construct (LP) and to ensure that we had a reliable instrument, we generated a sufficient number of items to develop 8 different forms. Based on analysis of pilot data, we developed four test forms, each containing 14 items for 6th and 7th grade, and four test forms, each containing 15 items, for 8th grade. Each test form contained three pairs of linking items for a total of six linking items. The different formswere used to adjust the overall difficulty of the tests. We eliminated items with a discrimination value of less than 0.30 unless we could identify a problem with the clarity and/or wording of the question or one of the possible answers. Assessment items can be considered distal (Ruiz-Primo, et al., 2002) and can serve as an independent measure of curriculum effectiveness, as they were designed to measure studentslevel on the LP and did not consider whether thoseideas were represented in the particular curriculum or how the curriculum sequenced those ideas.Because the items on the various forms were developed independently of the curriculum,
ISLS
317
ICLS2012
we were interested in determining how students performed on items that aligned with IQWST learning goals and those that did not. We classified each item either aligned or not aligned with IQWST. We then ran Multidimensional IRT analysis by treating each item group differently. This analysis indicates that IQWST students performed similarly on IQWST-aligned items and non-IQWST aligned items. The distributions of items are located between -4.2 and 2.9 logits. These findings indicate a wide range in the difficulty level that covered easy-to- difficult items that are appropriate for measuring 6th to 8th grade students chemistry idea.
Participants and Data Collection

Data are reported from 6 schools that used IQWST materials during national field trials but completed the assessments after their participation in field trials had concluded. Not all the schools involved in this study took part in the national field trials. Two of the schools represent middle- to upper-middle class socioeconomic status, and 4 of the schools were from neighborhoods that are lower-SES, with a majority of students eligible for free and reduced lunch.We administered the items in the fall, 2011.Seventeen teachers took part in the study; several taught more than one grade level. A total of 1519 students took part, with 481 students in 6th grade,who had only begun to study IQWST; 519 students in 7th grade, who had studied IQWST for one year, and 519 students in 8th grade, whohad studied IQWST for two years. The study design is cross-sectional. .
Results
We estimated students' latent ability parameter based on item response theory (IRT, McNamara, 1996) analysis. Students latent ability is considered a continuous latent variable using the partial credit model (PCM)(Masters, 1982). The raw score, which is the number of items correct, does not present a broad picture of test performance because it can be interpreted only in terms of a particular set of test questions. Latent student ability allows direct comparisons of student performance between specific sets of test items.Items were evaluated in terms of the goodness of fit (chi-square=9,487, df=120, p<.001). This shows that the items have different difficulty levels and that IRT is the appropriate method of analysis. First, we used fit index (using MNSQ, T-value >.40) and discrimination (r <.20) to identify poor items. The index provides information about whether items discriminate or not. In the IRT output, the mean square (MNSQ) of fit indices indicate the extent to which the item fits the item response model. The MNSQ, T-value, and discrimination are also good showing that the items are a valid measure. Second, we discussed items in face-to-face meetings to decide how to revise or to eliminate the items from future use of the instrument and further analysis. Table 1. Mean and S.D. by grades Grade levels 6th 7th 8th (* p<.001) N 481 519 519 Mean 454 503 540 (S.D) (84 ) (94 ) (102 ) Mean diff. (7th-6th) (8th-7th) (8th-6th) 49* 37* 86* Effect size 58 % 39 % 102 %
To facilitate interpretation, we determined students latent ability parameter to ability score to have an average score of 500 points and a standard deviation of 100(1),with about two-thirds of students scoring between 400 and 600 points. To answer the main research question regarding differences among the grade levels, we calculated whether there was a mean difference of students latent ability parameter to ability score to determine whether students improved across the years. We also calculated the effect size by dividing the mean differences by standard deviation of the lower grade (see Table 1). Significant differences were found between 6 7th, 7th 8th and 6th 8th grade(F=107, p<.001). From 6th to 7th grade, students' ability is improved by 58% of 6th grade standard deviation. The greatest difference was between 6th and 8th graders. Between 8th and 7th graders, the least difference occurred. We speculated that this occurred because the instrument contained few assessment items that focused on the chemical transformations, the focus of the 7th grade curriculum that would influence students performance on the fall, 8th grade assessment. Overall, however, these results support our hypothesis that students improve across grade levels. Figure 1 shows students ability score in a graphical representation, illustrating how students developed in their ability to respond to the chemistry items from 6th through 8th grade. The graph shows a shift in student ability score. Most students ability ranged from 300 to 750 with mean of 500. In 6th grade, most students were located from 300 to 600 with a mean of 454. In the 7th grade, the range was from 350 to 750, and 300 to 800 in 8th grade. The variance of 8th grade students (102 ability score) is wider than others. We speculated that this wider variance occurred because although the 7th grade curriculum supported students in understanding chemical transformations, the fall 8th grade assessment forms did not focus on chemical transformations. To explore the if there is a relationship in learning due to learning, we calculated the mean difference in students ability score to determine whether difference existed between genders. Table 2 shows the number of
ISLS
318
ICLS2012
students, mean, standard deviation, mean difference and t value for each category (male or female). There was no significant gender difference (t=1.39, p=.166). This is an important finding as it shows that IQWST supports student learning regardless of gender. Typically, particularly in urban schools, females outperform males (Geier, et al., 2008).
Number of Students
80 60 40 20
180 210 240 270 300 330 360 390 420 450 480 510 540 570 600 630 660 690 720 750 780 810 840
6th grade 7th grade 8th grade
Student ability score
Figure 1. Distribution of students by grades Table 2. Mean and S.D. by gender Gender Male Female Total N 710 761 1,471 Mean 505 497 500 (S.D.) (102) (79) (100) Mean diff. (B-G) -8 t 1.39 p 0. 166
To compare the effect of teacher, gender and school to students' ability score by each grade, we conducted ANOVA. The Table 3 shows the effect size of teacher, gender, and school by each grade. Overall, the results show that differences in teacher (F=35.3, p<.001) and school (F=52.5, p<.001) exist and that these tend to increase by grade. The differences in the teacher variable results in 16.2 standard deviations in 6th grade (F=16.5, p<.001), 17.5 in 7th (F=14.8, p<.001), and 21.2 in 8th (F=18.4, p<.001). The differences associated with school variable makes 16.4 standard deviation in 6th grade (F=19.8, p<.001), 16.6 in 7th (F=21.7, p<.001), and 21.5 in 8th (F=29.4, p<.001). These results indicate that teacher variables such as teaching strategies, teaching experience, or student-teacher relationships; and school characteristics such as environmental supports, study hours, or school culture may impact students' ability scores. Table 3. Effect size of variables on students' ability score by grade 6th grade 16.2% * 0.4% 16.4% * 7th grade 17.5% * 0.9% 16.6% * 8th grade 21.2% * -0.1% 21.5% * Total 26.5% * 0.1% 14.5% *
Teacher effect Gender effect School effect (* p<.001)
Discussion
Our work focuses on coherence around core ideas of science that arefoundational across science disciplines (i.e., biology, Earth science, physics, and chemistry) and important for all citizens to understand in a science- and technology-based society. These core ideas were the focus across IQWST units. Our hope in conducting this work was to illustrate that when coherence is built into curriculum materials, students subsequently can build integrated and deep understandings over time. Additionally, it was our hope that this study would provide support for the idea that curriculum coherence may increase student achievement and support students in building integrated understandings of core chemistry ideas over time. To test this, we administered assessment items developed and aligned with a learning progression for the transformation of matter. In developing the items, the sequence of ideas in IQWST was not considered. As such, these assessment items serve as an independent measure of student understanding (i.e., they are not linked to the curriculum), and can be considered distal from the curriculum and instruction specific to the IQWST classroom. Fourprimaryfindings resulted from this analysis. First, we demonstrate that students latent ability changes across grade level, supporting the hypothesisthat coherent curriculum materials can support students in developing integrated understanding, as evidenced by participants developing understanding of core chemistry ideas. This finding illustratesthe value of designing coherent curriculum that supportsstudents, over time,in building understanding of core ideas. It is important to realize, however, that the IQWST curriculum materials also made use of other important features, such as driving questions, to connect unit activities. Second, boys and girls did not perform differently. This differs from previous findings. Previous research has shown that the
ISLS 319
ICLS2012
science achievement of boys in urban schools often lags behind that of girls (Geier, et. al, 2008). Our findingsillustrate that the design features of IQWST, including coherence and the use ofdriving questions, supports both boys and girls in understanding core ideas of chemistry. Athird finding is that IQWST students performed similarly on items aligned to IQWST learning goals (e.g., particle model) and onitemsnot aligned to IQWST learning goals (e.g., electrical forces at the molecular level). It may be that using IQWST enabledstudents to read and have a general sense of the appropriate response by eliminating non-appropriate responses on the assessments. With a focus on core ideas and practices, the curriculum materials align closely with the new Framework (NRC, 2011). As such, we would hope to see this positiveeffect. A fourth finding is that teachers and school variables influence student achievement. The results providefeedback forboth the design of the curriculum materials and the design of learning progressions.The study has two keylimitations. First, we used a cross-sectional design rather than tracking students longitudinally. We are, therefore, continuing to collect data from the 6th and 7thgrade cohort in this study for future analysis. Second, teachersprofessional development experience varied. Because IQWST takes an approach to teaching and learning that is new for many, more thorough professional development and support mighthave led teachers to enact the materials with greater fidelity, which would influence outcomes. Despite these limitations, the results point to the importance of using coherent curriculum materials to support student learning.
Endnotes
(1) The ability scores t2 were normalized so that the mean and S.D of scores was 500 and 100 using the formula: , where t1 is the latent students ability parameter, and m1 and SD1 are its mean and S.D.
References
DeBoer, G.E., Herrmann Abell, C.F., & Gogos, A.,(2007, March-April). Assessment Linked to Science Learning Goals: Probing Student Thinking During Item Development. Paper presented at the NARST Annual Conference, New Orleans, LA. Geier, R., Blumenfeld, P., Marx, R., Krajcik, J., Fishman, B., &Soloway, E. (2008). Standardized Test Outcomes for Students Engaged in Inquiry-Based Science Curriculum in the Context of Urban Reform. Journal of Research in Science Teaching, 45(8), 922-939. Johnson, P. (2000). Children's understanding of substances: Part 1. Recognizing chemical change. International Journal of Science Education, 22, 719737. Kesidou, S., &Roseman, J. (2002). How well do middle school science programs measure up? Findings from Project 2061s curriculum review. Journal of Research in Science Teaching, 39, 522549. Krajcik, J., McNeill, K. L., Reiser, B., (2008). Learning-Goals-Driven Design Model:Developing Curriculum Materials that Align with National Standards and IncorporateProject-Based Pedagogy. Science Education, 92(1), 1-32. Krajcik, J., Reiser, B., Sutherland, L., & Fortus, D. (2011), IQWST: Investigatingand questioning our world through science and technology, (Middle School Science Curriculum Materials). Sangari Global Education/Active Science, USA. McNamara, T. (1996). Measuring second language performance. Addison Wesley Longman. Margel, H., Eylon, B., &Scherz, Z. (2008). A longitudinal study of junior high school students conceptions of the structure of materials. International Journal of Science Education, 45, 132-152. Masters, G. N. (1982). A Rasch model for partial credit scoring.Psychometrika, 47, 149-174. Nakhleh, M.B. & Samarapungavan, A. (1999). Elementary school children's beliefs about matter.Journal of Research in Science Teaching, 36, 777805. National Research Council, 2011.Framework for K-12 science education: Practices, cross-cutting concepts, and core ideas. H. Quinn, H. Schweingruber, and T. Keller (Eds.). Washington DC: National Academies Press. Roseman, J. E., Linn, M. C., &Koppal, M. (2008). Characterizing curriculum coherence. In Y. Kali, M. C. Linn & J. E. Roseman (Eds.).Designing coherent science education. Implications for curriculum, instruction, and policy (pp. 13-36). New York, NY: Teachers College Press. Ruiz-Primo, M. A., Shavelson, R. J., Hamilton, L., & Klein, S. (2002). On the Evaluation of Systemic Science Education Reform: Searching for Instructional Sensitivity. Journal of Research in Science Teaching, 39(5), 369-393. Schmidt, W. H., Wang, H. C., & McKnight, C. C. (2005). Curriculum coherence: An examination of U.S. mathematics and science content standards from an international perspective. Journal of Curriculum Studies, 37(5), 525-559. Shin, N., Stevens, S. Y., & Krajcik, J. (2011).Using Construct-Centered Design As ASystematic Approach for Tracking Student Learning Over Time. Taylor & Francis Group, London. Shwartz, Y., Weizman, A., Fortus, D., Krajcik, J., &Reiser, B. (2008). The IQWSTexperience: Using coherence as a design principle for a middle school sciencecurriculum. The Elementary School Journal, 109 (2), 199-219. Smith, C. L., Wiser, M., Anderson, C. W., & Krajcik, J. (2006). Implications of research on children's learning for standards and assessment: A proposed learning progression for matter and the atomic-molecular theory. Measurement: Interdisciplinary Research & Perspective, 4(1), 1-98.
ISLS
320
ICLS2012
Physical Activity Data Use by Technoathletes: Examples of Collection, Inscription, and Identification
Victor R. Lee & Joel Drake, Department of Instructional Technology and Learning Sciences, Utah State University, 2830 Old Main Hill, Logan UT 84322-2830, USA, victor.lee@usu.edu, joel.drake@sdl.usu.edu Abstract: The proliferation of physical activity data monitoring devices had led to an increase in technoathletesindividuals who combine athletic training and performance with the collection and evaluation of personally-relevant data in an effort to better understand their own abilities. We interviewed 20 technoathletes who were actively involved within either cycling or running communities. Qualitative vignettes of technoathletic engagement with data and the practice of data logging, in specific, are discussed and illustrated. Individual relationships that technoathletes have with their data are also examined. Through the examples, we highlight some commonalities in the data that were obtained and how various athletes represented that information. We also consider some of the tensions that technoathletes have with respect to the data they can obtain and how they saw themselves in light of their data and consider some implications for instruction.
Introduction
A distinguishing characteristic of research in the Learning Sciences has been a continual interest in how individuals develop proficiencies and become active participants in authentic practices. When researchers began to appreciate that cognition and learning were both inherently situated phenomena (Brown, Collins, & Duguid, 1989; Greeno, 1998; Greeno, Collins, & Resnick, 1996), they took notice of studies that showed the nuances of how people reasoned and used the associated disciplinary knowledge in real-world settings. For example, everyday mathematics research has illustrated nuances of computational processes of dairy work (Suchman, 1987), grocery shopping (Lave, Murtaugh, & de la Rocha, 1984), candy sales (Carraher, Carraher, & Schliemann, 1985; Saxe, 1988), basketball (Nasir, 2000), and the work of high status professionals (Hall, Lehrer, Lucas, & Schauble, 2004; Stevens & Hall, 1998). Considered together, this body of research related to everyday knowledge and everyday practices has helped to illustrate for us the breadth of prior understandings that individuals (often, but not always, children) possess. As learning scientists, we often maintain the beliefs that those everyday understandings should be harnessed and developed through instruction, or that they should serve to help us identify practices that can be targeted through instruction. Over the past few years, we have been involved in a research and development program that involves identifying data gathering tools and technologies developed for use by athletes and fitness enthusiasts and integrating them into K-12 instruction (Lee & DuMont, 2010; Lee & Thomas, 2011). These tools, which we refer to as Physical Activity Data (PAD) technologies, piqued our interest in part because they allow for flexible use in a variety of activity contexts. We also appreciated their ability to collect large volumes of data very quickly that could be re-represented in a number of visualized formats. However, our work thus far has only considered the use of these technologies within designed instruction. In the present paper, we are interested in better understanding their use in actual practice.
Theoretical perspective
In following with the work described above related to everyday and situated cognition, we are concerned largely with how people engage in the authentic ordinary practices (p. 34, Brown, et al., 1989) of athletic cultures. Furthermore, we consider ourselves as inquiring into and documenting routines within athletic communities of that embrace technoathleticisma strand of athletic activity that we see as melding physical training and exercise routines with the practices associated with collection and examination of information taken from data gathering technologies. For this work, we take seriously the notion that activities such as bicycling or running could be central and defining for particular athletic communities of practice (Wenger, 1998). An effort to understand these communities would require gathering reports and artifacts from informants who have maintained sustained involvement and participation, as well as records of knowledge sharing through sanctioned organizations and social groups. Therefore, thinking of technoathletes as residing within and as part of athletic communities of practice strongly influences our methodological approach. Part of our work here is to understand data collection, inscriptions, and identities that are formed within these communities, including potential regularities with respect to technoathletes use of data. We anticipated regularities with respect to how data are aggregated, organized, and represented. Furthermore, we hypothesized that it is during these acts of reflection and inspection that meanings were assembled in alignment with physical activity data such that new understandings about performance or selves could emerge (e.g., Author, 2010; Nemirovsky, 2011).
ISLS
321
ICLS2012
Data Sources and Analysis

Were we to follow some of the models of existing research related to everyday cognition (e.g., Wenger, 1991) then a long-term, ethnographic approach would have been ideal. However, because of the advanced skill level of many of our participants and the nature of their activities, which often involved major endurance activities over long distances (e.g., competitive marathon running, two hundred mile bicycle races), we opted to limit our data collection to interviews. We recruited 20 adult technoathletes to meet with a researcher for roughly one hour. These technoathletes were recruited through local sports equipment stores, running and cycling clubs, and word of mouth. All of the participants resided in and participated in athletic activities in Utah, a state in the US Rocky Mountain region known in part for abundant access to outdoor and endurance activities. The participants ranged in age from early-20s to mid-60s. In total, we recruited eleven technoathletes who identified primarily as bicyclists, eight who identified primarily as runners, and one who identified strongly with both activities. Ten males and ten females participated in this study. Most of the participants were college-educated. Prior to the hour-long videorecorded interviews, we arranged for the individuals to bring and share any devices or materials related to their use of physical activity data. During interviews, we asked these individuals to share their history leading to their involvement in their selected sport (running and/or bicycling). We then reviewed the PAD materials the individual brought. We also provided an internet-connected computer for any individuals who wished to show web services they used for tracking activity data. We also posed questions to these individuals about mechanical systems and comprehension of speed data, which we will report upon in future work. For this paper, we only report on the individuals described participation in their sports and their collection and use of physical activity data. After collecting the interviews, we examined the video records and any collected artifacts (i.e., computer files or training documents) and catalogued the technologies that these technoatheletes used, the data that they collected, and the manners in which they reported that they related to their data. For example, we documented instances of data organization and then examined if and when data collection was in service of training or a means of gaining personal insight. We identified themes salient to a subset of interviewees then engaged in a process of progressive hypothesis refinement (Engle, Conant, & Greeno, 2007Engle, Conant, & Greeno, 2007). As we broadened our examination to the entire corpus, we identified several representative transcript excerpts and subsequently subjected them to repeated video review and interpretation. In this paper, we present brief excerptsa collection of short case vignettesto illustrate some of our findings thus far.
Findings and examples

In this section, we present excerpts from interviews that illustrate some of the understandings and practices that our sample of technoathletes reported. However, to help orient the reader, we first provide some additional background about what initially attracted these individuals to their respective sports. The individuals in our sample typically established and maintained active participation in their sport for any of three, non-exclusive reasons. First, the individual had followed an established trajectory from earlier in their lives. Typically, these athletes had participated in an organized sport in high school or college and were already physically active; they sought to continue an active lifestyle as an adult through running or cycling. Another reason for involvement was that the individual was introduced to their current sport through a personal contact, such as a friend, family member, or neighbor. Half of the sample reported that a personal friend or acquaintance influenced their involvement. This influence varied in levels of intimacy. For some, it took the form of coworkers who suggested commuting to work together on bicycles. For others, it involved the desire to spend more time with significant others who were already active in the sport. Finally, some individuals reported becoming involved in their sport due to a perceived medical necessity. All of the participants participated in athletic community-sanctioned activities, such as annual races or riding/running groups that met weekly or monthly. All had also adopted some form of PAD technology. A few individuals simply used their iPods or smartphones to track their running time and distance, often with the aid of training or recording apps that could record splits or use GPS technology to map their trails. One individual used only an Omron pedometer in her training and daily exercise. Runners who had been participated in formal competitions often used watch-based device suites to track speed and distance. They placed a premium on speed and distance information. For cyclists, the device of choice was some form of bike computer, manufactured by companies such as Garmin or Blackburn, which could be attached to their bicycle to track speed, distance, and cadence. All but one cyclist in our sample owned or had used some form of cycling computer. Across both groups, there was a mixed opinion about heart rate monitors, as heart rate is understood as being strongly associated with specific training philosophies. Sixteen of the twenty participants (80%) reported experiences with data logging. These technoathletes either used web services that stored their information or created and maintained personal logs of their runs or rides. Sometimes these were in the service of a training regimen for a planned competition, but for others, it was simply a habitual activity to learn more about themselves. Below we present some examples to illustrate the design and use of these logs as tools for extracting meaning about physical activities and performance.
ISLS
322
ICLS2012
Additionally, we heard from these individuals about variable relationships they had with the data they collected. In some cases, the collection of data allowed them to view their capabilities or performance in a new light. In others, the data became a source of irritation that they sometimes had to separate from in order to enjoy their sports. Examples of both are provided in the sections below.
Physical Activity Data Logging: Collecting and Inscribing

Creating a wide range of inscriptions with data from physical activities was a common practice among technoathletes. Those who had only begun to use PAD devices in the past few years tended take advantage of software tools that were bundled with their PAD devices and simply upload their data into those systems. These tools could generate graphs, maps, and tables. Others, like Thomas (a cyclist in his 40s who had been riding competitively for several years), began with a detailed tabular logging system but later came to use simpler means. Thomas simply used a free wall calendar, which he filled in each day with route and distance information related to his cycling. Thomas did not consider himself an active user of these data. Rather, logging was simply a habitual activity for him that he continued throughout his years of cycling. However, even though he did not consider the logging as central to his cycling activity, he did maintain years of logs and reported occasions when he would still use them to make self-evaluations. T: There's times I've gone out and, especially at the beginning of a year and rode and thought oh man I'm just not up to snuff, or else I think I'm doing extra good but if I can go back and say well okay last year, you know, I was about in this range, about normal or I'm ahead or behind or whatever. It kind of helps me gauge where I'm at, you know, knowledge is power, I guess. And so I kind of always know where I'm at. For Thomas, the calendar and the data he kept in it served as an important artifact to help him recognize what was normal for him, particularly at the beginning of the training season when his most recent ride memories were of rides at his seasonal performance peak. It was a highly personalized evaluative tool against which he could compare himself relative to different times in the year. Another example of log use came from David, a competitive marathon runner in his 50s who used GPS technology and a heart rate monitor. He maintained two spreadsheet logs, one for his race times and the other for automatically recording his training runs. In his race log file, he noted the date, the name of the run, the type (10K, relay, marathon, etc.), his time, pace, placement in his division, and his placement overall. He also calculated and recorded his percentile rank relative to all other participants in the race. David also frequently recorded whether or not his time would qualify him for the Boston Marathon. Though he had no immediate plans to compete in the Boston Marathon, he viewed a qualifying time to be a notable achievement for a given race. As he described it, Boston was seen as an important venue within the distance running community, even though David was only participating in and had logged races in the Rocky Mountain area. Aside from using his log to evaluate his performance relative to peers and against an accepted community standard (Boston qualification times), David also used his log as a tool for setting his performance expectations. As it happened, our interview with him fell on the day before a local half-marathon. Int: Well, based on this, or whatever youve been doing, what do you think your Run the Rockies [the upcoming race] is going to turn out? D: The half [marathon] tomorrow? See, I Im probably, Im think around 1:35 [hh:mm]. Int: Whats making you think 1:35? D: Well, it was 1:38 last year [referencing spreadsheet], its a different course. Im not as, motivated for it as I am Arbor City [another race in his log] which is coming up in 3-4 weeks. Im more interested in racing the Arbor City half. I run this one because I like the course. I like it, I grew up here. But its more, I put it on the schedule because its a training run for the Dixie [another race]. So probably 1:35. Thats what I think. David felt comfortable making a forecast of his performance, in part by basing his estimates on his performance the previous year, his knowledge that the course was mapped differently this year, and his perceived interest. He also factored in his pace. In his race log, his pace (minute per mile) ranged from 7:08 to 8:31 across all of his races. For the current race year, his pace range tended to be between 6:42 and 7:45. As it turned out, Davids prediction was fairly accurate. He finished the half marathon in 1:32 the following day. Although the manner of logging for these two individuals differed, both Thomas and David were, like many others in our sample, habitual recorders of data. In the times that we spoke with them, they demonstrated how they each were able to inscribe different features of data and use them to make meaningful comparisons about themselves and reasonable forecasts for what they could accomplish.
ISLS
323
ICLS2012
Relationships with Data: Tensions and Realizations

In the examples discussed so far, there was a tendency to use data to aid in self-assessment. These individuals could comfortably use their data to situate their performance relative to the sporting season and to make predictions. However, technoathletes also used data to make judgments about their athleticism. In exploring these judgments further, we saw that for several individuals, this was also met with a tension between appreciation for and a general weariness with the data they collected. While valued, data were also seen as disruptive, as described by Tanner, a male cyclist in his 20s. T: When I started cycling earlier onI thought Oh, wouldnt that be cool to track my speed and know how fast I can go and track my miles, so I put a bike computer on both my bikes. I had it off within a week. I found that just having the numbers there, what Id end up doing was just be like, looking at my computer more than paying attention to what I was doing. It wasnt a danger thing, I was still paying attention where I needed to, but I felt like it robbed my experience from everything around me. So, yeah, I dont like having numbers facing me or beeping at me during a ride. Tanner was a bicycle enthusiast who was very knowledgeable about optimal bicycle components and loved the speed (especially downhill) he could reach while riding. But the data and their omnipresence were a distraction to him. As he described it, he felt that the technology robbed him of the experience. His solution was to stick his bike computer in a bag and bring it while riding. He was very interested in seeing some of the information, like his maximum speed, afterwards, but he did not want to see nor hear it while he was out enjoying his rides. This was a sentiment expressed in a number of ways by others. Times and speeds quantified experiences and were used to compare with biking or running friends and thus establish status within their respective training or competition groups. These tendencies to consider status, however, detracted from a more visceral enjoyment that they also associated with their sports. Yet it is important to note that nearly all users of PAD devices also expressed some positive aspects of their data, even when it was presented in the midst of their activity. Stacy, a woman in her mid-30s who struggled with a family history of obesity, had a personal anecdote that illustrated how data layered into her exercise helped change her self-perceptions in a very affirming way. S: There was this hill and this incline, and I would always be so out of breath when I got to that, and Id keep telling myself, Youre just being a wimp. You just need to buck up and do it. But every single time, Id start up on it, and then Id end up stopping and walking because it was so hard! And I just thought of myself as being a wimpand just not being tough and getting out there and run, you know? And so I kinda had a low opinion of myselfAnd then when I got this heart rate monitor, and I threw it on one day, and I was out running and all of a sudden, I get to that same point and I was about to walk, and I did walk, and I glance down at my heart rate monitor and my heart rate was 203! And I went OH! A) I should be walking and B) I am not a wimp! I need to, I need to walk through this. This is absolutely ridiculous to have it be that highAnd all of a sudden, it was, Im not a wimp. Im pretty tough if I can get it to that high. As she described it, Stacy was very critical of herself and her ability because she evaluated herself based on whether or not she could maintain a speed and run up a hill. She identified with being a wimp and had a low opinion of herself. But after she got her heart rate monitor and checked the data it was producing during that incline, she realized that by the metrics it was providing, she was working very hard. In fact, she thought the heart rate value was too high, and she was able to reframe her need to walk from being the result of wimpiness to being the result of an accelerated and unsustainable heart rate. Interestingly, after she saw that information, her athletic identity went from being a wimp to being pretty tough based on the heart rate metric that was made visible with her PAD device. Thus, physical activity data can also be a source of appreciation and positive identity development. These data can be used to form judgments about ones abilities, and that can be negative when that is not a desired focus during the activity. On the other hand, they can be a boon when an existing focus is causing distress. The data invite attention to certain features of a physical activity that are not made visible when doing the activity independent of any technology. As individuals who are interested in harnessing the technology to support learning, we believe that these and other anecdotes we heard from adult technoathletes ultimately highlight the importance of considering how learners might think about their own athleticism when they are given a chance to reflect on such personally-relevant information.
Summary and Instructional Implications
ISLS
324
ICLS2012
In this paper, we reported briefly how some technoathletes used PAD devices and the associated data that were produced. Logging practices were quite common among technoathletes, but there were still some individual nuances with respect to the media and forms in which data were represented and the ways in which those logging systems could be used. We also reported on how individuals felt, both positively and critically, about their activity data. The technologies they used were serving not only to create records, but also to make visible and prominent information about their experiences in ways that could be burdensome or redeeming. As one of our goals has been to understand authentic physical activity data use to inform our research and design efforts, the cases and examples here give us some ideas for ways in which PAD technology might be used in designed learning environments. Technoathletes frequently reflect individual data over time and that longitudinal slice might be one important way for students to think about physical activity data. This suggests PAD use in classrooms might require extended engagement and a large amount of sampling so that such individual trends can be obtained and considered. In our previous work, we were actually far more conservative with respect to how much time we asked students to use PAD technologies and tended to encourage students to look at data gathered from multiple people rather than just themselves. It might be more appropriate to create situations in which individual students can passively acquire individual data over several weeks and then encourage or design activities that require students to examine how they feel the data reflect upon them. Additionally, our conversations with technoathletes suggests that careful attention should be given to how data and technology can influence how individuals views of themselves and their experiences. We often engage in physical activities for the sheer enjoyment of being active and being mobile in the physical world. It is possible that turning such activities into occasions of data reflection, a major goal in our work, can detract or affect the enjoyment that we experience when we simply focus on the positive experiences associated with mobility. On the flip side, it may also be possible we can help individuals who do not think they can enjoy and engage in such activities and give them a means to understand and appreciate what they are capable of doing and experiencing. We look forward to exploring these issues more in the future.
References
Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18, 32-42. Carraher, T. N., Carraher, D. W., & Schliemann, A. D. (1985). Mathematics in the streets and in schools. British journal of developmental psychology, 3(21), 21-28. Engle, R. A., Conant, F. R., & Greeno, J. G. (2007). Progressive refinement of hypotheses in video-supported research. In R. Goldman, R. D. Pea, B. Barron & S. J. Derry (Eds.), Video Research in the Learning Sciences. Mahwah, NJ: Elrbaum. Lave, J., Murtaugh, M., & de la Rocha, O. (1984). The dialectic of arithmetic in grocery shopping. In B. Rogoff & J. Lave (Eds.), Everyday cognition: It's development in social context (pp. 67-94). Cambridge, MA: Harvard University Press. Lee, V. R., & DuMont, M. (2010). An exploration into how physical activity data-recording devices could be used in computer-supported data investigations. International Journal of Computers for Mathematical Learning, 15(3), 167-189. doi: 10.1007/s10758-010-9172-8 Lee, V. R., & Thomas, J. M. (2011). Integrating physical activity data technologies into elementary school classrooms. Educational Technology Research and Development, 59(6), 865-884. doi: 10.1007/s11423-011-9210-9 Nasir, N. i. S. (2000). "Points ain't everything": Emergent goals and average and percent understandings in the play of basketball among African American students. Anthropology & Education Quarterly, 31(3), 283-305. Nemirovsky, R. (2011). Episodic feelings and transfer of learning. Journal of the learning sciences, 20(2), 308337. Stevens, R., & Hall, R. (1998). Disciplined perception: Learning to see in technoscience. In M. Lampert & M. L. Blunk (Eds.), Talking mathematics in school: Studies of teaching and learning (pp. 107-149). Cambridge: Cambridge University Press. Suchman, L. A. (1987). Plans and Situated Actions: The Problem of Human-Machine Communication. Cambridge [Cambridgeshire] ; New York: Cambridge University Press. Wenger, E. (1998). Communities of Practice: Learning, Meaning, and Identity. New York: Cambridge University Press.
Acknowledgments
All participants are listed with pseudonyms. This work was supported in part by NSF grant no DRL-1054280. The opinions in this paper are those of the authors and not necessarily of the US National Science Foundation. Ani Aghababyan and Michelle Berry kindly assisted with collection and preparation of these interviews.
ISLS
325
ICLS2012
Locating the Development of Interest: Tools for Studying the Mutual Constitution of Persons and Cultural Practices in Places
William R. Penuel, University of Colorado, UCB 249, Boulder CO 80309, william.penuel@colorado.edu John H. Falk, Lynn D. Dierking, Julie Haun-Frank, Oregon State University, 237 Weniger Hall, Corvallis, OR 97331 falkj@science.oregonstate.edu, dierkinl@science.oregonstate.edu, haunfraj@onid.orst.edu Ben R. Kirshner, Adam J. York, University of Colorado, UCB 249, Boulder CO 80309 ben.kirshner@colorado.edu, adam.j.york@colorado.edu Abstract: In this paper, we propose a practice-based theory and new methods for the study of interest development. The theory incorporates individual psychological perspectives, but it also theorizes the place-based nature of interest development and seeks to give an account of how persons and practices are mutually constituted. It also identifies roles researchers can play in diagnosing inequity and informing efforts to improve coordination and enactment of opportunities to generate and develop interest in particular content areas. We describe how the Synergies project, a longitudinal study of elementary-aged childrens changing science-related pursuits and interests in a single community, is a context for developing this theory. Psychological theories frame the problem of interest development principally from an individual perspective. Hidi and Renningers (2006) four-phase model of interest development, for example, articulates a potential trajectory from situational interest in a topic toward a well-developed individual interest. They define the latter as a relatively enduring predisposition to reengage with particular classes of content over time (p. 115) that is accompanied by positive feelings toward engagement and supported by a strong knowledge base in the topic. Their own and others measures of interests, moreover, focus principally on individuals: they include self-report measures designed to locate students level of interest within the four-phase model (e.g., Linnenbrink-Garcia et al., 2010) and case studies that combine interviews with direct observations (e.g., Renninger & Hidi, 2002). Though psychological theories tend to focus on individual experience, these theories do assign a prominent role for social processes and context in interest development. Psychologists who study interest note that at all phases, interest can be generated, supported, and sustained by others (Hulleman, Durik, Schweigart, & Harackiewicz, 2008; Renninger & Hidi, 2002). The nature, depth, and longevity of interest, psychologists have observed, depend in part on favorable environmental factors, including social support (Nolen, 2007). Adults including teachers, parents, and unrelated adult guidesplay an important role in supporting interest development by brokering access to opportunities to pursue interest and creating opportunities for rich social interaction around specific topics (Pressick-Kilborn & Walker, 2002). Peers and siblings also are significant in supporting interest development (Azmitia & Hesser, 1993) What psychological theories tend to miss, however, is how individuals access to social supports for developing particular kinds of interest varies from place to place in ways that are consequential for, and constitutive of individuals developing interests. Interest develops within particular ecologies of support (Barron, 2006), and within what Azevedo (2011) calls interest relationships embedded in networks of practices. Within the institutionalized social practices of communities, Azevedo (2011) argues, individuals construct and pursue lines of practice in which their participation varies over time, and in which they participate with others to constitute these very practices. By foregrounding interest relationships within lines of practice, argues Azevedo (2011), the central problematic of research on interest development becomes to trace how the joint contributions of individuals and community practices within particular institutional contexts support persistent participation in elective pursuits (see also Pressick-Kilborn & Walker, 2002). In this reformulation of interest development research, one important question becomes how peoples participation simultaneously reproduces and changes those lines of practice. When the focus is on schooling practices, changes to those practices may be barely visible, and the timescales of radical change are long (Lemke, 2000). But lines of practice in communities associated with out-of-schooltime offerings, programs and services for youth and with voluntary activities, such as sports teams, may transform more frequently the opportunities they provide for interest development (e.g., Falk & Needham, 2011). For example, when sciencerelated museum exhibitions or library programs change in a community, they may introduce new topics and opportunities for families to engage with science, which may or may not develop into more enduring interests. As young people vote with their feet by choosing to engage in some activities and not others, they shape what happens within particular settings for learning (McLaughlin, Irby, & Langman, 1994). Young people involved in the arts and media production, for example, shape activity by their choices of performance and production topics. Even in places where youth have less formal authority to alter lines of practice, the constitution of the practice is still a joint accomplishment with those who design and lead activities (O'Connor & Allen, 2010).
ISLS
326
ICLS2012
This paper addresses the question of how we can theorize the simultaneous development of interest and practices and study it systematically. We offer as a guiding theoretical framework Packers (2010a) proposal for learning research as a reflexive science of constitution that enjoins researchers to take a critical stance of tracing the production of social orders and identities of orderers, as well as an emancipatory stance of seeking to inform concerted, collective efforts to transform practices in ways that expand human freedom. In order to show how such efforts play out in particular places, we describe two approaches we are employing in a study focused on science interest, the Synergies Project.
Theorizing Interest within a Reflexive Science of Constitution

Sociocultural theories of learning (e.g., Rogoff, 1995) and accounts of the microfoundations of institutions (e.g., Powell & Colyvas, 2007) both construe the development of persons and practices as mutually constitutive. That is to say, the forms of life in which we participate make us into certain kinds of people, by the tools and identities that practices make available for us to take up, and we make these practices by taking up, resisting, and adapting identities and tools that are available to them. Neither practices nor persons can be reduced to one another; nor can practices be viewed as factors that shape individual development. Rather, there is a productive tension (Wertsch & Penuel, 1996) between person and practice, one best characterized by viewing them as mutually constitutive. Packer (2010a) identifies three interrelated tasks of an educational researcher seeking to analyze the mutual constitution of persons and practices in a particular place: 1) Describing the order of a form of life; 2) Tracing the work of ordering by which that order is produced and reproduced, and 3) Interpreting the formation of orderers, that is, the identities of participants who carry out the work of ordering.
Describing the Order of a Form of Life

The order of a form of life includes the network of practices, tools, and people that relate to the focal phenomenon, which in our case is the development of interest. In sociocultural studies of the pursuit of hobbies and interests (Azevedo, 2011), this description has taken the form of describing the network of activities and cultural tools that make up communities of practice. These descriptions attempt to characterize the order at a particular juncture in history and time, as if that order is relatively stable, much in the way that Lave & Wenger (1991) represented the communities of practice in their seminal monograph.
Tracing the Work of Ordering

But tracing the work of ordering is a process by which the researcher seeks to explain how this order came to be, as well as how it is changing as people engage in that work. It considers communities of practice not as a container of qualities, but rather as dynamic intersections of people, tools, and activities that can be characterized in terms of flows, networks, and trajectories (Latour, 2005; Leander, Phillips, & Taylor, 2010; Packer, 2010b). An account of interest development, in this context, must take into account the ways that sponsors (Brandt, 1998) of interest development help young people deepen their interest by making it possible for them to move into new settings and help them navigate (Stevens, O'Connor, Garrison, Jocuns, & Amos, 2008) across institutional boundaries. It focuses also on how adults draw on their own networks to broker access (Barron, Martin, Takeuchi, & Fithian, 2009) for young people to adults who may be of assistance in pursuing an emerging interest. And it focuses on how interest develops over time, and across settings, as a function of prolepsis (Cole, 1996), that is, others projections of possible trajectories on young people, and as a function of persistence in participation in lines of practice (Azevedo, 2011).
Interpreting the Formation of Orderers

Interpreting the formation of orderers entails describing in detail the work of persons to construct motivations for their pursuits of particular lines of practice, that is, to develop individual interests in Hidi and Renningers (2006) formulation. This analysis of the production of orderers is likely similar to their case analyses (Renninger & Hidi, 2002) in that it highlights salient dimensions of individual development. However, Azevedos (2011) case study of David, a young person who developed and pursued a long-term interest in model rocketry, offers a better example, in that the case analysis focuses on the constraints and affordances of practices, and how individuals develop interests within the conditions of practice of a particular time and place. Individuals pursuit of these intereststhrough the appropriation of various tools or techniques of the self (Laidlaw, 2002)make them into particular kinds of orderers. Their contributions to practices, in turn, partly reproduce and also partly change the practices themselves, by the ways that they appropriate or make into their own the tools available to participants (de Certeau, 2002).
Realizing Critical Reflexivity

According to Packer, a science of constitution becomes critical when it takes as its primary interest the expansion of human freedom and diagnoses inequity within its account of mutual constitution. Being a critical researcher, however, of forms of life in which one is not a regular participant, is not straightforward.
ISLS
327
ICLS2012
Fieldworkers must continually negotiate access to practices (Harrington, 2003), establishing an acceptable identity within the terms of the form of life in which contact with participants enhances [their] identities (Packer, 2010a, p. 24). Navigating and giving an account of the practices of a community, moreover, requires a critical stance towards ones own theories and practices. That includes representing these theories and practices to others, partly as a form of member checking, but also to subject them to critique and challenge by participants. At the same time, this expression of reflexivity is intended to provide insight that is of use to participants in a particular form of life in the expansion of their own possibilities for action (Habermas, 1990).
Expanding Possibilities for Interest Development in a Community in The Synergies Project

The aim of the Synergies Project is to develop an innovative approach for empowering communities to envision and create a better, more effective community-wide STEM education system. The project, initiated by researchers and funded by the Noyce Foundation, is a collaborative effort among youth, adult community members, and representatives of institutions from a community that is focused on analyzing and improving STEM learning opportunities. The premise of the project is that if we can more fully understand how and why peoplein particular early adolescent children within a poor, underserved communitydevelop (or lose) STEM-related interests, it would be possible to create a more synergistic and effective STEM education system; a system that more successfully supported STEM learning for all. The approach we (the researchers) are employing is to study STEM interest development ecologically in everyday life and across multiple settings while also actively involving members of the community in a collective effort to enhance childrens STEM interest and engagement. In the project, we are employing two participatory research strategies that will enable us to explore and elaborate upon Packers (2010) outlines for a reflexive science of constitution. The first of the strategies is to engage adolescents as key informants and coresearchers with us in the task of addressing the question, When and where are science? (McDermott & Webber, 1998). The second is a community-centered participatory process of generating a theory of change that incorporates preliminary agent-based models as tools for interpreting the complex ways that interest develops across settings and times, and that seeks to answer the question, How does a community organize practices to form identities of youth who vary in their interest in science?
Strategy 1: Youth Participatory Action Research

One strategy we are using within Synergies is to engage adolescents in the neighborhood early on as key informants and as co-researchers. Youth themselves have been shown to be effective informants and coresearchers in collaborative community change initiatives (Kirshner, O'Donoghue, & McLaughlin, 2005). A particularly useful approach is to engage them in ethnographic research, in which they learn to document cultural and community practices through multiple media: writing, photography, interviews, and so on. By so doing, as outsiders to this neighborhood, we as researchers can craft identities for ourselves that are acceptable to participants as providers of an OST enrichment program that is teaching valued skills and providing meaningful experiences to youth in the community. By organizing the activities around the task of describing the organization of practices in which science interests might be generated and cultivated, we also sought to gain insight into the social orders and work of ordering within the neighborhood. Young people traverse multiple spaceson foot, by car, and by busthat allow them access to selected places and experiences. As part of the program activities, they have developed accounts of how lines of practice can emerge within particular settings, with whom, and with what support from adults and peers. In addition, a key idea of this aspect of the project is to broaden participants and our own conceptions of When and where is science? Several of the activities explicitly encourage youth researchers in the project to find science in everyday activities, even activities where the aim might not be science learning. Our goal is to enlist these youth as allies in identifying a broad range of activities within different settings where there are opportunities to develop interest in science. Over time, we imagine young peoples roles to transform into more activist roles within the project by serving as community ambassadors and guides for us and contribute to strategies that the community develops to improve coordination of learning opportunities.
Strategy 2: Participatory Modeling of a Theory of Change

It is typical at the outset of community-based initiatives to engage stakeholders from different institutions and interest groups in a process of articulating a theory of change (Connell & Kubisch, 1999). However, many of the representations of community theories of change may represent context in ways that do not reflect the ways that practices and persons are mutually constitutive. They fail, for example, to capture feedback loops within social systems, which can incline persons and practices under different conditions to develop along divergent trajectories (Fraser, Rademacher, & Sweeney, 2005). Complexity theory provides a framework in which the relationships between constructs at different levels can enables insights into the systems structure about how it operates and evolves (Gell-Mann, 2003). Agent-based modeling (ABM) tools are software programs that can
ISLS
328
ICLS2012
help to make mechanisms such as the kinds of dynamics that characterize complex systems (e.g., feedback loops) visible and thus more accessible (Wilensky & Reisman, 2006). Moreover, such tools also are potentially powerful tools for investigating alternate theories of learning and development, including the outcomes of designed learning environments (Blikstein, Abrahamson, & Wilensky, 2007). In the Synergies project, we are using these agent-based models to facilitate a process with the community of making visible the mechanisms and processes by which different stakeholders believe interestdeveloping opportunities are coordinated within the neighborhood. We will also use it as a way to foster reflexivity within our team by rendering visible some of the processes that social science theory posits are critical for interest development. We are still working out the precise details of how these tools might function in real-time, as part of a fast-paced meeting of stakeholders, but we are hopeful that these tools will be able to render and make useful for purposes of community organizing the conditions for learning, including inequities that lead some young people to develop a strong interest in science and others to become turned off.
How These Synergies Activities Inform Concerted Efforts to Expand Understanding of How to Promote Interest Development in the Community
The two strategies described above will inform two other strategies of the project that are designed to contribute to our collective knowledge of how to promote interest development and build community capacity. First, we plan to use both youths digital stories and the community modeling to inform the design of a longitudinal study of childrens changing participation in lines of practice and their interest development. This multi-method study will include surveys of all fifth-graders living in the community (including those attending public schools, private and parochial schools, and those who are home-schooled), in-depth interviews and collection of artifacts with children and their family members of a focused sample of 50 families, and longitudinal analyses of changes over time in outcomes, patterns of participation, and childrens social networks. At the conclusion of these analyses, we will work with community members to collaboratively design a series of educational interventions with our STEM partners (in school and out) that can then be implemented with additional funds raised prior to Years 3 & 4 and assessed using the ABM and on the ground data collection. We are at the beginning of this particular study, but its design illustrates a way that we can theorize and study interest development in a manner that contrasts sharply with past efforts, which largely develop accounts of interest development that focus on individual psychological processes and social dynamics, without attending to the structure of opportunities in particular places. By asking the questions, When and where are science? and How does a community organize practices to form identities of youth who vary in their interest in science? we aim to move theorizing about interest beyond simple apprenticeship models that presume stable practices and to foreground concerted, collective efforts to improve coordination of opportunities to learn.
References
Azevedo, F. S. (2011). Lines of practice: A practice-centered theory of interest relationships. Cognition and Instruction, 29(2), 147-184. Azmitia, M., & Hesser, J. (1993). Why siblings are important agents of cognitive development: A comparison of siblings and peers. Child Development, 64(2), 430-444. Barron, B. (2006). Interest and self-sustained learning as catalysts of development: A learning ecology perspective. Human Development, 49, 193-224. Barron, B., Martin, C. K., Takeuchi, L., & Fithian, R. (2009). Parents as learning partners in the development of technological fluency. International Journal of Learning and Media, 1(2), 55-77. Blikstein, P., Abrahamson, D., & Wilensky, U. (2007, April). Multi-agent simulation as a tool for investigating cognitive-developmental theory. Paper presented at the Annual Meeting of the American Educational Research Association, Chicago, IL. Brandt, D. (1998). Sponsors of literacy. College Composition and Communication, 49(2), 165-185. Cole, M. (1996). Cultural psychology: A once and future discipline. Cambridge, MA: Harvard University Press. Connell, J. P., & Kubisch, A. C. (1999). Applying a theory of change approach to the evaluation of comprehensive community initiatives: Progress, prospects, and problems. In K. F.-A. A. C. Kubisch, & J. P. Connell (Ed.), New approaches to evaluating community initiatives, Volume 2: Theory, measurement, and analysis. de Certeau, M. (2002). The practice of everyday life. Berkeley, CA: University of California Press. Falk, J. H., & Needham, M. D. (2011). Measuring the impact of a science center on its community. Journal of Research in Science Teaching, 48(1), 1-12. Fraser, E. E., Rademacher, I., & Sweeney, C. A. (2005, 2005, October). Toward the development of an evaluation approach for systems change programs. Paper presented at the Joint Conference of the Canadian Evaluation Society and the American Evaluation Association, Toronto, Ontario. Gell-Mann, M. (2003). Regularities and randomness: Evolving schemata in science and the arts. In J. Casti & A. Karlqvist (Eds.), Art and complexity (pp. 47-58). New York: Elsevier.
ISLS
329
ICLS2012
Habermas, J. (1990). Moral consciousness and communicative action (C. Lenhart & S. W. Nicholson, Trans.). Cambridge, MA: MIT Press. Harrington, B. (2003). The social psychology of access in ethnographic research. Journal of Contemporary Ethnography, 32, 592-626. Hidi, S., & Renninger, K. A. (2006). The four-phase model of interest development. Educational Psychologist, 41(2), 111-127. Hulleman, C., Durik, A. M., Schweigart, S. A., & Harackiewicz, J. M. (2008). Task values, achievement goals, and interest: An integrative analysis. Journal of Educational Psychology, 100, 398-416. Kirshner, B., O'Donoghue, J., & McLaughlin, M. W. (2005). Youth-adult research collaborations: Bringing youth voice to the research process. In J. L. Mahoney, R. W. Larson & J. S. Eccles (Eds.), Organized activities as contexts of development: Extracurricular activities, after-school, and community programs (pp. 131-156). Mahwah, NJ: Erlbaum. Laidlaw, J. (2002). For an anthropology of ethics and freedom. Journal of the Royal Anthropological Institute, 8, 311-332. Latour, B. (2005). Reassembling the social: An introduction to actor-network theory. New York: Oxford University Press. Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge, MA: Harvard University Press. Leander, K. M., Phillips, N. C., & Taylor, K. H. (2010). The changing social spaces of learning: Mapping new mobilities. Review of Research in Education, 34, 329-394. Lemke, J. L. (2000). Across the scales of time: Artifacts, activities, and meanings in ecosocial systems. Mind, Culture, and Activity, 7, 273-290. Linnenbrink-Garcia, L., Durik, A. M., Conley, A. M., Barron, K. E., Tauer, J. M., Karabenick, S. A., & Harackiewicz, J. M. (2010). Measuring situational interest in academic domains. Educational and Psychological Measurement, 70, 647-671. McDermott, R. P., & Webber, V. (1998). When is math or science? In J. G. Greeno & S. V. Goldman (Eds.), Thinking practices in mathematics and science learning (pp. 321-339). Mahwah, NJ: Erlbaum. McLaughlin, M. W., Irby, M. I., & Langman, J. (1994). Urban sanctuaries: Neighborhood organizations in the lives and futures of inner-city youth. San Francisco: Jossey-Bass. Nolen, S. B. (2007). Young children's motivation to read and write: Development in social contexts. Cognition and Instruction, 25, 219-270. O'Connor, K., & Allen, A.-R. (2010). Learning as the organizing of social futures. In W. R. Penuel & K. O'Connor (Eds). Learning research as a human science. National Society for Studies in Education, 109(1), 160-175. Packer, M. J. (2010a). Educational research as a reflexive science of constitution. In W. R. Penuel & K. O'Connor (Eds). Learning research as a human science. National Society for the Study of Education Yearbook, 109(1), 113-127. Packer, M. J. (2010b). The science of qualitative research: Towards a historical ontology. New York: Cambridge University Press. Powell, W. W., & Colyvas, J. A. (2007). Microfoundations of institutional theory. In R. Greenwood (Ed.), Handbook of organizational institutionalism (pp. 276-298). Thousand Oaks, CA: Sage. Pressick-Kilborn, K., & Walker, R. (2002). The social construction of interest in a learning community. In D. M. McInerney & S. Van Etten (Eds.), Research on sociocultural infuences on motivation and learning (pp. 153-182). Greenwich, CT: Information Age Publishing. Renninger, K. A., & Hidi, S. (2002). Student interest and achievement: Developmental issues raised by a case study. In A. Wigfield & J. S. Eccles (Eds.), Development of achievement motivation (pp. 173-195). New York, NY: Academic Press. Rogoff, B. (1995). Observing sociocultural activity on three planes: Participatory appropriation, guided participation, and apprenticeship. In J. V. Wertsch, P. del Rio & A. Alvarez (Eds.), Sociocultural studies of mind (pp. 139-164). Cambridge: Cambridge University Press. Stevens, R., O'Connor, K., Garrison, L., Jocuns, A., & Amos, D. M. (2008). Becoming an engineer: Toward a three dimensional view of engineering learning. Journal of Engineering Education, 97(3), 355-368. Wertsch, J. V., & Penuel, W. R. (1996). The individual-society antinomy revisited: Productive tensions in theories of human development, communication, and education. In D. R. Olson & N. Torrance (Eds.), The handbook of education and human development: New models of learning, teaching and schooling. (pp. 415-433). Oxford, England: Blackwell. Wilensky, U., & Reisman, K. (2006). Thinking like a wolf, a sheep, or a firefly: Learning biology through constructing and testing computational theories. Cognition and Instruction, 24(2), 171-209.
ISLS
330
ICLS2012
What to look for and what to do: Novice teachers abilities for noticing and responding to their students in-class inquiry
Louca, T. L., Skoulia, T., Tzialli, D., Department of Education Sciences, European University, 6, Diogenous Str., Engomi, 1516 Lefkosia, Cyprus, Email: Louca.L@cytanet.com.cy, T.Skoulia@euc.ac.cy, D.Tzialli@euc.ac.cy Abstract: Inquiry-based teaching requires that teachers pay attention to their students ideas and reasoning, and adapt their instruction accordingly. We analyzed 16 80-minute science lessons from 42 pre-service elementary teachers working in groups, in order to investigate teacher noticing and responding (TNR) abilities relating to student inquiry. Findings suggest that the pre-service teachers were able to identify and respond to a variety of aspects of their students inquiry, although we identified disagreements between what they responded to and what we considered important aspects of student inquiry. These findings highlight an ongoing disagreement with prior research, which suggests that teachers' TNR abilities develop with time and teaching experience. Consequently we propose a general need for a better understanding of teacher cognition and development.
Introduction
Scientific literacy includes more than just understanding the concepts of science. Current science education reform efforts, both in Europe and USA, emphasize promoting inquiry in science teaching and learning (EC, 2007; NRC, 2007). By inquiry, we are referring to the activities of students in which they develop knowledge and understanding of scientific ideas, as well as an understanding of how scientists study the natural world (NRC, 1996, p. 23). In order to be productive in promoting student inquiry, science education reform efforts emphasize the creation of classroom environments in which teachers make pedagogical decisions during ongoing instruction. This demands that teachers listen carefully to their students ideas and reasoning, and adapt their instruction accordingly (NRC, 2007; Smith, 1996), thus responding to and supporting student abilities for scientific inquiry. The ability to adapt instruction during teaching requires that teachers are able to notice and respond to aspects of classroom-based student inquiry (Sherin & Han, 2004; van Es & Sherin, 2002). To do so, teachers need to be able to invoke silence, listen to their students ideas and understand their thinking (van Zee & Minstrell, 1997). Moreover they need to indentify elements of scientific inquiry, be able to evaluate them in order to recognize what is important to attend and respond to (van Es & Sherin, 2002), and decide how to proceed based on an evaluation of what they have identified (Louca, Tzialli & Zacharia, in press). We refer to these abilities as teacher noticing and responding (TNR) abilities. Although noticing refers to a large arc of teacher practical reasoning, in this paper we focus on a small portion of these teacher abilities. Thus, we use noticing to refer to teachers abilities to attend to and reason about student inquiry which takes place during instruction. Responding refers to teachers abilities to evaluate in-class student inquiry and decide which teaching strategy is appropriate for supporting it.
Theoretical Framework
Pre-service teacher preparation for inquiry-based teaching in science
Despite NRCs (2007) current emphasis on student-centered instruction in science education, only a few programs have been designed to help pre-service teachers develop TNR abilities in order to respond to their students in-class inquiry (e.g., Louca, Santis, Tzialli, 2010; McDevitt et al., 1995). On the whole, teacher preparation for the teaching of science remains largely teacher-centered, focusing on the things novice teachers (should) do, such as the choice of instructional methods, management strategies, questioning skills, evaluation procedures, and planning processes. Thus, putting much of the emphasis of teacher preparation programs on helping teachers form a teaching identity (e.g., Freese, 2006). This teacher-centered approach of preparing pre-service teachers to teach science has been strongly influenced by stage-based accounts of teacher development (Kagan, 1992), which are based on the ontological premises of stage-based cognitive development. Teacher developmental stages refer to the stages through which teachers progressively gain professional knowledge, abilities and views about/for teaching (Nimmo, 1994). This approach suggests that during initial stages of their development, novice teachers need to construct their teacher self-image by developing routines related to classroom management, lesson design and preparation, and instruction and thus, beginning teachers concerns tend to focus primarily on their self-image as a teacher, rather than on students learning. Consequently, the level of effort and skill required to teach science using inquirybased approaches may not be feasible for many beginning teachers, resulting in a tendency to revert to authoritarian, teacher-directed approaches (Harlen, 1996) in such instances. Although this teacher-centered
ISLS
331
ICLS2012
approach of preparing pre-service teachers means that early in their professional development, teachers focus on their own behavior before they can attend to student learning, it is seen as a crucial step (Kagan, 1992). Once they have resolved an image of themselves, novice teachers can then shift their attention to attending to and responding to their students inquiry. Consequently, efforts to integrate aspects of teaching science as inquiry into their planning and instruction can be frustrating for novice teachers. Recently, a growing body of research has challenged stage-based accounts of teacher development, drawing attention to research data that show novice teachers TNR abilities for student inquiry invoked during early stages of teacher practice and professional development (e.g., Grossman, 1992; Louca, Santis & Tzialli, 2010; Olsen 2007; Zeichner & Gore, 1990). Levin, Hammer and Coffey (2009) provide an extensive review of the challenges to stage-based accounts of teacher development, indicating that this view may be misleading and steer teacher education in science in unproductive directions. They suggest that in a number of studies, preservice teachers were able to reflect on several issues related to the content of teaching (Grossman, 1992), and to attend to the substance of their students thinking (Davis, 2006), although Davis questions pre-service teachers abilities to reflect-in-action. Despite stage-based accounts implying that once classroom routines have been developed teachers focus their attention to student reasoning, Zeichner and Gore (1990) showed otherwise. They argue that novice teachers rapidly shift from progressive, student-centered attitudes formed during preservice training to traditional, teacher-centered approaches when confronted with the realities of the workplace. Furthermore, studies showed that teachers often become satisfied with their teacher-centered approach to teaching and are less likely to question their chosen routines (Grossman, 1992). Finally, Olsen (2007) indicates that such stage-based models focus on implementation and not necessarily on decision-making processes that take place during teaching, and that such work has been conducted primarily with in-service teachers. In agreeing with these challenges, we contend that the ability to attend to students inquiry is a critical aspect of the pedagogical content knowledge (PCK) which novice teachers need to develop as part of their preservice preparation (e.g., Davis & Smithey, 2008). We recognize that helping pre-service teachers develop rich pedagogical content knowledge prior to substantial teaching experience is a difficult task (van Driel, De Jong, & Verloop, 2002). Nevertheless, we agree with Davis and Smitheys suggestion that it is possible to provide preservice teachers with enough experiences to support them in developing PCK readiness that is the requirements for developing abilities to attend to their students inquiry.
Teacher noticing and responding abilities: What to notice and how to respond
Teacher TNR abilities require a number of skills and knowledge from teachers that are or at least should be an integral part of their PCK knowledge. Firstly, in order to be able to identify in-class student inquiry, teachers need to know what inquiry looks like in the classroom settings. This may include knowledge about the different elements of inquiry, both experimental and theoretical (e.g., mechanistic reasoning, analogical reasoning, argumentation, scientific explanations, abilities for the scientific methods etc (Louca & Hammer, 2007), the various levels of complexity of the different aspects of inquiry and their in-class form so that these might be recognized. Then, teachers need to have a well-developed view of inquiry in science and science education, and its function in the development of scientific knowledge, in order to be able to appreciate the importance of diverse elements of scientific inquiry and identify these within classroom discourse. Following the identification and evaluation of student inquiry, teachers need to choose the most appropriate teaching strategy for responding to elements of student inquiry. This requires that teachers have a repertoire of teaching strategies that they can invoke with very little time of reflection. A possible reason for the slow response from pre-service programs to help teachers develop such TNR abilities is the general need for a better understanding of teacher cognition, and more specifically of TNR abilities. Significantly, this has faced considerable challenges due to the on-going and complex nature of teaching (Berliner, 1994; Louca, Santis & Tzialli, 2010; Sandoval, Deneroff, & Franke, 2002; Sherin & Han, 2004). Our purpose in this paper is to investigate pre-service TNR abilities in order to contribute towards this more general need of investigating the development of teacher cognition. Our investigation takes the form of a comparison between what the studys pre-service teachers notice with what we, as researchers, identify as student inquiry in their teachings.
Methodology
As part of their course requirements, 42 pre-service elementary school teachers took a science methods course taught by the first author, and were randomly assigned to 16 groups. Towards the end of the semester, each group of teachers designed an 80-minute lesson plan, based on current trends of inquiry-based teaching and learning in science. They received detailed feedback from the course instructor, re-designed the lesson and resubmitted it for final approval. Then, one teacher from each group taught the lesson to an elementary class of students which was audio-taped and later transcribed. Working as a group, the teachers reflected upon the whole class conversations regarding: (1) what the teacher was responding to in terms of elements of student inquiry, and (2) how s/he responded to those elements. Teachers were not provided with a specific coding scheme to
ISLS
332
ICLS2012
analyze their transcripts. Rather they spent 6 hours watching an exemplary videotaped lesson with the instructor, pausing the tape and analyzing its transcript based on what the teacher was responding to and how. Following findings from similar studies (Sherin & Han, 2004; van Es & Sherin, 2002), and in order to facilitate their analysis, all teachers were provided with a matrix of four columns. The first column was used to identify the specific utterance(s) for each analysis (raw transcript data). The second called for the teachers interpretation of student inquiry (what was the teacher in the transcript responding to?); the third asked teachers to reflect on their evaluation of the particular student inquiry (i.e., what they thought at the time regarding the element of student inquiry they responded to) and the fourth asked for an analysis of the teacher response (how did the teacher respond?). After teachers submitted their assignments, the first and the second authors ran the same analysis based on the coding scheme that we have developed elsewhere (Louca et al, 2008). The scheme differentiates between what the teacher is responding to, concerning students (i) knowledge claims, (ii) reasoning and logic, (iii) everyday experiences, (iv) epistemology, and finally (v) the direction of the conversation. Based on the coding scheme, the teachers responses were identified as either (a) prompts, (b) clarifications (c) evaluations of student ideas or (d) a restatement of student ideas. Then, using the same coding scheme, we assigned a code to all teachers reflections, which then enabled us to compare (1) what the teachers thought they were responding to, with what we (the researchers) saw them as responding to, and (2) how the teachers thought they responded and how we saw them responding. Following our analysis, we compared the researchers coding with the teachers coding (using intercoder reliability test) to identify any significant differences. This comparison was done separately for what the teacher responded to and for how the teachers responded.
Findings
Findings revealed both similarities and differences between what the teachers identified, and what we found, regarding the inquiry elements to which they thought their fellow teacher responded to (see Table 1 for details of these findings). Firstly, all 16 groups were able to identify some of the conversational elements that they responded to as knowledge claims, although 12 groups identified more knowledge claims and 4 groups identified fewer knowledge claims than identified by the researchers. Secondly, 10 groups identified fewer conversational elements categorized as logic and reasoning than the researchers. Five groups reported more elements under logic and reasoning than the researchers and one group reported that the teacher did not respond to any logic and reasoning; however, in all those cases researchers identified some elements that teachers responded to and that fell in this category. Only 2 groups of teachers identified that they had responded to student experiences, whereas researchers found three additional groups that had responded to student experiences. Teachers and researchers agreed that 11 groups did not respond to any student experiences expressed in the lessons. In terms of identifying issues related to the conversational structure/direction, only 3 groups of teachers identified such elements, whereas researchers found an additional eight groups. Teachers and researchers agreed that 5 groups did not respond to any conversational elements related with the structure or the direction of the conversation. Lastly, no group identified any element from the epistemologies category, whereas researchers were able to identify four groups that responded to some such element. The intercoder reliability test between the teachers and researchers coding revealed an overall agreement of 66.44% and a Cohens Kappa of 0.247. Table 1: Crosstab of what pre-service teachers and researchers coded regarding what the teachers identified? STUDENTS Logic & Student Conversation Epistemologies Reasoning experiences 5.8% 9.4% 0.0% 0.6% 0.6% 1.2% 0.0% 0.0% 0.0% 0.0% 1.2% 0.0% 0.0% 1.1% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% (0.0-0.0%)
Knowledge claims Knowledge claims RESEARCHER Logic & Reasoning Student experiences Conversation Epistemologies Total 56.1%1 17.5% 3.5% 1.8% 1.2%
Total 64.3% (48.6-89.2%)2 26.9% (0.0-46.2%) 3.5% (0.0-22.5%) 3.5% (0.0-15.7%) 1.8% (0.0-3.1%) 100.0%
80.1% 16.4% 1.2% 2.3% (36.4-100.0%) (0.0-63.6%) (0.0-28.6%) (0.0-16.7%) 1 Percentages represent the mean percentage based on all the data 2 Parenthesis indicate the range of percentages found in all the data analyzed
ISLS
333
ICLS2012
In terms of how the teachers thought their fellow teacher had responded, findings revealed both similarities and differences between what the teachers identified, and what we found (see Table 2 for details). Specifically, findings were similar in terms of prompts: teachers responded mostly with prompts and less by making clarifications. On the other hand, 13.5% of their responses were identified by the teachers as evaluations of students ideas, whereas researchers identified only 9.1%. Three groups were identified by teachers and researchers as not using evaluation of student ideas or reasoning as a response, whereas agreement was found for only 2 of the groups. Lastly, 11.7% of teacher responses were identified by teachers as restatements of students ideas or reasoning, with 5 groups suggesting that they did not responded to any ideas or reasoning. The intercoder reliability test between the teachers and researchers coding revealed an overall agreement of 64.43% and a Cohens Kappa of 0.445. Table 2: How did the teachers respond? STUDENTS Clarifications Evaluation 5.3% 13.4% 0.0% 3.5% 2.9% 1.8% 7.6% 1.2% Total 53.2% (33.8%-74.2%)2 24.0% (8.3%-45.9%) 9.1% (0.0%-35.7%) 13.5% (4.3%-23.3%) 100.0%
Prompts RESERCHER Prompts Clarifications Evaluation Restatements Total 40.9%1 7.6% 0.6% 3.5%
Restatements 4.1% 1.2% 1.1% 5.3%
52.6% 22.2% 13.5% 11.7% (27.3%-78.6%) (0.0%-64.7%) (0.0%-35.7%) (4.3%-23.3%) 1 Percentages represent the mean percentage based on all the data 2 Parenthesis indicate the range of percentages found in all the data analyzed
Overall, our findings suggest that the pre-service teachers in the study were able to identify a variety of elements of student inquiry and respond to them, although in many cases there were significant differences between what the teachers and the researchers identified. This suggests that the pre-service teachers have some abilities for identifying elements of student inquiry both during teaching and when reflecting on their teaching based on the lesson transcript, although these abilities clearly need to be developed further. Additionally, although there is a variety of elements found in their lessons, the most prominent element of student inquiry both teachers and researchers found in the transcripts was knowledge claims, possibly showing a teacher emphasis on the content and knowledge, rather than on their students reasoning, epistemologies or experiences. In a similar manner, our findings suggest that teachers probably start developing a variety of teaching responses to student inquiry, as early in their teaching experience as those in the study. As the data shows, they may then use these accordingly based on their identification of student inquiry.
Conclusions and Implications

Our purpose in this paper was to investigate pre-service TNR abilities, by comparing what teachers identified as happening in their own audio-taped lessons and how they characterized their responses, with what we identified them noticing and responding to during their teaching. Thus we sought to provide further insight to novice science teachers cognition related to identifying, evaluating and responding to their students inquiry (van Es & Sherin, 2002). Our main argument is that our findings suggest that it is realistic to expect that pre-service teachers are able to attend to their students inquiry during teaching (Louca et al., 2010). This raises an important question related to teacher cognition. Despite views suggesting that pre-service teachers may be incapable of enacting teaching science as inquiry (e.g., McGinnis, Parker, & Graeber, 2004), is it more plausible to view pre-service teachers as being able to notice their students inquiry and respond to it? In this sense, we wonder whether it is not a matter of helping pre-service teachers develop (new) abilities for identifying and responding to student inquiry during teaching but rather, a matter of placing emphasis on helping teachers, in undergraduate methods courses (e.g., Davis & Smithey, 2008; van Driel, De Jong, & Verloop, 2002), further develop abilities they already have in order to recognize students in-class inquiry, as well as to develop a repertoire of teaching responses and the abilities to choose among those based on each in-class inquiry they respond to (e.g., Grossman, 1992; Louca, Santis & Tzialli, 2010; Olsen 2007; Zeichner & Gore, 1990;). Finally, our findings suggest that further, more detailed studies in this area are needed, in order to better understand the differences between what the teachers think they are doing (how they are responding to specific elements of student inquiry), and how they choose which elements of student inquiry to respond to.
ISLS 334
ICLS2012
References
Berliner, D.C. (1994). Expertise: The wonder of exemplary performances. In J.M. Mangier & C.C. Block (Eds.), Creating powerful thinking in teachers and students: Diverse perspectives (pp. 161-186). Fort Worth, TX: Holt, Rinehart, & Winston. Davis, E. A., & Smithey, J. (2008). Beginning Teachers Moving Toward Effective Elementary Science Teaching. Science Education, 93, 45-770. Davis, E. A. (2006). Characterizing productive reflection among preservice elementary teachers: Seeing what matters. Teaching and Teacher Education, 22, 281-301. European Commission (2007). Science Education Now: A renewed Pedagogy for the Future of Europe. A Report of the High-Level Group on Science Education Brussels. Freese, A. R. (2006). Reframing ones teaching: Discovering our teacher selves through reflection and inquiry. Teaching and Teacher Education, 22, 100-119. Grossman, P. L. (1992). Why models matter: An alternate view on professional growth in teaching. Review of Educational Research, 62, 171-179. Harlen, W. (1996) The Teaching of Science (2nd ed.) London: David Fulton. Kagan, D. M. (1992). Professional growth among preservice and beginning teachers. Review of Educational Research, 62, 129-169. Levin, D. M., Hammer, D. & Coffey, J. E. (2009). Novice Teachers' Attention to Student Thinking. Journal of Teacher Education, 60, 142-154. Louca, L. & Hammer, D. (2007). Elementary student nascent abilities for scientific argumentation. In S. Vosniadou, D. Kayser & A. Protopapas (Eds.) The proceedings of EuroCogSci07. The European Cognitive Science Conference 2007 (pp. 47-52). East Sussex, UK: Lawerence Erlbaum Associates. Louca, L. T., Santis, M., & Tzialli, D. (2010). Implementing a Lesson Plan Vs. Attending to student inquiry: The struggle of a kindergarten student- teacher between different frames during teaching science. Paper presented at the International Conference of the Learning Sciences. Louca, T. L., Tzialli, D., & Zacharia, Z. (in press). Identification, Interpretation Evaluation, Response: A framework for analyzing classroom-based teacher discourse in science. Manuscript accepted for publication in International Journal of Science Education. McDevitt, T. M., Troyer, R., Ambrosio, A. L., Heikinen, H. W. & Warren, E. (1995). Evaluating prospective elementary teachers' understanding of science and mathematics in a model preservice program. Journal of Research in Science Teaching, 32(7), 749-775. McGinnis, R. Parker, P., & Graeber, A. (2004). A cultural perspective of the induction of five reform-minded beginning mathematics and science teachers. Journal of Research in Science Teaching, 41, 720747. National Research Council [NRC] (1996). National science education standards. Washington, D.C.: NRC. National Research Council. [NRC] (2007). Taking science to school: Learning and teaching science in grades K-8. Washington, DC: Committee on Science Learning, Kindergarten Through Eighth Grade. Nimmo, G. (1994). The Idiosyncratic Nature of Beginning Teaching: Reaching Clearings by Different Paths. Paper presented at the Annual Meeting of the ATEA, Brisbane, Queensland, Australia. Olsen, J. K. (2007). Preservice Teachers Thinking Within a Research Based Framework: What Informs Decisions? International Journal of Science and Mathematics Education. 5, 49-83. Sandoval, W. A., Deneroff, V., & Franke, M. L. (2002). Teaching, as learning, as inquiry: moving beyond activity in the analysis of teaching practice. New Orleans, LA. Sherin, M. G. & Han, S.Y. (2004). Teacher Learning in the Context of a Video Club. Teacher and Teacher Education, 20, 163-183. Smith, J.P. (1996). Efficacy and teaching mathematics by telling: A challenge for reform. Journal for Research in Mathematics Education, 27(4), 458-477. van Driel, J., De Jong, O., & Verloop, N. (2002). The development of preservice chemistry teachers pedagogical content knowledge. Science Education, 86, 572 590. van Es, E. A., & Sherin, M. G. (2002). Learning to notice: Scaffolding new teachers interpretations of classroom interactions. Journal of Technology & Teacher Education, 10(4 van Zee, E. & Minstrell, J. (1997). Using Questioning to Guide Student Thinking. The Journal of the Learning Sciences, 6(2), 227-269. Zeichner, K., & Gore, J. (1990). Teacher socialization. In W. R. Houston, Handbook of research on teacher education (pp.329-348). New York: Macmillan.
Acknowledgments
The work reported in this paper was supported by a European Commission Coordination and Support Action under FP7, #SIS-CT-2009-234870
ISLS
335
ICLS2012
Developing a Technology-Enhanced Scientific Inquiry Curriculum in a Primary School: Outcomes of the School-Based Support
Yau-yuen YEUNG, Zhihong WAN and Winnie Wing-mui SO, The Hong Kong Institute of Education, 10 Lo Ping Road, Tai Po, N.T., HONG KONG Email: yyyeung@ied.edu.hk, wanzh@ied.edu.hk and wiso@ied.edu.hk Abstract: The current paper reports part of the findings of an ongoing two-year project on infusing digital technology in inquiry activities within General Studies lessons in a Hong Kong primary girls school. Subjects are 124 P5 students and their teachers. Data collected include tests of students knowledge on electricity, surveys of students attitude towards inquiry-based learning and the use of digital technology, as well as their corresponding skills, interviews on students and teachers experience, and observation of lessons. Results showed that students showed improvement in certain cognitive understandings; acquired basic skills in using digital technology in inquiry activities; were confident in using digital technology in their learning; and felt that both inquiry learning and digital technologies can increase their interest in science learning. The present work argues on the necessity of engaging teachers into authentic inquiry activities in conjunction with appropriate digital technologies for effectively developing students science process skills and subject knowledge.
Introduction
Information and Communication Technologies (ICT) have been increasingly introduced into the field of education in the past decades (Deng, Chen, Chai, & Qian, 2011; Roger, 2010). ICT can be used in school science as a research tool (e.g., data logger, spreadsheet, and other software or devices for capturing, processing, and interpreting data), a reference source (e.g., CD-ROMs, internet, and intranet), a means of communication (e.g., Word, interactive whiteboard, and other software or equipment for publishing and presentation, and computer projects), and a means of exploration (e.g., multimedia software for simulation of processes and conducting virtual experiments) (Ball, 2003). Among these applications of ICT for science teaching and learning, the first category is considered the most significant because it can support practical activities or scientific inquiry (Barton, 2004; Osborne & Hennessy, 2003). Although research findings on incorporating ICT into inquiry learning at the secondary school level have been widely reported, empirical evidence on the immersion of digital technologies into inquiry learning in primary schools is still lacking (Murphy, 2003). The current paper reports part of the findings of an ongoing two-year project on infusing digital technologies in inquiry activities within General Studies lessons in a Hong Kong primary girls school.
Why incorporate ICT in science lessons?

One of the commonly mentioned benefits of using ICT is the enhanced engagement of students (e.g., Deaney, Hennessy, & Ruthven, 2006; Osborne & Hennessy, 2003). Static numbers, text, and images are typical in traditional science teaching. When ICT is used in the classroom, movement, graphs, lights, sounds, and colors can all be incorporated in the process of presentation, making such lessons attractive and meaningful for stimulating students active learning. For example, teachers find that when a data logger is used by students in their science practical work, having something on screen acts as a magnet and helps them focus on their work (Deaney et al., 2006, p. 104). Similarly, after reviewing the effect of British government initiatives relevant to ICT on secondary schools, Ofsted (2004) states that the usage of ICT in science lessons can allow students to focus more quickly on the graphical results of their science experiments (p. 41). Aside from the novel and rich forms of presentation, students can generate cleaner and more immediate results from ICT, thereby enhancing students persistence and participation in laboratory activity (Osborne & Hennessy, 2003). Using ICT is also associated with the development of inquiry skills in the process of science learning (Newton, 2000; Weller, 1996). During the process of traditional experiments, the data collected by students may be insufficient or too messy. Moreover, traditional experiments are time-consuming because of the necessity of taking measurements, tabulating data, drawing graphs by hand, and executing multiple or difficult calculations. Software or electronic devices can increase the accuracy of data and reduce the effort that students must exert in collecting and then presenting data graphically (Newton, 2000). Given the increased productivity, students can have more time and energy to explore the existing data in different ways, considering various potential interpretations of the results, and even generate other assumptions to test. All these practices, in turn, contribute the development of inquiry skills. The use of digital devices in the school science laboratory also makes capturing transient phenomena and monitoring variables for a long period possible, thereby expanding the range of questions that can be investigated by students (Osborne & Hennessy, 2003; Singer et al., 2000). With an increased chance of doing science, student inquiry skills can be also improved.
ISLS
336
ICLS2012
The application of ICT has been widely reported to have the potential to enhance students conceptual understanding of science topics, such as force and motion (Tao & Gunstone, 1999), heat and temperature (Reid-Griffin & Carter, 2008), and titrations (Deng, et. al., 2011). One of the explanations for this finding is that, compared with manual pencil-and-paper methods, the graphs immediately generated by a computer can better facilitate an appreciation of the patterns embedded in the data to more efficiently link data obtained during practical work with the abstract ideas and concepts of science (Barton, 2004, p. 32). Moreover, the labor entailed by data collection and graph production is alleviated. Thus, students can focus more attention and time on thinking about the underlying scientific knowledge (Deaney et al., 2006). When ICT is used as a research tool or as a means of exploration, one set of equipment is commonly shared by a group of students. Given the collaborative relationship initiated by the use of ICT, meaningful interaction is established among students, thus facilitating an understanding of science (Mercer, 1994; Tao & Gunstone, 1999).
Research Methodology and Framework

The present study is based on an ongoing two-year project of infusing digital technologies in inquiry activities within General Studies lessons in a top-rank primary girls school, which is financially supported by the Hong Kong Quality Education Fund. The research team from the Hong Kong Institute of Education (HKIEd) is responsible for the school-based support of this project, consisting of (i) running workshops for teachers professional development; (ii) cooperating with teachers to adopt and implement some innovative teaching and learning activities or pedagogies (i.e. test-bed activities); (iii) conducting lesson studies with teachers to reflect on and enhance their practice; and (iv) evaluating students learning effectiveness and attitudes. The project follows a cross-sequential design in the present work. The project started in the 20102011 academic year with Grades 3 and 5. The same students undergo a follow up when upgraded to Grades 4 and 6 in the second academic year, i.e., 20112012. Through this design, the project will be conducted in four grades within two years, i.e., Grades 3 to 6. The topics selected for the infusion of digital technologies in inquiry learning from Grades 3 to Grade 6 are Living in Hong Kong, Sound and Light, and Electricity and Simple Mechanics. The current paper focused on the introduction of Electricity to Grade 5 pupils with the following research questions (after participating in the technology-enhanced scientific inquiry activities in this project): How well do they master basic operational skills of using digital technology in scientific inquiry? Are there any significant changes in students attitudes towards using digital technology in science learning? How well do students develop their cognitive understanding of science topics?
Subjects
All the General Studies teachers and students at Grade 5 participated in the present test-bed activities. There were a total of 124 students (in four classes) and four teachers and all of them are female. Students were engaged in four inquiry activities to investigate a closed circuit, conductor, fruit battery, and electricity generation using a magnet. Each inquiry activity was conducted in one double lesson. When students investigated the closed circuit, a digital camera was used to record and compare the brightness of the bulb. The data logger was used with current sensors to probe the conductor and fruit battery and with a voltage sensor to conduct the experiments in electricity generation using a magnet. To meet the requirement of primary students, a data logger called Addestation Amixer that can work without a computer was selected. Its touch screen feature can show the figure and generate an instant graph. Students can also manipulate the data logger using the touch screen. With reference to the constructive approach of teaching science, students were required to perform a Predict-observe-explain (POE) task (White & Gunstone, 1992) during the process of inquiry activities, in which they need to (i) make a prediction on the result of experiment; (ii) explain their prediction; (iii) conduct an experiment to observe and test their prediction; and (iv) reconcile any discrepancy between their prediction and the observation. In addition, during the lessons, the students cooperated in small groups of four or five. Each class has eight groups.
Data collection
Broadly speaking, four kinds of data were collected, including (i) tests of students scientific knowledge before and after activities, (ii) surveys of students attitude to and skills of doing inquiry and using ICT before and after activities, (iii) interviews of students and teachers experience interviews of students and teachers experience of using ICT in inquiry learning after activities, and (iv) observation of lessons of inquiry learning. No control group can be arranged because of the schools and parents concern or insistence to provide equal learning environment for all students. The third kind of data will be presented and discussed in another full paper. The pre- and post-tests both assess students conceptual understanding on a closed circuit (six items), conductor (10 items), fruit battery (four items), and electricity using a magnet (six items). For the questions on the conductor, fruit battery, and electricity generation using a magnet in the pre-test, students had to judge whether a certain variable can affect the current or voltage whereas they were required in the post-test to decide
ISLS 337
ICLS2012
on whether such an effect occurs as based on their own learning experience. In addition, three additional items in the post-test are used to assess students operational skills in using a data logger. The surveys before and after the inquiry activities both have five corresponding items on experience and attitude towards ICT, eight on skills of inquiry, and five on attitude towards scientific inquiry. The survey after the inquiry activities has an additional item on the relationship between using ICT and interest in learning science. All these items have four options, with 0 indicating strongly disagree and 3 indicating strongly agree. A total of 12 randomly selected students were interviewed after all the activities were completed, with each interview lasting approximately 20 minutes. All of the four P5 teachers attended a half-hour interview individually. A total of 16 double-lessons were video recorded, in which the activities of two randomly selected groups of students were also video recorded in each double lesson.
Data analysis
The students answers to the questions in the knowledge test were coded into three categories, i.e., right answer, wrong answer, and dont know, which were respectively scored 1, -1, and 0. Subsequently, the average of the scores on items of the same topic were calculated and used in the analysis. All the quantitative data were inputted in SPSS 18.0 for statistical analysis. Paired-sample t-test was conducted for the corresponding items in pre & post test and survey, in which average, t value, significant level and Cohens effect size were calculated. For those items that have no corresponding items in pre & post test and survey, descriptive analysis was made to calculate their means. These quantitative data were grouped and then reported in terms of three areas: skills of and attitude towards using digital technology, skills of and attitude towards scientific inquiry, and conceptual understanding of electricity topic. All the interview data were transcribed and coded. They were used to triangulate with the relevant findings from the questionnaire survey. The lessons observed were analyzed on the basis of field notes and video records.
Results
Skills of and attitude toward the use of digital technology
In the post-test, three items assessed students operational skills in using a data logger. An average of 74.37% students gave the correct answers to these questions. As indicated in the table below, after these inquiry activities, students felt that they have significantly more experience in using digital technology in conducting experiments (t=4.21, p<0.000, Cohens d=0.548) and know significantly more about how to use a digital camera to record data (t=5.46, p<0.000, Cohens d=0 .721). Although no significant increase in other three items was observed, students were found to already have very high confidence and a positive attitude toward using digital technology in the process of learning in the pre-survey. A total of 83.6% students agreed or strongly agreed that the use of digital technology in learning science increased their level of interest. Table 1 : Students experience of using digital technology and attitudes toward digital technology Questions I have experience of using digital technology to conduct experiments. I am confident in using digital technology in the process of learning. I know how to use a digital camera to record data. I like using digital technology in the process of learning. I feel that using digital technology in the process of learning is easy. Mean- MeanDif. Pre Post 1.97 2.29 1.93 2.36 2.33 2.4 2.46 2.48 2.38 2.34 .43 .17 .56 .03 .01 t Sig. Cohen's effect size .548 .236 .721 .027 .014
4.21 .000 1.83 .071 5.46 .000 .31 .757 .09 .927
Skills of and attitude towards scientific inquiry

According to students self-evaluation, after participating in these technology-enhanced inquiry activities, a significant increase in skills in terms of raising questions (t=2.04, p<.005, Cohens d=0.220), determining classifications (t=4.35, p<0.000, Cohens d=0.571), and distinguishing the variables in experiments (t=3.30, p<0.0.01, Cohens d=0.404) were observed. On the contrary, students negative perception that doing scientific inquiry makes science learning more difficult decreased significantly (t=-6.194, p<0.000, Cohens d=-0.738). Although the data did not indicate a significant increase in other aspects of attitude toward scientific inquiry, students scores have already been very high in the pre-survey. As the students articulated in the interviews after finishing all the inquiries, they are attracted by these activities (S3). If I read book, I can also learn about electricity, but when I have done the experiments by myself, I can have a clearer picture of how the result is obtained (S7).
ISLS
338
ICLS2012
Table 2: Items with significant increase in students self-evaluation of their own skills of scientific inquiry and their attitude toward scientific inquiry Questions I can raise questions for scientific inquiry. I know how to do the classification. Mean Mean Dif. -Pre -Post 2.03 2.19 .16 2.02 2.37 2.36 .35 .27 t 2.04 4.35 3.30 Sig. .044 .000 .001 .000 Cohen's effect size .220 .571 .404 -.738
I can distinguish the variables in experiments. 2.09 Doing scientific inquiry makes science learning 1.96 more difficult.
1.26 -.71 -6.194
Cognitive understanding on electricity

The pre- and post-tests have been conducted to assess students cognitive understanding of electricity. The result indicates that students attained a significant gain in their understanding of closed circuits. However, as previously mentioned, in the pre-test, students merely had to judge whether a certain factor affects the current or voltage generated for the questions on the conductor, fruit battery, and electricity generation using a magnet. On the contrary, they are required to decide on why such an effect occurs, if students think that it exists, in the post-test. In other words, compared with the corresponding questions in the pre-test, questions are more difficult in post-test. Therefore, a direct comparison of these questions cannot be made. As shown in Table 3, despite the increased difficulty, the development of students understanding on conductors is significant. However, the change in the two other aspects appears insignificant. Table 3: Students conceptual understanding on electricity Topic Closed circuit* Conductor Fruit battery Generating electivity by magnet Mean-Pre -0.057 0.253 0.242 0.229 Mean-Post 0.379 0.679 0.250 0.270
*Dif=0.437; t=8.178; Sig=.000; Cohen's effect size=1.088
Discussions and Conclusion

The present study, to some extent, echoes the argument that using ICT can enhance the engagement of students (Deaney et al., 2006; Osborne & Hennessy, 2003). Through the video recording, students were found to be enjoying their learning activities when they used a digital camera and data logger for scientific inquiry. The interview and survey also gave supporting evidence. We also observed some improvement in students skills in terms of raising questions, determining classifications, and distinguishing the variables in experiments, when digital technology is combined with constructive inquiry learning. These findings may partly support the relationship between the use of ICT and the development of inquiry skills stated in the literature (Newton, 2000; Weller, 1996). However, the survey results indicated that the development of other skills, such as designing experiments, using tables and charts to present findings, and discussing the results of experiments, is limited. Of course, the development of inquiry skills is a long-term endeavor that cannot be achieved within a short period. The literature indicated the potential of the use of ICT in science learning for enhancing students conceptual understanding (e.g., Deng, et. al., 2011; Reid-Griffin & Carter, 2008; Tao & Gunstone, 1999). Such potential may be partly revealed by the students learning of closed circuits and conductors in the present work. However, the levels of difficulty of the questions on the fruit battery and electricity generation using a magnet in the pre- and post-tests are different, so a direct comparison cannot be made. Thus, the absolute value in the post-test is not encouraging. In reality, the inquiry activities on these two topics are significantly more complex than those for the other two topics. When students use a fruit battery or magnet to generate electricity, they need to control numerous variables so that the data collected in different occasions can be comparable. On the contrary, such a requirement is a challenge for primary students. Table 4 and Table 5 showed the data collected by two groups of students in the activities of investigating fruit battery and generating electricity by magnet. These tables were showed by teachers in the blackboard. A considerable difference exists between the first number of the second row and the three other numbers in Table 4. In Table 5, a large variation exists between the first and second rows. The discrepancy within the data may interfere with students understanding of the underlying scientific ideas. Referencing Table 4, a teacher says, if we just focus on the first two digits of the numbers, are they close to each other? Yes, they are Therefore, will size of fruit affect the current generated by fruit battery? No significant effect is found here (T2). Another teacher, referencing Table 5, says, To compare the results, we
ISLS
339
ICLS2012
calculate the average of two groups The average for the triangle coil is 0.0175, and the average for the round coil is 0.014 No big difference (T4). In reality, the difference in the data collected by second group is significant, as shown in Table 4. The trends in the two rows are also different. The difference between the two groups in Table 5 is also very significant. By calculating the average in this situation, some very important information is overlooked. Thus, students should be guided to be more critical of the collected data by obtaining more data and discussing the underlying reasons for the inconsistency, rather than quickly drawing conclusions. A similar argument is emphasized by Newton (2000), stating that teachers need to teach pupils to notice events and data features and be critical of data (p. 1257). Such activities are especially important for students just starting to do scientific inquiry. Table 4: Data collected by two groups of students when investigating the effect of the size of the same kind of fruit on the current generated by a fruit battery Size of fruit Electricity (Current) Small Big Group 1 Group 2 0.912 0.992 0.932 0.922 Table 5: Data collected by two groups of students investigating the effect of the shape of a coil on the current generated by a magnet Electricity (Voltage) Group 1 Group 2 Shape of Coil Triangle 0.026 0.009 Round 0.024 0.004
In summary, the present study indicates that incorporating ICT in primary science teaching is possible. The lack of a control group is the key limitation of this study. More efforts should be exerted in developing relevant teaching resources and incorporating such resources into the formal curriculum. However, General Studies teachers may not be able to efficiently use the data generated by digital technologies in inquiry activities in developing students scientific ways of thinking. These teachers require more professional development to enhance their relevant teaching skills so that they can help students acquire a better understanding of the nature of inquiry by engaging them in authentic inquiry activities.
References
Ball, S. (2003). ICT that Works. Primary Science Review, 76, 11-13. Barton, R. (2004). Teaching secondary science with ICT. Buckingham: Open University Press. Deaney, R., Hennessy, S., & Ruthven, K. (2006). Teachers strategies for making effective use of datalogging. School Science Review, 88(323), 103-110. Deng, F., Chen, W., Chai, C. S., & Qian, Y. (2011). Constructivist-oriented data-logger activities in Chinese Chemistry classroom: Enhancing students conceptual understanding and their metacognition. The Asia-Pacific Education Researcher, 20(2), 207-221. Mercer, N. (1994). The quality of talk in childrens joint activity at the computer. Journal of Computer Assisted Learning, 10(1), 24-32. Murphy, C. (2003). Literature Review in Primary Science and ICT. A Report for NESTA Futurelab (no. 5) (Bristol, NESTA Futurelab). Newton, L. R. (2000). Data-logging in practical science: research and reality. International Journal of Science Education, 22(12), 1247- 1259. Ofsted [Office for Standards in Education]. (2004). ICT in schools: the impact of government initiatives secondary science. London: Office for Standards in Education. Osborne, J. & Hennessy, S. (2003). Literature review in science education and the role of ICT: Promise, problems and future directions. A Report for NESTA Futurelab (no. 6) (Bristol, NESTA Futurelab). Reid-Griffin, A., & Carter, G. (2008). Uncovering the Potential: The Role of Technologies on Science Learning of Middle School Students. International Journal of Science & Mathematics Education, 6(2), 329-349. Roger, Y. (2010). Enhancing learning: a study of how mobile devices can facilitate sensemaking. Personal and Ubiquitous Computing, 14 (2), 111-124. Singer, J., Marx, R. W., Krajcik, J. & Chambers, J. C. (2000). Constructing extended inquiry projects: curriculum materials for science education reform, Educational Psychologist, 35(3), 165178. Tao, P. K., & Gunstone, R. F. (1999). The process of conceptual change in force and motion during computer-supported physics instruction. Journal of Research in Science Teaching, 36(7), 859-882. Weller, H. G. (1996). Assessing the impact of computer-based learning in science. Journal of Research on Computing in Education, 28(4), 461- 485. White, R.T., & Gunstone, R. F. (1992). Probing understanding. London, UK: Falmer.
Acknowledgements
The financial support from the Hong Kong Quality Education Fund and HKIEd is gratefully acknowledged.
ISLS
340
ICLS2012
Towards Teaching Analytics: Repertory Grids for Formative Assessment

1
Ravi Vatrapu1, Peter Reimann2, and Abid Hussain1 Computational Social Science Laboratory, Copenhagen Business School, Denmark, {vatrapu, ah.itm}@cbs.dk 2 MTO Psychologische Forschung und Beratung, Germany, preimann.undefined@gmail.com Abstract: This paper introduces the Repertory Grid Technique (RGT), developed in personal construct psychology, for Technology Enhanced Formative Assessment (TEFA). Repertory Grid is a method for eliciting personal constructs of learners about elements belonging to the topic of study. The method of Repertory Grid is presented first followed by brief descriptions of two classroom studies and two eye-tracking controlled laboratory studies. Empirical findings contribute to a better understanding of the integration of Repertory Grid into teaching as an in-class learning activity or a take-home exercise, methodological support for teachers to designing and deploying RGT exercises, and computational support for visualizing the Repertory Grid data at the individual student level and whole classroom level for teaching analytics purposes. We outline the design and development of the Repertory Grids for Formative Assessment (RGFA) tool and conclude with directions for future work.
Introduction
Repertory Grid Technique (hereafter RGT) is a method for eliciting personal constructs of individuals about elements belonging to a topic of study. RGT is based on the seminal contribution of psychologist George Kelly (1963, 1992), Personal Construct Theory, and subsequent theoretical and methodological developments (cf. Adams-Webber, 2006; Fransella, Bell, & Bannister, 2003). RGT has been used by both researchers and practitioners in a wide variety of fields including psychotherapy (Winter, 2003), marketing (Frost & Braine, 1967), education (Bell & Harriaugstein, 1990; Mazhindu, 1992), and information systems (Cho & Wright, 2010; Tan & Hunter, 2002). RGT consists of a family of methods and variations involving the nature of the personal construct elicitation and the rating or ranking of elements in monadic, dyadic or triadic configurations (Fransella, et al., 2003). For the purposes of formative assessment, we have decided to start researching RGT with an implementation of the widely adopted method of triadic sorting of elements for personal construct elicitation and subsequent five-point Likert-item rating of the rest of the elements (Fransella, et al., 2003). Briefly put, the triadic sorting method consists of the participants being presented sets of three elements each. For a given set of three elements (e.g, Windows, OSX, Unix), the participant is prompted to select the element (e.g., Unix) that is different from the other two (Windows, OSX) and to state how it is different as the opposite construct (e.g., command line interface). Then, the participant has to state how the two remaining elements in the triad are similar to each other as the similarity construct (e.g., GUI). The rest of the elements (other operating systems, in our example) are then rated on a Likert-item scale ranging from the Opposite Construct (1) to the Similarity Construct (5). The participants repeat this process until all the triads of elements are sorted into different and similar and the elements for that comparison are rated. The outcome of this exercise is the Repertory Grid (RG) consisting of rows with triads, columns consisting of elements with the first column being the Opposite Construct and the last column being the Similarity Construct, and the cell values consisting of the ratings given for elements. Based on the RG, one can qualitatively appraise learners mental modelswhat they see as going together, and on what dimensionsand/or apply clustering methods or dimension reduction methods to derive quantitative measures of learners knowledge structures. The Methodology section next details two cycles of iterative research and development of RGT in a real classroom setting and subsequent eyetracking studies in the laboratory setting. There are three interdependent research and development objectives for the use of Repertory Grid Technique for Technology Enhanced Formative Assessment. (a) integration of Repertory Grid into the curriculum as an in-class learning activity or a take-home exercise, (b) methodological support for teachers to designing and deploying RGT exercises, and (c) computational support for visualizing the Repertory Grid data towards teaching analytics.
Methodology
In order to achieve these three interdependent objectives, we employed educational action research methods (Hartley, 2009) in a real classroom as described below.
Classroom Setting
Classroom setting for the in-class exercises was an undergraduate course Internet Marketing" at the Summer University program (ISUP 2011) of the Copenhagen Business School. Internet Marketing was taught in two sections of about 42 students each for 150 minutes on Tuesdays and Thursdays for five consecutive weeks.
ISLS
341
ICLS2012
Formative Assessment using Repertory Grid Online (FARGO)

Given the pedagogical coupling between the course curriculum and facebook with regard to social media, we decided to start our research and development of RGT for Technology Enhanced Formative Assessment (TEFA) using the facebook platform. A facebook application called Formative Assessment using Repertory Grid Online (FARGO) was developed towards this purpose. FARGO was implemented by Chris Teplovs while working as a Postdoctoral Research Fellow on the NEXT-TELL project at the Computational Social Science Laboratory (CSSL) of the Copenhagen Business School.
Study #1: Repertory Grid Classroom Exercise on Consumer Decision-Making

The topic for the repertory grid exercise was Consumer Decision Making. The eight elements were: car, laptop, beer, water, airline tickets, pair of shoes, pair of jeans , movie tickets. The elements were selected to range from fast moving consumer goods to potentially luxury goods (Veblen goods) and ranging from relatively inexpensive to relatively expensive purchases involving little or great consideration time, and personal taste vs. social influences. The teacher (and the first author) made the design decision to select 10 triads to include in the exercise. The selection criteria for the triads was that each element (i.e., product) should appear at least once and in different positions in the triad (first, second and third) and with as many different elements as possible. Based on the Comparative Method (Ragin, 1987), some triads were selected from the Most Similar Systems Design (MSSD) and other from the Most Different Systems Design (MDSD) perspectives. The order of presentation of the triads was randomized to control for practice effects. The in-class exercise was administered the week before the course module on Online Consumer Psychology consisting of a lecture and a take-home exercise on Decision Heuristics Simulations. After completing the facebook RGT exercise in-class, participants were provided with a repertory grid network diagram visualizing the relationship between the elements based on the ratings provided. Participants had the option of sharing their repertory grid network diagram and the repertory grid table with their classmates and other members of the facebook page for Online Marketing. Observations of the in-class exercise activity and student feedback indicated that ten triadic comparisons and ten sets of element ratings were tedious, tiresome and boring for the students.
Study #2: Repertory Grid Classroom Exercise on Online Marketing Topics

Based on the observations from the Consumer Decision-Making exercise, FARGO 2 implemented the critical design change of reducing the number of triads to five. The second half of the course was organized into special topics on eight different kinds of online marketing. The purpose of this repertory grid exercise was (a) to inquire into students conceptions about these eight different kinds of online marketing and adapt instructional content and delivery accordingly, and (b) to familiarize students of the existence of and the relationships between the eight different kinds of online marketing covered as special topics in the course curriculum. The in-class repertory grid exercise was administered at the midpoint of the course and the week before the special topics were scheduled. Students who didnt complete the exercise in the class were invited for an eye-tracking study in the laboratory.
Study #3: Eye-Tracking Laboratory Study of Repertory Grid Exercise of Online Marketing Topics
All students in the class were invited for an eye-tracking study in the laboratory. Study participation was voluntary and an online study registration form was used to collect students demographic data and availability. Students who didnt complete the Online Marketing Topics in the classroom setting were assigned to complete it in the laboratory setting.
Study #4: Eye-Tracking Laboratory Study of Multiple Representations of Repertory Grid Data for Consumer Decision-Making
Students who completed both the Consumer Decision Making and Online Marketing Topics Repertory Grid classroom exercises were assigned to the study of multiple representations of the Repertory Grid dataset for Consumer Decision-Making.
Results
The following subsections present select findings from the two eye-tracking lab studies on repertory grid exercise task and the teaching analytics task with multiple representations of the repertory grid exercise data.
Study #3: Eye-Tracking Laboratory Study of Repertory Grid Exercise of Online Marketing Topics
Six students (3 female and 3 male) participated in the first eye-tracking laboratory study of the repertory grid exercise with the eight online marketing topics as elements. The lab study exercise was identical to the in-class
ISLS
342
ICLS2012
exercise. The objectives of the laboratory study were (a) to investigate the time taken for construct elicitation and the subsequent elements rating for each of the five triads, and (b) to investigate the collective gaze behaviour of participants during the construct elicitation phase and the subsequent elements rating phase for each of the five triads.
Task Time: Construct Elicitation vs. Elements Rating

Except for a couple of instances, time taken for construct elicitation was higher than the time taken to rate the elements. In the two instances where the elements rating time was greater than the construct elicitation time, students had to relate opposite and similarity constructs that were specific to the three elements in the triad to the five other elements. One student commented that he would have chosen different constructs if he could go back. Analysis of the talk aloud and the structured interview data indicates that students spent more time on construct elicitation when one or more of the elements in the triad were unfamiliar to them (like advergaming, augmented reality advertising). No order effects were found.
Gaze Behaviour: Construct Elicitation and Elements Rating

Eye-tracking data analysis was conducted at the aggregate level for each of the five construction elicitation and elements rating tasks. An analysis of the aggregate heatmaps shows that the gaze distribution pattern is fairly similar across the five construction elicitation tasks. Students gaze is primarily allocated to the 3 elements in the triad and the text boxes for the opposite and similarity constructs. For this area of interest (triad radio buttons and construct textboxes), the average time to the first fixation (TTFF) ranges from 0.4 seconds to 1.6 seconds. The average time spent in this area of interest out of the total task time ranges from 24% to 32%. The elements ratings heatmaps show a greater variation in the gaze distribution on the webpage. There are two areas of interest. The first area of interest is the set of pull down list controls for rating the elements and the second area of interest is the elements names. Gaze is also allocated to the scale of the elicited constructs at the top. For the pull down list controls, the first area of interest, the average time to the first fixation (TTFF) ranges from 0.2 seconds to 0.9 seconds. The average time spent in this area of interest out of the total task time ranges from 28% to 43%. The Area of Interest (AOI) analysis was conducted on regions of the webpages that were of particular importance from a pedagogical and/or user interface design perspectives. Three areas of interest (AOI) were defined on the construct elicitations webpage corresponding to the following student actions: Selection of the different element in the triad (three radio button controls followed by the element names); Opposite construct elicitation (text box control); and Similarity construct elicitation (text box control). Results show that, on average, students gaze allocation was higher selection of the different element in the given triad with the opposite and similarity construct text boxes receiving roughly similar gaze allocation. Three Areas of Interest (AOI) were defined for analysis of eye gaze data for the elements ratings webpages. The three AOI were: Likert scale ranging from 1 (opposite construct) to 5 (similarity construct); Names of the eight elements; and Ratings region consisting of pull down list controls ranging from 1 (opposite construct) to 5 (similarity construct).Results show that, on average, only 1% of the gaze for total exposure time was allocated to the Likert scale with the ratings region and the element names receiving 21% to 15% gaze allocation on average. User interface and user experience design implications from the AOI analysis are discussed in the next section.
Study #4: Eye-Tracking Laboratory Study of Multiple Representations of Repertory Grid Data for Consumer Decision-Making
Four female and four male students participated in the second eye-tracking laboratory study. As mentioned earlier, the study consisted of three tasks presented in a random order. Due to space limitations, we only present selected results from the repertory grid dataset uploaded to IBMs Many Eyes website: Dynamic Word Cloud and Treemap. Analysis of gaze data for each of the above representations is presented below:
Dynamic Word Cloud of All Constructs Elicited

The Word Cloud representation is identical to the Static version of the task in terms of the horizontal layout and the alphabetical ordering of the words. The differences were that the Word Cloud had to be composed by the students, supported dynamic interaction and content was exhaustive of the constructs from all the ten triads of the Consumer Decision Making repertory grid exercise for all the students. The task consisted of initial construction of the word cloud, free range viewing subtask followed by word finding subtask. Heatmap analysis shows that the aggregate gaze of the participants was distributed around the most salient items (larger sized words). When compared to the heatmap of the Static Word Cloud, the dynamic visualization has more regions with high aggregate gaze. Area of Interest (AOI) analysis shows that the dynamic v visualization with the horizontal layout with alphabetical ordering of words results in more even gaze distribution. As it is to be expected, the most frequent word (decision) which has the largest size receives greater gaze allocation. A side-by-side Bee Swarm analysis of the Static Word Cloud and the Dynamic Word Cloud shows that unlike the
ISLS
343
ICLS2012
Static case, the initial gaze allocation of the students in the Dynamic case starts at different regions rather than at the largest word. This is mostly due to the fact that the Dynamic Word Cloud had many salient words.
Dynamic Treemap of All Elicited Constructs and Elements Ratings

Treemap visualizations were designed to provide a dashboard view of the entire repertory grid exercise. That is Treemap visualizations can provide a comprehensive view of the both the textual data (elicited constructs) and numeric data (element ratings). As with the Line Graph and Bar Chart visualizations, students were initially instructed to freely explore and interact with the Treemap visualization with subsequent instructions for guided interaction of the functionality of the Treemap visualization. Analysis of the talk-aloud and semi-structured interview data shows that students felt that the Treemaps were the most difficult visualizations to make sense of. Heatmap analysis shows that Treemap region corresponding to the first triad followed by the regions for the second, third and eighth triads accounted for the majority of the aggregate gaze distribution at 38%. Bee Swarm analysis shows that students gaze paths have a preference for the left half of the Treemap visualization. Students gaze paths start at different points of the Treemap visualization with some coalescing in the middle phase of the task.
Discussion
Based on the results reported in the previous section, we think that the repertory grid technique with triadic sorting is highly suited for technology enhanced formative assessment. A carefully designed repertory grid exercise provides insight into students personal constructs on a topic. An empirical finding from the laboratory study is that time taken for construct elicitation and elements ratings could provide another dimension for pedagogical decision-making. Eye-tracking results show that while aggregate gaze distribution varies for the elements rating phase, they remain fairly uniform for the construct elicitation phase. As for the representations of the repertory grid date, eye-tracking results combined with analysis of the verbal protocols and the semistructured interviews show that Word Clouds for constructs (text) and Line Graphs for element ratings (numbers) are effective visualizations. Interactive Treemap visualizations need to be better designed and endusers should be provided with training to comprehend and interact with the dashboard display. Implications for the different stakeholders are presented below.
Implications for Teachers

In designing repertory grid exercises, teachers should pay particular attention to the previous domain knowledge of students and to what extent the elicited constructs are grounded in the personal experience of the students compared to the domain knowledge. An ideal repertory grid exercise would involve 6-10 elements and 5-6 triads with each element appearing at least once and in different positions of the triad when a particular element features more than once across the different triads. The repertory grid exercise could be designed for individual students or as a computer supported collaborative learning (CSCL) exercise involving a small group of students. The pre-test and post-test paradigm could be applied to solicit individual or group repertory grids before and after a particular curriculum module has been taught. Further, the teacher can make his or her own repertory grid available to the students for reflection, and repertory grids of domain experts for benchmarking and guided inquiry. Post repertory grid exercise tasks could include asking the individual students or groups to reflect on their own repertory grids, to inspect the repertory grids of their peers or domain experts, and/or to inspect the visualizations of the repertory grids for the entire class. An additional implication from the classroom exercises and the eye-tracking laboratory studies is that teachers could also learn about students current understanding based on the time taken for construct elicitation and element rating. With regard to formative assessment, teachers can inspect the constructs or the Word Cloud representations of the individual or collective constructs and discern students level of domain knowledge. Similarly, teachers can scrutinize the elements ratings to discern students ability to distinguish between the different concepts. With necessary training, teachers can make use of Treemap or some other visualization of the entire repertory grid exercise to adapt the content and didactics for that particular curriculum module. Apart from the classroom usage scenario, another usage scenario for teachers is to employ the repertory grid exercise as lightweight appraisal method for informal learning tasks. We will research this usage scenario in future work with teachers.
Implications for Students

Repertory grid exercises on topics not familiar to students either from prior formal learning settings or from personal experience seem to be perceived as challenging and engaging. That said, a well-designed repertory grid exercise on the familiar and lived practice would allow students to externalize their implicitly held constructs. Students should then be motivated and guided to reflect on their intuitions and connect their personal constructs to domain concepts. Students should also be able to co-design repertory grid exercises with peers and teachers. Co-designing a repertory grid exercise would require students to select the topic, the elements, and the number, content and order of triads. This in itself could be pedagogically effective. Finally, students should be given the
ISLS
344
ICLS2012
option of sharing their repertory grids with their classmates and within their social networks. Students should be able to interact with their visualizations of their individual repertory grids and those of their peers and the classroom level repertory grid. Moreover, students should be able to upload their repertory grid exercises to their e-portfolios and integrate them with their open learner models.
Implications for Researchers

From a learning sciences research standpoint there are at least two lines of inquiry to pursue. The first line of research is to build on existing work in personal construct psychology in understanding the underlying psychological processes of the repertory grid technique (RGT). The second line of inquiry is into research and development of methodological and computational support for teachers to design and evaluate RGT exercises for formative assessment purposes. Particular attention should be paid to the time on task for construct elicitation and element ratings phases of the RGT exercise in addition to the personal constructs and the ordering of elements on the bipolar scale of the opposite and similarity constructs. Change over time in the repertory grids of students as they progress through curriculum and acquisition and development of professional vision (Goodwin, 1994) for teachers are two important research considerations.
Implications for Design

The findings from the classroom exercises and the eye-tracking laboratory studies were used in the requirements gathering phase for the design and implementation of the software application, Repertory Grid for Formative Assessment (RGFA, http://cssl.cbs.dk/software/rgfa). Release One (R1) of RGFA is designed to provide a simple interface for teachers to design a repertory grid exercise and deploy it to a group of students. RGFA is designed from the scratch to integrate with a wider technology infrastructure developed in the NEXT-TELL project (www.next-tell.eu). RGFA also incorporates proper instrumentation for research purposes and interactive visualization functionalities to be developed and deployed in upcoming releases. RGFA is our first step towards a comprehensive research program on teaching analytics. Teaching Analytics seeks to gather, archive, process, model, analyse, and visualize data from teaching and learning activities in informal and formal learning settings to empower teachers dynamic diagnostic decision-making for formative assessment purposes and for individual and collaborative learning purposes.
Acknowledgements
This work is supported by the NEXT-TELL - Next Generation Teaching, Education and Learning for Life integrated project co-funded by the European Union under the ICT theme of the 7th Framework Programme for R&D (FP7).
References
Adams-Webber, J. (2006). Reviews of A manual for repertory grid technique. Journal of Constructivist Psychology, 19(4), 351-353. doi: 10.1080/13854040600689133 Bell, W., & Harriaugstein, E. S. (1990). The Repertory Grid as a Medium for Investigating Teacher Pupil Perspectives on Educational Software. Computers in Education, 333-338. Cho, V., & Wright, R. (2010). Exploring the evaluation framework of strategic information systems using repertory grid technique: a cognitive perspective from chief information officers. Behaviour & Information Technology, 29(5), 447-457. doi: 10.1080/01449290802121206 Fransella, F., Bell, R., & Bannister, D. (2003). A Manual for Repertory Grid Technique (2 ed.): Wiley. Frost, W. A. K., & Braine, R. L. (1967). Application of the Repertory Grid Technique to Problems in Market Research. Journal of the Market Research Society, 9(3), 161-175. Goodwin, C. (1994). Professional vision. American Anthropologist, 96(3), 606-633. Hartley, J. (2009). Sage handbook of educational action research. British Journal of Educational Technology, 40(5), 966-967. Kelly, G. A. (1963). A theory of personality: W. W. Norton & Company. Kelly, G. A. (1992). The Psychology of Personal Constructs: Volume Two: Clinical Diagnosis and Psychotherapy (New ed.): Routledge. Mazhindu, G. N. (1992). Using Repertory Grid Research Methodology in Nurse Education and Practice - a Critique. Journal of Advanced Nursing, 17(5), 604-608. Ragin, C. (1987). The Comparative Method: Moving Beyond Qualitative and Quantitative Strategies: University of California Press. Tan, F. B., & Hunter, M. G. (2002). The repertory grid technique: A method for the study of cognition in information systems. MIS Quarterly, 26(1), 39-57. Winter, D. A. (2003). Repertory grid technique as a psychotherapy research measure. Psychotherapy Research, 13(1), 25-42. doi: 10.1093/ptr/kpg005
ISLS
345
ICLS2012
The teachers balance between structure and flexibility in the technology-enhanced collaborative inquiry setting
Marjut Viilo, Pirita Seitamaa-Hakkarainen, Department of Teacher Education, University of Helsinki, P.O. Box 8, 00014 University of Helsinki, Finland marjut.viilo@helsinki.fi, pirita.seitamaa-hakkarainen@helsinki.fi Kai Hakkarainen, Department of Education, University of Turku, Assistentinkatu 5, 20014 Turku, Finland, kai.hakkarainen@utu.fi Abstract: The focus of this paper rises from the dilemma created by the promises and demands of students technology-enhanced collaborative inquiry learning on the one hand and lack of competencies in carrying the collective cognitive responsibility of inquiry on the other hand. In this study, we analyze the teachers balancing between structure and flexibility when aiming towards supporting students in taking their own responsibility for collaborative technology-enhanced inquiry practices in an ordinary Finnish elementary-level school. The aim of the Artifact project - the Past, the Present, and the Future was to support students (N=32) understanding of the diversity of artifacts. For our analysis, we selected all the videorecorded episodes that represented teachers and students joint whole class activities. It is concluded that in the joint activities, the responsibility for directing the inquiry varied from collectively developed to teacher directed.
Introduction
The studies from the successful knowledge creation and inquiry-based classrooms have been reporting promising results for the purpose of education in the knowledge age (Zhang, Scardamalia, Reeve & Messina, 2009; Hmelo-Silver et al. 2007). In inquiry and knowledge creation based learning approaches the aim is to engage classroom community in tackling with open-ended complex problems and exploiting collaboration. The aim is to support handling the ideas and knowledge continually improvable and raise the students own ideas and questions at the center of working (Scardamalia 2002). This could be promoted by getting community to define their own learning goals and get them engaged in collective cognitive responsibility of their own process (Scardamalia 2002; Zhang, Scardamalia, Reeve & Messina 2009). However, if the local classroom culture is colored with traditions or passive voices, and the students are expecting the teacher to lead their knowledge acquisition, the implementation of totally new kind of working practices is evident in order to reach collective cognitive responsibility for the advancement of collective knowledge and inquiry. Challenge for the teacher, when directing the classroom from traditional teacher-driven approaches towards shared inquiry aims, is to tolerate openness. When the process and the object of inquiry are designed in collaboration with students, the outcome of the inquiry cannot be fully known beforehand neither the phases of the process nor the content to be studied. In addition, although aiming towards collective responsibility and shared aims, the students should be offered enough support (Hmelo-Silver, Duncan, & Chinn, 2007). The role of the teacher in the classroom is often portrayed either as a transmitter of knowledge or facilitator of learning. Instead of just using dichotomies, we need to understand the diversity of roles that teacher plays within the variety of day-to day activities in a classroom. Simultaneously with expecting students to take responsibility for their own work, they have to be deliberately guided to follow certain rules and standards, such as setting up productive research questions and making plans of inquiry (Olson 2007). In progressively implementing inquiry-learning practices and facilitating self-regulative and collaborative learning activities, the teachers still need to set requirements for the students in their inquiry and set up some procedures and rules how to proceed. According to Olson (2007), only when the students are familiar with the aims and have undertaken the responsibility for the teacher-set inquiry obligations, are they able to set requirements for themselves and then adapt and reshape the rules if needed. Supporting students generation of their own ideas and promoting collaboration within a classroom community is not possible without supporting routines and structures that channel the students activities in a way that elicits participation in inquiry. Having too rigid structures may, however, undermine inquiry efforts altogether. If the teachers prepare or follow strongly scripted curricula or fixed routines, their own creativity and ability to respond to the specific needs of certain classroom are limited (Roth, 2002; Sawyer, 2004). A scripted approach clearly specifies tasks and learning activities, the order and form they should take. In scripted teaching, the teacher is the one who controls the flow of the class, limiting when students can talk and how much impact their statements can have on the collective encounters (Sawyer, 2004). To continue, there are serious concerns that too-strong structuring undermines higher-level objective of inquiry learning because it
ISLS
346
ICLS2012
makes peripheral the pursuit of advancement of the students own ideas (Lakkala, Muukkonen, Paavola, & Hakkarainen, 2008). Several researchers emphasize (Sawyer 2004; Lakkala, & al., 2008) that the most effective classroom interaction will balance structure with flexibility and improvisation. Cazden (2001) advocated eschewing an either-or approach and emphasized that teachers should have a repertoire of lesson structures and teaching styles. Thus, effective teaching gives students freedom to construct their own knowledge, while providing and shifting between carefully chosen elements of structure; these include scaffolds, activity formats or pedagogical frameworks that support the co-constructive process (Sawyer, 2004). The shifts themselves are the teachers improvisational responses to the unique needs of that class and those students (Sawyer, 2004). Consequently, the challenge for the teacher is to orchestrate the inquiry process and practices where the students are utilizing their own ideas and strengthening their own community, and promote sustained, collective, pedagogical settings in which idea improvement is the central focus rather, than specific learning tasks or activities (Zhang, Hong, Scardamalia, Teo, & Morley, 2011).
Aims and scope of the study

This paper focuses on the dilemma created by the promises and demands of students collaborative inquiry learning on the one hand and lack of competencies in carrying the collective cognitive responsibility of inquiry on the other hand. In this study, we analyze the teachers balancing between structure and flexibility when aiming towards supporting students in taking their own responsibility for collaborative technology-enhanced inquiry practices in an ordinary elementary-level school. The research question is: How was the responsibility of directing inquiry distributed between the teacher and students within the different collective activities?
Data collection and analysis

The context for the study was The Artifact Project The Past, the Present and the Future at Laajasalo Comprehensive School, Helsinki, Finland (Seitamaa-Hakkarainen, Viilo, & Hakkarainen, 2010). This intervention has been inspired by the knowledge-building and progressive inquiry approaches which aim at engaging both teachers and pupils in building new knowledge and understanding (Scardamalia, 2002; Hakkarainen, 2003). The project was designed together with the researchers and class teacher, who is familiar with previous approaches and have been implementing them in her class. In the project, 32 students participated. The project started in the beginning of the second term of fourth grade and ended at the end of the fifth grade. The teacher had begun working with this particular class only in the first term of their fourth grade. The students had learned in their earlier school years the teaching and learning activities, where they followed traditional schooling culture with teacher-directed activities and pre-given problems. The aim of the Artifact project was to support students understanding of the diversity of artifacts in their cultural context. The students were asked to analyze artifacts within a cultural context, to study physical phenomena (such as electricity) related to artifacts, and to design future artifacts. In the project, the collaborative learning environment, Knowledge Forum, provided tools for visualizing and building knowledge (http: www.knowledgeforum.com). During the longitudinal project, we collected 56 lessons of video data (i.e. one lesson lasted 45 minutes). The project activities were concentrating on continuous elaboration of ideas and themes within the project frames. The organization of the project broke down the lesson limits so that the projects themes and activities were often continuing through several lessons. In addition, the teacher did not follow basic lesson structures (i.e., teachers lesson introduction, lesson body, and closing), even in several occasions the teacher needed to take care of lesson transitions with introducing the continuing theme or closing the working time. Thus, for the analysis, the video data from Past and Present phase were divided into distinguishable episodes. An episode was considered distinguishable when the aim, the social organization, or the supporting tools (for example use of KF) changed during the activity, or the break recessed the activity. Each collective episode lasted approximately 11 minutes. The Future phase was excluded from the analysis, because the teacher remained background and the professional designer guided the students design process. For the present analysis, we selected all those collective episodes (N=34) where both, the teacher and students, were present. The collective episodes were analyzed from the two perspectives: (1) Responsibility for directing the inquiry, and (2) Type of activity. In the responsibility for directing the inquiry, the episode was classified as a) teacher directed, when the teacher alone directed the strategy and the students were listening, b) teacher promoted when the teacher carried the responsibility for maintaining the inquiry but the whole class participated, and c) collectively developed when the whole class community developed the strategy together. We also analyzed the type of activity in the episode to understand when the collective activities involved either a) planning, or b) reflecting on strategy; or, alternatively when the activity concentrated on c) improving ideas. All the episodes were further divided in events (N=116) according to topics of discussion in the activity. These topics served different functions for advancing the activity and were categorized for a) Discussing content, b) Discussing strategy and deepening inquiry, c) Discussing previous phase, d) Inquiry management and summarizing, e) Teacher guidance, or f) Teacher organization.
ISLS
347
ICLS2012
Results
The results indicate (see Figure 1) that most of the cases where the teacher alone took the responsibility for directing the inquiry were idea improvement episodes. In those episodes the teacher either took care of the lesson transitions or structured the project activities, such as provided short guidelines regarding what they were supposed to be doing or requested to deepen their thinking. In addition, these included a couple of teacherextended questions that the students would not have been able to consider themselves, but the teacher thought would provide a good opportunity to connect. In the teacher directed episodes the topics of discussion were in general concentrating around short teacher organization functions.
Figure 1. Different types of activity (planning, reflecting strategy, and idea improvement) in video recorded episodes according to responsibility for directing inquiry The teacher-promoted collective episodes occurred when the students were not able to take the cognitive responsibility or carry on the strategic directing of the process. In those episodes, the students did not always have tools to participate although the teachers aim was to turn the strategic activity to the students. In some cases, the teacher had a preliminary idea what they could achieve and chose to promote the inquiry further. When planning or reflecting strategy, the teacher, for example, assisted students in implementing the suitable strategy to plan their inquiry (i.e., determine how to use the collected information), or then she promoted students in reflecting the used strategy at meta-level (i.e., what kind of research questions are valuable). It is noteworthy that the teacher needed to promote the discussions when the used strategy was reflected. She needed to socialize the students understanding of how to observe, create questions, or use KF for building on knowledge. In teacher-promoted idea-improvement episodes, they were developing the inquiry ideas further, but the teacher needed to promote the discussion. The teachers support and inducement was, on the one hand, essential for creating discussion; on the other hand, she often had some background idea what they could consider. In the teacher promoted idea improvement episodes, the functions of the different events were concentrating on developing and discussing content together. The talk about content was often preceded with discussions of previous phase represented by Knowledge Forum. The following excerpt is from the longer event where they thought back the issues about how they had made the lamp light. In the excerpt, the teacher creates ground for deepening the inquiry about electricity towards explaining what happens inside the electricity cable. The teacher had a background idea herself how they could continue, and wanted to connect the students present wonderings to the current. Students have not been able to deepen their inquiry themselves, thus the teacher promoted it based on the students KF wonderings: Excerpt 1, Discussing previous phase Teacher: [searching Theos note from KF and opening it] So what was it about the system that, like over here Theo said that the power is conducted. And Theo already talks about how the power circuit doesnt conduct energy if the power circuit isnt closed. What does it mean? What do you mean by that? Theo: It means that plus and minus go into one another. Teacher: Yeah, but where does it go and how? Theo: It goes so that there goes a cable from the minus end to the lamp and from the lamp a cable goes to the plus end. Teacher: So you mean that something happens inside the lamp? Well, what happens in the lamp? Theo: Well, the current goes through it. Teacher: Well how can it go through? What transports it? Theo: Well its there. Its trying to get to the minus end all the time. Teacher: What does it transmit along? Whats inside the cable?
ISLS
348
ICLS2012
Theo: Copper wires. Teacher: Good, now were at the heart of the issue. So theres something inside the wire. The analysis revealed (see figure 1), further, that whole class collectively took responsibility for directing the inquiry when the process needed planning (i.e., they planed how to conduct the project activities, such as creating research plan, or own research questions). These episodes took place usually in the beginning of the new phase of the project. However, there were no collective episodes of idea improvement or reflecting strategy that would have involved collectively shared responsibility for directing inquiry. Idea improvement mainly took place in episodes of team-based and individual working, where the students were carrying the inquiry further by themselves. The topics of discussion in these episodes served discussing strategy and deepening inquiry three times more often than in other episodes. During these discussions, the collective tried to plan how to continue like in the following extract where they were collectively developing the inquiry and planning how to study artifacts: Excerpt 2, Discussing strategy and deepening inquiry Teacher: So these are the ways, for example, how to continue the study further and further. Do we have any other ways to study things, Something else than these that we have already earlier used. Nina? Nina: We could go some place out and study them. Teacher: We can go out to some place and study. Where? Put the thing away, please. Where? What did you mean? If we think about the artifact? Nina: Well. Teacher: If it is the question about some concrete artifact, well, where one could go? Where would you go? Amy? Amy: To the museum. Teacher: aha Sam: But it costs. Teacher: It costs. For example, to the museum. Didnt we went to Heureka (science museum) when wondering about flying and space. Lara. Lara: Well then, for example, the artifact project We could look at where they are made, those eq Teacher: But that could be interesting! To go some place, yes, where they are made. Different types of artifacts. Heidi. Heidi: To the library. Teacher: yes, what else? Dane? Dane: If we study some artifact, if someone would have the artifact herself, it could be brought here, to the classroom. Teacher: mm, We could study our own depositories as well! Artifacts that we own, that is also possible. Do you have more ideas? Michael. Michael: Well, I had approximately the same idea, that if there would exist some old staff that could be asked about from grandparents or others that would they have some similar artifacts, if they would know, or some other friends could be asked about if they would have the artifact. If they would know? Teacher: Do you know what it is called about? What are we doing then? What they are doing when asking about something? Well, Michael? Michael: Is it some gallup? Teacher: Well, nearly so, gallup yes, maybe the concept do not support the very idea, but what are they called about? We are doing what? When you are asking about something I am just searching the concept, we will learn new concepts at the same time. Well, isnt it an interview? Well make an interview, well interview some expert. Good idea. Good idea, very good ideas you have. I save quickly your ideas here that they wont run away anywhere. Lets put them here. What would be the good title for these? Dane? Dane: Studying artifact.
Conclusion
Three types of collective working episodes were distinguished regarding responsibility of directing inquiry in practice. Firstly, we identified episodes in which the teacher and students were equally involved in developing the inquiry. In these episodes, the strategic plans and goals of the project were determined jointly between the teacher and students although the teacher sometimes needed to facilitate and nourish the discussion from her part. Secondly, it was distinguished episodes in which the teacher deliberately either promoted discussions towards suitable strategy for further inquiry, or carried idea improvement towards understanding that the students would not have been able to reach themselves. Such teacher promoting was needed when students
ISLS
349
ICLS2012
were not able to express their own ideas or spontaneously emerged patterns of classroom discourse did not lead the community to a direction that would have assisted in obtaining higher-level objectives of the project. Thirdly, there occurred episodes where the teacher controlled joint activities so as to reach deeper level of explanation. In these cases the teacher considered it necessary to directly lead the community to using appropriate inquiry strategies. The present project was a collective object oriented process in nature and intended to support students in appropriating inquiry practices that resemble those that experts functioning in knowledge-creation communities enact (Bransford et al., 2006; Edelson, Gordin, & Pea, 1999; Carey & Smith, 1995; Bereiter & Scardamalia, 2003). Based on the results, the teacher guided students to bring up ideas relying on their own experiences and background knowledge, and her aim was to support students knowledge building and taking of responsibility for idea improvement. Yet, there were no whole class episodes, where students made most of the initiations regarding advancement of inquiry and the teacher only chaired the session. However, it still is very typical to Finnish school that students are not oriented toward sharing their spontaneously generated ideas in the social space of the classroom (Hakkarainen, Jrvel, Lipponen, & Lehtinen, 1998). Moreover, the studentdriven collaborative activities took frequently place in the context of team work. On the other hand, the present teacher was students co-research in the project; she was responsible to its strategic guidance together with students. The present type of investigative study project does not simply represent the teachers world because of being co-figured with students and their interest. Neither did the project represent students world; it constitute a collective space that emerged through joint efforts of the participants. Cazdens (2001) claim that differences between learning in teacher-led lessons and learning in peer groups are becoming less marked, when the work in progress can be shared for whole classroom community through technology, is supported in the project at hand. Differently constructed episodes created all common ground and took productively part in the collective inquiry.
References
Bereiter, C. & Scardamalia, M. (2003). Learning to work creatively with knowledge. In E. de Corte, L. Verschaffel, N. Entwistle, & J. van Merrinboer (Eds.), Powerful learning environments. Unraveling basic components and dimensions. Oxford: Elsevier Science. Bransford, J. D., Barron, B., Pea, R. D., Meltzoff, A., Kuhl, P., Bell, P., Stevens, R., Schwartz, D., Vye, N., Reeves, B., Roschelle, J., & Sabelli, N. H. (2006). Foundations and opportunities for an interdisciplinary science of learning. In R. K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (pp. 1934). Cambridge: Cambridge University Press. Carey, S. & Smith, C. (1995). On understanding the Nature of scientific Knowledge. In D. Perkins, J. Schwartz, M. West, & M. Wiske (Eds.), Software goes to school (39-55). Oxford: Oxford University Press. Cazden, C. B. (2001). The classroom discourse The language of teaching and learning. Heinemann: Portsmouth. Edelson, D., Gordin, D. & Pea, R. (1999). Addressing the challenges of inquiry-based learning through technology and curriculum design. The journal of the learning sciences, 8(3&4). 391-450. Hakkarainen, K. (2003). Progressive inquiry in computer-supported biology classroom. Journal of Research in Science Teaching, 40(10), 1072-1088. Hakkarainen, K., Jrvel, S., Lipponen, L. & Lehtinen, E. (1998). Culture of collaboration in computersupported learning: Finnish perspectives. Journal of Interactive Learning Research, 9, 271-288. Hmelo-Silver, C. Duncan, R., & Chinn, C. (2007). Scaffolding and Achievement in Problem-Based and Inquiry Learning: A Response to Kirschner, Sweller, And Clark (2006). Educational Psychologist, 42(2), 99107. Lakkala, M., Muukkonen, H., Paavola, S., & Hakkarainen, K. (2008). Designing pedagogical infrastructures in university courses for technology-enhanced collaborative inquiry. Research and Practice in Technology Enhanced Learning, 3(1), 33-64. Olson, D. (2007). The self-ascription of intention: Responsibility, obligation and self-control. Synthese, 159, 297-314. Roth, W.M. (1998). Designing Communities. Boston: Kluwer. Roth, W. M. (2002). Being and becoming in classroom. Westport, CT: Ablex. Sawyer, K. (2004). Creative Teaching: Collaborative Discussion as Disciplined Improvisation. Educational Researcher, 33(2),12-20. Scardamalia, M. (2002). Collective cognitive responsibility for the advancement of knowledge. In B. Smith (Ed.), Liberal education in a knowledge society (pp. 6798). Chicago: Open Court. Zhang, J., Hong, H., Scardamalia, M., Chew, L., & Morley, E. (2011). Sustaining knowledge building as a principle-based innovation at an elementary school. Journal of the Learning Sciences, 20(2), 262-307. Zhang, J., Scardamalia, M., Reeve, R. & Messina, R. (2009). Designs for Collective Cognitive Responsibility in Knowledge-Building Communities. The Journal of the learning sciences, 18(1). 7-44.
ISLS
350
ICLS2012
Scaffolding and Assessing Knowledge Building among Chinese Tertiary Students Using E-portfolios
Chunlin Lei, Carol K.K. Chan, The University of Hong Kong, Pokfulam, Hong Kong, SAR, China leichl@hku.hk; ckkchan@hku.hk Abstract: The goal of the study was to develop and examine a computer-supported knowledge building environment, incorporating e-portfolio assessment to scaffold and assess collaborative knowledge building. It involved a quasi-experimental design in a Chinese tertiary setting to understand the characteristics of and relations among knowledge building participation, collaboration, and collective knowledge growth. Students reflected and documented their knowledge building practice in the e-group-portfolios, which helped to characterize and scaffold knowledge building discourse. Students in the KB environment were found more actively engaged in Knowledge Forum and high-level discourse moves. Future research may place emphases not only on forum participation, but also on how high level discourse moves can be fostered to advance collective knowledge towards knowledge creation.
Introduction
Learning is now conceived as a collaborative endeavor (Cummins, 2002), and assessment, especially formative assessment should be an integral part of the instructional process (Shepard, 2000) and be aligned to enhance learning (Biggs, 2003; Black and Wiliam, 2004; Broadfoot 1996; Rushton, 2005). With the upsurge of information technology, computer-supported collaborative learning (CSCL) has emerged and provides benefits and potential for twenty first century education (Stahl, 2006). Although the key focus in paradigms of learning has turned to social and collective aspects and assessment is for enhancing learning, important questions regarding how to develop a social constructivist classroom, and how to identify, scaffold, and assess collective learning and collaboration remain to be investigated. CSCL is promising, but putting students in a CSCL environment does not necessarily lead to meaningful collaboration and learning (Kreijns, Kirschner, and Jochems, 2003). There is in general a lack of alignment in learning, assessment, technology and collaboration in higher education. Although there is much calling for developing higher-order thinking in higher education reforms across the world, very little research on how to assess those higher-order competencies has been conducted (Chan and van Aalst, 2004). This study aimed to adopt the knowledge building approach to examine collective learning, formative assessment, and collaboration in the context of higher education.
Theoretical Perspectives
To align learning, collaboration and assessment in a technology-enhanced learning environment, this study is premised theoretically on the knowledge-building model of Scardamalia and Bereiter (2006). As a forerunner of CSCL, knowledge building emphasizes knowledge creation as collective work of a learning community. To support student discourse, Knowledge Forum (KF) is designed to objectify the creation and improvement of ideas manifested in the form of notes. Scardamalia (2002) has proposed a system of 12 knowledge building principles including improvable ideas, epistemic agency, and collective responsibility to facilitate and examine the socio-cognitive and socio-technological dynamics of knowledge creation. Sheperd (2000) has proposed a constructivist framework on assessment which emphasizes the link between assessment and ongoing instruction, and the need for student to develop critical thinking, problemsolving, and meta-cognitive abilities. As a method for formative assessment, a portfolio can be a showcase for a students best work (Roberts, 2005). When a portfolio is used for assessing students learning, it usually engages a student in higher order thinking through the use of inquiry and reflection (Johnson et al., 2010). Recent studies in knowledge building have focused on implementing knowledge building principles and e-portfolio assessment for knowledge building (Lee et al., 2006; van Aalst and Chan, 2007; Zhang et. al., 2007). Lee et al. (2006) selected five key facets of knowledge building principles and guided students to use portfolio to assess individual and community knowledge advances in computer forums. Zhang et al. used principles, but not formative assessment to analyze students growth in domain understanding. Other studies (Chan and Lam, 2010) used reflective assessment to foster students conceptual change. Although many studies of knowledge building have been conducted in schools (Chan, 2011; Hakkarainen, 2003, 2004; Zhang et al, 2007), there is a lack of systematic research on knowledge building in tertiary settings. Few studies have addressed assessment for learning under CSCL in Mainland China. Ge (2011) reported a net-based peer assessment for improving Chinese adult learners English writing ability. The findings showed participants got satisfactory results, yet collaboration was very difficult to foster.
ISLS
351
ICLS2012
Research has identified that knowledge building discourse can be distinguished as knowledge sharing, knowledge construction and knowledge creation (van Aalst, 2009); it is useful to see how the discourse patterns may be manifested among students. In addition, earlier studies examined portfolio on individual basis and portfolio in this study are on group basis, which may also be better able to capture the collective work in the community. Therefore, the goal of this study is to design and examine a knowledge building environment and in particular, use portfolio assessment to scaffold, characterize and assess collective knowledge building. The research questions were: 1) What are the characteristics of student knowledge building participation, collaboration, and collective knowledge growth in the e-portfolio-augmented knowledge building environment? 2) How are such knowledge building dynamics related to one another?
Methods and Design

Participants and Procedure
Participants were two intact classes of first year tertiary students in a renowned university in Shanghai, China. It adopted a quasi-experimental design examining the effects of a knowledge-building environment with principle (KB, n= 30) and a non knowledge-building environment (NKB, n = 30). The KB environment was characterized by knowledge building pedagogy, principles, and the technological platform, KF. The NKB environment was a typical teaching environment consisting of teachers lecture and students discussion; it was not governed by knowledge building principles, but KF also was utilized. The consideration for employing KF to both classes was that we would examine more closely whether it was merely the novel effects of inclusion of technology or the deeper impacts via knowledge building theory-pedagogy-technology that made changes. Both classes were taught by the same teacher who had 12-year experience of teaching, accessed a huge amount of knowledge building database, and observed knowledge building teachers meetings frequently. The course was titled Introduction to Business, conducted in two semesters of academic year of 2010-11. Each semester lasted 14 weeks and each week had 2 lessons of 1.5 hours. Students had face-to-face discussion and inquiry in class and wrote notes on KF after class. The KB students were taught the knowledge-building principles explicitly, while the NKB students were not taught them but were encouraged to use KF as a new technological medium to communicate and learn.
Design of a Knowledge Building Environment

We designed a leaning environment based on knowledge building pedagogy to facilitate collaboration and deep learning. (1) Forming a collaborative learning culture. In the first few sessions, students were provided with learning experiences which familiarized them with the technology and acculturated them into the practices of collaborative inquiry. Focus was placed on making ideas public on KF (Fig.1a). (2) Developing knowledgebuilding inquiry. Students through face-to-face and online discourse, elaborated what they know about the topic, set forth their theories, searched for useful resources, and explored answers in a cyclical and deepening way. (3) Emergence and rise-above. Agency was turned over to the students and they were able to define goals and activities as the questions emerged in class or online. They assumed responsibility for their own learning and helped each other for knowledge advancement. The teacher acted as a facilitator, scaffolding the discussion towards a continuous rise above way. (4) Portfolio assessment. Formative assessment was used to scaffold as well as to characterize collective knowledge. In class and on KF, students tracked and reflected their inquiry trajectory to identify what were the promising questions, and what needed further inquiry. During semester 2, students in both classes were divided into 6 small groups to do projects; each group was then required to submit 3 e-portfolios (Fig.1b) during week 4, 7, and 9 respectively. Students were suggested to include 4 best clusters of notes from KF into a portfolio and to provide explanations simultaneously to justify their selection. For the KB environment, students adopted knowledge building principles such as improvable ideas, collective responsibility, epistemic agency, and constructive uses of authoritative information to direct their writing on KF, which intended to refine their discourse towards knowledge creation.
Figure1a. A discussion view of Knowledge Forum

(the red tiny icons represent students notes)
Figure1b. An example of students e-portfolio

(putting our knowledge together)
ISLS
352
ICLS2012
Data Sources
Data included assessment applets and Analytic Toolkit (tools embedded with KF) indices that utilize server log data to represent students participation patterns on KF. Other important data were students e-group-portfolios submitted on KF based on the approach developed by van Aalst and Chan (2007) and Lee et al. (2006). In both the KB and NKB classes, students formed into small groups and wrote group portfolios. Each portfolio consisted of four clusters of best notes from KF. A cluster of notes refers to notes addressing the same principle topic or problem in the communal space; they are put together to form an inquiry thread (Zhang et al. 2007). In preparing their group portfolio, students documented knowledge advances in the community; they developed meta-discourse to reflect on the collective knowledge growth. The e-portfolios were employed as data sources to track collective knowledge building. In the present study, group rather than individual portfolios were employed. According to the design, each of the KB and NKB class produced 18 e-group-portfolios, totaling 36 portfolios.
Analyses and Results

Community Awareness and Connectedness
We employed two KF Applet indices Notes Read Density and Notes Build-on Density to measure student involvement in knowledge building community. The assumption is that if a student reads more of others notes, he or she may develop stronger community awareness, and the same is true regarding building on others notes and community connectedness. By the end of instruction, applet data showed Read Density and Build-on Density for both environments (Fig 2a): 91.2% and 63.4% for KB; and 87.8% and 22.2% for NKB. The results, especially the Build-on Density, were encouraging compared to those of previous studies (Chan and Fu, 2011; van Aalst and Truong, 2011). It indicated the KB environment (Fig 2b) students responded more directly and actively than the students in the NKB environment (Fig 2c).
100% 80% 60% 40% 20% 0% Read Build-on KB NKB
Fig 2a. Read & Build-on Density (KB & NKB)
Fig 2b. Build-on (KB)
Fig 2c. Build-on (NKB)
KF participation and Engagement

We then examined students overall participation and engagement in KF via Analytic Toolkit (ATK), which provides sever log information on notes written, notes read, notes linked, notes revision, keywords scaffolds used, etc. The knowledge building literature suggests that these quantitative indices indicate students collaboration and meta-cognition for intentional learning. The results of ATK indices for both KB and NKB environments were as follows: number of notes written, 57.8 (18.9) and 24.4(9.8); notes revisions, 16.8 (12.8) and 7.6(9.1); scaffolds used, 42.5 (26.5) and 5.9 (8.5); percentage of notes linked, 78% (11%) and 52%(16%); notes read, 47% (15%) and 24% (13%);and keywords,56%(23%) and36%(20%). According to previous studies on computer-supported learning in higher education, fragmented contributions and limited thread growth are common (Hewitt, 2005). The present data indicated both KB and NKB classes had high levels of contribution and connectedness. To provide more coherent analysis, six ATK indices were combined using factor analysis (see Lee et al., 2006). Two factors emerged productivity (write, read, revision, scaffolds) and collaboration (linked, keywords). Statistical analysis indicated significant differences between two environments, for ATK Productivity, F (1,58) = 73.7, p<.001 and for ATK Collaboration, F (1,58) = 44.5, p<.001.
Characterizing Knowledge-Building Dynamics

To better understand the knowledge building process and characterize the dynamics of students collective work, students e-group-portfolios were analyzed. Portfolio discourse objectified their perspectives about knowledge and knowledge building. For example, at the lower level, they might think its just fine to reproduce what others think and draw some notes together to finish a task. Or at the middle level, they might want to understand what other people say and if possible, join in the discussion and enrich their personal minds. While at the much higher level, they needed to demonstrate higher order abilities to synthesize and evaluate, to assume collectives responsibilities and view the class as a learning community; therefore, discourse was geared to address collective knowledge gaps and dedicated to something new and meaningful for the community. Qualitative analyses were conducted on the portfolios to identify and characterize collective knowledge building dynamics, nine categories of discourse processes with descriptions and examples were listed below (Table 1).
ISLS
353
ICLS2012
Table 1: Categories, Description, and Examples of Discourse Moves Category Description

#1. Listing or Paraphrase (LP) #2. Brief Summary (BS) #3. Interpretation or Elaboration (IE) #4. Problem or question-based (PB) Copy information from others notes or repeat the information in a very close way; put notes together without explanations Summarize a few notes shortly and incompletely Interpret others notes information with different wording or extend information by examples or evidence See the discussion as problem-based and a deepening process of seeking answers or solutions to questions/problems
Examples (excepts from e-portfolios)

And ZBW said HR training is not onlybut also*. * copy of ZBWs note info. A good manger must have these principles. First, Second,. Third, LW got some info indicating that more than half people believeThat is to say it is . FXX brings forward her question, how to establish a system. LCh gave her answerthen WY gave an answer in another situationYIM brings an idea which is totally different from othersCXB gave us some info. The main question was raised by JHLmany students took part in the discussion actively. ZLY said and summarized her idea However, CXB seemed to [differ] with her. She thoughtand put up a further question This problem was intractable. CJR gave a general answer this answer didnt respond to the question directly, we work on this later. Another chain began with WY was confused about CXB gave another aspect of There is a chain of notes began with LWs questionL built on and asked CMW answered the question she used the knowledge learnt from the Market Leader course, which shows At the beginning of this semester, we knew little about HRM We now showed a great understanding of it. This topic is comp up with XB, I think its interesting when she wrote a note Lets start a new topic! This is like a class without a teacher. She studies some HR theories and tries to justify her ideas by these theories. Theory Y is one of the most appropriate Based on these expanded ideas, we actually have created a theory on what makes an excellent manager L raised new questionsW didnt answer them immediately; instead, 5 other students gave their explanations to the questions Two of the 5 notes are even further discussed. LC thought WQs theory cant explain her questions. I hope we classmates can solve the problems This topic has not been widely discussed yetit is quite new and interesting Ultimately, we arrived at a new conclusion
#5. Emergent Questions (EQ)
Keep asking related questions, showing doubts or seeking clarification; responses and explanations are intertwined progressively in the discussion
#6. Constructive use of Information (CI)
Use information, either from experts, books, internet, or other related courses, life experience, etc. to justify or deepen their ideas
#7. Meta-Cognition (MC)
Realize high points in the discussion; and selfdefine goals and tasks of exploration
#8. Theory-Building (TB)
Focus on theories while developing the discourse; use theories to explain business phenomena with an attempt to create new theories
#9. Meta-Discourse (MD)
Reflect on discourse goals and directions; adopt a collective we perspective to show collective responsibility in advancing knowledge and tackle difficult/ important issues which may be neglected by the community
Using this coding scheme, we examined discourse patterns in 36 group portfolios. Following the procedures of quantifying qualitative data (Chi, 1997), we computed the mean scores for different categories of discourse processes for the two classes (Table 2). Multivariate analyses conducted indicating significant results followed by univariate analyses. Significant differences were obtained on: Listing & Paraphrase, F(1,58)=25.0, p<.001; Brief Summary, F(1,58) =17.7, p<.001; Emergent Question, F(1,58)=72.5, p<.001; Theory Building, F(1,58)=14.8, p<.001; Meta-discourse, F(1,58)=38.4, p<.001; and Meta-cognition, F(1,58)=3.4, p<.08; while no significant difference on Interpretation and Elaboration, Problem or Question Based, and Constructive Use of Information. These results suggested that NKB students were more involved in activities such as listing, paraphrasing and briefly summarizing when conducting the e-portfolios; while KB students were more engaged in high level processes of offering explanations, reflecting directions of their discourse, addressing collective knowledge gaps and attempting to create new knowledge or theories.
ISLS
354
ICLS2012
Table 2: Comparison of Discourse Patterns between KB and NKB

1.LP 2. BS 3. IE 4. PB 5. EQ 6. CI 7. MC 8. TB 9. MD KB M(SD) .17 (.38) .67(.76) 1.2(1.1) .83(.70) 2.3(.76) 1.2(1.1) 1.2(.91) 1.2(.91) 1.8(.70) NKB M(SD) 1.0 (.83) 1.5(.78) 1.2(.70) 1.2(.91) .67(.76) .83(1.1) .83(.38) .33(.76) .76(.75) Note: 1.Listing or paraphrase; 2.Brief Summary; 3 Interpretation or Elaboration; 4. Problem or Question-Based; 5. Emergent Question; 6. Constructive use of Information; 7. Meta-Cognition; 8. Theory-Building; 9. Mate-discourse
Relations between KB Participation and Discourse Moves

To further understand knowledge building dynamics, we investigated the relationships among different variables in the learning environments. We obtained students high-level discourse move scores by combining categories #4 to #9. Pearson correlation analyses showed significant positive relations among the measures (Table 3). ATK productivity was correlated with collaboration. It indicated that students contributed more notes to KF were also likely to interact more with the community members. Importantly, there was strong positive relationship between students high level discourse move and forum engagement. It suggested that students more engaged in KF might also be able to move to higher level of discourse, thus directing themselves towards emergent knowledge creation. Table 3: Correlations among ATK Productivity, Collaboration and High Level Discourse Moves 1 2
1.ATK Productivity 2.ATK Collaboration 3. High Level discourse Moves Note: N=60, **p<.01;
.400** .386**
.392**

This study developed a computer-supported knowledge building environment, augmented with e-portfolio assessment, to scaffold and assess collective knowledge building. By adopting a quasi-experimental design, this study showed students in the KB environment were more connected to one another and more engaged in the knowledge building process. Technology such as KF did help students in both classes, but it was the underlying knowledge building principles and pedagogy that had contributed to the different effects. E-portfolio assessment helped to scaffold and assess collaborative knowledge building, revealing more knowledge building dynamics and the way students were engaged in the knowledge building process. We developed a new coding scheme to characterize and assess e-group-portfolio discourse in the area of business studies, enriching the knowledge building literature on differentiating knowledge (van Aalst, 2009). Positive correlations were found between knowledge building engagement and high-level discourse moves. Therefore, future research might lay emphases not only on forum engagement, but also on how high level discourse moves can be fostered to advance collective knowledge towards knowledge creation.
References
Chan, C. (2011). Bridging research and practice: Implementing and sustaining knowledge building in Hong Kong classrooms. International Journal of Computer-Supported Collaborative Learning, 6(2), 147. Cummins, J. (2002). Foreword. In P. Gibbons (Ed.), Scaffolding language, scaffolding learning: Teaching second language learners in the mainstream classroom (pp. v-viii). Portsmouth, NH: Heinemann. Lee, E. Y. C., C. K. K. Chan, et al. (2006). Students assessing their own collaborative knowledge building. International Journal of Computer-Supported Collaborative Learning 1: 278-307. Hewitt, J. (2005). Toward an understanding of how threads die in asynchronous computer conferences. The Journal of the Learning Sciences, 14(4), 567-589. Scardamalia, M. (2002). Collective cognitive responsibility for the advancement of knowledge. In B. Smith (Ed.), Liberal Education in a knowledge society (pp. 67-98). Scardamalia, M., & Bereiter, C. (2006). Knowledge building. In R.K.Sawyer (Ed.), The Cambridge handbook of the learning sciences (pp. 97-115). Cambridge: Cambridge University Press Shepard, L.A (2000). The role of assessment in a learning culture. Educational Researcher, Vol.29, No. 7, pp. 4-14. van Aalst, J. (2009). Distinguishing knowledge-sharing, knowledge construction, and knowledge-creation discourses. International Journal of Computer-Supported Collaborative Learning (4), 259-287. van Aalst, J., & Chan, C. (2007). Student-directed assessment of knowledge building using electronic portfolios in Knowledge Forum. The Journal of the Learning Sciences, 16(2), 175-220. Zhang, J. W., M. Scardamalia, et al. (2007). Socio-cognitive dynamics of knowledge building in the work of 9 and 10-year-olds. Education Tech Research Dev. 55: 117-145.
ISLS
355
ICLS2012
Contribution of Motivational Orientations to Student Outcomes in a Discovery-Based Program of Game Design Learning
Rebecca Reynolds, School of Communication & Information, Rutgers University Ming Ming Chiu, Graduate School of Education, University at Buffalo, SUNY Email: rebecca.reynolds@gmail.com Abstract: This paper explored relationships between middle school and high school teams of students motivational orientation and team outcomes in a guided discovery-based context in which students learned while designing web games. Teams of students with greater initial intrinsic motivation or a greater increase in intrinsic motivation during the activity had higher programming scores. Wiki activity contributed to outcomes. Age was negatively associated with outcomes indicating possibly the program is more conducive to middle school students. Findings contribute to scholarly debates on incompatibilities between discovery learning due to cognitive load, and self-determination theory.
Introduction
This paper reports findings from the 2009/2010 school year of a Constructionist digital literacy project being conducted with students and educators in the state of West Virginia involving a state-wide network of classrooms engaged in game design learning and purposeful social media use among youth. Middle school, high school and community college students (N = 386) enroll in a blended learning game design elective course offered daily, for 1 or 2 semesters, credit and a grade. The paper explores teams motivational orientation and active program work in relation to their learning outcomes, in the context of a guided discovery-based curriculum in which students and educators co-learn together in this social learning system.
Literature Review
Computational game making activities can enhance students meta-cognition, self-regulation, and computational thinking (e.g., Harel & Papert, 1991; Harel, 1991; Kafai, 1995; Kafai & Ching, 1998; Papert, 1980). Research supporting these findings has been conducted under the educative framework for action Constructionism (Papert, 1980; Harel and Papert, 1991). Constructionism draws upon both Piagets constructivist theory and Vygotskys social constructivist theory. Adherents design and implement learning innovations and environments that foster learners conscious creation of a meaningful, computational public artifact (e.g. a game), created and shared in a reflective, workshop environment of peer and expert-guided scaffolding (Harel and Papert, 1991). This study centers on a program of game design that employs a range of web 2.0 technologies including Flash and a wiki e-learning platform. In the co-learning model, the students engage in self-driven learning while their educators are still novices, using the provided resources. This activity can be characterized as guided discovery-based learning.
Discovery-based learning
While some argue that students can feel frustrated during discovery-based learning, others argue that students can enjoy it (Reynolds & Harel Caperton, 2011). Reynolds & Harel Caperton (2011) find contrasts in student perceptions of both enjoyment and challenge in the discovery-based context investigated here during the prior school year, given that occasionally novice learners must seek out learning supports to meet design needs in the moment. Kirschner, Sweller & Clark (2006) criticize discovery-based learning for creating excessive cognitive load due to lack of structure and distraction in the search process that reduces student motivation. In contrast, self-determination theory (SDT, Deci & Ryan, 2008) argues that three primary constructs underlie intrinsically motivated human behavior, and are innate needs: the need for autonomy (to have choice and control over ones life), for competence (to be effective), and for social relatedness (to feel connected to others, loved, and cared for) (Deci & Ryan, 2008; Ryan & Deci, 2000; Deci & Ryan, 1985). Level of autonomy afforded by a given environment, and the intrinsic and extrinsic motivational orientation of individuals play roles in their experience and fulfillment. Given the autonomy-supportiveness of the designed intervention, we propose the following hypotheses. Hypotheses. H1. Teams with higher mean intrinsic motivation have higher team scores. Given the guidance, scaffolding and support from both teachers and non-profit staff, along with resources provided in the blended e-learning environment, we expect that even those students who are more controlled in their selfregulation style will also succeed and perform well in the program because the necessary supports have been provided for those who might need them. H2. Teams with more extrinsically-oriented self-regulation have higher team scores. Furthermore, in addition to the generalized holistic inventories outlined above, student motivation towards the specific contextual range of activities in which they engage might be related to subsequent team scores. H3. Teams with greater increases in motivation towards program activities have higher team scores. Finally, given students active engagement and use of a wiki, we explore the extent to which such activity may contribute to game design outcomes. H4. Teams with more wiki activity have higher team scores.
ISLS
356
ICLS2012
We also control for a range of other variables that might contribute to student outcomes, including student demographics and teacher demographics.
Method
Intervention. In brief, a non-profit provided students and teachers digital learning supports via a wiki-based social media platform called MyGLife.org, in-person twice-annual teacher training, and ongoing virtual webinars with students and teachers. Individual students develop games, interacting with each other, teachers, and content resources on the wiki and in class, following a blended learning curriculum daily, for up to 90 minutes per session, across either a semester or a full year. The program applied many of the attributes of Constructionist design workshops described in Harel & Papert (1991) and Seely Brown (2005). Data sources. Data sources have been combined into a single dataset and include pre- and postprogram student survey data; educator progress reports bearing reflections and official student lists; student wiki activity metrics provided in MediaWiki e-learning platform, MyGLife.org; wiki user gallery and team pages identifying what individuals are on a given team; content analyzed student final game project files in Flash formats of .FLA and .SWF. Surveys were conducted online in August of 2009, January of 2010, and May/June 2010, depending on student participation modality (first semester only, second semester only, or full year). Links were distributed to students via each pilot location wiki, with educator administration. Research was conducted with full parental consent and child assent, and IRB approval. Participants. Out of 386 middle and high school student participants in the 2009/2010 school year, a total of 368 completed the pre-survey (95%), and 277 completed the post-survey (72%). A total of 64 middle school and 322 high school students participated. The drop off from pre-survey to post-survey is due to a range of factors, including student voluntary opt-out, student absences at the end of the school year, and student discontinuations in the program, changing of schools, etc. Findings reported here reflect those middle and high school students who completed both pre and post surveys. Participation was voluntary, and we acquired signed parent/guardian university IRB-standard permission forms for all students.
Dependent variables
Game quality. For the dependent variable, we conducted content analysis of all teams final games; hence, team is the unit of analysis. We evaluate functionality built into students completed games (mechanics), as well as the games cultural content and design. The purpose for evaluating games is to better understand the range of game mechanics and messages students achieved in their particular school setting, identify patterns, and explore explanations. We also evaluate games to better understand the extent of knowledge students are gaining in the dimensions we explore. Coding Scheme Development. We define "game" as: a file that goes beyond a mere image, to include some level of interactivity, in which, at minimum, the file provides response to the player, based on a player action. Defining a game at this minimal level of interactivity allows us to code the full range of game files created by students, basic to advanced. The format of the game files students post online include both .SWF (Small Web Format / Shockwave Flash) and the .FLA project file format. The final coding scheme enables evaluation of Actionscript programming codes that could reasonably be expected from introductory game design students (1=present, 0=absent), and, evaluation of design attributes built into the game (visual and sound design elements, game play experience, concept development, genre) (1=Not present / insufficient representation; 2=basic / introductory representation; 3=well-developed representation). The team scores ranged from 16 to 61. To test inter-rater reliability, we computed the kappas for each section of the coding scheme: actionscript programming evaluation, 0.85; Visual and sound design evaluation, 0.81; Game play experience evaluation, 0.87; Concept development evaluation, 0.75.
Independent variables
Independent variables were tested at three levels of analysis: school, classroom educator, and student game design team. All variables are from surveys unless otherwise noted. School level variables comprise grade level (middle school vs. high school) and school mean of parent education. At the educator level, we aggregated educator time on task, calculating the average number of teaching hours, administration hours, and self-learning hours spent by educators in an average quarter, using a standardized metric controlling for the N of progress reports provided and drawing upon quarterly self-reported teacher progress reports as the data source. We have also calculated the average word count of their progress reports in each quarter and N of years teaching game design (including the present year). The number of students on a team ranged from 1 to 6. Thus, the individual data were aggregated to the team level and its mean and standard deviations were computed (Chiu, 2008; Chiu & Khoo, 2005). Individual student variables included N of participation months, girl, age, self-reported grade, parent education, motivation measures and psychological needs measures. All of these variables were self-reported by each student except N of participation months. While most variables were measured through surveys before the activity, intrinsic and extrinsic motivation and psychological needs variables were measured using previously validated instruments
ISLS
357
ICLS2012
through both pre-activity and post-activity surveys (Black & Deci, 2000; Williams & Deci, 1996) and indices were created (Joreskog & Sorbom, 2004). Alpha reliabilities for intrinsic and extrinsic motivation were 0.80 and 0.75 respectively. We also computed differences between pre- and post-variables for individuals, and then each teams mean of these values were computed. Team mean change in motivation = mean of each team members (post-activity intrinsic motivation pre-activity intrinsic motivation). Indices of enjoyment (Ryan, Koestner, & Deci, 1991) towards a range of program activities within each of the contemporary learning abilities categories above were also created via factor analyses; for each category, combinations of at least 3 survey items reflecting enjoyment towards program activities achieved eigenvalues >1. Wiki activity indicates collaboration, project management, file sharing and publishing online. MediaWiki tools used provide an archive. The N of Wiki edits tell us how many times students edit and save pages. N of file uploads indicate how many project file uploads were made by students as an indicator of their productivity in game design. N of blog posts were also counted (students use blogs as a designers notebook, to reflect on process, free write, and exchange social messages). Each team game was coded for its genre: social issue, 1; educational, 2; entertainment, 3; mixed, 4; other, 5. Social issue games reflected a greater level of open and free choice by students, and include issues like poverty, teen pregnancy, nutrition, etc. Educational games were didactic and meant to teach the player something. These were more often linked to the core curriculum so their creation was more structured (e.g., science games were required by some teachers because the game design class was offered as a supplement to a science class). Entertainment games often reflected pop culture themes and did not contain more serious content. Some games fell outside these categories and were labeled as other.
Analysis
We modeled the team scores of programming, design, game play, and concept with a multivariate outcome, multilevel model (Goldstein, 1995). A variance components model tested for significant differences at each (1) level: Team_Scoreyij = y00 + eyij + fy0j The vector Team_Scores outcome variable y (Programming, Design, Game_play, or Concept) of team i in classroom j had a grand mean intercept y00, with team- and classroom-level residuals (eyij and fy0j). Explanatory variables were entered in sequential sets to estimate the variance explained by each set (Kennedy, 2008). School characteristics might affect teacher characteristics. Teacher characteristics might affect team members. All of these might affect team processes. Hence, we entered the variables as follows: school, teacher, team member characteristics, and team processes: Team_Scoreyij = y00+eyij+fy0j+y0vSchooly0j+y0wTeachery0j+yxjTeam_membersyij+yzjTeam_Actionsyij (2) Schoolv: a school characteristics vector; high school and school mean parent education. Teacherw: a vector of w variables at the teacher level; teacher gender, average teaching hours, average administrative hours, average self-learning hours, average word count of progress reports, and number of years teaching in Globaloria. We tested whether sets of predictors were significant with a nested hypothesis test (2 log likelihood, Kennedy, 2008). Non-significant variables were removed. Team_membersy: a vector of x team variables; number of team members, number of participation months, proportion of girls in team, mean age, mean self-reported grades, standard deviation (SD) of self-reported grades, mean parent education, the means and SDs of motivation and enjoyment measures before game design (intrinsic motivation, extrinsic motivation, game enjoyment, search enjoyment, surf enjoyment, social media enjoyment, idea planning enjoyment, enjoyment of creating ideas, enjoyment of publishing, and basic psychological needs) and the means of the changes in the above motivation and enjoyment measures during game design. Team_Actionsz: a vector of z team processes; game genres, mean and SD Wiki edits per month, mean and SD FLA project uploads per month, mean and SD SWF flash uploads per month, mean and SD blog writes per month. We used multi-level mediation tests across the above vectors (Krull & MacKinnon, 2001). Then, we did a 2-level path analysis (Goldstein, 1995). We reported how a ten percent increase in each continuous variable above its mean was linked to the team scores (result=b * SD * [10% / 34%]; 1 SD 34%). As percent increase is not linearly related to standard deviation, scaling is not warranted. We used an alpha level of .05. To control for the false discovery rate (FDR), we used the two-stage linear step-up procedure, which outperformed 13 other methods in computer simulations (Benjamini, Krieger & Yekutieli, 2006). The small sample of classrooms (N = 18) limits identification of non-significant classroom-level results (for a 0.4 effect size at p = .05, statistical power = 0.19; Konstantopoulos, 2008). We analyzed residuals for influential outliers.
Results
The participants created 139 games (28 individual games). Explanatory Model. The results show substantial variance both across classrooms and within classrooms (programming: 43% across classrooms and 57% within classrooms; design: 60% and 40%; game play: 43% and 57%; concept: 52% and 48%; full tables provided in full paper). All results discussed below describe first entry into the regression, controlling for all previously included variables. Path Diagrams. Detailed path diagrams of standardized final 2-level model predicting team programming, design, game play
ISLS
358
ICLS2012
and concept scores are presented as an appendix at the following link: http://db.tt/fqS11bIU. To aid reader understanding, the single path analysis is split into four separate diagrams, one for each outcome variable. Programming. Teachers characteristics, team member characteristics, and team actions were linked to teams programming scores. If a teacher spent 10% less time on self-learning, his or her teams of students averaged 5% higher programming scores. Teachers self-learning hours accounted for 45% of the classroomlevel variance and 19% of the total variance of teams programming scores. Teams of students who were one year younger on average scored 6% higher in programming. Meanwhile, teams with 10% higher mean intrinsic motivation before the game design activity scored 2% higher in programming. Teams whose members averaged a 10% increase in intrinsic motivation during the activity had 2% higher programming scores. Team member characteristics accounted for an extra 19% of the variance in teams programming scores. Teams that created mixed genre games averaged 28% higher in programming. Furthermore, teams with 10% more SWF flash uploads scored 2% higher in programming. Team actions accounted for an extra 7% of the variance in teams programming scores. Design. Team member characteristics and team actions were linked to design scores. Teams whose members pre-survey variation of enjoyment of the activity brainstorming and planning your game exceeded the mean by 10%, had 1% higher design scores. Team member characteristics accounted for 9% of the variance in teams design scores. Teams that created mixed genre games averaged 25% higher design scores, and teams with 10% more SWF flash uploads had 2% higher design scores. Team actions accounted for an extra 18% of design score variance. Game play. Teacher gender, team member characteristics, and team actions were linked to game play scores. Teams taught by female teachers averaged 34% higher game play scores than those taught by male teachers (model 1). Teacher gender accounted for 50% of the classroom-level variance and 21% of the total variance of game play scores. Teams of students who were one year younger on average had 4% higher game play scores (model 2). Meanwhile, teams whose self-reported grades averaged one letter grade above the mean had 9% higher concept scores. Teams whose members pre-survey variation exceeded the mean by 10%, in the area of enjoyment of the activity brainstorming and planning your game, had 2% higher game play scores. Moreover, teams whose members variation in enjoyment of the activity brainstorming and planning your game before the activity exceeded the mean by 10% had 3% higher game play scores. Team member characteristics accounted for an extra 18% of the variance in teams game play scores. Teams that created mixed genre games averaged 22% higher game play scores (model 3). Furthermore, teams with 10% more FLA project uploads had 2% higher game play scores. Team processes accounted for an extra 6% of the variance in teams game play scores. Concept. Team member characteristics and team processes were linked to teams concept scores. Teams whose members variation in enjoyment of the activity brainstorming and planning your game before the activity exceeded the mean by 10% had 2% higher concept scores (model 2). Teams whose basic psychological needs before the activity exceeded the mean by 10% had 1% higher concept scores. Meanwhile, teams with 10% less variation in team members idea planning enjoyment before the activity had 1% higher concept scores. Team member characteristics accounted for 11% of the variance in teams concept scores. Teams that created educational or mixed genre games had 20% or 44% higher concept scores (model 3). Furthermore, teams with 10% more FLA project uploads had 3% higher concept scores. Team processes accounted for an extra 32% of the variance in teams concept scores. Other variables (notably, parent education) did not show significant effects. Mediation tests showed no significant mediations across significant explanatory variables.
Discussion
Unlike extrinsic motivation, intrinsic motivation is related to team programming outcomes, supporting H1 but not H2. For the computational activities that required resourcefulness in finding support resources (especially for novice educators), this guided discovery-based program of learning may support the autonomous natures of students with intrinsic motivational orientations. The positive intrinsic motivation results support selfdetermination theory (e.g. Ryan & Deci 2000) and suggest that the Globaloria programs affordance of perceived autonomy and actual opportunity for autonomous engagement aids fulfillment of this basic psychological need for intrinsically motivated students. Teams with greater variation in their members enjoyment of the activity brainstorming and planning your game had higher design, gameplay and concept scores, showing some support for H3. Past case study research of game design processes (e.g., Reynolds, 2010) indicates that students often self-organize into the following team roles: programmer, designer, researcher of info for the narrative concept, and project manager. While team roles are shared, shift over time, and are predicated on N of students, pre-program variation in this category (some high and some low) may indicate a variation in preferences for certain roles, which may have contributed to more felicitous team dynamics and game design processes. This is consistent with past results showing that teams with a subset of individuals who identified with and enjoyed this over and above others had more coherent storylines and smoother design processes.
ISLS 359
ICLS2012
Teams with more wiki activity had higher team scores, supporting H4. These results suggest that using wiki resources and the wiki platform to share files and manage their project and process, contributes directly to outcomes. Also, younger teams of students outperformed older teams of students, possibly because younger found the activities more engaging. At the teacher level, results of a negative contribution for self-learning hours may indicate that students of more novice educators who required more self-study time did not perform as well. These results lead to practical recommendations for the organization, as well as offering insight into the extent to which critiques on discovery-based learning may need to be qualified in relation to autonomy. For instance, the autonomous versus controlled motivational orientation instrument could be used up front to identify students who may need more guidance and structure. This may require a greater teacher training and preparation. Further, the wiki may need to be further optimized to help students find and use resources. One limitation is that this data set only included assessments of teams. Future studies can also include assessments of individuals. More research is needed to understand the contribution of implementation context variables in the intervention that our study does not precisely measure (e.g., game subject focus; a classs integration with core curricular domain classes; sequence of learning activities and varying implementation contexts by school) and other individual- team- and teacher-level variables that might influence student outcomes. We will analyze the data in the current school year, which includes such additional variables. Further we have proposed grant funding to conduct qualitative case study research in West Virginia schools to better understand the mechanisms of teacher, team, and curriculum dynamics, and generate new hypotheses. These results may extend beyond the learning sciences into the larger arena of scholarship on socio-technical system design in general, as individual intrinsic motivation may contribute to the success of user experiences in sociotechnical environments when such environments are less-structured and require autonomous engagement. This finding could hold importance given the growing level of expectation in our society for individuals to have/develop digital expertise and actively use technology in order to participate as citizens.
References
Benjamini, Y., Krieger, A. M., & Yekutieli, D. (2006). Adaptive linear step-up procedures that control the false discovery rate. Biometrika, 93, 491-507. Black, A. E., & Deci, E. L. (2000). The effects of instructors' autonomy support and students' autonomous motivation on learning organic chemistry. Science Education, 84, 740-756. Deci, E. L., & Ryan, R. M. (2008). Facilitating optimal motivation and psychological well-being across lifes domains. Canadian Psychology, 49, 14-23. Goldstein, H. (1995). Multilevel statistical models. Sydney: Edward Arnold. Harel, I, & Papert, S. (1991). Constructionism. Norwood, NJ: Ablex Publishing Harel Caperton, I. (2010). Toward a theory of game-media literacy: Playing and building as reading and writing. International Journal of Gaming and Computer-Mediated Simulations, 2(1), 2010. Hmelo-Silver, C. E., Duncan, R. G., Chinn, C. A. (2007). Scaffolding and achievement in problem-based and inquiry learning: A response to Kirschner, Sweller, and Clark (2006). Educational Psychologist, 42, 99-107. Kafai, Y. B. (1995). Minds in play: Computer game design as a context for childrens learning. Mahwah, NJ: Lawrence Erlbaum. Kafai, Y. B., and Ching, C. C. (1998, December). Talking science through design: Childrens science discourse within software design activities. Paper presented at the ICLS, Atlanta, Georgia. Kennedy, P. (2008). A guide to econometrics. Cambridge: Blackwell. Kirschner, P.A., Sweller, J. & Clark, R.E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry based teaching. Educational Psychologist, 41, 75-86. Konstantopoulos, S. (2008). The power of the test in three-level cluster randomized designs. Journal of Research on Educational Effectiveness, 1, 66-88. Krull, J. L., & MacKinnon, D. P. (2001). Multilevel modeling of individual and group level mediated effects. Multivariate Behavioral Research, 36, 249-277. Papert, S. (1980) Mindstorms. New York: Basic Books, A Division of HarperCollins Publishers, Inc. Reynolds, R., Harel Caperton, I. (2009). The emergence of 6 contemporary learning abilities in high school students as they develop and design interactive games. Paper presented at AERA, April 2009. Reynolds, R & Harel Caperton, I. (2011). Contrasts in student engagement, meaning-making, dislikes, and challenges in a discovery-based program of game design learning. Educational Technology Research and Development, 59(2): 267-289. Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55, 68-78. Williams, G. C., & Deci, E. L. (1996). Internalization of biopsychosocial values by medical students: A test of self-determination theory. Journal of Personality and Social Psychology, 70, 767-779.
ISLS
360
ICLS2012
The Knowledge Connections Analyzer

Jan van Aalsta, Carol K. K. Chana, Stella Wen Tiana, Christopher Teplovsb, Yuen-Yan Chana, Wing-San Wana, a Faculty of Education, the University of Hong Kong, Hong Kong S.A.R., CHINA b OISE/University of Toronto, 252 Bloor Street W., Toronto, ON, CANADA E-mail: vanaalst@hku.hk, ckk@hku.hk, tianwen@hku.hk, christopher.teplovs@gmail.com, yychan8@hku.hk, wan.zero@gmail.com Abstract: We describe the development of an SQL-based formative assessment system, the Knowledge Connections Analzyer (KCA), which is designed to provide evidence on four general questions that students may have about their work in an asynchronous online discussion environment: (1) Are we collaborating? (2) Are we putting our knowledge together? (3) How do ideas develop over time? (4) What is happening to my stuff? These questions are inspired by Scardamalias knowledge-building principles. The KCA first converts a Knowledge Forum tuplestore database to SQL format, and then executes queries relevant to these four questions. It Students and their teacher can employ it to self-assess their knowledge building. This paper elaborates upon a conceptual framework underlying the system design, describes the KCA, and reports the results of several rounds of usability testing involving teachers and students.
Introduction
In the last decade there has been much interest in education in the use of discoursewriting, reading, and other actionsin Web-based environments. The best known environments inlcude learning management systems like Moodle and WebCT, and more specialized inquiry environments such as Knowledge Forum. Knowledge building is one of the most developed educationals models that involves computer-supported discourse; one of its most important features is that students efforts are directed at advancing the collective knowledge in a community (Scardamalia, 2002). Students are not just trying to understand things for themselves but aim to add something new to what is known in the community. In this context, Knowledge Forum is used to support participatory practices (Hickey et al., 2010) where participants share and collaboratively improve and synthesize ideas. Recently, interest in the large-scale implementation of approaches like knowledge building, which aim to help students to develop 21st century skills such as collaboration, ability to deal with novel situations, and self-regulated learning, has been mounting. However, one important challenge is that assessment tools need to become widely available that can be used by teachers and students to self-assess their knowledge building efforts. For example, to what extent can we say that there are collaborative dynamics, synthesis and rise-above, and improvement over time, and what are individual students roles and accomplishments? Such questions access both individual and collective aspects of knowledge building, and process as well as accomplishments. Ability to have data in hand to reflect on such questions is important for the development of knowledge-building discourse. The literature on asynchronous online discussions overwhelmingly demonstrates that such discussions have disappointing rates of participation, depth of inquiry, and knowledge advancement (Pifarre & Cobos, 2010); they often remain at the level of conversations in which students share opinions and ideas, and engage in superficial arguments, without advancing the collective knowledge of the community. Although we have made progress in conceptualizing the nature of discourse required (Scardamalia & Bereiter, 2006; van Aalst, 2006), better assessment tools are needed. Currently assessments of online discourse consist of content analyses that examine specific features of the discourse (e.g., how knowledge is constructed, the epistemic levels of students questions and ideas, and how ideas are diffused), but these types of analysis are too labor-intensive to inform students efforts while they are in progress. To scale up approaches like knowledge building, it is essential to develop assessment technologies that can provide students and teachers with useful information about their discourse and what it is accomplishingand that such tools can be used by teachers and students themselves. For the knowledge-building community the problem is urgent because the Analytic Toolkit for Knowledge Forum is becoming technically outdated, and does not provide all of the analyses that are now needed. This paper provides a brief report on the development a formative assessment tool, the Knowledge Connections Analzyer (KCA), which focuses on the use of Knowledge Forum server data to provide students and teachers with information about their knowledge building dynamics for self-assessment and reflection. The KCA provides analysis on four general questions students may have about their online discourse. It is hoped that our conceptual framework behind these designs contributes to the presentation of server log data as evidence to support the efficacy of self-assessment, reflection, and work improvement in an online environment.
ISLS
361
ICLS2012
Meanwhile, we hope the development of the system represents an on-going effort to implement and refine our formative assessment framework for web-based discussions.
Formative Assessment
In various forms, assessment drives educational practice (Biggs, 1996). Interest in formative assessment received a boost after the major review by (Black & Wiliam, 1998), which showed substantial positive impacts of formative assessment on learning. However, these practices seem to focus on such things as providing feedback on student work (e.g., tests and projects) and in-class questioning. Knowledge Forum is recoganized as a typical form of computer-supported collaborative learning (CSCL) environments (Salovaara & Jrvel, 2003; Scardamalia & Bereiter, 1996). An extensive review of measures and assessment in CSCL research shows that after collaboration measurement has played a dominating role, while there is an insufficient collection of tools and measures for examining processes involved in CSCL (Gress et al., 2010). To avoid such an afterwards style and to capture the knowledge building dynamics, we adopt the perspective of formative assessment (Scriven, 1967) to frame our work: the tool should provide assessment for students and teachers to reflect on knowledge building, when it is still in progress, and that is used to enhance knowledge building. We developed a web-based system for teacher-driven and student-driven formative assessment. Knowledge-building theory emphasizes that assessment is an integral component embedded in knowledge building and transforms it: it constitutes collaborative inquiry into the nature of the communitys work and its progress, and leads to new actions designed to enhance both (Scardamalia, 2002). Thus, formative assessment is not epistemologically distinct from knowledge building, except that the domain of the inquiry is not subject matter (e.g., science concepts) but the process of knowledge building.
Embedded Knowledge Building Principles

Scardamalia (2002) proposed a 12-principle system to describe the socio-cognitive and technological dynamics of knowledge building (the explanations of the principles are omitted due to the length limit). In practice, however, many teachers find this system complicated, in part because the principles overlap and lack the precision necessary to distinguish knowledge building from other approaches. For example, many educational approaches involve inquiry into authentic problems, idea diversity, metacognition, the constructive use of authoritative sources, and the democratization of knowledge (Kolodner et al., 2003; Polman, 2000). The KCA is thus designed around four general questions that students may have about their online work: (1). Are we collaborating? (2). Are we putting our knowledge together? (3). How do ideas develop over time? (4). What is happening to my stuff? Implementing Knowledge Forum environment has shown impact on student, teacher and school development at various levels of educational context (Chan, 2011). Although we cannot take for granted that students development of conventional proficiencies has been well addressed by the implementation of Knowledge Forum, we would like focus on assessing the aforementioned four aspects in this paper. On the one hand, the assessment is to help students reflect on their knowledge building work that is still in progress, rather than what proficiencies they have acquired. On the other hand, reflection on the knowledge building principles is the underlying rationale for the four questions. Table 1 lists the key purpose, knowledge building principles, and the major analyses that have been implemented in the KCA system. Table 1: Embedded Knowledge Building Principles in Four Questions. Questions Are we collaborating? Key Purpose A community-oriented question that asks whether collaboration is a welldeveloped practice in the community To explore the extent to which synthesis and riseabove are occurring from collective aspect To explore the extent of community engagement, Knowledge Building Principles Democratizing knowledge, Idea Diversity, Improvable Ideas, Epistemic Agency, Rise Above Community Knowledge, Collective Responsibility, Idea Diversity Improvable Ideas, Epistemic Agency Major Analyzing Indicators To what extent students have an audience for their work in the form of read, build-on, etc. to their notes The percentage of build-on links and the percentage of refer-to links The percentage of notes that have received a certain level of
Are we putting our knowledge together? How do ideas develop over
ISLS
362
ICLS2012
time? What is happening to my stuff?
and the emergent of new ideas from collective aspect To reflect on ones own note-writing efforts from individual aspect
Embedded, Concurrent and Transformative Assessment
interaction, keywords introduced, scaffolds used A ranked list of ones notes that prompted a given type of interaction with specified frequency, and the details of each interactions
The Knowledge Connections Analyzer

A special feature is that while many CSCL analyses tools are designed for researchers, this system is designed for teachers and students. Therefore, the KCA employs representations such as pie charts and bar graphs that are familiar to secondary school students and teachers. For example, we use pie charts and bar graphs to present the linkage of collaborators rather than employ complex social network grams that often used by researchers. To provide a better picture of the KCA system, we present sample results from KCA analyses for three of the questions.
Are We Collaborating?
Figure 1 presents sample results from a class of 38 secondary school students. The pie chart shows that 53% of students had at least five fellow students build on at least one of their notes. The bar graph on the right shows the exact number of students who built onto the notes of each of these students (e.g., six students built onto the notes of TKP 1A01, and 22 onto the notes of TKP1A33).
Figure 1. Sample view of Are we collaborating analysis results.
Are We Putting our Knowledge Together?

The pie chart in Figure 2 (next page) shows the results for a database in which 10% and 36% of notes contained and were used as references, respectively. The table in the lower part of the figure reveals that many of the links were created in an effort to provide synthesis (e.g., Summary The set of notes begins and as I go through all the comments from teachers.). By providing both statistics on and the details of notes in which references occur, we hope to make possible conversations about both the extent and quality of referencing practices.
How Do Ideas Develop over Time?

The pie chart in Figure 3 shows the results for a database in which 96% of notes were read by at least 2 times. The list on the top right shows the keywords emerged during selected period with the frequency of usage in the bracket. Clicking any notes that within the 96%, we could see the detail of the notes, in which keywords are highlighted and scaffolds in use are shown.
ISLS
363
ICLS2012
Figure 2. Sample view of Are we putting our knowledge together analysis results.
Figure 3. Sample view of How do ideas develop over time analysis results.
Testing and Improvement

We carried out three phases of usability testing on the KCA to check the accuracy of the analyses and determine whether teachers and students find the system and its analyses useful. First, the accuracy of the SQL database was compared with the results of an older tool written in Perl, the Analytic Toolkit (ATK) for Knowledge Forum (Burtis, 1998), which has been used extensively in the past decade and garnered considerable confidence in its accuracy among teachers and researchers. The SQL results had an overall acceptable error rate of less than 5%. Second, we invited teachers feedback at several points and made improvements to the KCA based on their comments. For example, we added a function to enable teachers to export results files to an electronic spreadsheet. Third, we conducted a small-scale field test in two classrooms, involving 10 randomly selected students from each class. The teachers liked the interface and found the KCA easy to use, informative, and efficient. The students found the tool easy to use and gave them fresh perspectives for assessing their performance on Knowledge Forum. They said during interviews that they could interpret the results in relation to their knowledge building dynamics. Due to the length limit, detailed findings and discussions are not adequately addressed in this paper.
ISLS
364
ICLS2012

This paper reports on the development a formative assessment tool which focuses on the use of Knowledge Forum server data to provide students and teachers with information about their knowledge building dynamics for self-assessment and reflection. The results from the preliminary testing of the KCA with a small sample of middle-school students are encouraging. After improving the system design based on usability testing feedback, we are currently using the system in larger classrooms to study how teachers and students use it to improve their knowledge building. Based on proposed conceptual framework, KCA provides analysis on four general questions students may have about their online discourse. The four questions discussed herein are motivated by knowledgebuilding theory, but their application is not limited to it. Having an audience, synthesis, idea improvement, and understanding why some contributions make an impact and others do not are relevant to almost any approach to CSCL. However, they are crucial to knowledge building whose discourse occurs primarily (although not exclusively) online and over a long period of time. Knowledge-building inquiries can be sustained for months (Zhang et al., 2007), whereas typical problem-solving episodes in CSCL approaches require considerably less time. Further, although the SQL database is based on a Knowledge Forum tuplestore, it would not be difficult to create versions that can be applied to other environments, such as Moodle.
References
Biggs, J. (Ed.). (1996). Testing: To educate or to select? Education in Hong Kong at the crossroads. Hong Kong, China: Hong kong Educational Publishing Company. Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education: Principles, Policy, and Practice, 5, 7-74. Burtis, P. J. (1998). Anallytic toolkit for Knowledge Forum. Toronto, ON, Canada: Author. Chan, C.K.K. (2011). Bridging research and practice: Implementing and sustaining knowledge building in Hong Kong classrooms. International Journal of Computer-Supported Collaborative Learning, 6, 147-186. Gress, C. L. Z., Fior, M., Hadwin, A. F., & Winne, P. H. (2010). Measurement and assessment in computersupported collaborative learning. Computers in Human Behavior, 26, 806-814. Hickey, D. T., Honeyford, M. A., Clinton, K. A., McWilliams, J., Shute, V. J., & Becker, B. J. (2010). Participatory assessment of 21st century proficiencies. In V. J. Shute & B. J. Becker (Eds.), Innovative assessment for the 21st century Supporting educational needs (pp. 107-138). New York, NY: Springer. Kolodner, J. L., Camp, P. J., Crismond, D., Fasse, B., Gray, J., Holbrook, J., et al. (2003). Problem-based learning meets case-based reasoning in the middle-school science classroom: Putting Learning by Design into Practice. The Journal of the Learning Sciences, 12, 495-547. Pifarre, M., & Cobos, R. (2010). Promoting metacognitive skills through peer scaffolding in a CSCL environment. International Journal of Computer-Supported Collaborative Learning, 5, 237-253. Polman, J. L. (2000). Designing project-based science: Connecting learners through guided inquiry. New York, NY: Teachers Colledge Press. Salovaara, H., & Jrvel, S. (2003). Students strategic actions in computer supported collaborative learning. Learning Environments Research, 6, 267-284. Scardamalia, M. (2002). Collective cognitive responsibility for the advancement of knowledge. In B. Smith (Ed.), Liberal education in a knowledge society (pp. 67-98). Chicago, IL: Open Court. Scardamalia, M., & Bereiter, C. (1996). Student communities for the advancement of knowledge. Communications of the ACM, 39, 36-38. Scardamalia, M., & Bereiter, C. (2006). Knowledge building: Theory, pedagogy, and technology. In R. K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (pp. 97-115). New York, NY: Cambridge University Press. Scriven, M. (1967). The methodology of evaluation. In R. Tyler, R. Gagne & M. Scriven (Eds.), Perspectives on curriculum evaluation. Chicago, IL.: Rand McNally and Co. van Aalst, J. (2006). Rethinking the nature of online work in asynchronous learning networks. British Journal of Educational Technology, 37, 279-288. Zhang, J., Scardamalia, M., Lamon, M., Messina, R., & Reeve, R. (2007). Socio-cognitive dynamics of knowledge building in the work of 9- and 10-year-olds. Educational Technology Research & Development, 55, 117-145.
Acknowledgments
This research was supported by a Seed Funding project from the University of Hong Kong (Project #20091115918) and the Seed Grant from the Strategic Research Themes - Sciences of Learning, University of Hong Kong. The authors thank all teachers, researchers, and students who provided comments on the system.
ISLS
365
ICLS2012
Effects of Argumentation Scaffolds and Problem Representation on Students Solutions and Argumentation Quality in Physics
Carina M. Rebello, University of Missouri, Science Education Center, cp5xc@mail.missouri.edu Eleanor Sayre, Kansas State University, Department of Physics, esayre@phys.ksu.edu N. Sanjay Rebello, Kansas State University, Department of Physics, srebello@phys.ksu.edu Abstract: Prior studies have revealed students difficulties in problem solving with multiple representations due to poor reasoning skills. Research has shown that inclusion of argumentation tasks can improve these skills. We investigate alternative forms of argumentation scaffolds (to construct or evaluate an argument) integrated within physics problems, and their impact on students solutions and argumentation quality. Results suggest that the scaffolds, not mere inclusion of argumentation, can improve students argumentation skills. Additionally, in verbal and graphical representation problems, evaluation of an argument with prompts improves students conceptual quality of the solutions.
Introduction
Since the early 1980s, there have been numerous studies regarding college students difficulties with conceptual understanding and problem solving with multiple representations in physics due to poor reasoning skills (Hsu, Brewe, Foster, & Harper, 2004). Innovative pedagogical approaches such as Physics by Inquiry (PbI) (McDermott, 1996) have emerged within the physics community to help enhance students scientific reasoning skills. PbI employs both open-ended question tasks and hypothetical student debate tasks requiring application of conceptual knowledge and reasoning skills. Such debate tasks provide a useful context for emphasizing reasoning skills and argumentation in the curriculum. Prior studies have shown that embedding argumentation activities in problem solving can improve problem solving abilities, ways of thinking, and conceptual change (Jonassen & Kim, 2010). Argumentation activities can enhance scientific reasoning, facilitate construction and enhancement of knowledge (Driver, Newton, & Osborne, 2000), improve proficiency in advancing, critiquing and justifying claims, and facilitate formative assessment (Osborne, Erduran, & Simon, 2004). An emphasis on argumentation is consistent with the goal of improving students reasoning in problem solving and proficiency in advancing, critiquing, and justifying claims (Kuhn and Udell, 2003). Research also suggests that the development of argumentation skills may lead to deeper conceptual understanding. However, the relationship between argumentation and conceptual understanding is not straightforward (Dawson & Venville, 2009). In this study we examined the effects of alternative forms of argumentation on students physics solutions. The PbI problem format was modified to scaffold students construction or evaluation of arguments and compared to typical PbI question formats. Additionally, the goal of this study was to examine the effects of argumentation and quality of solutions for problems consisting of different forms of representations.
Literature Review
Argumentation is a critical thinking used in formal and informal contexts to resolve questions, make decisions, formulate ideas, and solve well- and ill-structured problems (Cho & Jonassen, 2002; Jonassen, 1997). It has been used to assess problem solving abilities (Jonassen & Kim, 2010). Kuhn (1991) identifies five key argumentation skills: generating causal theories offering supporting evidence, envisioning conditions that undermine ones theory, generating alternative theories, and rebutting alternative theories. Successful argumentation requires a problem solver to develop and articulate a reasonable solution, support the solution with data and evidence, and identify alternative solutions (Cho & Jonassen, 2002). The strength of an argument depends upon the context and nature of the task (Newton, Driver, & Osborne, 1999. Studies have found that most U.S. students lack argumentation skills and have difficulty comprehending arguments, writing persuasive essays, differentiating theory and evidence, and producing evidence, alternatives, counterarguments, or rebuttals (Reznitskya et al., 2001). Zeidler (1997) identified several students problems with argumentation, such as: selecting only evidence that supports their claim or my-side bias, reliance on personal beliefs rather than counter evidence, overgeneralization, and making unsupported assertions. Additional studies found that students tend to rely on data to support their claims but frequently do not include warrants or backings (Bell & Linn, 2000). These findings have implications for the design of classroom activities to improve students argumentation. Research on students problem solving reveals that most students use a means-end approach and have difficulty solving problems with multiple representations such as graphs (Meltzer, 2005; Rosengrant, Etkina, & van Heuvelen, 2006). To become better problem solvers students must improve their conceptual understanding, learn to solve problems using multiple representations (Rosengrant et al., 2006), and must be engaged in tasks that require meaningful justifications requiring application of argumentation during problem
ISLS
366
ICLS2012
solving (Jonassen et al., 2009). Argumentation consists of two skills constructing and evaluating arguments. Effective argumentation requires evaluating, weighting evidence and counterevidence, and considering arguments, counter-arguments, and rebuttals to arrive at a solution (Jonassen et al., 2009). Here we explore how introductory physics students can be supported to achieve superior argumentation skills in problem solving.
Toulmins argumentation pattern (TAP) (1958) which is commonly used to assess and/or instruct learners about argument structure describes argument features as (i) claims conclusions or assertions, (ii) data facts that provide the foundations of the claim, (iii) warrants reasons, rules, or principles that provide connection between data, claim, or conclusion, (iv) backing assumptions used to provide justification of the warrant, (v) qualifiers conditions or limitations under which the claim is true, and (vi) rebuttals conditions when the claim is not true (Driver et al., 2000). In this study a rubric adapting TAP is used to assess the structures of arguments offered by students.
Research Questions
We addressed the following research questions in the context of solving conceptual physics problems. What level of argumentation and conceptual quality do introductory college physics students, without training in argumentation, demonstrate in their solutions to physics problems? How do the argumentation and conceptual quality of students solutions for these problems change based on o the representation verbal or graphical in which information is provided in the problem? o whether the problem requires learners to construct or evaluate arguments? o the use of specific prompts designed to scaffold the construction and evaluation of arguments?
Research Context
This study was conducted at a large, US public university, in a semester-long (15 week) course offered in the physics department. Participants were enrolled in an algebra-based physics course. The course lecture met three times weekly for 50 minutes each and one laboratory (110 minutes) and one recitation (50 minutes) a week. Topics covered include kinematics, dynamics, work and energy, rotational motion, oscillatory motion, fluids, wave and sound, and thermodynamics. The course is taken primarily by life/health science majors with an enrollment of about 400 students and employs a textbook and online homework system. Homework, recitation problems, and exams tend to emphasize quantitative problem solving rather than argumentation.
Data Sources
We created two problems problem 1 (in verbal representation) and problem 2 (in graphical representation) both targeting a key kinematics misconception (Trowbridge & McDermott, 1980; Beichner, 1994). Each problem was presented in four conditions construct, evaluate, control construct, and control evaluate with argumentation prompts shown in Table 1. Evaluate and construct prompts were adapted from Jonassen et al. (2009) and Mason and Scirica (2006). Control prompts were similar to those used in PbI. Table 1: Prompts provided in each of the condition Construct Prompts Construct an argument to justify your answer. Explain your position clearly and completely by providing all reasons that support your conclusion. Remember to consider: What evidence supports your reasons? One of your classmates may disagree with your conclusion. What might they think is the alternative conclusion? What reasons would your classmate provide to support their conclusion? What would you reply to your classmate to explain that your position is right? Control Construct Prompts What is your answer? Explain your reasoning. Evaluate Prompts Which statement (of the ones provided above) best describes the physical phenomenon? Or do you have another argument? Explain, elaborate, and justify your preferred solution. Remember to consider: What evidence and reasons support your selection? Explain your reasoning for not choosing the alternative solution(s). What are the weaknesses in the alternative argument(s)? How might a classmate supporting the other solution disagree with your preferred solution? What would you reply to your classmate to explain that your position is right? Control Evaluate Prompts Which statement do you agree with? Or do you have another answer? Explain your reasoning.
ISLS
367
ICLS2012
The study was completed four weeks into the semester. The problem versions were administered randomly to students in 10 out of 19 laboratory sections, each enrolling approximately 20 students. Participants were offered extra credit as incentive and assured confidentiality. Response rate was at least 95% for each of the participating laboratory sections yielding a total of 198 respondents. Figure 1 shows a flow chart of the design.
198 respondents in 10 out of 19 lab sections
Construct (N = 65) Prob1 (verbal) [Construct] Prob2 (graphical) [Construct] Evaluate (N = 68) Prob1 (verbal) [Evaluate] Prob2 (graphical) [Evaluate] Control (N = 65) Prob1&2 (verbal & graphical) [Control Const] [Control Evaluate] (graphical) [Evaluate] Figure 1. Flow chart showing the design of the study
Data Analysis
After reading through students written responses, we created three scoring rubrics two for conceptual quality and one for argumentation quality. The first two rubrics were designed to analyze students physics conceptual quality of their problem solutions for each problem. The holistic conceptual quality rubric for Problem 1, which had two possible answers, was on a six-point scale: incorrect answer, does not recognize either of the possibilities, with little or no reasoning (0); incorrect answer does not recognize either of the possibilities but has some good scientific reasoning (1); recognized one of the possibilities with incomplete or no reasoning (2); recognizes one of the possibilities with fully correct reasoning (3); recognizes both possibilities with reasoning, but does not fully justify both possibilities (4); and recognizes both possibilities with correct reasoning (5). The holistic conceptual quality rubric for Problem 2 was an adaptation of the first rubric, but because there was only one scientifically acceptable solution, the rubric had only four levels: incorrect answer with little or no scientifically correct reasoning (0); incorrect answer with some good scientific reasoning (1); correct answer with some incorrect scientific reasoning (2); and correct answer with correct scientific reasoning (3). To assess argumentation quality of students written responses we used a rubric adapted from Sadler and Fowlers (2006) Argumentation Quality Rubric, which in turn was adopted from Toulmins (1958) TAP framework. The rubric allows for analysis of how claims are justified on a five-point scale: no justification provided (0), to one in which claims are justified with elaborate grounds and a counter position, in which contradictory evidence is also evaluated (4). Grounds refer to possible supports for a justification including: data, backing, and warrants. This rubric was adapted into a six-point rubric to allow not only the inclusion of a counter position(s) in the justification statement but also the inclusion of a rebuttal(s) in their solutions to address counterclaim concerns (5). Both conceptual and argument quality were coded by two independent raters (including the author) who were familiar with the design and purpose of the study. After independent coding, the raters discussed all codes to reach 100% agreement. In order to compare the quality of arguments offered by participants responses in each of the four conditions (construct, evaluate, control-construct, and control-evaluate), a multivariate analysis of variance (MANOVA) was performed. The four conditions (construct, evaluate, control construct, and control evaluate) served as the independent variable and the quality of conceptual quality scores and quality of argumentation scores as the dependent variables. Univariate analysis of variance (ANOVAs) was conducted to determine if the conditions have a significant effect on each dependent variable (conceptual quality and argumentation quality).
Findings
Participants from all conditions demonstrated variability in conceptual and argumentation quality in response to the verbal and graphical problem statements. The distribution of scores appears to vary across both problems. Means and standard deviations of conceptual and argumentation quality for each problem are shown in Table 2. Table 2: Means and standard deviations for conceptual quality and argumentation quality on problems 1 and 2. Conceptual Quality Argumentation Quality Problem 1 Problem 2 Problem 1 Problem 2 Version (verbal) (graphical) (verbal) (graphical) Mean1 SD1 Mean2 SD2 Mean1 SD1 Mean2 SD2 Construct 2.062 1.01 0.862 0.63 4.462 0.90 4.369 1.15 Evaluate 2.368 1.53 1.691 1.11 3.824 0.73 3.456 0.97 Control Construct 1.569 1.03 1.200 1.16 2.339 0.76 2.108 0.94 Control Evaluate 1.508 1.40 1.477 1.23 2.554 0.81 2.231 0.67 The MANOVA analysis using conditions (construct, evaluate, control construct, and control evaluate) as independent variables and conceptual quality scores and argumentation quality scores as the dependent variables revealed a statistically significant difference among the four conditions for the two provided problems [Wilks = 0.347, F(12.0, 677.6) = 27.8, p < .001, 2 = 0.297]. Univariate ANOVAs were explored to test if
ISLS 368
ICLS2012
the independent variable has a significant effect on each dependent variable. Results (see Table 3) show significant differences between conceptual and argumentation scores for the two problems. Table 3: Univariate ANOVAs. Problem 1 (Verbal) Argumentation Quality Problem 1 (Verbal) Conceptual Quality Problem 2 (Graphical) Argumentation Quality Problem 2 (Graphical) Conceptual Quality F 104.97 6.94 84.05 7.74 df 3, 259 3, 259 3, 259 3, 259 p .000 .000 .000 .000 Effect Size .549 .074 .493 .082
Follow-up Tukeys HSD analysis with an overall alpha level of .05 revealed that for problem 1 (verbal) argumentation quality, there is a significant difference between scores on evaluate and construct conditions, the latter having a greater argumentation quality. Scores on the evaluate condition were also statistically greater than scores on the control conditions. Additionally, scores on the construct condition were statistically greater for argumentation quality than scores on the control conditions. Thus, results for argumentation quality in problem 1 (verbal) indicate that the construct condition with prompts yields a higher argumentation quality than the evaluate condition with prompts or either of the control conditions. Also the evaluate condition with prompts tends to yield higher argumentation quality than either of the control conditions as well. Results suggest that the construct and evaluate prompts used in problem 1 do help produce higher argumentation quality than the control problem statements, which are similar to problem statements used in the PbI curriculum. Also for the verbal problem statement, a construct argument condition with prompts produced higher argumentation quality. For problem 1 (verbal) conceptual quality in problem solutions, scores on the evaluate condition were statistically greater than scores on the control conditions. Thus in the evaluate condition, if prompts are provided students have a higher conceptual quality in their problem solutions, but problem format (construct or evaluate) does not make a statically significant difference on the conceptual quality of problem solutions. For problem 2 (graphical) argumentation quality scores on construct were statistically higher than scores on control conditions. Also there were significant differences for scores on evaluate and control conditions. Scores on the construct condition were also statistically higher than evaluate conditions. Thus, results for argumentation quality in problem 2 (graphical) indicate that prompts are important, particularly for the construct condition rather than for the evaluate condition. Yet, problem format does not make a difference on students argumentation quality. Finally, for problem 2 (graphical) the evaluate conditions yielded a higher conceptual quality in the problem solution than construct conditions. Also, there was a statistically significant difference between evaluate and control-construct conditions and between construct and control-evaluate conditions. Thus, results for conceptual quality in problem 2 (graphical) indicate that prompts do not matter for the graphical question to improve conceptual quality. If prompts are given, they are more effective in the evaluate condition than the construct condition but the use of prompts is not more effective than the corresponding control conditions.

Our results indicate there are statistical differences between the four conditions for each problem. Both problem 1 (verbal) and problem 2 (graphical) assess the same conceptual topic (kinematics) and were prone to the same misconception. Though their representations were different, the results of the argumentation qualities of the two problems were very similar across the different conditions. Results for argumentation quality in problem 1 indicate that if prompts are provided, the construct condition produces higher argumentation quality. Additionally, for argumentation quality in problem 2 the construct condition produces a higher argumentation quality than the evaluate or control conditions. Hence, results regarding argumentation quality suggest that the problem format (construct/evaluate) does not make a difference but whether or not prompts are used does make a difference. Similar to Jonassen et al.s (2009) results, argumentation quality for each problem representation suggests that using construct and evaluate alternative argument tasks with prompts produce higher argumentation qualities. In fact, typical problem statements (explain your reasoning) used in the PbI and other curriculum tend not to produce higher argumentation quality if students are not appropriately guided to provide reasons. Hence, the open-ended and hypothetical student debate problems as used in the PbI curriculum may not produce higher argumentation level such as considering possible counter-arguments and rebuttals, without appropriate scaffolding to do so. However, unlike Jonassen et al.s (2009) results, our results indicate that there is a significant difference between construct and evaluate conditions for stronger argument structures. There were also notable similarities between the problems in conceptual quality. It seems that the evaluate condition with prompts for problems 1 (verbal) and 2 (graphical) can have a greater impact on conceptual quality. For problem 1 (verbal), the construct condition does not outperform the evaluate condition on conceptual quality. Specifically, the use of a hypothetical student debate problem used in PbI may not produce higher conceptual quality justification used in arguments without suitable scaffolding to appropriately
ISLS
369
ICLS2012
evaluate arguments and justify a position. However, results indicate that there was no difference in conceptual quality between construct and evaluate conditions indicating that argumentation formats do not make a difference. Also, there are no conceptual quality differences between construct and control-construct conditions. Thus, the use of prompts does not make a significant difference in conceptual quality in the construct condition. Finally, results for conceptual quality in problem 2 (graphical) indicate that the evaluation condition yields a better conceptual quality problem solution than the construct condition or the control-construct condition. This result could be attributed to the presentation of two alternative graphical solutions, one with a misconception and the other scientifically correct, for students to evaluate and choose from as opposed to constructing justifications entirely on their own. Interestingly for problem 2 (graphical), prompts do not improve conceptual quality. Yet if prompts are provided, they are more effective for the evaluate condition than the construct condition. The difference in the effectiveness of the prompts depending upon problem representations has not been probed previously. Thus in light of the results obtained here, more research in this area is warranted to study the issue further.
References
Beichner, R. J. (1994). Testing student interpretation of kinematics graphs. American Journal of Physics, 62, 750-762. Bell, P., & Linn, M. C. (2000). Scientific arguments as learning artifacts: Designing for learning from the Web with KIE. International Journal of Science Education, 22, 797 817. Cho, K. L., & Jonassen, D. H. (2002). The effects of argumentation scaffolds on argumentation and problem solving. Educational Technology Research and Development, 50, 5-22. Dawson, V., & Venville, G. J. (2009). High-school students informal reasoning and argumentation in biotechnology: An indicator of scientific literacy? International Journal of Science Education, 31, 1421-1445. Driver, R., Newton, P., & Osborne, J. (2000). Establishing the norms of scientific argumentation in classrooms. Science Education, 84, 287-312. Hsu, L., Brewe, E., Foster, T. M., & Harper, K. A. (2004). Resource letter RPS-1: Research in problem solving. American Journal of Physics, 72, 1147-1156. McDermott, L. C. (1996). Physics by inquiry: An introduction to physics and the physical sciences. New York: J. Wiley. Meltzer, D.E. (2005). Relation between students problem-solving performance and representational format. American Journal of Physics, 73, 463-478. Jonassen, D. H. (1997). Instructional design model for well-structured and ill-structured problem-solving learning outcomes. Educational Technology Research and Development, 45, 65-95. Jonassen, D. H., & Kim, B. (2010). Arguing to learn and learning to argue: design justifications and guidelines. Educational Technology Research and Development, 58, 439-457. Jonassen, D. H., Shen, D., Marra, R. M., Cho, Y-H., Lo, J. L., & Lohani, V. K. (2009). Engaging and supporting problem solving in engineering ethics. Journal of Engineering Education, 98, 235-254. Kuhn, D. (1991). The skills of argument. Cambridge, UK: Cambridge University Press. Kuhn, D., & Udell, W. (2003). The development of argumentation skills. Child Development, 74, 1245-1260. Mason, L., & Scirica, F. (2006). Prediction of students argumentation skills about controversial topics by epistemological understanding. Learning and Instruction, 16, 492-509. Newton, P., Driver, R., & Osborne, J. (1999). The place of argument in the pedagogy of school science. International Journal of Science Education, 21, 553-576. Osborne, J., Erduran, S., & Simon, S. (2004). Enhancing the quality of argumentation in school science. Journal of Research in Science Teaching, 41, 994-1020. Reznitskya, A., Anderson, R.C., McNurlin, B., Nguyen-Jahiel, K., Archodidou. A., & Kim, S.Y (2001). Influence of oral discussion on written argumentation. Discourse Processes, 32 (2&3), 155-175. Rosengrant, D., Etkina, E., & Van Heuvelen, A. (2006). An overview of recent research on multiple representation. Paper presented at Physics Education Research Conference, Syracuse University, Syracuse, NY. Retrieved from http://dx.doi.org/10.1063/1.2508714. Sadler, T. S., & Fowler, S. R. (2006). A threshold model of content knowledge transfer for socioscientific argumentation. Science Education, 90, 986-1004. Trowbridge, D. E., & McDermott, L. C. (1980). Investigation of student understanding of the concept of velocity in one dimension. American Journal of Physics, 48, 1020-1028. Toulmin, S. E. (1958). The uses of argument. Cambridge, England: Cambridge University Press. Zeidler, D. L. (1997). The central role of fallacious thinking in science education. Science Education, 81, 483496.
ISLS
370
ICLS2012
Inevitable Breakdowns in Putting Argumentation into Practice in Classroom Talk

Baruch Schwarz, The Hebrew University, Jerusalem, msschwar@mscc.huji.ac.il Nitsa Shahar, Levinski College, Tel Aviv, nitzas@netvision.net.il Abstract: The present paper reports on a history course in which the teacher implemented argumentative practices that integrated dialogic and critical aspects. The analysis of some successive activities shows that the establishment of argumentative norms in the class was far from smooth, although the teacher attempted to encourage collaboration among students. We show that when it is about blending dialogic and critical aspects of argumentation, breakdowns are inevitable but coping with them successfully leads to highly productive classroom discussions. The state of affairs on the effect of certain forms of classroom talk on learning and development is extremely complex. This complexity comes first from what researchers by desirable forms of talk. For example, exploratory talk (Mercer, Wegerif, & Dawes, 1999), collaborative reasoning (Reznitskaya, Anderson, & Kuo, 2007) or dialogic teaching (Burbules, 1993) are as many forms of talk for which power relations are flexible, authority is shared among group members, questions are open or divergent, and products and processes of discussions are continually scrutinized by the discussants. Research done on the effects of these forms of talk shows inconsistencies: For example, Mercer, Wegerif and Dawes showed that prolounged engagment in exploratory talk results in development of reasoning skills while Reznitskaya, Anderson, and Kuo showed lack of effect of collaborative reasoning on argumentative writing, or argument recall. These inconsistencies may originate from the fact that the forms of talk are not identical. It seems that while all the forms above share dialogicity, some stress criticality as another characteristic of desirable talk. The problems with research on the effects of certain forms of classroom talk on learning and development are not only ontological, though, but pertain also to pedagogy and to methodology. From a pedagogical point of view, teacher-led discussions are always part of larger activities in which the approach adopted by the teacher in the discussions can be understood. For example, a more authoritative style may be justified, from the point of view of a teacher whose general approach is dialogical, to summarize ideas previously reached in small discussions. As Mercer (2008) posited, classroom talk needs temporal analysis, continuity that helps capitalizing on ideas sown in one activity and that germinate in another. Several studies have shown that effects that seem negative after a short time seem turn to positive after more time. For example, Morehouse and Williams (1998) evaluated the effects of a 3-year instructional intervention using P4C pedagogy (a form of dialogic teaching) on argumentative writing. Performance declined, during the 1st year of the program. However, during subsequent 2 years, P4C students showed a significantly better ability to construct written arguments, compared to other students. But, and this is a major methodological problem, long term interventions in schools cannot rely on the exclusive enactment of desirable forms of talk. What is measured in the long run is the result of the enactment of multiple practices whose diversity is inevitable in classrooms. In this context, measuring effects of forms of talk on learning and development in an experimental way is very difficult to conduct. It is vain to attempt to disentangle their specific contributions in the blended character of classroom activity. It is then not surprising that the general picture in research findings on the effects of the enactment of specific forms of talk in classrooms mentioned above is quite inconsistent. We do not utterly oppose to such studies but opt for a vigilant position concerning the use of inferential statistics for studying classroom talk. We agree with Mercer to additionally opt for qualitative methods that help tracing successive various activities in a temporal analysis. In the background of this complex picture on argumentation teaching in classrooms, the Kishurim group has initiated a design research program dedicated to foster both dialogic and dialectical aspects of argumentation in classrooms. The group first relied on educational research to recognize that dialogic and dialectical aspects of argumentation are difficult to sustain and that argumentation should be facilitated. The Kishurim group constructed several tools, among them Digalo (Schwarz & de Groot, 2007), a graphical tool that affords dialogic and dialectical moves in synchronous e-discussions (described further on). The tool was extensively used in small group discussions. In parallel, the Kishurim group implemented in schools argumentative practices that resembled in some way P4C practices: critical discussions around texts, resolution of dilemmas with societal impact, argumentative writing. Also, teachers were trained to lead critical discussions, to encourage collaboration, to integrate technologies tools in their teaching, and to blend different practices to foster argumentation in their classes.
The educational setting

ISLS 371
ICLS2012
11 Grade 9 students (6 females and 5 males) from a junior high-school of a large city in Israel participated in a course consisting of twenty-two 90 min. long meetings on the theme Childhood in wartime with special attention to years 1933-1940. The course focused on activities in history, argumentation and inquiry learning. These activities were first introduced separately to teach their basics, but became intermingled later on. For example, in session 11, the tension during wartime between the will to become an adult and to remain a child was discussed. Different texts (written documents, pictures and maps) were at students disposal. Means of communication varied: face-to-face, synchronous communication in the same computer room, or while each participant being at home through graphical software Digalo (Schwarz & Glassner, 2007).
Figure 1. A Digalo Discussion Figure 1 displays a Digalo discussion around the dilemma children during the thirties in Nazi Germany. It is in a form of a map with graphical and textual features. Turn 2 is embodied in a form of a question in which the teacher Yoram presents the topic of the discussion. The content of the question appears in the lower right window. Five students (Sima, Ben, Yossi, Shir and Ron) whose identity is represented in an icon in the lower left window, participate in the discussion. The different shapes of the map represent argumentative functions (argument, explanation, question, support, opposition, etc.) students created from their ontology tool bar. At the end of the course, the teacher a former high-tech specialist, received a prize on his teaching and on the site he constructed to scaffold learning. Digalo was used intensively in 14 out of 22 sessions. The teacher designed the activities, the materials and the program. He provided multiple texts that suggested different perspectives with which students were encouraged to engage in dialogic argumentation. At the end of the course, the students were invited to engage in a dyadic inquiry-based activity that included the formulation of a research question, the evaluation of sources and the co-writing of an argumentative essay.
Tracing the establishment of argumentative norms in successive activities

The present paper is aimed at observing how dialogic and dialectical norms were instilled in the course and to show that this instilment was far from being smooth. Let us see first describe a Digalo discussion. We present here the textual translation of the argumentative map displayed in Figure 1. Chosen forms are underlined and titles italicized: 2 Question Yoram German children during the thirties in Germany From Samuel Kelners testimony, we learn that impressive ceremonies, the flags of the German Reich, marches, and the soldiers that paraded in the streets in uniforms deeply impacted on German children. Many of them turned to admirers of the regime and of its ideology. Why do you think that these demonstrations of power impressed children so much? Did the demonstrations impressed children more than adults? What do you learn from this? 3 Claim Ben The strong influence of demonstrations on children In my opinion, the demonstrations of power by the Nazis influenced children because its easy to teach children to believe in something, and in addition, the Nazis exposed their ideas convincingly and in a rational way that added and strengthened the childrens opinions 4 Claim Yossi Arrow of support to Ben in 3 The strong influence of demonstrations on German children In my opinion, the demonstration of power of the Nazis had a deep influence on children because children admire strong characters (the Nazis invaded many regions in Europe and were very strong). For example, small children admire figures such as Superman and Spiderman because they are strong. At that time the Nazis showed they were strong and children were influenced. 5 Claim Shir Arrow of support to Ben in 3 and to Yossi in 4
ISLS 372
ICLS2012
Children and their parents I think that children were very much influenced from all the fuss that the demonstrations did [] In contrast to them, their parents could have seen demonstrations without being drafted to the anti-Semitic movement. Unfortunately a part of them were influenced by the demonstrations and collaborated. 6 Claim Sima Arrow of opposition to Ben at 3 Hitlers demonstrations In my opinion, children were more easily influenced from Hitlers demonstrations of power because they look for characters with whom they can identify, and Hitler was presented as a hero. 11 Information Sima Without any arrow Education in Germany In Germany, Jewish children were driven out from schools and pupils that remained began learning Nazi principles. The educational system coerced on children Hitlers positions and by such created a distorted perspective on Jews and other people. 12 Claim Shir Arrow of opposition to Ben at 3; Arrow of support to Sima at 11 Not sure You think that children were more influenced than parents? In my opinion, this is not true. Its a fact that many adults supported the Nazi ideology as a result of the demonstrations 14 Argument Yossi Neutral link to Ben at 7 Your answer is not clear. Could you explain it? 15 Question Yoram Without arrow So childrens ability to judge limited so that this is the role of adults to supervise them. Do you agree with this? 19 Claim Shir Arrow of opposition to Yoram at 15 The problem is that adults too were drafted by the anti-Semitic movement The role of the adults is indeed to supervise children and to teach them not to believe to whatever they come across, but in the case of the numerous demonstrations and the activities of the Nazi ideology, the adults too were fooled by the Nazi ideology. We can see that in turns 3-6, the students chose the form of an argument in order to give a reasoned answer to the teachers question. We also see that in their reasoned positions, the discussants refer also to their peers through support (e.g., in 5 where Shir refers both to Ben and Yossi) or through opposition (e.g., in 6 where Sima opposes to Ben). After her opposition is rejected, Sima adds information as objective data (hence without any arrow) in 11. This information conveys a strong argument concerning the role of education in the Nazi indoctrination of children in Germany which led Shir to agree with Sima and to oppose Ben in 12. Yorams question in 15 is in a disguised challenge to the agreed argument that judgment abilities are limited among children. Shir reacts in 19 by the counterchallenge that adults too were equally influenced. The double characterization of Digalo communication the argumentative function by choosing adequate graphical forms, and their content lead the discussion to take place in two planes, argumentative and content-related. Scrutiny over this isolated discussion may lead to the conclusion that students were encultured to argumentative practices combining dialectical and dialogical aspects. But does enacting specific practices mean that the participants have appropriated the norms that these initiated practices seem to convey? The teacher Yoram provided numerous occasions for testing appropriation of norms. For example, just after the Digalo discussion above, he asked one discussant from each discussion group to summarize the discussion and to present it to the other group. Sima was designated in the first group and Neta in the second. Such a task seemed anodyne. Lets see how the teacher handles this activity in the second group. Karin is appointed head of her group, but just after she writes the summary she extracted from her Digalo argumentative map, and reads it aloud to her peers, Liron reacts vehemently: 18Yo: I try to understand the dispute 19Lir: I claim that she still contradicts herself 20Yo: I propose not to focus on this dispute! The discussion was far wider than your dispute! 21Li: No, I explain to you 22Yo: Well, go on, ignore this dispute and go on, so that people know what happened in this discussion. For example, you have what Neta wrote, what Maya said. One referred to what you said, Ron? [Liron gets crazy that Karin sits down and types on the computer and the rest of her group listen to her] 23Yo: Liron, do you have another idea? 24Li: For, every comment she writes, she should first tell it to all of us, and we all will summarize our views, and then everybody will write an integrated answer to turn it to something acceptable for all of us! 25Yo: Karin, you should tell to the other what you write, to turn it to acceptable for all 26Li: But this is nonsense if she is the only one to write 27Yo: When Karin formulates something, she should present it to everybody. Does somebody want to add 28Li: Why shouldnt we add to the formulation, we belong to the group too! 29Yo: Somebody should gather and organize things. When you will be a group leader, youll organize, too 30Li: Neta is the group leader, not Karin! 31Ka: Neta gave me the opportunity to concentrate, because she had difficulties doing it
ISLS
373
ICLS2012
32Li: [stands up in front of the group] I waste my time when sitting here. [Turns to the teacher] You said that the goal here was to collaborate with each other. So, when she writes formulations, she must collaborate! 33Ka: You quarrel on something unimportant! 34Li: This is surely about principles! I can sit wherever I want [she moves away from the group], and then you say that I dont collaborate! Whats the benefit of simply sitting, idle? She should collaborate with us, she should say things aloud! This dispute overtly concerned social issues: individual writing of a summary of a discussion in the presence of her peers undermines the value of collaboration fostered by the teacher in Digalo discussions. Liron argues with the teacher (19, 21, 24, 26, 28, 30) more than with her peers. It appears that the head of the group was replaced (30-31), and this fact irritates Liron who feels that she could serve (better) as the head of the group. It seems that the teacher did not know how to handle events in which collaborative values were at odds with other social goals. The dispute with Liron shows that the practice of summarizing conflicted with critical reasoning and dialogicity, in her harsh protestations. Liron makes it clear to the teacher and her peers that discussions does not mean only sitting and chatting together, through agreement with the head of the group (32, 34). Rather, each participant has the right to plead his/her case (24, 28, 32, 34). This fact exemplifies a deep change in the behavior of discussants in capitalizing on their discussion space. Liron brings to surface the duality of the right and the duties of participants in dialogic argumentation: on the one side, the liberty to decide when and how to intervene, and on the other side, the commitment to enable others to intervene and to disagree even when what is said does not fit own standards. She clearly expresses that the norms of collaboration the teacher championed were violated (24, 26, 28). In 32 she affirms You said that the goal here was to collaborate with each other. So, when she writes formulations, she must collaborate! to express that she takes seriously the teachers advice, not as an isolated script, but as a norm of behavior. In 33, the teacher misses the point and misinterprets Lirons obstinacy as something unimportant. He probably links this behavior to a trait of Lirons personality, the fact that she is a rebel. This is probably true but she is right and beautifully articulates in 34, that her rebellion in the present case reflects a principle, a norm of behavior that she (so to speak) appropriated: This is surely about principles! I can sit wherever I want, and then you say that I dont collaborate! Whats the benefit of simply sitting, idle? Many cases of tension originating from the collision between norms of critical reasoning and norms of dialogicity occurred. The reiteration of tensions led to a special atmosphere in the class, in which criticality and dialogicity cohabitated. This is a healthy and unsolved tension! For example, the following protocol presents a discussion between the teacher, Liron and her pals during lesson 5 about the principles of dictatorship: 1Li: You said that dictatorship is not good. As soon as you say this, you behave as a dictator! 2Yo: What you say is bright! 3Li: So now we have a bit dictatorship, and this is a proof that dictatorship is not necessarily something bad 4Ne: I did not understand what you said 5Yo: What Liron said is that I and you look like somebody who supports democracy, but in fact you are a dictator: You keep me silent, and you tell me this is good, it puts some order 6Li: [in an ironic tone] Fairly true! 7Yo: The school acts in a democratic spirit, it is not entirely democratic but the atmosphere is democratic! 8Ne: 9Li: 10Yo: 11Si: But were here in the class. Its not the same Its also democracy; it can be clean up the floor, arrange the chairs! Sima, what did you want to say? I think that this isnt a democracy. Second, you dont make me feel angry, there are norms established by people, so that will not be considered as dictatorship! 12Yo: You mean that rules in school set the[Liron tries to interrupt Yoram]. Could you listen, Liron? You gave a good comment, Sima commented on you and I want to add something. In dictatorships, basic rights are not respected, such as human dignity and human freedom (Liron tries again to break out)... In schools, there are rules. In democracies, too, people are obedient to rules. Otherwise, its impossible to administrate a country. In schools, rules are a bit harsher. [] I can give you some extra work, castigate you, and write a letter to your parents but to beat you, to lock you up things that people did once in non-democratic countries [Liron smiles]. You got it? Schools breathe democracy. They are not utterly democratic, but live in a democratic atmosphere 13Li: I dont say this is bad, I say that a bit of dictatorship puts some order! Its possible to establish some dictatorship in the government! The discussion stresses a tension, and this time a positive one, between dialogicity and critical thinking: The teacher praises Liron (2) for her challenging question (1). In (3), she elaborates her challenge by arguing in favor of a bit of dictatorship. This sophisticated argument is not well understood by Neta (4). The teacher clarifies Lirons challenge (5), and by doing so, establishes an expression of closeness that gives her lot of satisfaction (6). In (7), the teacher tries to temperate Lirons challenge. This intervention opens the dialogue
ISLS
374
ICLS2012
with the whole class and Neta counter-challenges Lirons argument by noticing that it is irrelevant to classroom reality (8) to which Liron reacts in (9). In (10), the teacher orchestrates again the dialogical-critical discussion by inviting Sima to contribute after he noticed that she wants to speak. Sima opposes Lirons challenge from a new direction societal norms: what happens in classrooms cannot be called dictatorship. At this point, the teacher decides to develop a long argument that integrates both Liron and Simas arguments about rights and the necessity of law. This development is accompanied by strong emotional reactions from the part of Liron but is finally accepted. It seems that not only Liron but also the teacher and the other participants felt a lot of satisfaction as provocative interventions resulted in a reasonable solution reflecting a variety of ideas, and most importantly and integrating the opinions of the discussants. This solution could not be appreciated if it would not have been the conclusion of a dialogical process.
Putting into practice argumentation as a solution or as a problem to solve?

The course Childhood in wartime was organized as a series of activities during which the teacher implemented various argumentative practices: from guided discussions on what a good argument /discussion is, to extraction of arguments from a text, unguided Digalo discussion, summarizing discussions, presentations or writing of argumentative essays. The high level of the outcomes we presented is surprising: the teacher is not always at the center of this community. Students ask for explanations, challenge, and do not accept the teachers ideas as such. Beyond all that, it appears that the students are emotionally moved and dialogicity reigns over all discussion: all discussants refer to each other respectfully and right to the point in reasoning chains; also, discussants are critical as they rely on historical documents, compare conflicting sources, challenge each other (including the teacher) with harsh questions, but attempt to reach reasoned conclusions. It seems then that the students constitute a community of learners dedicated to argumentation. The origins of this success are multiple. First, students internalized argumentative norms: In 14 of the 22 lessons, Digalo was used. And since students had to comply with the ontology proposed by Digalo, they progressively internalized argumentative norms. Another reasonable factor of the success of the course in turning the group of students into a community of learners complying with argumentative norms is the design of the course that included contents in history, inquiry, and argumentation, first separated and explicit, then combined and generally implicit: At the end of the course, activities relied on multiple sources, and students had to search for relevant information, to test its accuracy by consulting other sources and to discuss best interpretations, etc. However, the tool and the design did not lead only to solutions, to smooth to compliance with argumentative norms and to high-quality discussions but also to tensions: inviting to dialogicity was problematic when choosing between ideas or between people to fulfill a role was at stake. The teacher often found himself in big difficulties and sometimes failed in supporting a weak student in summarizing a discussion in the presence of her peers, in justifying why one student should take the lead in a collaborative activity because of the high standards of criticality he himself demanded from his students. Also, the most critical students used the tools he taught them to undermine his authority. More seriously, smugly knowledgeable students like Liron were very aggressive like when they overtly stressed that other students were not clear about their own opinion (so that their opinion could be neglected). Yoram did not know how to cope with Liron. The high standards of criticality set by the teacher led him and the students to blunt attacks on what others claimed. These attacks were harmful and showed that argumentation can destabilize human relationships. The teacher did not know in advance how to cope with these difficult situations but he recognized that this is an inevitable problem that he must face without escaping to authoritativeness. The repeated attempts of this excellent teacher to bridge between dialogicity and criticality turned the moments of friction, as constitutive moments of the community of learners.
References
Burbules, N. (1993). Dialogue in teaching: Theory and practice. New York: Teachers College Press. Mercer, N. (2008). The Seeds of Time: Why Classroom Dialogue Needs a Temporal Analysis. Journal of the Learning Sciences, 17(1), 33-59. Mercer, N., Wegerif, R., & Dawes, L. (1999). Children's talk and the development of reasoning in the classroom. British Educational Research Journal, 25(1), 95-111. Morehouse, R., & Williams, M. (1998). Report on student use of argument skills. Critical and Creative Thinking, 6(1), 14-20. Reznitskaya, A., Anderson, R. C., & Kuo, L. (2007). Teaching and learning argumentation. Elementary School Journal, 107(5), 449-472. Schwarz, B. B. (2003). Collective reading of multiple texts in argumentative activities. The International Journal of Educational Research, 39, 133-151. Schwarz, B. B. & Glassner, A. (2007). The role of floor control and of ontology in argumentative activities with discussion-based tools. International Journal of Computer Supported Collaborative Learning, 2(4), 449478.
ISLS
375
ICLS2012
Multiple solutions and their diverse justifications to the service of learning in early geometrical problem solving
Naomi Prusak, The Hebrew University, Jerusalem, Israel, inlrap12@netvision.net.il Rina Hershkowitz, The Weizmann Institute of Science, Israel rina.hershkovitz@weizmann.ac.il Baruch B. Schwarz, The Hebrew University, Jerusalem, Israel, msschwar@mscc.huji.ac.il Abstract: The goal of this paper is to show that argumentation gains from being multimodal in learning geometry, especially at elementary levels. Grade 3 students participated in a year long course designed to foster mathematical reasoning. The course combined problem solving in dyads, peer argumentation and teacher-led discussions. We focus on one activity: identifying the types of solutions, the kinds of reasoning and the kinds of non-verbal actions (gestures, drawings, folding etc) used. We show how gestures and other non-verbal actions were interwoven with childrens verbal peer argumentation and led them to new insights on the concept of area.
Introduction and Theoretical Framework

In recent decades, there has been a growing emphasis in mathematics education on fostering thinking and reasoning abilities. Leading researchers have pointed at special forms of talk that may support mathematical reasoning. Argumentative forms of talk have been favored by several researchers. In the realm of unguided small group talk, Schwarz and Asterhan (Asterhan & Schwarz, 2007, 2009; Schwarz & Linchevski, 2007) showed that argumentative talk may lead to conceptual learning and change. However, they also showed that it is not easily triggered and that several conditions (e.g., the presence of devices for testing hypotheses, or the timely introduction of specific argumentative scripts) are crucial for the emergence of productive argumentation (see also Howe et al, 2000). In mathematics education, teacher-led reflective talk and argumentative talk have been shown as crucial for the establishment of socio-mathematical norms (Cobb et al., 1997). Other researchers have studied conditions for unguided productive argumentative talk to show the role of principled design (Schwarz, Hershkowitz & Prusak, 2010). However, recently mathematical problem solving is recognized as based on multiple channels of communication gestures, drawings, bodily actions, or manipulation of software. These channels of communication provide precious information on mathematics learning. As Radford (2009) claims: "Mathematical cognition is not only mediated by written symbols, but that it is also mediated, in a genuine sense, by actions, gestures and other types of signs" (p. 112). Similarly, we show in this paper that the deployment of mathematical reasoning is often multi-channeled. In particular, we show the centrality of multi-modality in early geometry learning. More specifically, Duval (2006) has indirectly provided an interesting research direction. He linked argumentation in geometry to the mrological decomposition of shapes divisions of the whole into parts with the aim of reconstructing another figure, which allows the detection of geometrical properties. This mrological decomposition can be done materially (by cutting and reassembling), graphically (by drawing lines that reorganize the shapes) and by looking. In our working hypothesis, we adopted Duvals theoretical idea that mrological decompositions involve productive argumentation. We will show that multimodality is a central characteristic of this argumentation and in general to early geometry problem solving.
Description of the experiment The problem solving course

Three groups of 20 talented 3rd graders participated in a special enrichment program in mathematics over three successive years. The students attended 28 meetings over a whole academic year. The program was designed to develop problem solving strategies and practices such as drawing a diagram, adopting trial and error methods, identifying patterns and putting into action deductive considerations. About 25% of the activities dealt with issues related to the geometrical concepts of area and perimeter and their relationships. Typically, in each lesson (75 minutes long), the teacher first initiated a 15 min. long discussion to ground necessary understandings to be shared. Then, the teacher distributed worksheets, and encouraged students to work in dyads (up to 40-50 minutes). The teacher passed by the dyads to answer questions if needed. At the end of the activity, the teacher orchestrated a reflective discussion on the activity. The activities were designed to trigger productive argumentation (Andriessen & Schwarz, 2009; Hadas, Hershkowitz & Schwarz, 2002). Three design principles were adopted: a) creating a conflict situation in which students were confronted with unexpected data or divergent opinions; b) encouraging participants to collaborate;
ISLS
376
ICLS2012
c) providing an environment for raising and checking hypotheses. The activity sharing a cake (Fig. 1) fulfilled these three design principles.
Figure 1. The first two tasks of the "Sharing a cake" activity The goal of the first task is to encourage students to provide diverse solutions and diverse justifications. The students are explicitly required to explain and justify in writing each of the solutions they chose to draw. Nine grid squares representing the cake were given to students in their worksheets in order to encourage them to find many diverse solutions, and to provide a proper context for comparing areas of various shapes (especially non-congruent) created on the square grid. The goal of the second task is to trigger a cognitive conflict in which a non congruent partition of the square is presented. Moreover we introduced a text by the imaginary student Mindy to bring to the surface current misconceptions regarding the concept of area.
Methodology
The students' work and the classroom's work as a whole were observed, videotaped, transcribed and analyzed. We used four cameras: one directly on the teacher presenting the task; and conducting the whole class discourse at the end of most lessons. Two other cameras documented a dyad or triads throughout the whole meeting (75 minutes) and the fourth camera was mobile and documented small episodes of various interactions within small groups in the class.
Data analysis of the sharing a cake activity

Like in all the other meetings, students were asked to collaborate in pairs. They were asked to negotiate and reach agreement. A first analysis revealed that this task resulted in multiple and creative solutions. We identified four types of solutions (see Fig. 2). In Type A, all four shapes were congruent and were created by simple partitions: drawing diagonals, perpendicular bisectors or segments parallel to one side. 95% of the students proposed all three Type A solutions. In Type B, all four shapes were congruent but were created through more sophisticated partitions. For Type C, solutions consisted of two different pairs of congruent shapes. In Type D, there was no more than one pair of congruent shapes (all four shapes could be non-congruent see Fig. 2). 100% displayed at least one Type A solution, 87% found at least one Type B solution, 71% found at least one Type C solution, and 60% of the students found at least one Type D solution.
Figure 2. The four types of solutions and their subtypes
ISLS
377
ICLS2012
Table 1 shows that 84% of the students produced at least three types of solutions. In fact, they produced at least one solution in which not all of the four parts are congruent. Our goal to lead students to understand that not only congruent shapes have the same area was then achieved. In addition to the solutions drawn on the worksheets, scrutiny over the videotapes revealed that these drawings were the result of a rich interactive activity during which children justified their solutions and convinced their peers in various ways. In our analysis, we aimed at uncovering the actions and justifications involved in the solutions. Table 1: The percentage of dyads that found more than one type of solution Number of solutions Type of solution A and B Two types of solutions A and C A and D A and B and C Three types of solutions A and B and D A and C and D Four types of solutions A and B and C and D Percent 8% 3% 5% 29% 16% 5% 34% 34% 50% 16% Total
Some examples are shown in Table 2. The first column refers to the type of solution peers elaborated on. The second column in Table 2 includes protocols of the verbal and non verbal action and solutions' justification. The third column includes the analysis of the justification deployed (in italics) and the non-verbal actions (underlined) used. Table 2 shows that for each solution in the diagram, non-verbal actions are central to the reasoning process. Table 2: Reasoning examples accompanied co-constructions of solutions Type of Protocols of actions and justifications solution Don: Each part is the same square The four parts 2 are of the same sizes and with the same shape [Points with his finger along the division's lines] Justifications' and actions' analysis Congruency justification Drawing a virtual perpendicular bisector with finger to explain solution. logical justification + Congruency
Articulating deductive reasoning verbally while pointing at a figure, to make clear the object of reasoning
c4
Harry: This we know is right and this also is right so I divide the square in half and each half I divide in two equal parts, so I'm sure my solution is right
c1ii
Lital: this and this these are equal[points to squares with her pencil], but the other two are different [points to the rectangles with her pencil] Ophir: but I know why these two are equal [pointing at a rectangle and a square]. One square is equal to one rectangle because it is" as if" it is divided and these two are equal to this. [ Pointing with her pencil on the two parts of the divided rectangle, moving her figure along an imaginary line] Lital: Now I understand Shay: these two are the same, they have three squares
[Pointing with his pencil on the two congruent shapes]
Composing and decomposing Transforming a figure (here decomposing or folding it) to (counter)-challenge an argument in a logical way
d2
Counting Justification Pointing at elements of a figure to support deductive reasoning (here through counting),
And here and here this is two halves now the square is 4 squares And this [pointing at the triangle above] 2 squares and 1/2 and 1/2 and 1/2 and 1/2 all together 4 squares.
ISLS
378
ICLS2012
The first kind of justification we observed was visual direct strategies based on a holistic intuitive perception of congruent shapes. To justify his Type a 2 solution (see Table 2), Don declared: The four parts have the same sizes and the same shape (he meant congruent). He accompanies his verbal justification by a pointing gesture and by the virtual drawing of a segment. We called this type of justification Congruency. The second type we observed is the composing and decomposing justification. It consists of transforming one figure into a different one. The C1ii type in Table 2 is a solution for which the composing and decomposing serves as a dialectical move to handle a disagreement. Ophirs gesture of moving of a figure around an imaginary line helps Lital being convinced by figuring out a transformation. The third type we observed is counting justification. It consists of counting square units or dots. The counting is usually accompanied by a gesture of one-to-one pointing, as in Shay justified his solution d 2 : these two are the same, they have three squaresAnd here and here this is two halves. As shown in Table 2, Harry justified his solution c 4 by providing a logical argument. We analyzed the distribution of written justifications to find that the most popular justification was based on congruency (49%), then counting (39%), leaving composing and decomposing to only 12% of the written justifications. We then examined relationships between the type of solution and the type of justification: The Congruency justification dominates Type A and Type B solutions (about 70%). This is not surprising as for these types, all shapes are congruent. For Type C solutions, the most frequent kinds of justifications were compose and decompose and counting with around the same percentage (about 40% each). For Type D solutions, the most frequent kind of justification was the counting one (about 85%). It is reasonable to assume that the 15% who used compose and decompose for Type D solutions were led by the need to have congruent shapes. We found that a significant correlation between the solution type and its justification. We investigated whether students' justifications belong to various types or they hold on one type. We found that more than 70% of the dyads use at least two types of justifications, and 14% of them used all three categories of justifications. Task 1 helped students apprehend the problem of dividing a geometrical shape in parts whose areas are equal. However our hypothesis was that the compose/decompose strategy enables further investigation of the concept of area. But we found that only solutions whose type is B or C yield compose/decompose justifications. Task 2 (see Fig. 1).was designed to afford composing and decomposing. In the following section, we show for one dyad how Task 2 (1) afforded the emergence of Compose/Decompose strategies, (2) was accompanied by a rich array of non-verbal actions, and (3) led students to new insights upon the concept of area.
The case of Harry and Larry

In addition to the fact that argumentation in Task 2 may stem from the composition and decomposition of geometrical figures, Task 2 also provides an opportunity for creating a cognitive conflict; because many of the Grade 3 students identified equality of areas with congruency of shapes. In addition the text offered opposite alleged opinions held by two imaginary students, which was intended to lower the chances of avoiding facing the cognitive conflict. In fact, both opinions were plausible and deciding whether Danny is right or wrong was a real challenge that led the Grade 3 students to engage in argumentation. Harry and Larry, are in a sociocognitive conflict: Harry claims first that Danny is wrong and Larry claims the opposite. We depict here the multimodal argumentation that led them to jointly reach the correct solution. Non-verbal actions are in brackets: Larry 4: I think Danny is correct the four parts are equal Harry 5: Only two parts [Points at 1 and 2 with two fingers, stares at drawing, then shows with two fingers that he measures the length of a first segment in part 2 and then measures the same length on the corresponding segment in part 1. Harry draws squares to divide each part to square units] (see Fig. 3). Interviewer 6: Yes, so you claim that Danny is wrong? Figure 3: Harry's squares Harry 7: There are here [pointing on partitioned squares he drew] (see Fig.3) more parts partition so this is larger [pointing with his finger moving around the boundary of part 3.] Larry 8: But look, all these here [composes Parts 1 and 2] and here all this [composes Parts 3 and 4] is equal. All this is equal Larry 9: because here it is half [composes parts 1 +2] and here it is also half. [Larry brings a model of the cake he cut off from the appendix] Larry 10: Because it seems to me like a half that way [makes an encompassing gesture on parts 1 + 2] and it is here...I [rotates the model of the cake and draws the following broken line]. (See figure 4) Larry cuts the model of the cake with Figure 4: the broken scissors along the broken line to convince Harry that part (1 + 2) is equal line in Larry's drawing to part (3 + 4)]. Harry 13: [looks carefully at Larry's efforts] I have an idea [Harry cuts part 4 and
ISLS 379
ICLS2012
reassemble it to look exactly like part 3] (See Fig. 5) Larry 14: I told you they are the same. Danny is right. Figure 5: Harry's compose and decompose manipulation A first analysis of this protocol helps showing that the composing and decomposing justification is central here (in Larry 8, 9, 10 and in Harry 13). It is involved in a rich argumentative activity which leads Harry to be convinced: Harrys interventions in 5 and 7 are challenges to Larry in 4. Larry 8, 9 and 10 are counterarguments. The composing and decomposing justification is not fully verbal: Harrys moving around the boundary of part 3 (in Harry 7) precedes Larrys implementation of compose and decompose justification (in Larry 8); the encompassing gesture and the rotation in Larry 10 are ways to organize compose and decompose justification. The actions function as mental planning and monitoring of their strategies toward the solution of Task 2.
Some conclusions
The design of activities such as Task 1 afforded collaboration and experiencing problem solving processes which led to many solutions and to various types of justifications. And indeed the analysis of data in Task 1 shows that the young participants were challenged and produced many surprising solutions and justifications. The socio-cognitive conflict designed in Task 2 triggered the enactment of non-verbal actions that palliated the difficulty to articulate verbal justifications. With the help of these multiple channels, we observed new insights in the comprehension of geometrical areas the fact that the shape of a geometrical figure can be changed without modifying its area. We also observed the seeds of deductive considerations as the compositions and decompositions were accompanied by verbal justifications including terms such as because alongside with non-verbal actions. This semiotic bundle (Arzarello, 2006; Radford, 2009) helped Harry and Larry to orchestrate reasoning in rich argumentative processes. We suggest that through these actions between the material (seeing, touching, and modifying) and the mental, children could function at an intermediate level to monitor and especially regulate their solutions.
References
Arzarello, F. (2006). Semiosis as a multimodal process, Relime, Numero Especial, 267-299. Andriessen, J. E. B. & Schwarz, B. B. (2009). Argumentative design. In N. Muller-Mirza and A. N. PerretClermont (Eds.), Argumentation and Education Theoretical Foundations. Springer Verlag. Asterhan, C. S. C. & Schwarz, B. B. (2007). The effects of monological and dialogical argumentation on concept learning in evolutionary theory. Journal of Educational Psychology, 99(3), 626-639. Asterhan, C. S. C. & Schwarz, B. B. (2009). Argumentation and explanation in conceptual change: Indications from protocol analyses of peer-to-peer dialogue. Cognitive Science, 33, 374-400. Cobb, P., Boufi, A., McClain, K. & Whitenack, J. (1997). Reflective discourse and collective reflection. Journal of Research in Mathematics Education, 28(3), 258-277. Duval, R. (2006). Les conditions cognitives de l'apprentissage de la gomtrie: dveloppement de la visualisation, diffrenciation des raisonnements et coordination de leur fonctionnement, Annales de Didactique et de Sciences Cognitives, 10, 5-53. Goldin-Meadow, S. (2003). Hearing gesture: How our hands help us think. Cambridge, MA: Harvard University Press. Hadas, N., Hershkowitz, R., & Schwarz, B. B. (2002). Between task design and students explanations in geometrical activities. Canadian Journal of Research in Mathematics Education, 2, 529-552 Howe, C., Tolmie, A., Duchak-Tanner, V., & Rattay, C. (2000). Hypothesis-testing in science: Group consensus and the acquisition of conceptual and procedural knowledge. Learning & Instruction, 10, 361-391. Mercer, N., Wegerif, R., & Dawes, L. (1999). Children's talk and the development of reasoning in the classroom. British Educational Research Journal, 25(1), 95-111. Radford, L. (2009). Why do gestures matter? Sensuous cognition and the palpability of mathematical meaning. Educational Studies in Mathematics, 70, 111-126 Reznitskaya, A., Anderson, R. C., & Kuo, L. (2007). Teaching and learning argumentation. Elementary School Journal, 107(5), 449-472. Schwarz, B. B., & Linchevski, L. (2007). The role of task design and of argumentation in cognitive development during peer interaction. The case of proportional reasoning. Learning and Instruction, 17, 510-531. Schwarz, B. B., Hershkowitz, R. & Prusak, N. (2010). Argumentation and mathematics. In C. Howe & K. Littleton (Eds.), Educational Dialogues: Understanding and promoting productive interaction. Routledge.
ISLS
380
ICLS2012
ChemVLab+: Evaluating a Virtual Lab Tutor for High School Chemistry

Jodi L.Davenport, WestEd, Oakland, CA, USA, jdavenp@wested.org Anna Rafferty, UC Berkeley, Berkeley, CA, USA, rafferty@eecs.berkeley.edu Michael J. Timms, Australian Council for Educational Research, Melbourne, Australia, mtimms9@gmail.com David Yaron, Carnegie Mellon University, Pittsburgh, PA, USA, yaron@cmu.edu Michael Karabinos, Carnegie Mellon University, Pittsburgh, PA, USA, mk7@andrew.cmu.edu
Abstract: Conceptual understanding in chemistry requires students to connect quantitative calculations, microscopic chemical processes (e.g., atoms and molecules), and macroscopic outcomes (e.g., concentrations, color, temperature). The ChemCollective Virtual Lab's interactive simulation environment fosters the mapping between these representations by providing students with a fully functional online chemistry lab with realistic chemicals, glassware, and equipment. In the ChemVLab+ project (www.chemvlab.org), we enhance the virtual lab by developing intelligent tutoring for this open-ended simulation environment and creating embedded formative assessment activities. Students have opportunities to apply chemistry knowledge to meaningful contexts and receive immediate, individualized feedback as the system estimates their proficiency based on their actions. We report findings from a classroom field study of ChemVlab+ activities. Data from classroom observations, preposttest learning gains, detailed log file analyses, and teacher interviews suggest that students were actively engaged, that students improved their understanding of chemistry, and that teachers found the activities worthwhile.
Introduction
Typical high school chemistry instruction consists of quantitative problem solving activities with the implicit assumption that students are learning core concepts in chemistry through the manipulation of numbers and symbols. Successful performance on complex calculations is taken as evidence of student mastery. However, research in chemistry education questions whether quantitative ability reflects conceptual understanding. Students have great difficulty connecting the mathematical representations with the underlying chemistry concepts and even high achieving students may lack basic knowledge of core principles (e.g., Bodner & Herron, 2002; Gabel & Bunce, 1994; Nakhleh & Mitchell, 1993; Smith & Metz, 1996). For instance, Smith & Metz (1996) found that students who performed well on traditional assessments in acid/base chemistry failed to identify strong versus weak acids when shown diagrams, suggesting that definitions and terms were used without true comprehension. Similarly, Nakhleh & Mitchell (1993) found that students given both conceptual and algorithmic items paired for identical concepts were much more successful on solving algorithmic items. Half of the students with high algorithmic performance had low conceptual performance. The current emphasis on algorithmic problem solving does not adequately prepare students with the conceptual understanding they need to reason in chemistry.
Theoretical and Methodological Approaches

Although chemists require quantitative calculations and a deep understanding of representations and notational systems, current instruction fails to connect the calculations of chemistry with their use in authentic chemistry practices. For instance, a study by Evans, Leinhardt, Karabinos, and Yaron (2006) found that the primary aims for practicing chemists are to explain phenomena, analyze substances to reveal their chemical make up and synthesize new materials. However, Evans found that textbooks focused equally on explanation and calculations with little coverage of analysis or synthesis activities. This large disconnect between authentic practice and the content of typical chemistry instruction suggests that many students fail to acquire a realistic sense of chemistry as a domain. When learning is disconnected from intellectual and practical use, the knowledge is not readily accessible, and not very memorable or robust across applications (e.g., Bransford, Brown, & Cocking, 2000). The design of the embedded formative assessment modules for the ChemCollective Virtual Lab embodies what has been learned from cognitive and educational measurement research about the power of quizzing and formative assessment for learning. Pashler et al., 2007 cites multiple studies showing that quizzing helps students to remember key information over longer periods (e.g., Roediger & Karpicke, 2006). Further, Black and Wiliam (1998) suggested that effective formative assessment that provides short term feedback so that obstacles can be identified and tackled (Black, 1998) produces effect sizes ranging between .4 and .7. Our embedded formative assessments deliver just-in-time coaching as students undertake the activities. Similar
ISLS
381
ICLS2012
contingent coaching that includes explanations has been shown to promote student achievement (BangertDrowns et al., 1991; Dassa et al., 1993). We hypothesized that the combination of formative assessment with the ability of the ChemCollective Virtual Lab to engage students in meaningful problem solving in chemistry will enhance student learning of complex chemistry concepts. We explored student engagement and learning by using a mixed-methods approach that involved classroom observations, pretests and posttests, log file analyses, and teacher interviews.
Method
Participants
Sixty-nine students from three classrooms at two California high schools completed the online chemistry modules as part of normal classroom instruction.
Design and Procedures

Students individually completed pretests during the class period before the activities began. Over the next three to four class sessions, two researchers performed classroom observations as students completed three online chemistry modules working either alone or in pairs. Computer log files recorded all student actions on the computer activities. After completing all three modules, students individually completed the posttest.
Materials ChemVlab+ Activities

Three online activities included intelligent tutoring and served as learning materials with embedded formative assessments. All activities addressed core issues in stoichiometry (i.e., the relationship between relative quantities of reactants and products in chemical reactions) and the central idea of conservation of matter. Each activity consisted of a series of tasks in an authentic context that exposed students to techniques and tools in analytic chemistry. The first stoichiometry activity focused on the issues of concentration and dilution in the context of preparing drinks with differing concentrations. Students completed interactive activities to sort drinks according to concentration, used a spectrometer in the virtual lab to determine concentration, and created solutions with specified concentrations. The second activity focused on the concepts of dilution and molar mass as students determined which factories along a river were polluting. Students determined the molar masses of different compounds and used the virtual lab to create dilutions and evaluate the water samples. Finally, the third activity focused on balancing chemical equations and using stoichiometry to determine unknown entities as students used the analytic technique of gravimetric analysis to determine whether samples of drinking water met EPA guidelines. Students used the virtual lab to determine which reactions formed precipitates, balanced chemical equations, and carried out inquiry to evaluate the concentration of sulfates in water. See Figure 1.
Figure 1. Students combine chemicals in the virtual lab to determine how the chemicals react.
Embedded formative assessments and Tutoring

Prior studies using the ChemCollective virtual lab environment suggested that many students spent a lot of time unsure of what action to take. To keep students in the zone of proximal development (Vygotsky, 1978), where students are challenged, but not bored or frustrated, the activities offered tutoring to students when they clicked the HINT button or attempted to move on with incorrect responses. Feedback appeared as a dialog box that provided information about the nature of the error, and visual elements that highlighted areas for students to attend to. See Figure 2.
ISLS
382
ICLS2012
Figure 2. Example of hint messages in activity 3. As our embedded formative assessment modules included a range of activities in open-ended environments, we used constraint-based modeling (CBM) to identify errors using logic that allowed students the freedom to use a variety of strategies (Mitrovic, 1994; Ohlsson, 1994). The logic that underlies our hinting was created based on data mining of detailed log files of 450 college students solving problems using the ChemCollective virtual lab. The constraints produced tutoring that breaks down larger problems into subgoals. Feedback was graduated and ranged from simple prompting to think about the rule or concept that applies, to reminding them of the applicable rule or concept, to showing them how the rule or concept is used to solve the problem. As a simple case, in the example above the system uses logic to determine whether a student has typed in the appropriate numbers for each atom in order to calculate the atomic mass. The first time a student makes an error, the system flags the error by through a visual indicator of where the mistake occurred (the triangle with exclamation point) and a message appears that the student can click the triangle indicator for additional help. The next level of help gives the student the rule for how to approach the problem. In this case, the message would state, The chemical formula for ammonium nitrate is NH4NO3. Look at the formula to determine the number of each type of atom in the molecule. Finally, if the student requires the maximum level of help, the message student receives the prompt that The chemical formula of ammonium nitrate is NH4NO3. In each molecule of ammonium nitrate there are 4 atoms of hydrogen, 3 atoms of oxygen and 2 atoms of nitrogen. Please put the correct values in the highlighted boxes.
Data Analysis
We used a mixed-methods approach to evaluate whether the online chemistry activities promoted student engagement and learning. We analyzed data from four sources: classroom observations, pre and posttest results, computer logs and teacher interviews. To ensure that students were actively engaged in the activities, two researchers conducted classroom observations as the students completed the assessments. To measure student learning as a result of the activities, students completed a pretest before completing the three chemistry activities and completed the same test as a posttest. The pre-post measure consisted of both researcher developed items as well as items from released versions of the American Chemical Society exam. To measure student learning during the activities, we analyzed the log files. As the activities were designed to give students multiple opportunities to practice the same skills, evaluated the log data generated by the online activities to determine whether students made fewer errors over time on similar items. Finally, to establish whether teachers found the activities to be usable and feasible for classroom use, we carried out teacher interviews with the two participating teachers.
Results
Classroom Observations
The classroom observations revealed that students spend the majority of time engaged and on-task as they completed the activities. Overall, student conversation was related to the chemistry activities. The observations also revealed usability issues of the software and tutoring. During the first activity, a number of students had difficulty pouring and finding glassware in the virtual lab. However, students rapidly learned to use the interface and did not have subsequent problems on the following activities. Further, two hints related to molar mass were not interpreted by the students in the way we expected. We use these data to iteratively refine the activities and will conduct another round of classroom tests throughout the 2011-2012 school year.
Pre-Post Test Analyses

As one indicator of student learning from the ChemVLab+ activities, all students took a pretest and a posttest. A paired-samples t-test comparing student pre and posttest scores found that students posttest scores (M= 16.8, SD=5.2) were significantly higher than their pretest scores (M= 15.4, SD=4.8), t(68) =3.7, p = .001. The Cohens d effect size for this analysis was .28.
ISLS 383
ICLS2012
Log file analyses

In addition to the pre-post test measures, we also investigated whether the process data provided evidence of student learning throughout the activities. Our activities with intelligent tutoring provided increasing amounts of help until students correctly complete each part of the activity. Data mining the log files generated by students using the chemistry virtual labs, we were able to investigate whether students learned during the activities themselves. Here we provide two examples that demonstrate student behavior shifting over the course of a single activity.
Activity 1: Matching Concentrations

In activity 1, students were asked to create drinks by adding water and a drink mix to match the concentration of another students drink. In the activity, students were first provided scaffolds in the prompt as they matched the concentration of Diegos drink, later in the activity they were asked to match the concentration of Mias drink. A prototypical strategy for solving this problem is to add a fixed amount of water to a vessel and to slowly add the drink mix until the concentration (as measured by the spectrometer) matches the desired concentration. A detailed analysis of the log files reveals student learning as they progressed through the activity. Initially, despite scaffolding in the prompt, students struggled to match the concentration of Diegos drink. Only 24% of the students were able to match the concentration using a single vessel, and the log files revealed many exploratory and incorrect behaviors in the virtual lab. For instance, rather than adding additional water when the concentration exceeded what was desired, many students abandoned the flasks and started again. Further, 25% of students continued to add mix to flasks that already exceeded the desired concentrations. With the automated feedback and tutoring, all students were eventually able to create the correct concentration. Performance on a subsequent parallel task, to match the concentration of Mias drink, suggests that students learned over the course of the assessment. In finding Mias concentration 61% of students (compared with 24% for the previous task) came up with the prototypical solution of matching the concentration using a single vessel. Further, students were less likely to make errors such as persisting in adding drink mix to a vessel that exceeded the desired concentration. Overall, only 10% of abandoned vessels exceeded the desired concentration (compared with 23% in the prior task). Taken together, the log file analyses suggest that students applied their learning from the tutoring on prior screens to modify their behaviors.
Activity 2: Making Dilutions

A detailed log file analysis of activity 2 also provides evidence of student learning in the course of the activity. In activity 2 students were first asked to make a dilution of 2:1, and later asked to make a dilution of 6:1. In the first task, to make a 2:1 dilution, the log files revealed a range of error types. Many students (63%) failed to take into account the volume of the beakers; 46% of students attempted to pour more liquid in a beaker than it could hold. A number of students, 27% (including some over-pourers) attempted to pour more liquid out of a vessel than actually existed. In a dilution, there is an amount of substance (the chemical you care about) in a solution (typically water). Thus to make a 1:2 dilution, the students must consider the amount of substance in 1 part water, compared with the same amount of substance in 2 parts water. The correct answer is to double the current volume of the solution by adding equal parts water and the to-be-diluted sample. In contrast, 37% of students erroneously poured in twice as much water as the sample. The second task, to make a more complex 6:1 dilution revealed significant student learning. Students made fewer mistakes related to the volume of the vessels with only 30% of students attempting to overpour (compared with 63% in the previous task). Further, 89% of students made a correct dilution by pouring the water and sample only once; and those that did not generally poured the sample once and the water multiple times. This pouring behavior was more focused than on the previous dilution as shown by the number of prototypical solutions. In general, students were given less scaffolding in the prompt and required fewer hints to get the correct solution, showing they had modified their behavior based on the prior task.
Teacher Interviews
After the classroom observations were complete, researchers carried out teacher interviews with the two participating teachers to gauge their overall reaction to the activities and establish the usability and feasibility of the assessments. Both teachers responded very positively to the ChemVLab+ activities and planned to use them again with their classes. Teachers said they thought the activities made sure students were learning the material instead of going through the motions, that the students were really involved and that the activities were a good challenge. Further, the teachers felt the intelligent tutoring was good at identifying student errors and offering guidance. For instance, on teacher said, Kids that normally have trouble with unit conversions and molar mass still had trouble in the lab. But the lab does a good job hand-holding them and showing them how to do it. Finally, the teachers said their students enjoyed the problem solving contexts. Students liked using the
ISLS
384
ICLS2012
real settings and tying the chemistry to new contexts. Overall, the reactions of the teachers suggest that the ChemVlab+ activities were usable and feasible for classroom use.
Significance and Conclusions

Our projects goals are to create and test activities that teach and assess core ideas in chemistry while providing students the opportunity to carryout open-ended investigations and make connections between microscopic, macroscopic and quantitative representations. Findings from our field test suggest that the activities promoted student engagement and learning of chemistry concepts. Classroom observation data suggested that the majority of students were actively engaged and on topic as they worked through the activities. Students scores were significantly higher on the posttest than on the pretest, suggesting overall learning gains as a result of the activities. Analyses of student process data from the computer log files suggest that students make fewer errors on subsequent items that address similar skills as they work through the ChemVlab+ activities. Finally, teacher interviews suggest that our system is usable, feasible, and considered worthwhile The ChemVlab+ activities embody the key features of formative assessment with feedback and coaching, realistic problem contexts, and activities that allow students to connect multiple representations and explore the same concepts in a variety of contexts. Our results provide preliminary evidence that using researchbased design principles to create interactive online activities improves student understanding of difficult scientific concepts. Further, our mixed-methods approach provides a range of data that allow us to capture student engagement and learning over the course of the activities, as well as after the activities were completed. The current work demonstrates that it is possible to provide effective intelligent tutoring for activities in an open-ended simulation environment. Further, our mixed-methods approach provides a framework for the evaluation of intelligent tutors that tackle complex subjects. The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305A100069 to WestEd. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.
References
Bangert-Drowns, R. L., Kulik, C. C., Kulik, J. A., & Morgan, M. T. (1991). The instructional effect of feedback in test-like events. Review of Educational Research, 61(2), 213-238. Bodner, G. M., & Herron, J. D. (2002). Problem-solving in Chemistry. In J. K. G. e. al. (Ed.), Chemical Education: Towards Research-based Practice (pp. 235-266). Norwell, MA: Kluwer Academic Publishers. Black, P. J. (1998). Testing, Friend Or Foe?: The Theory and Practice of Assessment and Testing: Routledge. Black, P. J., & Wiliam, D. (1998). Inside the Black Box: Raising Standards Through Classroom Assessment. Phi Delta Kappan, 80(2), 139-144. Bransford, J. D. E., Brown, A. L. E., & Cocking, R. R. E. (2000). How People Learn: Brain, Mind, Experience, and School. Expanded Edition: National Academies Press, 2102 Constitution Avenue N.W., Washington DC Dassa, C., & et al. (1993). Formative Assessment in a Classroom Setting: From Practice to Computer Innovations. Alberta Journal of Educational Research, 39(1), 111-125. Evans, K. L., Leinhardt, G., Karabinos, M., & Yaron, D. (2006). Chemistry in the field and chemistry in the classroom: A cognitive disconnect? Journal of Chemical Education, 83(4), 655. Gabel, D. L., & Bunce, D. M. (1994). Research on problem solving: Chemistry. Handbook of Research on Science Teaching and Learning, 301-326. Mitrovic, A., (1998). Experience in Implementing Constraint-based Modeling in SQL-Tutor. Proc. ITS'98, pp 414-423. Nakhleh, M. B., & Mitchell, R. C. (1993). Concept-Learning Versus Problem-Solving - There Is a Difference. Journal of Chemical Education, 70(3), 190-192. Ohlsson, S. (1994). Constraint-Based Student Modeling. In Student Modeling: The Key to Individualized Knowledge-Based instruction. J.E. Greer and G. McCalla, pp 167-189. Pashler, H., Bain, P., Bottge, B., Graesser, A., Koedinger, K., McDaniel, M., et al. (2007). Organizing instruction and study to improve student learning (NCER 2007-2004). Washington, DC: National Center for Education Research. Roediger, H.L., and Karpicke, J.D. (2006b). Test-enhanced learning: Taking memory tests improves long-term retention. Psychological Science, 17, 249-255. Smith, K. J., & Metz, P. A. (1996). Evaluating Student Understanding of Solution Chemistry through Microscopic Representations. Journal of Chemical Education, 73(3), 233-235. Vygotsky, L. S.: Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard University Press (1978).
ISLS
385
ICLS2012
Legitimate Peripheral Participation in Academic Communities of Practice How Newcomers' Learning is Supported in Student Councils
Julia Eberle, Karsten Stegmann, Frank Fischer, Chair of Education and Educational Psychology, Ludwig-Maximilians-University Munich, Leopoldstr. 13, 80802 Mnchen, Germany Email: julia.eberle@psy.lmu.de, karsten.stegmann@psy.lmu.de, frank.fischer@psy.lmu.de Abstract: A rather small part of lifelong learning happens in settings that are intentionally designed for learning. Much more knowledge is acquired through participative learning, i.e. social interaction within specific groups or communities. So far participative learning has only been vaguely described as legitimate peripheral participation, but facilitating influences of this process have not yet been investigated. In this paper the activities of senior members of communities of practice (faculty student councils) towards newcomers, so called socialization tactics, were examined. Twelve distinct socialization tactics were identified in interviews with 14 expericenced members. Some of them were used in all communities of practice, some in none of them, and for some the use varied across the communities. Finally, a model of the effect of socialization tactics relative to community size, and time of exposure on newcomers level of participation is presented, using HLM on data of 68 newcomers in 14 communities.
Legitimate Peripheral Participation in Communities of Practice

Learning has been described using several metaphors (Sfard, 1998; Paavola, Lipponen & Hakkarainen, 2004); one of them is the participation metaphor. This metaphor of participative learning views learning as a process of becoming a member of a certain community. This, for instance, requires the active use of the communitys language during communication with other members of the community. Participative learning can also be found within more traditional learning settings, which are not explicitly designed for participative learning; student governments, student associations, and student councils are examples within higher education settings. Lave and Wenger (1991) conceptualize this type of informal learning that can occur within such participative settings as legitimate peripheral participation in communities of practice. Wenger, White, and Smith (2009) describe a community of practice (CoP) by three dimensions: domain, practice, and community. The domain is the shared interest and focus of attention of the CoP members. It defines the reason why the CoP is built and defines its identity for internal and external purposes. The domain is constantly discussed between the members and controversies can lead to a split up into separate CoPs. Practice as the second dimension describes the activities and thinking common to CoP members that is based on their shared experience and knowledge of the domain. Community describes the aspect that members of a CoP over time build a relationship of trust and mutual engagement. The members can take over different roles, from passive members in the periphery to more active ones; very important is the existence of legitimate and capable leading persons that hold the community together. Newcomers who join a CoP undergo a complex process of learning and enculturation that shapes their identity as an active member of the community. Lave and Wenger (1991) describe this learning process as legitimate peripheral participation (LPP). The main focus of this perspective on learning is an increase of newcomers level of participation within a specific CoP. A newcomer starts as a passive outsider and becomes over time a more capable and active member, accompanied by a transformation of the identity. This transformation includes a change in cognition, communication, and behavior towards the shared practices of a CoP and is based on a CoP-specific, socially shared learning history. An important aspect of this kind of learning is the question of legitimation of newcomers participation. Lave and Wenger (1991) assume that there are more or less strict entry criteria or barriers in each CoP that are of great importance of newcomers learning as they regulate the possibility of a newcomer to get a chance to learn. Only legitimate newcomers get access to community activities, information, and resources and those are important for peripheral participation. Peripheral participation describes the increasing involvement of a newcomer, starting with rather passive activities like observing senior members, then going on to take over simple and easily manageable tasks and later take over more responsibility and more complex tasks as full members. The approach of legitimate peripheral participation in CoPs has attracted much attention and the community of practice concept has developed over time (see for example Barab & Duffy, 2000). Also a substantial scientific knowledge base on CoPs has been developed. However this research has focused so far mainly on two areas: (a) identifying CoPs in real world settings (see for example Safran (2009) for a community of home schooling parents); (b) investigating how CoPs can be initiated and facilitated, for example with focus on optimizing communication in organizations (Wenger, McDermott, & Snyder, 2002), in and around the classroom (Olitsky, 2007) or for workplace learning (Boyd & Lawley, 2009). There are only few systematic
ISLS
386
ICLS2012
studies about the core element of the initial theory: legitimate peripheral participation. We still know little what the underlying processes of this type of learning are and what factors influence them.
Socialization tactics of Senior Members and Other Influences on Newcomer Learning

The time of exposure to a CoP can be assumed to play an important role for the learning process of newcomers: The more time a newcomer spends in a CoP and with its members, the more opportunities for learning can occur. But it also can be assumed, that LPP may not be appropriately understood as mere exposure effect, in which only the amount of time spent by a newcomer matters. In contrast, Lave and Wenger (1991) name some examples of possible influences within the process of LPP like mentoring or exchange with other members of the CoP. Also empirical evidence for instructional influences that shape and influence the LPP process can be found. Lambson (2010), for example, showed in a case study on three newcomers in a CoP of literacy teachers that legitimate peripheral participation was influenced by strategies of an experienced senior CoP member. But so far, instructional strategies during the process of LPP have hardly been studied in a systematic way. Support strategies have, however, been studied in another research field, namely the field of organizational socialization. The socialization process of new employees in an organization has been found to be actively shaped by the members of the organization, the senior employees, and the newcomers by using socalled socialization tactics (Levine & Moreland, 1991). Several organizational socialization tactics which are used by senior employees to influence the learning process of newcomers have been identified by these authors: Positive and negative welcoming strategies were identified which are used in the very beginning of group entry to encourage newcomers to learn more, either because they are treated nicely or because they are harshly confronted with their shortcomings. Also several forms of patronage have been identified: senior members serve as models, trainers, sponsors, masters or mentors for newcomers. Encapsulation is another socialization in which newcomers are encouraged to spend their time for and with the group. Consistent training is a socialization tactic which is dependent on the senior members who send similar signals to newcomers and the tactic to assess newcomers knowledge about the organizations culture can help senior members to decide how to shape the further progress of the newcomers learning. Another possible influencing factor on newcomer learning in CoPs might be the size of the community. Group size has repeatedly been identified as an important factor for social interactions (Moreland, Levine, & Wingert, 1996): For example, larger groups usually have access to more resources, which can increase their effectiveness and make them more resilient against worsening circumstances. On the other hand, compared to smaller groups, larger groups have more problems with coordination, misunderstandings, overlooking of situations in which individual members need help, or passivity as well as lower commitment and motivation of individual members. Those aspects seem also to be relevant for newcomer learning in CoPs, so CoP size could also be considered to be an influencing factor on legitimate peripheral participation.
Research Questions
So far, the role of socialization tactics in CoPs for the process of legitimate peripheral participation has not been investigated systematically. Hence the research questions of this study are: 1. What socialization tactics are used by senior members in academic CoPs? We expect to find some of the socialization tactics already known in organizational settings (Levine & Moreland, 1991). It also seems likely that additional socialization tactics which are specific to CoPs can be identified. For a better understanding of the LPP process, also the relation of the identified socialization tactics to the level of participation of newcomers within their CoPs is of interest: 2. To what extent can the use of socialization tactics predict newcomers level of participation in academic CoPs beyond time of exposure and CoP size? We expect to be able to model the relation between newcomer learning and the socialization tactics, including an evaluation regarding the usefulness of the single tactics.
Faculty Student Councils as Communities of Practice

For an investigation of socialization tactics in CoPs, faculty student councils as they can be found in many university systems, seem an ideal setting. The integration of newcomers plays an important role as membership is restricted to students before their graduation. This means that time of being a member is limited. For the survival of the community, the continuous integration of newcomers is essential and has to be done efficiently. Often student associations also have difficulties to attract newcomers, so they have to be especially careful to integrate the few who are interested in becoming members. In the German university system faculty student councils (FSC) are well-established and most of them show the general characteristics of CoPs.
ISLS
387
ICLS2012
Domain. A FSC usually consists only of students of one particular discipline. The main goal of a FSC is to influence, improve, and shape the study conditions in their school, according to the needs and wishes of the student body. For this reason, students negotiate among the members on what levels they are able and willing to act and what tasks and projects they want to work on. FSC activities range from organizing student parties to more abstract engagement in university policy-making. Practice. The FSCs interact on a regular basis, mostly face-to-face but also online. The members of an FSC structure their work and community completely independent; some have leaders, assign specialized roles to individual members or even split up into specialized sub-communities. Other FSCs decide to make all decisions together. But in every case the work done by a FSC depends on the interest and engagement of its members. Within each FSC some kind of micro-culture develops which results in specific language and use of specialized terms to describe their activities and tasks as well as in shared opinions about relevant events and practices to deal with occurring situations. Community. Student engagement in the FSCs involved in this study is voluntary and takes place during students spare-time without any study-related reward. In most FSCs elections by the student body legitimate the FSC members as student representatives, but it is usually possible to be a member without being officially elected; also, some FSCs exist entirely without any legitimation by elections.
Method
Participants and Design of the Study
Fourteen faculty student councils (FSCs) of a German university voluntarily participated in this study. The number of members in the individual FSCs ranged from 6 to 22 with a mean of M = 14.9 members (SD = 5.2). All together 208 persons participated, 68 of them newcomers, who had been members of their particular FSC for less than 12 months. The disciplinary scope of the participating FSCs reached from disciplines in humanities to social science and science. The study was designed as a descriptive field study based on interviews and social network questionnaires.
Data sources and instruments

Socialization tactics. Socialization tactics were assessed using semi-structured, problem-centered interviews with 14 experienced senior members (one per FSC). Based on an interview guideline the senior members were first asked to describe their own position in the FSC and important aspects of their FSC like its history, aims and structure as well as the number of members. The second part of the interviews dealt with the FSCs socialization tactics. Here we used a two-fold procedure: First, the participants were asked to describe how the members usually influence the learning process of a newcomer in their FSC. After this open question, the senior members were directly asked about the use of each of the socialization tactics described by Levine and Moreland (1991). A qualitative content analysis was performed on the transcripts of the interviews. For that purpose a coding scheme was developed using a mixture of theory-driven top-down process and data-driven bottom-up process. The final coding scheme consisted of the following twelve categories: (1) possibilities for (peripheral) participation, (2) legitimation, (3) positive welcoming strategies, (4) negative welcoming strategies, (5) model, (6) sponsoring, (7) master/mentor, (8) encapsulation, (9) consistent, (10) knowledge assessment, (11) accessibility of community knowledge, (12) providing information before community entry. After training two coders reached an overall intercoder-agreement of = .80. After performing this qualitative analysis a score was built for each FSCs use of each socialization tactic, based on the amount of identified sub-aspects of the socialization tactics. Level of participation. Newcomers level of participation within the FSC was measured by an onlinequestionnaire. Participants were asked to indicate how intensively the participant had collaborated with the other FSC members during the last three months. A list of all FSC members was provided and the participant was asked to indicate the collaboration with each of them on a 10-point scale. Each newcomers level of participation was computed as the mean of the ratings that this person received from the senior members. Time of exposure. The time of exposure was also taken from the online-questionnaire, in which all members were asked since when they had been a member of their FSC. CoP Size. CoP size was reported by the experienced senior members in the interviews.
Statistical Analysis
A statistical analysis with respect to research question 2 was performed. Hierarchical linear modeling was used as the data had a 2-level nested-structure level 1 for the individual newcomer, and level 2 for the FSC. The only level 1 variable was the time of exposure to the CoP. Level 2 variables were CoP size and the use of the identified socialization tactics specific to each FSC. In a step-wise procedure potential level-2 predictors were included into the model based on t-to-enter statistics; always the variable with the highest t-value was included. Variables were grand-mean centered and all statistical tests were performed on the 5% level of significance.
ISLS
388
ICLS2012
Results
RQ1: Identification of socialization tactics
A qualitative content analysis of interviews with experienced members of 14 CoPs lead to the following results: The classical CoP socialization tactics providing possibilities for (peripheral) participation and legitimation are used to a great extent in most FSCs. They are part of the original LPP approach (Lave & Wenger. 1991) and their importance is supported by our findings. But also socialization tactics that have so far only been identified in research on organizational socialization (Levine & Moreland, 1991) are used in CoPs. However, the use of positive welcoming strategies, encapsulation and consistent training varies more between the FSCs than the use of classical CoP socialization tactics. Except for positive welcoming strategies, the organizational socialization tactics were usually not spontaneously reported by the FSC members. In the case of encapsulation and consistent training there are substantial differences between FSCs regarding the concrete implementation. Encapsulation is usually only a by-product of other activities and also consistent training is not sophisticatedly planned within the FSCs. When FSC members talked about aspects of consistent training, it became obvious that most FSCs had hardly ever discussed how to deal with newcomers, what problems they might have and how they could be supported. However, most of the FSCs had discussed how newcomers could be attracted. Therefore, parts of the sponsoring tactic that are also important to attract newcomers are commonly used in FSCs. Also the newly identified socialization tactic of providing information before community entry has its main purpose in the attraction of newcomers. Consistent with the finding that FSCs have a positive attitude towards newcomers and strongly welcome them, they report not to use tactics which could cause newcomers to leave the CoP like negative welcoming strategies or knowledge assessment. In general there is a lack of awareness regarding newcomers need for support among experienced members. Consistent with that finding, socialization tactics which require much coordination and planning among FSC members are also not found, especially serving as a model and/or master/mentor. Also the socialization tactic of providing access to community knowledge which also requires effort of senior members is hardly reported.
RQ2: Effects of socialization tactics on newcomers level of participation beyond time of exposure and community size
An exploratory HLM model was computed to describe the effects of time of exposure, CoP size, and the use of socialization tactics within a CoP on newcomers level of participation. This model included as predictors for newcomers level of participation time of exposure as a positive significant influencing factor ( = 0.210; SE = 0.043; p = .014) and CoP size, which had a negative significant effect ( = -0.135; SE = 0.034; p = .003. Regarding the effect of socialization tactics on newcomers level of participation a significant positive interaction effect of sponsoring and time of exposure ( = 0.669; SE = 0.185; p = .004) was found. From all other identified socialization tactics only the newly identified CoP specific tactics accessibility of community knowledge ( = 3.704; SE = 0.791; p = .001) and providing information before community entry ( = -2.123; SE = 0.703; p = .014) were included as significant predictors, with the first having a positive effect and the latter having a negative effect on newcomers level of participation. This model explained 90% of the variance between FSCs and 31% of the variance between individual newcomers.
Discussion
Qualitative findings. Indeed, we were able to identify a range of socialization tactics that are used in CoPs. However, some of those that are widely used in organizational settings seem not to be used at all in academic CoPs. The used socialization tactics are characterized by two aspects: Their underlying idea is positive towards newcomers and they are easy to use. Some of the used socialization tactics have been known as socialization tactics from organizational contexts before (Levine & Moreland, 1991), some others seem to be unique for CoPs and more related to the concept of legitimate peripheral participation (Lave & Wenger, 1991). Additionally, the analysis showed that senior CoP members often are not aware that they have possibilities to support the learning processes of newcomers actively. The difference between the use of socialization tactics in organizational settings and those in CoPs is another interesting finding. Members of CoPs are voluntary and not paid for contributing, which makes it easier for them to leave at any time compared to employees in work groups. So in CoPs senior members think that it is more important that newcomers feel welcome and are not discouraged, resulting in the use of positive socialization tactics. Another difference between CoPs and organizational work groups is that in work groups the task of a particular new employee is usually predefined, whereas in a CoP newcomers have to identify themselves why it is meaningful for them to be part of the community and what tasks they want to take over. These needs seem to result in special socialization tactics for CoPs that are tailored to this problem. Quantitative findings. Based on a quantification of the qualitative results an exploratory HLM model was developed to test the relative impact of the different socialization tactics, and to provide guidance for further research on legitimate peripheral participation. The identified model included beyond time of exposure
ISLS
389
ICLS2012
and CoP size three socialization tactics: if providing information before community entry was done elaborately in a FSC, this was related to a lower level of newcomer participation. However, if the FSC focused on providing access to community knowledge, this was related to a higher level of newcomer participation. Sponsoring, on the other hand, had a positive effect on the change of newcomers participation level over time. This model explains the major part of the variance between the CoPs, Additionally it explains a substantial share of the variability between individuals. These results can be seen as evidence that the LPP approach by Lave and Wenger (1991) could be improved by including the socialization tactics of senior CoP members and aspects that are known to be important in other settings and from other areas of research as the size of a CoP. This leads to a better understanding of the underlying mechanisms of learning processes in CoPs. Limitations of the study. Two limitations of this study have to be addressed: first, the measurement of the socialization tactics and second, the sample size. Measuring the use of socialization tactics with interviews can not be as accurate as observations of real behavior and it can be called into question how well the interviewees were able to observe and report about the socialization tactics that are used in their FSC. However, the interview partners were selected based on their experience within the FSCs and proved to be quite sensitive to the topic. Most of them were absolutely aware of the learning process of newcomers and had no problems to answer the questions. When they were asked about the different possible socialization tactics, many of them said at one point that they felt as if supporting their newcomers not enough, now as they were aware of all the possibilities; this is an indicator that social desirability did not play a big role during the interviews. Of course, some of the socialization tactics might be hard to observe and could be used mainly on an implicit level, like serving as a model, and so they could not be identified. In general, the interviews can be assumed to be trustworthy and that our findings might rather underestimate the existence and importance of the tactics. This is the same for the second limitation the sample size, which is rather small for using HLM analysis. However, the small sample size would rather lead to an underestimation than to an overestimation of the effects. Nevertheless, this study is a first step towards a better understanding of the role of socialization tactics in LPP and future research with more accurate measurement techniques and a bigger sample size is needed. Practical implications. One of the most important practical implications for FSCs and comparable CoPs might be to foster awareness among senior members that the learning process of newcomer can actively be shaped using socialization tactics. But not every possible socialization tactic seems to be helpful for newcomer learning. According to our model sponsoring should be fostered, as well as strategies that make community knowledge accessible to newcomers, for example by collecting helpful information and using supportive software like Wikis.
References
Barab, S. A., & Duffy, T. M. (2000). From Practice Fields to Communities of Practice. In D. H. Jonassen (Ed.), Theoretical foundations of learning environments. Mahwah, NJ: Erlbaum. Boyd, P., & Lawley, L. (2009). Becoming a Lecturer in Nurse Education: the work-place learning of Clinical Experts as Newcomers. Learning in Health and Social Care, 8(4), 292300. Lambson, D. (2010). Novice teachers learning through participation in a teacher study group. Teaching and Teacher Education, 26(8), 16601668. Lave, J. & Wenger, E. (1991). Situated Learning: Legitimate peripheral participation. Cambridge: Cambridge Univ. Pr. Levine, J. M. & Moreland, R. L. (1991). Culture and Socialization in Work Groups. In L.B. Resnick, J. M. Levine, & S. D. Teasley (Eds.), Perspectives on socially shared cognition (pp. 257-279). Washington, DC: American Psychological Ass. Moreland, R. L., Levine, J. M., & Wingert, M. L. (1996). Creating the Ideal Group: Composition Effects at Work. In E. H. Witte (Ed.), Understanding group behavior / Erich H. Witte: Vol. 2. Small group processes and interpersonal relations (pp. 1135). Mahwah, NJ: Erlbaum. Olitsky, S. (2007). Promoting student engagement in science: Interaction rituals and the pursuit of a community of practice. Journal of Research in Science Teaching, 44(1), 3356. Paavola, S., Lipponen, L., & Hakkarainen, K. (2004). Models of Innovative Knowledge Communities and Three Metaphors of Learning. Review of Educational Research, 74(4), 557-576. Safran, L. (2009). Situated adult learning: the home education neighbourhood group. Journal of Unschooling and Alternative Learning, 3(6), 1436. Sfard, A. (1998). On two metaphors for learning and the danger of choosing just one. Educational Researcher, 29(4), 4-13. Wenger, E., McDermott, R., & Snyder, W. (2002). Cultivating communities of practice: A guide to managing knowledge. Boston, Mass.: Harvard Business School Press. Wenger, E., White, N., & Smith, J. D. (2009). Digital habitats: Stewarding technology for communities. Portland, OR: Cpsquare.
ISLS
390
ICLS2012
Characterizing Collaboration with Metapragmatics: Using PreK Virtual Manipulatives on a Multi-Touch Tabletop
Michael A. Evans, Department of Learning Sciences, Virginia Tech, mae@vt.edu Kamala Russell, McNeill Gesture Lab, University of Chicago, kamala@uchicago.edu Michaela Hnizda, McNeill Gesture Lab, University of Chicago, mhnizda@gmail.com David McNeill, McNeill Gesture Lab, University of Chicago, Abstract: This paper is a microethnographic comparison between a pair of PreK (ages 4-5) girls and triad of boys working collaboratively on the same mathematical task. The children, matched by gender homogenously, are charged with completing a tangram puzzle on a SMART Table, a multi-touch tabletop that supports virtual manipulatives. Tasks are constrained by specific rules to encourage communication. Normally, collaboration can be investigated in terms of the task: did participants finish the puzzle efficiently? In contrast, we approach collaboration analytically as a way of acting. We define acting collaboratively as consistent alignment as to the focus of talk, and taking turns to complete the task while following the specified rules. We use the notion of metapragmatics to capture how a participants actions (talk and gesture) are meaningful for the consequences of interaction.
Overview and Purpose

The purpose of this study was to examine the degree to which social constraints placed on PreK students (ages 4-5) working in pairs or triads to solve transformational geometry problems affected the quality of discourse. We applied a multimodal (talk, gesture) analysis technique to characterize a stage in the development of problem-solving and communication skills along with mathematical reasoning. This works extends prior efforts working with grade 2 students (ages 7-8), allowing us to compare findings as we establish a developmental perspective on mathematical reasoning in early childhood mathematics with manipulative technologies.
Impetus
Social Considerations in Education. According to the U.S. National Council of Teachers of Mathematics (NCTM), communication is a highly necessary component of mathematical comprehension (NCTM, 2000). The authors of NCTMs Principles and Standards for School Mathematics (2000) write, Students who are involved in discussions in which they justify solutions especially in the face of disagreement will gain better mathematical understanding as they work to convince their peers about differing points of view (p. 60). This communication-centered framework for understanding, which advocates a learner-centered pedagogy, differs noticeably from transmission and acquisition metaphors, is not only gaining attention in the learning sciences, but also in public school systems and teacher education programs. Previous studies have described communication as necessary for facilitating mathematical learning as collaborative, positive, and academically relevant. Researchers define this type interaction in terms of sharing, turn taking, and active talking or gesturing of instructions to peers. In contrast, this study seeks to bring a finer-grained notion of collaborative behavior, taken from linguistic anthropological literature, to bear on childrens performance in such mathematical tasks. We examine the structure of their interactions on the whole, in both talk and gesture, looking for cohesive structures of speech and behavior, as an indication of collaboration and interactions that we suspect will facilitate greater learning. Directly related to communication is the idea of social constraint within the classroom. Social constraints are often embedded in instructional technologies, but not always incorporated into designed classroom activities. Implementation of these constraints, however, can foster communication; as the level of constraint increases, communication becomes increasingly necessary amongst the students (Saxe, 2002). As a means to understand how constrains influence communication, and subsequent attempts to collaboratively solve a mathematics problem, we have devised a taxonomy that is also meant to approximate certain constraints and enablers of physical and standard desktop virtual manipulatives. In this study we create an environment where children are each given a subset of the pieces for themselves. While our primary goal is to understand how communication, social constrains, and current technologies support the development of mathematical thinking, we are also interested in deriving requirements for future applications that may be developed for interactive surfaces, including multi-touch tabletops and tablets. Mathematical Thinking in PreK Geometry. According to NCTM (2000), PreK-2 students should be engaged in learning geometric foundations. More specifically, children at this age should work to use direction, distance, location, and navigation skills to build mental maps through participation in hands on activities such as solving puzzles, individually or in small groups, with manipulatives. Additionally, children should move beyond simple trial and error strategies, and instead begin to develop the visualizing, describing, and justifying skills that will serve them as they progress through their studies of mathematics (van Hiele, 1985). With these mathematical goals in mind, a base-line ability for PreK students can be established.
ISLS
391
ICLS2012
Use of Manipulatives in Early Childhood Mathematics Education. As previously stated, NCTM (2000) recommends the use of manipulatives as PreK-2 students learn the basics of geometry. Furthermore, Tapper (2007) points out manipulatives, like tangrams, help students build on prior knowledge and expand both their math content knowledge and their problem solving skills (p. 11). Given that young children have limited abilities to mentally transform shapes, activities allowing them to experience and perform such transformations on physical and virtual objects can contribute to development and refinement of this ability. Additionally, virtual tangram puzzle activities provide learners with the opportunity to create and share artifacts socially, and thus, by their very nature, create the opportunity for communication. Given the importance of communication,, we find it pertinent to advance how learning scientists characterize collaboration as socially-oriented, rather than task-oriented. In the following, we suggest that metapragmatic speech may prove fruitful, leveraging this construct to analyze data from a study on PreK-2 virtual manipulatives developed for an multi-touch tabletop.
Characterizing Collaboration via Metapragmatic Speech

Metapragmatic speech refers to speech about the occurring discourse. Metapragmatic speech may be more or less explicit such as markers of reported speech (explicit), performative verbs like say, opine, or yell (explicit), or metrical structures like pair-part turn-taking and repetition of poetic form (implicit). In the implicit case, the particular way speech is structured as opposed to all other possibilities comments on what sort of speech is occurring. Metapragmatic speech functions to characterize the present interaction as an interpretable sort of action. For example, the way speech is reported characterizes the interaction in which it originally occurred. When someone says, Sally reported that she ran into a wall with her car, a listener has an idea of the interaction in which Sally did so as a presentation of an informational account of some event, especially one that some listener awaited or was entitled to (taken from Lucy, 1993, p.96). We understand this as such because the explicit metapragmatic reported entails that which Sally said had the socially understood features of a report. Growing out of the sum of previous characterizations in that interactions relevant past, the metapragmatic elements of an utterance regiment the references and predications of the pragmatic aspects of speech with respect to the future course of the interaction. They function in a dialectic among the history of what has been achieved in an interaction, the immediate description of the interaction, and what has now been done by describing the interaction in such a way (Silverstein, 1993). Consequently, collaborative interactions should not be called so simply because multiple participants are engaged in the same object task. Interactions are collaborative because participants act in a mode socially understood as collaborative, a fact that is analytically describable only with metapragmatics. Collaboration emerges when participants explicitly describe themselves as collaborating, or implicitly fit themselves into the structures we associate with collaboration as a way of interacting such as turn-taking structures marked by time and repeating poetic structures from turn to turn. Metapragmatic speech that functions to make an interaction coherently collaborative is a co-referential string (McNeill et. al., 2010). Such an interaction is composed of parallel structures across orderly turns that establish the same referential goal. A metapragmatic structure is maintained by repeated instantiations in talk and gesture. Because collaboration operates on a scale larger than a moment, an emergent collaborative character of what we are talking doing now depends on the chain linkages of collaborative metapragmatics throughout an interaction.
Methods
Participants. For this study, participants consisted of three PreK girls and three PreK boys from a child development center for learning and research located on the campus of a large research university in the midAtlantic United States. During the virtual manipulative phase of the study, only two of the three girls were present due to fluctuations in summer attendance. Location. The child development center describes its mission as to provide a model preschool program and leadership for local, state, and national early childhood communities in teaching, learning and research as well as to address the service missions of the university. Although a separate room was used when conducting research, the overall setting remained within the PreK classroom atmosphere. Students were not removed from the center, but instead led by a teacher to an adjacent classroom dedicated to participation in this project. Setup. Video equipment, consisting of two cameras, was set up to capture the students faces and hands. Next, a SMART Table was placed in the corner of the room, with students being arranged around it. Finally, windows were covered with paper and a screen was placed behind the researchers to reduce distractions. Task. The task imposed social constraints on participants in three stages. This study focuses on the divided ownership stage, where each student has control over a subset of puzzle pieces only. This mode is meant to mimic a policy-heavy constraint that can be reproduced with physical or virtual manipulatives.
Metapragmatic Speech in Two Groups

Here we present two groups (depicted in Figures 1 and 2) showing significant differences in the way that they negotiated the task at hand. These differences can be located in the relative frequency of object co-references
ISLS
392
ICLS2012
and task co-references, and also in the function of metapragmatic speech, either to assert ones presence and correctness or to co-create a coherent structure for the interaction. Accordingly, Group 1 (girls) and Group 2 (boys) also differed in the way that they involved the teacher in their interaction. Group 1 made few object references in speech, none of which were negative. Gwen used object references only on one occasion to suggest where the other girl could put her piece next. Direct references to the task occurred rarely. Furthermore, most of their object references were non-verbal: gestural indications to objects or spaces, or uses of a point handshape to pull the piece around the tabletop. By contrast, Group 2 made frequent object references, as they pointed out where pieces are and where they should go, to themselves or their teacher, rather than to each other. These object references were not co-references as they were not chains of repeated reference to the same spaces or objects across turns and participants, but instead were disconnected. Most of their object references were verbal, and often used to undermine each others actions. In a rare co-reference, one participant may convey that a piece should go here, rather than there, to which another replies that does go there, ignoring the previous suggestion and continuing to move the piece in the intended direction, whether it fit there or not. Group 2 primarily discusses the objects in front of them, while Group 1 references them only infrequently. Group 1_Stage2 00:00:43:00 00:01:02:00 Time 0:430:46 0:47 0:490:50 0:54 0:540:55 0:560:58 0:591:02 Name Emily Ms. Lisa Ms. Lisa Emily Ms. Lisa Emily Ms. Lisa Talk Hey can you move your stinky head? Thats not nice. Is she teasing you Gwen? (both girls giggle) Yeah. You guys are a big tease. We are. We always tease each other. Hey, there you go. Good job! Gesture Emily touches red piece
Emily moves hands to green piece, moves it to the puzzle Gwen touches blue piece, moves it to waiting space, and orients it to the puzzle
(0:54) Gwen moving the blue piece to waiting space from her corner
(0:55) Gwen holding the blue piece in waiting space Figure 1. Transcript excerpt for Group 1 (Gwen and Emma) working with teapot puzzle.
Emily moves green piece on puzzle, trying to get it exactly right. Emily and Gwens pieces switch (computer error)
Group 1 makes mostly explicit metapragmatic references to their interaction as being collaborative. Emily remarks we always tease each other and we always know where it goes. These characterizations of their relationship have two functions: the first allows them to address each other throughout the interaction as butthead, stinky butt, stinky head and allows for these terms of address to not be considered offensive; the second describes their interaction as a collaborative one, at a time when there has been a break in their cooperation. When they hold two pieces too close together, the computer switches them and they jump apart. This gives Emily Gwens piece and Gwen Emilys piece. Emily says we always know where it goes right after this break in their cooperative pair-part structure. By predicating both in the first person plural, Emily reaffirms that they are collaborating and are correct in how to do it. Group 1 only referred to the rules of the task at the very beginning of the interaction, when Emily asks why is no one sharing the green? The girls used explicit metapragmatics to describe themselves as in alignment. Group 2 (boys) however, predominantly made references to the task, reinforcing the rules imposed by the teacher in a competing manner. The few instances when they do speak about things other than the pieces in front of them are when a participant accidentally touches anothers piece. Even though he may already have recognized his mistake in touching the wrong piece, the other boy reinforces the rule by saying You can never touch any of these. While these instances are still metapragmatic, members of Group 2 use the rules of the task to assert their rights over the rights of the others. Only one boy uses we once, to say I know we turn it this way, after an open-ended I know by another boy, apparently describing how he is turning the piece to fit it in the puzzle. They do not talk about themselves as part of a group.
Discussion
ISLS
393
ICLS2012
Group 1 developed a distinction within the tabletop between their space, the waiting space, and the puzzle. Their metapragmatic strings of co-references created the turn taking structure that they used to divide labor in their virtual tangram puzzle. They developed a spatial regimentation of the board, where each participant had her own corner of the board, and an area between the puzzle outline and the corners served as waiting space. When Gwen or Emily moved a piece from their personal space to the puzzle they took control of the turn, the other would then move a piece from their space to the waiting space. This continued in turns. They also used their teasing words as ways of enforcing their turn taking. Early in the interaction, Emily uses the formula move your ___ as she is beginning her turn of moving a piece. Later, Gwen says move your poop head to which Emily responds move your butthead and move your stinky butt. These happen when she is taking a particularly long time to orient a piece perfectly in the white background. This is not immediately effective, and Gwen finally ends this with a move your hand and grabbing Emilys hand out of the way. After the first iteration of this formula, Emily describes it dismissively as typical of their friendly interactions, making it noninsulting. By using it as she takes her turn with the puzzle, the parallel structure of this formula functions as an implicit metapragmatic indication that they are taking turns, not simply talking. When Emily takes too long on her turn later in the interaction, she and Gwen use this formula with neat parallelism to determine whether her turn with the puzzle is over. Gwen ends by breaking slightly with the structure and talking about Emilys hand, not addressing her with a silly name. Group 1creates a turn taking structure by continuing to act it out nonverbally throughout the interaction, which is mirrored in the parallelism in their speech. This metapragmatic chain of co-references in action and speech instantiates their explicit metapragmatics of collaboration. Group 2_Stage2 00:01:02:000 - 00:01:15:000 Time 1:021:03 1:041:05 1:061:07 1:071:08 1:091:11 1:121:13 1:131:15 Name Jason Rico Jason Charlie Jason Talk How do I do this? Hey how did this happen? How did this happen? Rrrrr Hey move, silly. Hey move, silly. Gesture Jason looks at his piece and starts moving it toward the puzzle. Rico attempts to move triangle piece with index finger. Notices piece to be upside down. Moves piece rigorously. Uses index and middle finger at first in an attempt to rotate piece, then rigorously attempts move it with the index finger. Attempts to move piece into place, now using index and middle finger. (1:09) Each of the boys is struggling with the technical aspects of the puzzle. While their speech and gesture show a degree of parallelism, reflected in the repetition of similar words and gestures, it is directed at each of their puzzle pieces, rather than each other.
Rico Charlie
Move the Still moves piece rigorously, right way, attempting to rotate it. silly. Figure 2. Transcript excerpt for Group 2 (Jason, Rico, Charlie) working on teapot puzzle.
Group 2 also makes an exchange with parallel structure, but we cannot call it a chain of co-references. They are not participating in a turn taking structure; rather each mimics the speech of the other while performing their own task. As they all struggle with the technical aspects of the puzzle, one of the group members utters in frustration move silly, which triggers a chain reaction of the other two members repeating the same or a similar utterance. Comparable scenarios occur frequently throughout that groups interaction, most of which related to their inability to solve the puzzle, rather than indicating attempts at solving the puzzle in a cooperative manner. While their speech and gesture suggest a form of parallelism, it is directed at each of their puzzle pieces, rather than at each other and can therefore not be characterized as the sort of cooperation we find productive. They are not taking turns at the time, and because their gaze and focus of action continues to be on their pieces on the board, this parallelism is not part of a chain of co-references to the same puzzle piece and cannot be seen as metapragmatically instantiating cooperation. Groups 1 and 2 also interacted differently with the teacher present in the elicitations. Group 1 seemed to be interacting with each other into the structure of which the teacher would interject. The teachers comments were often giggled at and not responded to directly. The individual participants in roup 2 however, used the teacher as their interlocutor, rather than each other. The teacher frequently had to encourage and remind them to help each other, which created an artificial sense of cooperation in certain instances. Most importantly, group 2 actively sought recognition from their teacher for accomplishing tasks. If one boy moved the piece into the right place, the teacher would praise his efforts, which in turn seemed to motivate the other boys to put pieces into the right place as well, in order to be praised and
ISLS
394
ICLS2012
recognized by the teacher. Consequently, group 2 mostly addressed the teacher, and not each other. Overall, group 1 used speech and gesture to create a turn taking structure integrated with the space on the board, through which they collaborated on their task. They both used implicit and explicit metapragmatics in the same way, to achieve alignment. Group 2 in contrast, worked mainly individually, in occasional dialogue with the teacher or competitively with each other. They accomplished the task; however, it took them 3:52 min. to complete their puzzle, whereas it took Group 1 2:15. Even though Group 1 was composed of girls, and Group 2 of boys, we do not intend to use these data from only two elicitations to draw comparisons along gender lines. It is suggestive that the two groups behaved so differently in these cases, but there are other factors that may condition this. There were only two girls in Group 1, while Group 2 had three boys. It may have been easier to maintain turn taking and negotiate space on the table with only two participants. Either way, these two groups exemplify different types of metapragmatic behavior that have different functional consequences in their interactions. Group 1 co-created a coherent poetic and metrical structure that puts them in alignment (even when they tease each other), whereas Group 2 competed for praise and the floor. With reference to mathematical learning, we believe that the coincidence of a coherent interactional structure characterized by a metapragmatics of collaboration and relative ease with completing a task are not coincidental. The group that collaborated implicitly negotiated the task through the structure of their talk and action. The group that did not collaborate explicitly discussed the task, but their talk seemed to be for themselves or the teacher, and got in the way of attention to the accuracy of their movements with relation to the mathematical goal of the task. We believe that collaboration in this sort of task can then be seen as an indicator of attention and mathematical ability.
Conclusion
In this paper, we present a rationale for investigating PreK mathematics education, focusing on the social, cognitive, and technological priorities as advocated by national standards organization. This study is significant in that it may serve learning scientists interested in characterizing collaboration and early childhood educators intending to enhance communication as a pedagogical value (Ares et al., 2009; Clements et al., 2004). Additionally, this study involves extending our framework for multimodal analysis of nonverbal and verbal deixis to apply to the PreK age range. In addition to generating findings about which virtual and physical modalities are most successful for different types and levels of PreK learners, we are also generating more general information about communication habits of students in this age range and their potential for distributed cognition. Given a larger goal of developing additional software for multi-touch and interactive surfaces, these results begin to lead to new requirements for instructional strategies and technologies. Additionally, the system of coding we are continuing to develop for this data may be useful to other researchers seeking to find complex patterns in multimodal data, especially in work pertaining to early childhood education.
References
Ares, N., Stroup, W., & Schademan, A. (2009). The power of mediating artifacts in group-level development of mathematical discourse. Cognition and Instruction, 27(1), 1-24. Clements, D., Wilson, D., & Sarama, J. (2004). Young childrens composition of geometric figures: A learning trajectory. Mathematical Thinking and Learning, 6(2), 163-184. Lucy, J. (1993). Metapragmatic presentationals: reporting speech with quotatives in Yucatec Maya. In John Lucy (ed.) Reflexive Language: Reported Speech and Metapragmatics. 91-125. Cambridge: Cambridge University Press. McNeill, D, Duncan, S., Franklin, A., Goss, J. Kimbara, I. Parrill, F. Welji, H., Chen, L. Harper, M. Quek, F. Rose, T. & Tuttle, R. Mind-Merging. In a Festschrift for Robert Krauss, Ezequiel Morsella (ed.), Expressing oneself / expressing one's self: Communication, language, cognition, and identity. London: Taylor and Francis. 2010 National Council of Teachers of Mathematics (2000). Principles and standards for school mathematics. Reston, VA: National Council of Teachers of Mathematics. Saxe, G. (2002). Children's developing mathematics in collective practices: A framework for analysis. The Journal of Learning Sciences, 11(2&3), 275-300. Silverstein, M. (1993). Metapragmatic discourse and metapragmatic function. In John Lucy (ed.) Reflexive Language: Reported Speech and Metapragmatics. 33-58. Cambridge: Cambridge University Press. Tapper, J. (2007). Tangrams and geometry concepts. Connect Magazine, 20(4), 7-11. van Hiele, P.M. (1985). Structure and insight: a theory of mathematics education. Orlando, FL: Academic Press.
Acknowledgments
National Science Foundation (Awards 0736151, 0832479); SMART Technologies
ISLS
395
ICLS2012
Development and Validation of a Scale to Place Students along a Learning Progression

Namsoo Shin & Shawn Y. Stevens University of Michigan, 610 E. University Ave., Ann Arbor, MI Email: namsoo@umich.edu, sstevens@umich.edu Abstract: The learning progression (LP) is a construct recently adopted by the science education community as a potential way to create a coherent science curriculum to support student understanding of a few core ideas. However, the field does not yet have sufficient evidence-based LPs for educators to use. We describe efforts that are part of a project that aims to develop and empirically test a hypothetical LP related to the transformation of matter for grade 612 students. In particular, we present the design research methodology and progress towards developing and validating an instrument developed based on a LP. The resulting instrument can locate where students fall along a hypothetical LP and characterize how a variety of factors (e.g., school, context, gender) affect students performance. These results will inform the development of curriculum materials and instruction that better support student understanding of ideas important for science literacy. Recently, science educators have been working to change the focus of the curriculum to the development of an integrated understanding of a relatively small number of core ideas instead of a shallow coverage of a broad range of isolated concepts (Duschl, Schweingruber, & Shouse, 2007). To further this effort, the science education community has begun to employ the idea of learning progressions (LPs) to put emphasis on how understanding of these core ideas develops over an extended period of time. (National Research Council, 2010; Smith et. al., 2006). LPs are research-based descriptions of how students build their knowledge, and gain more expertise within and across disciplines over a broad span of time, that provide a means of organizing and aligning the science content, instruction and assessment. As of yet, the field does not have sufficient evidencebased LPs for educators for use in educational settings. Our research group is working to develop and empirically test a LP for grade 612 students that emphasizes characterizing and measuring the development of connections between ideas within and across topics that are required to develop understanding of transformations of matter. The main research goal is to develop and validate an assessment to act as a ruler to place students on the hypothetical LP that can discriminate students performance by a variety of factors (e.g., school, curriculum, gender). We use a principled and systematic design process to guide the development and validation of assessments associated with the LP. The developed instruments are intended to monitor how grade 6-8 students may develop conceptual understanding of and apply concepts related to the structure, properties and behavior of matter to explain transformation phenomena. This research will inform the revision of the hypothetical LP as well as the associated assessment items and enable the empirical testing of a portion of the LP.
Learning Progressions
A LP addresses a range of content within a discipline defined by a lower and upper anchor (Duschl, et al., 2007). The lower anchor provides a description of the knowledge and reasoning that should be held by students prior to beginning to develop understanding of concepts contained in the LP. The knowledge and skills students are expected to develop by the end of the progression, the upper anchor, are drawn from what educational research has defined to be developmentally feasible, and are also related to goals of science literacy and societal needs and expectations (Mohan, Chen, & Anderson, 2009). Each step, or level, along the progression should be logical, comprehensible and measurable. Many factors determine the path that students may follow, including the context, instruction, curriculum materials, and students prior knowledge and experiences. Therefore, we expect there could be multiple paths between levels defined on a LP. An instrument designed to measure progress along a LP must be flexible enough to monitor progress along multiple paths. Explaining many important phenomena requires incorporating ideas from many topic areas (e.g., structure of matter, energy, conservation). We hypothesize that focusing on the growth of sets of ideas instead of individual topic areas over the course of the LP, will better support the development of integrated knowledge structures (Stevens, Delgado, & Krajcik, 2010), which in turn will result in learners who can better select and apply the appropriate ideas more consistently to explain a broader range of phenomena and to solve new problems. Using this model, levels of the LP do not progress in sophistication through the addition of a new type of idea (e.g., adding conservation or energy to understanding of the structure of matter) as every level
ISLS
396
ICLS2012
incorporates multiple topic areas. Thus, assessments must measure how students use and connect the ideas contained on each level of the LP.
Construct-Centered Design Process (CCD)

The process and methodology of the development of LP should be iterative and process-oriented for investigating student learning over a long period time. A reliable and valid instrument is necessary to track how students develop competencies over time as we improve teaching and learning strategies, and develop new materials. The complex process of developing and empirically testing a LP requires the collection, analysis and organization of an extensive amount of data, which is consistent with a design research approach (Collins, Joseph, & Bielaczyc, 2004). Such research involves the development of products (e.g., learning progression, instructional materials, technology tools) and explores the relevance of the products on learning rather than simply examining isolated variables within laboratory contexts (Brown & Collins, 1992; Barab & Squire, 2004). The Construct-Centered Design (CCD) process (Shin, Stevens, & Krajcik, 2010) provides an approach flexible enough to guide the development of a diverse set of research products, including assessments, instructional materials and the LP itself using a design research approach. The first step of the CCD process involves defining the construct(s). A construct includes the ideas or concepts that we wish to learn about and measure (Wilson, 2005). This process involves explicitly specifying all of the content that is necessary to develop understanding of the aspects of the construct to be addressed by the LP. This also includes identifying potential student difficulties and alternative ideas related to the specified content, which is useful for creating distracters during item development. The next step involves developing a set of claims and defining the evidence that support the claims based on the relevant content for each of the constructs (Mislevy, et al., 2003; Mislevy & Riconscente, 2005). The claims describe what students should be able to do with the knowledge at a particular level in the progression. The evidence describes the ideas, skills and connections within and between constructs that students need in order to demonstrate that they have a particular level of understanding. The claims and evidence guide the development of the LP and the associated assessment tasks.
Methodology
Design research is by nature an iterative process. As such, there are several phases in the development an empirical testing of a LP. The first phase involves developing a hypothetical LP based on empirical research and the logic of the discipline. Cross-sectional data is collected to support and fill in any gaps in the research literature. Phase II involves the developing an instrument to measure student progress along the LP. This phase also involves collecting cross-sectional data to validate the instrument. The final phase involves empirically testing the LP through collection of longitudinal data to measure how student learning progresses. Here we discuss efforts related to Phase II.
Developing the Instrument

Following the model of learning and the CCD process, we developed assessment items to create a scale that will enable us to locate where students understanding falls along the LP. We believe that students at all levels can integrate ideas from multiple topic areas when explaining phenomena. Its just the complexity and scientific accuracy of the models that they use in their explanations that will increase in sophistication as they move along the hypothetical LP for the transformation of matter. We tried to design items that require students to incorporate ideas from multiple topic areas regardless of the LP level with which it is associated. Middle and high school, and college students (grades 6-16; N=792) from four secondary schools and three universities in Southeast Michigan pilot tested 112 items to create a bank of items that concentrates on Levels 1-3 of the LP since that is the portion we are empirically testing (grades 68). More detail on this part of the process will be published elsewhere (Stevens & Shin, 2012). For the field-test data collection, we used items selected from or revised from the pilot testing. For example, we found that open-ended items generally did not provide much information. Figure 1a provides an example of an open-ended item that we modified. We found that by asking students to critique each model they did not pick encouraged them to incorporate more ideas than the original item asking them to explain why they chose their answer. On other occasions, we modified towards more closed items. For instance, instead of an open-ended item, we generated a series of true/false questions that address all of the topic areas we wished to measure so that we could obtain some information on whether students did not know how to use the ideas or just did not use them in the open-ended format. This type of item also allows us to measure multiple levels on the LP within the same item (see Figure 1b). We found multiple true/false items to be useful for other reasons as well. Higher performing students found them more difficult than typical multiplechoice items because they had to read and evaluate each statement and gained no advantage from the test-taking skills this population generally possesses. Lower performing students liked this type of item because it is not all or nothing. If they do not know one part, they still have a chance to answer other parts correctly so this type of item is less stressful for them. In the pilot, we gave the choice of not sure for this type of item, but did
ISLS
397
ICLS2012
not include that choice in the field test. We found that the inclusion of not sure provided cleaner data by eliminating much of the guessing issues that often accompany true/false items. We created four test forms (1A, 2A, 3A 4A), each containing 14 items for the 6th and 7th grade students, and four test forms (1B, 2B, 3B, 4B), each containing 15 items, for the 8th grade students. The A- and B-version of each form differ by three items to adjust the overall difficulty of the tests. Each test form contained three pairs of linking items for a total of six linking items. Linking items were chosen to cover a range of topic areas along Levels 1-3 on the LP. The A- and B-versions of each test form contained the same linking items.
Figure 1. (a) Example of modified open-ended item.
(b) Example of item that measures LP multiple levels.
Participants
We collected cross-sectional data from 37 teachers from nine schools in four different states representing a varied demographic profile including schools in urban (N = 3), suburban (N = 5) and rural (N = 1) settings. Approximately 3800 students in 6th, 7th, and 8th grades from public (N = 7), private (N = 1) and charter (N = 1) schools participated in the field test in Fall, 2010.
Data Analysis
We used 52 multiple-choice items and 10 open-ended or short-answer items in this data analysis. The total scores differed between forms because the multi-part and open-ended items had different total scores. We eliminated the students who responded to none of the items in their test. There were a total of 3405 valid subjects after this data cleaning and recapture. Data analysis was performed using ConQuest software (Wu, Adams, Wilson, and Haldane, 2007). We used Item Response Theory (IRT) to ensure the revised items were effective at measuring the construct and to create a scale to measure student progress. Differential Item Functioning analysis (DIF) and Unidimensional Latent Regression (LR) were used to characterize the different factors that could affect measurement of student performance.

Creating a scale to measure student progress
The weighted means square (MNSQ) of fit (> 1), T-value (> 3), reliability, and discrimination (< 0.3) outputs from IRT analysis were used for judging the quality of each item. In the IRT output, the fit indices indicate the extent to which the item fits the item response model. Typically, when the fit index is much greater than 1, the item is generally less discriminating than the model predicts and is therefore a potential candidate for deletion. In addition, when items exhibit a large difference between the unweighted and weighted fit value, it indicates that they lack generalizability and may have to be revised or deleted. When items did not fit the model well, we evaluated the items themselves and the scoring approach to decide whether to revise the scoring or remove the item from analysis and future data collection. We also removed items for which 90% of students answered correctly or incorrectly to account for ceiling and floor effects. After deleting the problematic items, we ran the IRT analysis again to ensure that the rest of the items fit the model well. A Wright map of the remaining items indicated that the items cover a full range of difficulty level well (-3 to 2 logits) to provide a measure of progress in 6th 8th graders understanding of the transformation of matter.
Characterization and validation of the assessment

In order to evaluate the sensitivity of our instrument to various factors, we applied DIF and LR analyses. First, we wanted to ensure that our instrument could distinguish between schools. Due to the range of characteristics and contexts of the schools, we expected a large range in student performances. Using individual IRT scores, we calculated the average ability scores for students at each school (see Table 1). The higher logit values represent
ISLS 398
ICLS2012
higher ability level. We see a wide range of difference in ability levels, which suggests that we can reliably discriminate between schools. LR performed pair-wise between groups of Table 1: MAa levels by school and gender difference. suburban, urban and rural schools indicated that Schoola MA SEc Gender Diff. SE students at our urban and rural schools performed similarly (urban, 0.09 logit [SE=0.06] lower than the U1 -0.96 0.15 0.02 0.05 rural school). The students at the suburban schools U2 -1.28 0.15 0.02 SE on average performed over 0.3 logit higher than U3 -0.65 0.07 0.08 0.03 either the urban or rural schools. This result is R1 -0.81 0.10 -0.42 0.08 consistent with our prediction that the schools with S1 -0.15 0.08 0.19 0.07 higher social economic status than the rural and the S2 -0.32 0.06 0.03 0.06 urban schools would score higher than other S3 -0.60 0.15 0.10 0.06 schools. S4 -0.55 0.09 0.15 0.03 DIF analysis indicated that male students S5 -0.48 0.08 0.10 SE a exhibited slightly higher, but statistically significant MA = mean ability logit ;bS = suburban; U = urban; R = rural; c SE = average ability levels than the female students (0.15 standard error of measurement logit, SE=0.03). This difference is generally observed for middle school students (Marx, et al., 2004). We performed DIF analysis to examine differences in performance by gender at each of our schools (see Table 1). Several schools showed no significant difference in ability levels of male and female students. Interestingly, at R1 the difference was reversed with females exhibiting a significantly higher ability than male students. We will perform a detailed analysis of the curriculum materials to try to identify what might be responsible for this trend. The degree of variation between male and female performance at different schools suggests that our assessment does not have an inherent gender bias. These results provided evidence validity of the LP assessment in terms of the gender variable. We performed LR analysis between the grades over all schools to determine whether we could measure progress along the LP using this assessment. We observed that overall, students in 8th grade showed an increase in average ability level of approximately 0.55 logit relative to the 6th graders. The increase in ability level was not uniform (0.34 logit, SE=0.02 between 6th and 7th graders; and 0.21 logit, SE=0.03 between 7th and 8th graders). To further characterize differences between schools, we compared a subset of schools. U1 and S1 both have a similar resources and science curriculum as it relates to the LP, but a very different student population. S1 and S2 have similar student populations, but differ in curriculum and resources. We observed an increase in mean ability levels at all three schools (see Table 2). As with all schools combined, the amount of progress differed from 6th to 7th grade as compared to 7th to 8th grade for the individual schools. The degree of progress at each point in the curriculum and in total differs. In addition, the total and step-wise progress differed between schools. Although the absolute ability scores appear to be correlated to SES, other factors appear to affect the degree of progress students make. We expected a smaller increase in ability level for S2 students because content associated with our LP is introduced primarily in grade 8 so at this point in the year even 8th grade students had little instruction on relevant content. In contrast, students at U1 and S1 both receive instruction on content related to the LP throughout grades 68. Thus, we predicted a larger increase in mean ability levels for these two schools relative to S2. A detailed analysis of the curriculum and observation of instructional practices will help us delineate the other factors affecting the students progress. Overall, the scores increase with grade, although there are variances among schools in the amount of learning progress and when it occurs, which is expected due to differences in context and curricula. These results suggest that our assessment can discriminate students performances across the grade levels. Table 2: Comparison of increase in mean ability levels. Schoola U1 S1 S2
;a
6th to 7th grade (logit) 0.55 0.38 0.34

cb
SEb 0.10 0.10 0.06
7th to 8th grade (logit) 0.35 0.72 0.09
SE 0.11 0.11 0.06
S = suburban; U = urban;
SE = standard error of measurement

This paper illustrated how we created and validated an instrument that allows us to create a detailed characterization of student performance. We will use these items examine the effect that various factors (e.g., gender, school context, curriculum materials) that can affect student learning related to explaining transformations of matter. This is a necessary step for empirically testing a LP. Many factors (e.g., context, instruction, curriculum materials, and students prior knowledge and experiences) affect how and when students can learn new ideas and connect them to other related ideas. Learning research suggests that learners may follow different pathways as they work to make sense of ideas and
ISLS
399
ICLS2012
develop integrated understanding. Thus, multiple instructional strategies and curriculum materials may be successful in helping students to develop more sophisticated knowledge and skills, depending on their previous academic and life experiences. We will use this validated scale developed to locate students along the LP and relating student performance to determine how these factors may affect student performance. In addition, we are analyzing the curriculum materials and classroom observation data to more fully characterize students instructional experiences to help us connect various factors with students measured progress. These results will inform the development of curriculum materials and instruction that better support the development of student understanding of ideas related to the structure, properties and behavior of matter. We are in the process of collecting longitudinal data to track student learning along the LP across three years using the validated items (Phase III). In addition, we are running confirmatory factor analysis and multidimensional IRT to measure the validity and reliability of the LP items and to determine if the items clustered as defined in the LP.
References
Barab, S. A. & Squire, K. D. (2004). Design-Based Research: Putting Our Stake in the Ground. Journal of the Learning Sciences, 13(1), 1- 14. Brown, A. L. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. The Journal of the Learning Sciences, 2(2): 141-178. Collins, A., Joseph, D. and Bielaczyc, K (2004). Design research; Theoretical and methodological issues, Journal of the Learning Sciences, 13(1), 15-42. Duncan, R. G., Rogat, A. D., & Yarden, A. (2009). A learning progression for deepening students understandings of modern genetics across the 5th-10th grades. Journal of Research in Science Teaching, 46(6), 655-674. Duschl, R. A., Schweingruber, H. A. and Shouse, A. (eds) (2007). Taking science to school: Learning and teaching science in grades K-8, Washington, D.C.: National Academy Press. Krajcik, J., McNeill, K. L, & Reiser, B. J. (2008). Learning-goals-driven design model: Developing curriculum materials that align with national standards and incorporate project-based pedagogy. Science Education, 92(1), 1-32. Marx, R.W., Blumenfeld, P.C., Krajcik, J.S., Fishman, B., Soloway, E., Geier, R., & Tal, R.T. (2004). Inquirybased science in the middle grades: Assessment of learning in urban systemic reform. Journal of Research in Science Teaching, 41(10), 10631080. Merritt, J., Krajcik, J., & Schwartz, Y. (2008). Development of a learning progression for the particle model of matter. Proceedings from the 8th International Conference of the Learning Sciences in Utrecht, the Netherlands. Mislevy, R. J. and Riconscente, M. (2005). Evidence-centered assessment design: Layers, structures, and terminology, Menlo Park, CA: SRI International. National Research Council (NRC; 2012). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas. The National Academies Press: Washington, D.C. Roseman, J. E., Caldwell, A., Gogos, A., & Kurth, L. (April, 2006). Mapping a coherent learning progression for the molecular basis of heredity. Paper presented at the annual meeting of the National Association for Research in Science Teaching, San Francisco, CA. Shin, N., Stevens, S. Y., & Krajcik, J. (2010). Using Construct-Centered Design as a systematic approach for tracking student learning over time. In (Ed). S. Rodrigues, using analytical frameworks for classroom research: collecting data and analysing narrative, Routledge, Taylor & Francis, London. Smith, C. L., Wiser, M., Anderson, C. W. and Krajcik, J. (2006). Implications of research on childrens learning for standards and assessment: A proposed learning progression for matter and the atomic molecular theory, Measurement: Interdisciplinary Research and Perspectives, 4(1), 1 98. Stevens, S. Y. & Shin, N. (April, 2012). Developing and Validating a Ruler to Locate and Follow Students Along a Learning Progression. Poster to be presented at the American Educational Research Association Conference, Vancouver, BC. Stevens, S. Y., Delgado, C., & Krajcik, J. S. (2010). Developing a hypothetical multi-dimensional learning progression for the nature of matter. Journal of Research in Science Teaching, 47(6), 687-715. Wilson, M. (2005). Constructing Measures: An Item Response Modeling Approach. Mahwah, NJ: Erlbaum. Wu, M. L., Adams, R. J., Wilson, M. R., and Haldane, S.A. (2007). ACER ConQuest Version 2: Generalised item response modelling software. Camberwell: Australian Council for Educational Research.
Acknowledgments
This work was funded by the National Science Foundation (grant numbers 0822038 and 0426328). Any opinions expressed in this work are those of the authors and do not necessarily represent those of the funding agency. We also wish to thank the participating schools, teachers and students, and our research team.
ISLS
400
ICLS2012
Learners as Informants of Educational Game Design

Beaumie Kim, Lynde Tan, Mi Song Kim National Institute of Education, Nanyang Technological University 1 Nanyang Walk, Singapore 637616 beaumie@gmail.com, lynde.tan@nie.edu.sg, misong.kim@gmail.com Abstract: This paper describes an educational game design approach that attempts to position learners as the best resources in knowing what engages them in a learning environment. This approach, known as informant design approach, enables young people to draw on their cultural experiences in learning and gaming as resources for game development. In the threeyear research and development project, we conducted five progressive informant design workshops with twenty adolescent learners, where their conceptions of earth science as well as ideas for game design are considered at its various stages of educational game development. Our findings from the qualitative analysis of video and artifacts draw attention to the learners cultural models of learning and gaming and discuss the importance of engaging learners as informants of educational game design.
Introduction
Young people may often be thought to be ignorant of what is best for them. This paper attempts to position learners as the best resources in knowing what engages them in a learning environment, for educational game design. Such a disposition to game development is inspired by Gees (2003) argument about cultural models and the role they play in learning. According to Gee (2003), cultural models are: stories or images of experience that people can tell themselves or simulate in their minds, stories and images that represent what they take to be normal or typical cases or situations We act with others and attempt to make sense of what they are doing and saying. We interact with the media of our society and attempt to make sense of what is said and done there, as well. (p. 146) Bartlett and Holland (2002) would concur that learners bring their cultural models to learning environments. Using the term cultural worlds, Bartlett and Holland argue, cultural worlds are continuously figured in practice through the use of cultural artefacts or objects inscribed by the collective attribution of meaning" (p.12). It was our aim to involve learners in our game design by providing them with opportunities to reflect on their cultural model of the way they think about learning and gaming and use their funds of knowledge (Gonzlez, Moll, & Amanti, 2005) as resources for the design. In this paper, we describe how the design approach, known as the informant design approach, has helped us understand young peoples cultural models of learning with games. In this approach, we were able to create commonalities of experiences among the learners so as to help them reflect upon their cultural models of learning and game play. We were also able to explore ways to better encourage learners to use their own funds of knowledge for learning and designing by crossing the boundaries between their formal classroom learning and their cultural practices. The findings highlight the use of their cultural models as resources for the design as well as their cultural models being challenged by the constraints of designing an educational game, which they also felt should be different from commercial games they were familiar with.
Methodology: Informant Design Approach

With the emergence of new technologies, how you learn in and out of the classroom is drastically changing. At the same time, todays youths are not only the users of emerging technologies but also creators of many contents occupying them. In recent years, the idea of involving the target users of educational resources or tools during the development stages has been cited as a more efficient approach in meeting the pedagogical needs of those who may use the resources or tools. Many studies have reported the merits of such an approach, namely to allow the users contributions to be effected into the redesign processes (Scaife & Rogers, 1999) to gain access to the first-hand knowledge of the users needs and preferences (Antle, 2003; Scaife & Rogers, 1999); to tap on the creativity of the users (Antle, 2003; Rudd, Colligan, & Naik, 2006; Scaife & Rogers, 1999); and to help very young children to express their ideas (Walsh et al., 2010). Of note is Facer and Williamsons (2004) argument that when co-designing educational technologies with their target users, the following benefits of such an approach are evident: (i) it offers first hand experience of the needs, interests and requirements of end-users; (ii) it enables developers to be surprised by users and to avoid creating formulaic work; (iii) it allows developers to avoid costly mistakes and identify design difficulties at an early stage; and (iv) it offers the opportunity to create resources that are embedded in teaching strategies and educational contexts, and which consequently, actually achieve their educational aims (p. 1). Depending on the roles of the young people, the processes are called user-centered design, participatory design, and informant design approaches (Facer & Williamson, 2004). User-centered design views young
ISLS 401
ICLS2012
people as testers for designs to assess whether their needs are met (Norman & Draper, 1986). However, the limited involvement often does not allow timely input into the redesign process (Scaife & Rogers, 1999). Participatory design considers young people as partners throughout the design process by assigning them more equal and responsible roles and empowering them to make design decisions (Druin, 1999). The true partnership, however, is unlikely: young people do not see adults as equal in making design decisions; and adults do not see that young people can control the design directions (Scaife & Rogers, 1999). Informant design approach (Scaife & Rogers, 1999) takes a moderate stance in terms of the young peoples role in the design: they play various roles in each stage, depending on when researchers believe they can give appropriate information and ideas. Facer and Williamson (2004) highlighted that in informant design approach children or teachers are seen as experts or native informants informing designers of key issues related to their experience, helping to develop early design ideas and testing prototypes in development (p. 1). In our study, we have found three compelling reasons for using informant design approach to develop a game for learning earth science concepts (Kim, Tan, & Kim, 2010): (i) it creates an intertextual link with other sources of the students knowledge; (ii) it creates deeper meaning making for the learning of earth science concepts; and (iii) it taps on their funds of knowledge, bridging their school and out-of-school practices (p. 2). In our designs, we have adopted a similar stance as Facer and Williamson (2004). A key goal of our project is to develop a 3D multi-user game to recreate and replay parts of Earths history to use within the school Geography curriculum to address contents related to the Earths processes. We call this game Voyage to the Age of Dinosaurs or VAD for short. This is because the theme of dinosaurs is found appropriate for approaching the topics of the Earths processes focusing on fossilization, which has the potential to provide learners with alternative ways of experiencing Earth processes as part of a complex whole and to support understandings of the relationships among geological events. Therefore, dinosaurs as extinct creatures and their fossils as traces of Earths history were considered as the conceptual and motivational anchors of VAD. Figure 1 shows the scenes in one of the VAD prototypes where players interact with dinosaurs in the past world (early Cretaceous) where there exists both abundance and danger (i.e., fertile lake with volcano and volcanic eruptions). These dinosaurs are the same kinds as the fossils they search in the present world in the VAD game.
Figure 1. Navigating using compass bearings & interacting with dinosaurs in the past world Over the course of three years, we conducted five progressive design workshops as our informant design approach. There were twenty-two students between the age of thirteen and fifteen (equivalent to US Grade 7-10) from two secondary schools participated as users and design partners; their experiences with technology center around online communications activities and online games, such as MapleStory and World of Warcraft based on our initial survey. These workshops helped us establish a framework for collaborative design practices with students (as suggested by Brown, 1992; Druin, 1999; Scaife & Rogers, 1999). We started by the effort of listening to students voices and continued to encourage their agency in the design activities. Each workshop can be characterized as: (Workshop I) Understanding Learner Conceptions about the Earth; (Workshop II) Creating Learners Narratives about Earth and Dinosaurs; (Workshop III) Dinosaur Game Play and Ideas (Prototype I Testing); (Workshop IV) Experiencing the Game Narrative (Prototype II Testing); and (Workshop V) Designing Specific Parts of the Game (Prototype III Testing) (refer to Figure 2.). For this paper, the data is drawn from the student-created artifacts (20 individual and 16 group) and 16 hours of video recordings of their group work. The artifacts and videos were annotated for various activities they were involved and then analyzed for themes. We started with an open, vertical coding for look out for emerging themes for each group by data types (i.e., artifacts and video). We then conducted horizontal coding where we compared the themes across the groups (Lincoln & Guba, 1985). We discuss various themes of students ideas and cultural models around the issues of learning and gaming. In their discussions during the design workshops, they made use of their prior knowledge from what they had seen in movies and other media, and from their gaming experience. Drawing on Gees (2003) cultural models, we present the three categorizations of the learners cultural models of learning in education games: 1) ideas as model answer, 2) gaming as non-learning, and 3) discovery or forced reading. Their pre-existing cultural models on how learning only took place through overt instruction by the adults (such as teachers and parents) had been continuously surfaced and challenged during the informant
ISLS 402
ICLS2012
design workshops. In the following, we describe how four groups of teenage students (4 or 5 persons in one group) shared (and transformed) their cultural models about games and learning through these design workshop. We present themes that emerged consistently across all four groups. In the interest of the space, only illustrative examples are presented in our discussion below.
Learners Cultural Models of Learning in Educational Game

Ideas as the Model Answers
In our efforts to listen to learners own ideas about educational game design, we were able to understand their cultural models on how they had been gaining knowledge and understanding. Specifically, when we asked the learners for possible game ideas, they regarded the notion of ideas as information they could recall. During Workshop I, learners often looked for correct information in their mind from what they remembered from books, media and remarks by authoritative figures such as teachers and parents, and tried to reproduce them. During the discussion on how volcano erupts, one of the groups was engaged in a conversation on whether or not their teachers had taught this topic and how they could not remember even though they thought they had heard something about it before1. In developing a story and a short film about dinosaurs and their fossils during Workshop II, some initially looked for good answers by suggesting imitating what the research team had provided as an example.
Gaming as Non-Learning
In Workshop III, there were conflicting discourses about how learners could learn in games. When asked to design the hottest game in town, they were focused on how they should have challenging tasks to improve their skills and acquire more advanced tools. This means that they want to become a better game player by playing the game. On the other hand, as a suggestion to improve VAD prototype I2, some proposed having many stages to complete by answering questions correctly, which mirrors knowledge testing culture of schools (and/or many educational games). While designing games for learning, students were making attempts to bridge their game play (outside school) with school learning (i.e., by suggesting question-andanswer part for the game). For them the intertexuality of gaming (i.e., learning skills by playing) suggests a separate regime from that of learning (i.e., gaining knowledge to be tested). During Workshop IV, students played and made suggestions for VAD prototype II. In this prototype, players could earn Experience Points and lose Health and Life points - if players lose life, they were sent to the Health Camp, where they could consult a Dinopedia and complete quizzes on volcanoes and rock types while "recovering," before they returned to active play. Again, some criticisms pointed to the overt nature of the pedagogy in the Healing Camp (i.e., solving quizzes and puzzles to restore their lives). On the other hand, some suggested using questioning of knowledge in the context of players advancing within the quest or to the next level. One group suggested, Assemble the fossils after the excavation. Take them to Dr. KongLong. He will test your knowledge on fossil and dino rocks and dinos. If you gain xx points, you will advance to some time-travelling expedition. But if you don't, you will lose all your points and fossils. They also wanted the Healing Camp to be a more of training ground where they could practice shooting dinosaurs, which would help them do better in the game. What they had suggested indicated how learning happens incrementally through practice and multiple achievements (Gee, 2003), where learners repeat the same activities (i.e., searching & assembling fossils, being tested by Dr. KongLong, keeping themselves from dying, and shooting dinosaurs) until they are successful so that they earn points, improve skills, and move on to next levels.
Discovery or Forced Reading

Additionally in Workshop IV, students suggestions were looking for more challenges and discovery, which would allow them to have different achievements depending on how they play in the game. For example, one group suggested a narrative where the [t]ime Machine spoil: everyone find materials to build time machine, whereas another group wanted to discover various things as they go by having [h]idden passages, traps, hidden eggs, time-traveling expeditions, weapons to be found on the floor. For the things they needed in order to advance and gain points for the game play, they expected themselves to explore the environment and find these resources themselves. On the other hand, students suggested the objects, characters, and the environment to embody information for educational purposes. They said, Upon capturing of dino, the info will be in the logbook/displayed on screen for 2-5 min (forced to read) and Labeling species of dinos, which resonate with Gees (2003) Material Intelligence Principle of learning (i.e., knowledge is stored in the environment/objects). These learners were suggesting that the main thing to learn in the game was the information about various kinds of dinosaurs; such fact finding practice was integral to the gaming experience outside of school. For instance, in many online game forums for monster-appearing games that they had played, one of the main information shared and sought for were the list of different kinds of monsters. Dinosaurs species are also important information in understanding evolutions of various animal species and highlighted in many existing media (e.g, books, TV shows, movies, games, museum displays). At the same time, this shows their conflicting cultural models about learning in that the reading needs to be forced in order for it to happen (and perhaps
ISLS
403
ICLS2012
they think it is important that they read information provided even though they do not like to do so). In the short excerpt3 of students conversation below, it is notable that in the midst of their effort to bringing in educational purpose in the game ideas, they kept drawing on their cultural model of dinosaur as a gigantic and violent creature that is a good fit for their monster image of games: 1. Ken: Perhaps this can also be education. Like, for example, you gather the DNA of different dinosaurs then introduce dinosaurs. 2. Nick: But, then when you get back to the present right, then a GIANT dinosaur follow you, then it is the boss stage4 (waving hands around to show that it is big). 3. Facilitator: Boss stage some more? Haha (all start laughing) Although students tend to get more excited about the prospect of fighting bigger and stronger dinosaurs (turn #2), some have started to show some agency in trying to reconcile the perceived divide between gaming and learning, by suggesting how to earn the information (they deemed important) themselves in the game (turn#1).
Implications for Game Design

We learn to think, feel and act in certain ways through our cultural experience in our lifeworlds. These experiences may allow us to form nave ideas, but at the same time, become generative resources for learning when we take them seriously (i.e., situating the concepts/ideas in ways that consider learners cultural models) (Kim & Kim, 2010). Students, when we initially asked them to produce their ideas about Earths dynamics and stories, felt obligated to produce model or correct answers. At the same time, their suggestions for gaming tasks (i.e., more actions and achievements) and learning tasks (i.e., more knowledge-based) were mostly separate within their game ideas. Although they have applied a different mechanism to what they have thought educational in the game (e.g., displaying information so that they are forced to read it) might mean, they have started bridging the two (e.g., players need to earn such information by performing tasks) over the course of five workshop participations.
Figure 2. Five informant design workshops Our informant design approach with five workshops might have gradually changed the role of students. The change is in the design of the workshops, the development phase of the game, students rapport with each other and the research team, and especially students attitudes toward their own ideas. Figure 2 epitomizes the five workshops, their designs and activities. We started from understanding their cultural models related to the concepts and stories about the Earth and dinosaurs, and shifted toward asking them to expand the story and concepts in the prototype using their own ideas. We see that our effort has been also shifting from hearing learners voices to relying on learners agency. Voice has different connotations for different research context (see, Fielding, 2007), but the research often involves the intention that it matters what young people think. For us, it started with trying to understand their cultural models. We often heard their competing voices in their discourse, with which they mixed ideas from adults with their own interpretations and/or sought for correctness or appropriateness of their ideas in the earlier workshops. The learner agency we saw from the participating students was their use of their cultural models about gaming and learning in their design ideas to merge them together, which is similar to what Greeno and van de Sande (2007) called, conceptual agency: choosing or adapting concepts/strategies/methods for use in problem solving or understanding. As mentioned earlier, students were also challenged with their cultural model on how they see the game disconnected from the real world, especially from learning. Through the design and development of the
ISLS
404
ICLS2012
workshop and the game, and the workshop implementation with students, we were able to explore various ways to embody concepts within the game environment as well as to realize how difficult (and/or inappropriate) it is sometimes to do so. Our best interest also gradually shifted from game design itself to focusing on connecting what games can embody with what physical world (including people) embodies. Especially in workshop IV, we translated tasks or concepts we intended in the game into the hands-on activities. Such workshop helped us to see what might actually better represented in the game, what might be better as hands-on activities, and how we might help them link their experiences in and out of the game and in and out of the classroom. Immersion into the game (or other) activities are important, but sitting back, reflecting on it, and linking it to concepts and other experiences are much more important. In conclusion, our design approach, as we were designing workshops and conducting them, taught us how we might better understand and challenge learners and our cultural models, and how we together develop as better designers and learners.
References
Antle, A. (2003). Case study: The design of CBC4kids story builder. Paper presented at the Interaction Design and Children 2003, Preston, England. Kim, B., Tan, L., & Kim, M. (2010). Unpacking learner voices for educational game design. In Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2010 (pp. 3370-3375). Chesapeake, VA: AACE. Kim, B., & Kim, M. S. (2010). Distributed emotions in the design of learning technologies. Educational Technology, 50(5), 14-19. Brown, A. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. The Journal of Learning Sciences, 2(2), 141-178. Druin, A. (1999). Cooperative inquiry: Developing new technologies for children with children. Paper presented at the CHI'99, New York. Facer, K., & Williamson, B. (2004). Designing educational technologies with users. Retrieved March 20, 2009, from http://www.futurelab.org.uk/resources/publications-reports-articles/handbooks/Handbook196 Fielding, M. (2007). BeyondVoice: Newroles, relations, and contexts in researching with young people. Discourse: studiesinthecultural politicsof education, 28(3), 301-310. Gee, J. P. (2003). What video games have to teach us about learning and literacy. New York: Palgrave Macmillan Gonzlez, N., Moll, L., & Amanti, C. (2005). Funds of knowledge: Theorizing practices in households, communities, and classrooms. New Jersey: Lawrence Erlbaum Associates. Greeno, J. G., & van de Sande, C. (2007). Perspectival Understanding of Conceptions and Conceptual Growth in Interaction. Educational Psychologist, 42(1), 9 - 23. Lincoln, Y. & Guba, E. (1985). Naturalistic Inquiry. London: Sage. Norman, D. A., & Draper, S. W. (Eds.). (1986). User centered system design: New perspectives on humancomputer interaction. Hillsdale, NJ: Lawrence Erlbaum Associates. Rudd, T., Colligan, F., & Naik, R. (2006). Learner voice. Retrieved March 20, 2009, from http://www.futurelab.org.uk/resources/publications-reports-articles/handbooks/Handbook132 Scaife, M., & Rogers, Y. (1999). Kids as informants: Telling us what we didn't know or confirming what we knew already. In A. Druin (Ed.), The Design of Children's Technology (pp. 28-50). San Francisco, CA: Morgan Kaufmann. Walsh, G., Druin, A., Guha, M. L., Foss, E., Golub, E., Hatley, L., Bonsignore, E., & Franckel, S. (2010). Layered elaboration: a new technique for co-design with children. In Proceedings of ACM CHI2010 (pp. 1237-1240). Atlanta, GA. Wang, X., Kim, B., & Kim, M. (2011). Extrapolating from students preconception to scientific consensus: Singapore secondary students conversation on Earth. The Asia-Pacific Education Researcher, 20(2), 261-275. Wikipedia. (2011). Boss (video gaming). Wikipedia. Retrieved from http://en.wikipedia.org. Education.
Endnotes
(1) However, some students were able to engage in deeper discourse on how things happen based on their ideas even though some could not let go of discussing their lack of correct knowledge. Details about their discourse are not reported in this paper. See, Wang, Kim, & Kim (2011) for reference. (2) The first prototype was developed within the storyline of searching the fossils and seeing dinosaurs in action similar to those of students stories from Workshop II. Players meet Dr. Konglong who asks their help to save Dilong from early Cretaceous period. They first collect fossils to open the portal to the past, where they find information about volcanoes structure and kinds from the past, and save Dilong and themselves from the volcanic eruption. (3) We have used pseudonyms to protect confidentiality and removed any parts of Singapore Colloquial English that affected the intelligibility of the language used. (4) They are referring to a battle or fight with a character, which is generally seen at the climax of a particular section of the game, usually at the end of a stage or level. The boss enemy is generally far stronger than the opponents the player has faced up to that point (Wikipedia, 2011).
ISLS
405
ICLS2012
The Role of Gesture in Solving Spatial Problems in STEM

Matthew E. Lira, Mike Stieff, Stephanie Scopelitis, University of Illinois at Chicago, 1007 W. Harrison Street, Suite 2048; Chicago, IL 60607 Email: mlira2@uic.edu, mstieff@uic.edu, scopelitis@gmail.com Abstract: Gestural activity is an important component of teaching and learning in STEM disciplines. Gestures are most often observed in social settings where two or more interlocutors are engaged in natural conversation. Empirical investigations, however, suggest that gestures also play a role in individual cognition. We conducted two experiments that investigated the functional role of gestures as a strategy for solving translation problems in organic chemistry. Gestured-instructions did not promote gesture-use as a strategy. Removing instructional diagrams from view did promote gesture use. Students who did gesture did not perform better than those who did not; however, students who received gestured-instructions appeared more likely to adopt the spatial perspective of the instructor.
General Introduction
Gestural activity is an important component of teaching and learning in STEM (Science, Technology, Engineering and Mathematics) disciplines (Alibali & Nathan, 2007; Radinsky, 2008; Roth, 2000). The role of gestures to support communication and meaning making in STEM is most often observed in social settings where two or more interlocutors are engaged in natural conversation. Empirical investigations suggest that gestures also play a role in individual thinking (Rauscher, Krauss, & Chen, 1996; Roth & Hwang, 2006), learning (Goldin-Meadow, Alibali, & Church, 1993), and problem-solving (Hegarty, 2004; Schwartz & Black, 1996). Moreover, gestures can function pedagogically (Ehrlich, Levine, & Goldin-Meadow, 2006) by augmenting reasoning and problem-solving. Recent studies investigating the functional role of gesture in tasks such as mental rotation (Chu & Kita, 2011), mathematical equation solving (Alibali & Goldin-Meadow, 1993), Piagetian conservation tasks (Church & Goldin-Meadow, 1986), Tower of Hanoi tasks (Garber & GoldinMeadow, 2002), and gear-rotation tasks (Perry & Elder, 1997) suggest that gesture not only supports communication, but also plays an important role in individual cognition. Although not a primary focus of analysis in studies of individual gesture use, one feature common to the problem-solving tasks in these studies is that each involves reasoning about dynamic, spatial-relational information. Gestures are complex motor movements that occur in 3-dimensional space. Importantly, gestures afford the individual the ability to represent dynamic, spatial-relational information with the hands. Indeed, the history of gesture analysis suggests that gestures convey the imagistic component of thought (McNeill, 1996; McNeill, 2005). Theoretically, then, when an individual displays gestures while problem solving it offers evidence that imagistic reasoning is used as the primary problem-solving strategy (Stieff, 2011; Stieff & Raje, 2010). Nevertheless, the precise functional role of gesture in problem solving remains unknown. For example, gesture use may allow the problem solver to offload spatial information onto the hands and thus free up cognitive resources (Goldin-Meadow, Nusbaum, Kelly, & Wagner, 2001). Alternatively, gestures may cue visual or motor schema that provide important insights into spatial relationships. Then again, gestures may be epiphenomenal and simply indicate that other unseen processes are at work. The present study offers initial insight into the functional role of gesture in spatial problem solving in a STEM discipline. Specifically, we present our first efforts to understand the effect of gesture use on solving representational translation problems, or translations between molecular diagrams, in organic chemistry. Organic chemistry is a domain heavily focused on the study of spatial relationships in molecules and is an opportune discipline to study the role of gesture use during individual problem solving. As part of the course of study in organic chemistry, students must learn to represent three-dimensional structures with two-dimensional diagrams, such as Dash-wedge, Newman, and Fischer Projections (see Figure 1). To successfully translate between these diagrams, students must attend to the precise way that each diagram depicts spatial-relational information differently. Previously, Stieff and colleagues (Stieff, 2011; Stieff & Raje, 2010; Stieff, Ryu, & Dixon, 2010) showed that self-reports of using imagistic strategies to solve such problems occur contemporaneously with the use of spontaneous gestures during problem-solving activities. In the present study, we examined whether gestured problem-solving instructions can (1) improve students achievement on translation tasks and (2) increase students use of gestures (and presumably imagistic reasoning strategies) during problem solving.
ISLS
406
ICLS2012
Figure 1. Three commonly used molecular diagrams representing different perspectives on the same molecule.
Methods & Analysis

We conducted two experiments that investigated the functional role of gesture in problem-solving. In the first experiment, we recruited 20 participants enrolled in either a first- or second-semester undergraduate organic chemistry course. Participants were randomly placed into one of two groups: No Gesture Instructions (NG) or Gestured Instructions (G). In the NG-group (10 participants), problem-solving instructions and short explanations of each depicted diagram were presented via written text and diagrams only. Students were asked to read the directions/explanations on their own, and the instructions and accompanying images were made available to NG-group participants during problem solving. In the G-group (10 participants), the same instructions and explanations were presented by the experimenter via talk and gestures, which were produced over the relevant molecular diagrams presented on an 8.5 X 11 page (see Figure 2). Instructional gestures were based on gestures observed in use by organic chemistry lecturers teaching at the participants university. The molecular diagrams were made available to participants during problem solving, but the written instructions were removed. Following instruction, participants in both groups completed an 18-item organic chemistry representation translation assessment. Each item presented one representation (Fischer, Newman or Dashwedge) of a molecule and instructed the student to draw the molecule using one of the other two representations. Upon completing the assessment, participants were debriefed, paid $20 USD, and dismissed.
Figure 2. An example of a modeling gesture used during instruction. In the second experiment, we followed a similar method with 24 additional participants (12 NG-group, 12 G-group). Here, we altered the protocol by removing the molecular diagrams after providing instructions and explanations to examine how the availability of the diagrams affected gesture production and achievement. For this analysis, we exclusively coded the problem-solving segments and did not analyze instances of instruction or debrief. Videos were reviewed for instances of gestural activity and instances related to depiction were identified. A single instance of gestural activity was defined as the time from which the hands first engaged in gesturing to the time they came to rest. We define depictive gestures to include all gestural actions that invoked representations of an object or idea, such as molecular shape or structure. For example, participants could use their hands to depict a molecule by extending two fingers to invoke nearby bonds as in Figure 2. Similarly, participants could simulate performing an action upon a referent, such as grabbing a depicted bond and rotating it clockwise. We did not analyze personal (e.g., face-scratching), deictic (e.g., pointing at the paper or diagram), or pre-writing (e.g., tracing a structure on the paper) gestures. Coded gestures were labeled either gross or subtle: gross refers to large, full hand actions and subtle refers to small, abbreviated actions. Student responses to each assessment item were coded as absolutely or relatively correct. Absolutely correct responses included only those drawings that rendered the molecule in the exact spatial perspective and conformation as it was presented in the given diagram. Relatively correct responses included any drawing that
ISLS
407
ICLS2012
preserved the correct internal spatial-relations, but did not render the exact spatial perspective and conformation of the given diagram. For example, a relatively correct response might include drawings of the correct molecule rotated 180 degrees. Additionally, each response was coded for directionality, orientation, and conformation. In terms of directionality, responses were coded as rendered from the left or right depending upon the given diagram. In terms of orientation, responses were coded as rendered as right-side-up or upside-down. In terms of conformation, responses were coded as rendered with all internal spatial-relationships preserved as in the given diagram or modified. Importantly, the gestures produced by the interviewers were systematically presented from the right side direction with a right-side-up orientation toward the given diagrams.
Results & Discussion

First, we address the question, Does gestured instruction improve achievement? Based on prior research involving the functional role of gesture during problem solving, we predicted that gestured instruction would improve achievement on the representational translation task. In addition, we are also able to examine the interaction of diagram availability and gestured instructions on achievement because we manipulated the availability of molecular diagrams across experiments. Because the instructional diagrams depict important spatial relationships between each representation, we predicted that achievement would be higher when diagrams are made available. To test these predictions, we analyzed the total number of correct items via a 2 (Gestured Instruction, No Gesture Instruction) x 2 (Diagrams Available, Diagrams Removed) ANOVA. The analysis revealed that there was no effect of instruction, F(1, 44) = .32, p = .27, or diagram availability, F(1, 44) = .13, p = .72, on achievement. Additionally, no interaction was observed. We next asked, Does students gestural activity increase or decrease after gestured instructions? As above, we were again able to examine the effect of diagram availability on students gesture use. To answer this question we analyzed the total number of gestures produced by students via a 2 (Gestured Instruction, No Gesture Instruction) x 2 (Diagrams Available, Diagrams Removed) ANOVA. The analysis revealed that there was no main-effect of gestured instruction, F(1, 34) = .98, p = .33. We did, however, observe a main-effect of diagram availability, F(1, 34) = 4.7, p = .04. As seen in Figure 3, removing the diagrams from view did produce more gestural activity by students.
Figure 3. Participants mean number of gestures by instructions and diagram availability. As a follow up to both questions, we explored the relationship between individual gesture use and achievement. Achievement was equivalent between those students who used at least one gesture and those who produced no gestures at all during problem-solving, F(1, 34) = .67, p = .42. Among students who did gesture during problem-solving, however, a moderate correlation was observed between gesture use and achievement that is likely driven by three outliers who produced extremely high numbers of gestures during problem-solving, r(23) = .58, p = .004.
ISLS
408
ICLS2012
Given that our instructional manipulation did not improve achievement or increase gestural activity during problem solving, we further examined the possible effect of our instructional conditions on the quality of diagrams produced by participants. Recall that the gestures produced by the interviewers during instruction always depicted a right-hand direction and right-side-up orientation toward the molecular diagrams. Specifically, we were interested in knowing whether participants were more likely to adopt an instructors perspective when learning from gestures. First, we examined the directionality of drawn structures as above. Interestingly, we observed a trend in the dataset that did not meet statistical significance at alpha = .05: a greater number of right-side approach diagrams were observed when instructions were accompanied by gesture, F(1, 43) = 2.34, p = .10 (see Figure 4). Although gestured instructions produced no observable increase in gesture use or achievement, it did appear to affect the students perspective when drawing molecular diagrams even in a study with low N, as here.
Figure 4. Participants mean number of right-side depictions by instructions and diagram availability.
Scholarly Significance & Implications

Previous investigations have demonstrated that gestures maintain spatial imagery during communicative activities (Wesp, Hesse, Keutmann, & Wheaton, 2001). Our investigation suggests that this previous finding may extend to contexts in which individuals are reasoning about spatial-relationships in isolation. Moreover, our results indicate an important interaction between gesture use and external representations. That is, individuals appear to use gestures to represent spatial information when external representations of that information are unavailable. Here we observed that when such diagrams are removed from sight, students produced significantly more gestures than when the diagrams were made available. However, many students used gestures while problem solving in an idiosyncratic fashion across experiments: Some participants gestured infrequently, some gestured frequently, and still others gestured randomly during the experiment. Of note, the highest achieving students were observed to produce the largest number of gestures on nearly every task. Despite the increased gestural activity in the diagrams removed condition, we did not observe a large effect of gesture use on achievement across conditions. Given that we observed a moderate correlation between gesture use and achievement, it nevertheless remains a possibility that increased gesture use may improve achievement. In this investigation, we observed an appreciable amount of variance in gesture production and it may be that participants did not produce a sufficient amount of gestures to support problem solving. Participants may have not produced gestures for several reasons, one being that they have pre-existing strategies for solving translation problems, as has been documented elsewhere (cf., Stieff & Raje, 2010), which they favored over the strategy presented during instruction. As it stands, these results suggest that our instructional intervention may not have been robust enough to encourage students to use gestures during problem solving, which warrant future investigations that include more extensive interventions.
ISLS
409
ICLS2012
References
Alibali, M.W. & Goldin-Meadow, S. (1993). Gesturespeech mismatch and mechanisms of learning: What the hands reveal about a childs state of mind. Cognitive Psychology, 25, 468-523. Alibali, M. W., & Nathan, M. J. (2007). Teachers gestures as a means of scaffolding students understanding: Evidence from an early algebra lesson. Video Research in the Learning Sciences. Mahwah, NJ: Erlbaum, Chu, M., & Kita, S. (2011). The nature of gestures beneficial role in spatial problem solving. Journal of Experimental Psychology: General, 140(1), 102-116. Church, R.B, & Goldin-Meadow, S. (1986). The mismatch between speech and gesture as an index of transitional knowledge. Cognition, 23, 43-71. Ehrlich, S. B., Levine, S. C., & Goldin-Meadow, S. (2006). The importance of gesture in children's spatial reasoning. Developmental Psychology, 42(6), 1259. Garber, P., & Goldin-Meadow, S. (2002). Gesture offers insight into problem-solving in adults and children. Cognitive Science, 26(6), 817-831. Goldin-Meadow, S., Alibali, M. W., & Church, R. B. (1993). Transitions in concept acquisition: Using the hand to read the mind. Psychological Review, 100(2), 279. Goldin-Meadow, S., Nusbaum, H., Kelly, S. D., & Wagner, S. (2001). Explaining math: Gesturing lightens the load. Psychological Science, 12(6), 516. Hegarty, M. (2004). Mechanical reasoning by mental simulation. Trends in Cognitive Sciences, 8(6), 280-285. McNeill, D. (1996). Hand and mind: What gestures reveal about thought. University of Chicago Press. McNeill, D. (2005). Gesture and thought. University of Chicago Press. Perry, M., & Elder, A. D. (1997). Knowledge in transition: Adults' developing understanding of a principle of physical causality* 1,* 2. Cognitive Development, 12(1), 131-157. Radinsky, J. (2008). Students' roles in group-work with visual data: A site of science learning. Cognition and Instruction, 26(2), 145-194. Rauscher, F. H., Krauss, R. M., & Chen, Y. (1996). Gesture, speech, and lexical access: The role of lexical movements in speech production. Psychological Science, 7(4), 226. Roth, W. M. (2000). From gesture to scientific language. Journal of Pragmatics, 32(11), 1683-1714. Roth, W. M., & Hwang, S. W. (2006). On the relation of abstract and concrete in scientists' graph interpretations: A case study. The Journal of Mathematical Behavior, 25(4), 318-333. Schwartz, D. L., & Black, J. B. (1996). Shuttling between depictive models and abstract rules: Induction and fallback. Cognitive Science, 20(4), 457-497. Stieff, M., & Raje, S. (2010). Expert algorithmic and imagistic problem solving strategies in advanced chemistry. Spatial Cognition and Computation, 10(1), 53-81. Stieff, M., Hegarty, M., & Dixon, B. L. (2010). Alternative strategies for spatial reasoning with diagrams. In A. K. Goel, M. Jamnik, & N. H. Narayanan (Eds.), Diagrammatic Representation and Inference. (pp. 115127). New York: Springer. Stieff, M. (2011). When is a molecule three dimensional? A task-specific role for imagistic reasoning in advanced chemistry. Science Education, 95, 310336. Wesp, R., Hesse, J., Keutmann, D., & Wheaton, K. (2001). Gestures maintain spatial imagery. American Journal of Psychology, 114(4), 591-60
Acknowledgement
The preparation of this paper was supported by the National Science Foundation, NSF Award #0723313 (UMD). We would like to gratefully acknowledge Lianne Schroeder for her assistance with data collection and data analysis.
ISLS
410
ICLS2012
Audience Effects: A Bidirectional Artifact Analysis of Adolescents Creative Writing

Alecia Marie Magnifico, University of Illinois at UrbanaChampaign, amagnif@illinois.edu Abstract: Much of the literature that developed rhetorical and cognitive viewpoints on audience suggests that when writers write with an audience in mind, imagined readers affect the writing. This paper investigates these audience effects by examining a creative writing workshop and asking how conversations with readers lead to revisions. Drawn from a case study of an 11th grade classroom, this bidirectional artifact analysis traces writing through students drafts, revisions, and feedback. The resulting work shows how revision and writing development ties to interactions with readersmembers of the students audience.
Introduction and Research Questions

Much of the literature that developed rhetorical (e.g. Ong, 1975; Berkenkotter, 1981) and cognitive (e.g. Bereiter & Scardamalia, 1987; Flower & Hayes, 1981) viewpoints on audience includes an implicit assumption: Real and imagined readers affect writers writing. Such studies have led to consequences in school writing curricula, including pre-writing (e.g. Pressley, 2005), writers workshop (e.g. Atwell, 1998), and authentic pedagogies that connect students with real-world audiences (e.g. Duke, 2010). Similar to Learning Sciences work that aims to link classroom learning with authentic practices and contexts (e.g. Brown, 1992; Shaffer, 2006), these pedagogies posit that writing for readers reinforces literacy as communicative and helps writers plan, draft, and revise. This paper, drawn from a case study of an 11th grade English classroom, analyzes students experiences with one of these pedagogies: writers workshop. I employ bidirectional artifact analysis (Halverson & Magnifico, under revision) to trace students writings through individual drafts, revisions, and feedback from peers and teacher. This work shows how writing development (or lack thereof) was tied to interactions with readersmembers of the students audience. I take up two research questions: (1) In the context of a creative writing community, how do readers impact participants development of their written pieces? (2) What changes and revisions do participants make when they write for and converse with their readers?
Theory
Ideas about audience have shifted as new literacies have pushed writers into a world where they can write and read, as Lunsford & Ede put it, among the audience (2009) (cf. Duke, 2010). Such studies suggest a role for the audience that is both cognitivein that writing for an audience forces writers to consider readersand socialin that sharing work opens up communication (Brandt, 1992; Magnifico, 2010). While cognitiveprocess traditions show that writers consider questions about readers as they work (e.g. Flower & Hayes, 1981), audience members play a literal distributed and dialogic role in collaborative writing (e.g. Bakhtin, 1976/1994). Conversation, social knowledge construction, and perspective negotiation may result from these interactions, (Nystrand, Gamoran, Kachur, & Prendergast, 1997), helping students see writing as a communicative practice.
Methods
This bidirectional artifact analysis (Halverson & Magnifico, under revision) marries elements of narrative analysis (e.g. Labov & Waletzky, 1967/1997), discourse analysis of feedback and conversation (e.g. Wood & Kroger, 2000), and artifact analysis of students written texts. This combination is useful because it attends to narratives key paradox: Good stories are acceptable to readers, so they must be credible. At the same time, they must be reportable; unusual enough to reach beyond known tales. When they responded to each others pieces, the young writers in this study noted literary elements, and, as readers, explained the effects of these choices. While the reportability/credibility paradox was not invoked directly, student critics discussed whether their writings had achieved this balance. Was it a good story or poem? How did the piece achieve goodness? Further, workshop writing relies on writers and readers critique: a comparison of drafts to discuss how revisions contribute to (or detract from) creative works. This analysis follows the logic of the critique process, examining the interplay of students drafts and conversations in a bidirectional way. Instead of examining successive representations, bidirectional analysis reaches backwards and forwards in time, working to understand how new representations are built as revisions of prior iterations (Enyedy, 2005).
Participants, Data, and Setting

This case follows nine students (six female, three male) and their teacher, Mr. Caswell (all names are pseudonyms), through a creative writing workshop in their International Baccalaureate English classroom. Their school, a K-12 college preparatory school, is located near a medium-sized Midwestern city. Observations totaled
ISLS 411
ICLS2012
15 hours over ten weeks, and interactions were recorded and transcribed. Students worked through three cycles of creative writing, drafting alone and then critiquing their work in small groups. In Cycle #1 and #2, students wrote a prose piece and a small poetry collection, revising one of these works in Cycle #3. For all cycles, students also submitted analytical writing detailing how they used literary techniques in their writing. My own role in this space shuttled between observation and participant observation. I observed and recorded field notes when the students were writing quietly, but consulted with students who needed advice as well.
Data Analysis
The data for this analysis are drawn from the students drafts and small-group workshop critiques. A timeline provides the central metaphor for this analysis, and thus, I created timeline representations for each piece of writing, chronologically arranging all of the collected artifacts: students drafts, written feedback, and transcripts of their workshop conversations. From these timelines, I worked backwards and forwards to code how the writings changed over time (Halverson & Magnifico, under revision; Enyedy, 2005): I marked revisions, traced each change (where possible) to a feedback suggestion, and examined uptake in future drafts. The drafts and critiques pictured in Figure 1 represents Noah and Kiras rich, complex critique over time. The goal of this messy representation was to trace how writers revised in response to feedback.
Figure 1. Photographic representation of bidirectional artifact analysis
WRITTEN DRAFT 1
FEEDBACK CONVERSATION
WRITTEN DRAFT 2
Figure 2. Graphical representation of bidirectional artifact analysis The graphical representation (Figure 2) shows the logic of this method. Solid arrows represent critiques that refer back to the text of prior drafts (e.g. Your word choice in the first line is confusing), while dashed arrows represent suggestions that point ahead to potential revisions (e.g. In the next draft, try using more description). In an effective critique, both of these elements are present in the feedback.
Results
The results I present focus on the case of one student, Noah, who was chosen because his work with different partners provides representative examples of both predominant critique patterns in the classroom. Noah wrote five poems over the course of Cycles #2 and #3 and discussed two of these poems extensively in class. He focused on Pitter Patter, a poem that describes a rainstorm, through two conversations with Kira, and After the Battle, a poem that recasts gym class dodgeball as an epic battle, through two conversations with Nasha.
Noah and Nasha: Abstract Critique

Most of the students had little experience with creative writing workshops, and often commented on each others work in a general, abstract way. Noah paired up with Nasha during two class periods to discuss After the Battle. These meetings featured vague criticisms, arguments about words and style, and few direct references to the text. When Nasha challenged his word choice, for example, Noah often argued to justify his ideas:
ISLS
412
ICLS2012
Nasha: I noticed that you... um. You use big words that dont really fit in the context. Noah: Godddd. I like this one. Theres a continuity to this one. Nasha: Your rhyming stinks. Noah: I dont try to. Nasha: If youre going to make this a lyrical poem, and Im assuming thats what it is, is that what is this? Noah: It doesnt have to be, it doesnt have to rhyme... Nasha: Exactly, so dont make it rhyme. Disregard rhyming entirely, because its so awful. (Classroom transcript 4/24/2009.) Nashas troubleNoahs use of big wordssurfaces in many critique conversations about Noahs poetry and invokes poetic genre; for instance, this criticism cites the potential of mak[ing] this a lyrical poem, which Nasha assumes is After the Battles genre. The critique challenges Noah to conform more closely to traditional poetic language, but Nasha remains abstract, providing neither context nor textual references. She never explains which big words dont really fit in the context, or which rhymes are awful. Noah argues, again at the level of genre, that the words have continuity and that lyric poems [dont] have to rhyme. Lacking clear referents in the text, the critique becomes defensive. These characteristics represented much of Noah and Nashas conversation over two class meetings, as demonstrated by the descriptive statistics in Table 1. Table 1: Descriptive Statistics: Noah and Nashas Critique Conversations
Category Total time available Written suggestions Oral suggestions with clear textual antecedents Oral suggestions that refer generally to the poem Total revisions with clear antecedents in suggestions Measure 1.00.33 (two 30-min conversations) 2 6 4 3
Over the course of two meetings (1:00:33 total time), Nasha makes two written suggestionsword substitutions. She provides six oral suggestions with clear referents in After the Battle, and four criticisms that generally refer to the poem. As a result, Noah makes three revisions, two of which are Nashas suggested word substitutions. Given the high-level comments and arguments about genre that characterized much of their conversation, it is possible (although not clear from the data) that Nasha led Noah to consider poetic language more carefully. What is clear is that these abstract discussions led to little textual change over time: Despite Nashas concerns, Noah revises little.
Noah and Kira: Concrete Critique

Some small-group partnerships offered specific context and suggestions for revision. For the duration of Cycle #2, Noah and Kira worked together twice, closely reading each others work. While they did challenge each others interpretations, they often resolved these arguments with textual analysis. Preceding this segment, Kira wrote comments on Noahs poem, showing she did not understand the setting: driving during a downpour: Kira: I wrote a lot of stuff, did you have any questions? Noah: Yeah. Um... Yeah... I guess. Im in a car. Kira: Yeah... like, I get that, I found out that you were in a car... Noah: Down here. Kira: Here. I was like, oh! Wait! Youre IN the car, okay. So maybe like up here, to like create that perspective right from the top, say like of my rubber tires, or like, do something to say that you ARE part of this, youre in this car, not just... because when I read this, I just imagined this, like, a street and a car, like a car going by on a street, like I didnt imagine you were in the car... . Noah: That works too... . Kira: [Then] the car came back, and I got it! ...Um, so yeah, so I wrote that here, perspective, create your perspective from the beginning and that makes the poem easier to sorta picture. (Classroom transcript, 4/1/2009). Kira notes the poems unclear setting and premise, and in response, suggests creating [the] perspective from the beginning. She points to clear consequences, invoking herself as a reader who had to [find] out that you
ISLS
413
ICLS2012
were in [the] car, and who thus misunderstood the context. Kira pinpoints her lack of clarity, noting that Pitter Patter does not mention the speaker until late in the poem. It shows the downpour to readers, but not the speakers perspective on the scene. To remedy this lack of clarity, Kira suggests revising the first stanza to like create that perspective right from the top... do something to say that you ARE part of this. She notes that because she didnt understand this perspective, she failed to understand the poems scopeNoahs experience, not an external description of a scene. While most of Kiras suggestions change few words, they add sensory details, precise language, and Noahs personal perspective to Pitter Patter. In contrast to his collaboration with Nasha (Table 1), his critique conversations with Kira (Table 2) contain explicit suggestions and justifications for revision. As demonstrated in the quotation above, Kira aligns her critiques with Noahs text, pointing out textual elements and the consequences of his choices. Table 2: Descriptive Statistics: Noah and Kiras Critique Conversations
Category Total time available Written suggestions Oral suggestions with clear textual antecedents Oral suggestions that refer generally to the poem Total revisions with clear antecedents in suggestions Measure 1.18.42 (two 38-min conversations) 6 9 0 12
Over the course of their two critique conversations (1:18:42 total time), Kira makes six explicit written suggestions that include word substitutions, image clarifications, and perspective suggestions; nine oral suggestions that have clear antecedents in Pitter Patter, and no general criticisms. As a result of these exchanges, Noah makes 12 revisions, including new or expanded stanzas, several clarified images, and a strengthened first-person perspective. In contrast to his collaboration with Nasha, Noahs conversations with Kira led to many more revisions. Tracing Noahs work shows that while most students made relevant comments (ones that were potentially useful to learning about concepts like genre), not all of these suggestions led to real revision in future drafts. Rather, Noahs responses to critiques present a clear pattern. He revises when a reader shows that his language does not convey his ideasfor example, Kiras notes about Pitter Patters perspectiveand suggests useful changes. In this way, audience response seems critical to Noahs creative and representational processes.
Limitations
Any analysis is necessarily limited by available data. Because bidirectional artifact analysis uses artifacts and transcripts as data sources, it cannot capture the full range of Noahs revisions (some of which are not rooted in the feedback), or individual cognitions. While I cannot claim that these critiques fully account for the development of Noahs poems, tracing the artifacts of and conversations around this work point to a link between feedback structures and revision patterns, as well as pedagogical strategies for creative writing.
Discussion and Significance

Much early cognitive and rhetorical work on audience claims that attention to audience is one hallmark of expert writers (e.g. Berkenkotter, 1981), who ask hypothetical questions about their audience as a planning strategy (e.g. Flower & Hayes, 1981). Extending this claim to literacy learning suggests that learning to write for an audience contributes to writing skill. This paper examines the effects of writing for and interacting with readers and represents an attempt to learn more about how audience carries this weight. In this classroom, a workshop design allows for social and dialogic audience feedback (Bakhtin, 1976/1994), and makes the contributions of students readers more clearly evident. This bidirectional analysis draws concrete links between students feedback and revisions, and thus, broadly maps the affordances of a workshop design in a classroom space: Discussions with real readers proxy for the reactions of outside readers (Lunsford & Ede, 2009) and help learners to see their writing as communicative and meaningful. Classroom writing can become a design experiment that teaches authentic practice (Brown, 1992). Broadly, Mr. Caswells classroom shows that a collaborative workshop, even one housed entirely within a school space apart from the real world, can create a real audience. The most successful groups came to understand that writing is a tool to share meanings, images and ideas. Before Kira pointed out Noahs failure to clarify his perspective, he was likely thinking of his poem as an assignment, not a representation of his experience to share with readers. This analysis shows that Noah was more likely to revise around these perspectival critiques, perhaps because they changed his understanding of the poem itself. These insights about
ISLS
414
ICLS2012
the communicative nature of writing are important and rare in schools, where most traces of writer and reader have been subsumed in right and wrong answers to teachers questions (Nystrand, et. al., 1997). At the same time, the open workshop format became a constraint for many students who did not ask questions, negotiate meanings, or justify reasonable paths for revision self-sufficiently. Examining these conversations in a bidirectional way highlights productive critique moves and shows that some students (Kira) constructed these on their ownand consequently inspired significant revision. Despite similar classroom resources, Nashas feedback was never as concrete or generative as Kiras. This analysis thus suggest that (1) mentors should model revision techniques, and (2) mentors should model critique techniques for students to use (e.g. mirroring outside readers). Examining how critique stretches between and across drafts and students suggests how pedagogy can encourage a productive, revision-focused writing environment.
Audience as a Social and Cognitive Tool

These data show that these young, learning writers individual cognitions only took them so far. Whereas cognitive process theorists (Flower & Hayes, 1981) found that the abstract sense of audienceto whom am I writing?helped experts to begin their work, social contact with real readers aided these students. There is little evidence that these young writers thought about audience before they began to write, but many used the designed practice of getting feedback from readers to identify problem areas, refine ideas, and revise. In short, workshop writing teaches young writers that writing is more than what a writer thinks that she is saying, but what other readers and listeners perceive as well. Developing this understanding motivates an orientation of questioning meaning, revising to feedback, and refining ideas through conversation. Real-world, disciplinary knowledge is built in this way as wellbut these techniques are unavailable in schools when communicative, social paradigms for learning are absent.
References
Atwell, N. (1998). In the middle: New understandings about writing, reading, and learning. Portsmouth, NH: Boynton/Cook Publishers, Inc. Brown, A.L. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. Journal of the Learning Sciences, 2(2), 141-178. Bakhtin, M. (1994). Discourse in life and discourse in art (I. R. Titunik, Trans.). In P. Elbow (Ed.), Landmark essays: Voice and writing (pp. 3-10). Davis, CA: Hermagoras Press. (Reprinted from Freudianism: A Marxist critique, by V. N. Volosinov & N. H. Bruss, Eds., 1976, New York: Academic Press.) Brandt, D. (1992). The cognitive as the social: An ethnomethodological approach to writing process research. Written Communication, 9(3), 315-355. Bereiter, C. & Scardamalia, M. (1987). The psychology of written composition. Hillsdale, NJ: Lawrence Erlbaum. Berkenkotter, C. (1981). Understanding a writer's awareness of audience. College Composition and Communication, 32(4):388-391. Duke, N.K. (2010). The real-world reading and writing U.S. children need. Phi Delta Kappan, 91(5), 68-71. Enyedy, N. (2005). Inventing mapping: Creating cultural forms to solve collective problems. Cognition & Instruction, 23(4), 427-466. Flower, L. & Hayes, J.R. (1981). A cognitive process theory of writing. College Composition and Communication, 32(4), 365-387. Labov, W., & Waletzky, J. (1967/1997). Narrative analysis: Oral versions of personal experience. Journal of Narrative and Life History, 7(14), 338. Lunsford, A.A. & Ede, L. (2009). Among the audience: On audience in an age of new literacies. In M.E. Weiser, B.M. Fehler, & A.M. Gonzales (Eds.) Engaging audience: Writing in an age of new literacies. (pp. 42-69). Magnifico, A.M. (2010). Writing for whom: Cognition, motivation, and a writers audience. Educational Psychologist, 45(3), 167-184. Halverson, E.R. & Magnifico, A.M. (under revision). Bidirectional artifact analysis: A method for analyzing digitally-mediated creative processes. Under revision for Handbook of Design in Educational Technology. Nystrand, M., Gamoran, A., Kachur, R. & Prendergast, C. (Eds.). (1997). Opening dialogue: Understanding the dynamics of language and learning in the English classroom. New York, NY: Teachers College Press. Ong, W. (1975). The writer's audience is always a fiction. PMLA, 90(1), 9-21. Pressley, M. (2005). Reading instruction that works: The case for balanced teaching (3rd ed.). New York, NY: Guilford Publications. Shaffer, D. W. (2006). How computer games help children learn. New York, NY: Palgrave Macmillian. Wood, L.A., and Kroger, R.O. (2000). Doing discourse analysis: Methods for studying action in talk and text. Thousand Oaks, CA: Sage Publications.
ISLS
415
ICLS2012
The use of text and process mining techniques to study the impact of feedback on students writing processes
Rafael A. Calvo(1), Anindito Aditomo(1,2), Vilaythong Southavilay(1), Kalina Yacef(1), University of Sydney, Australia, Email: rafael.calvo@sydney.edu.au, aadi4954@uni.sydney.edu.au, vstoto@gmail.com, kalina.yacef@sydney.edu.au
(1) (2)
The University of Sydney, Australia, The University of Surabaya, Indonesia
Abstract: Understanding the impact of feedback in complex learning activities, such as writing, is challenging. We contribute a combination of writing environments and data and process mining tools that can provide new ways of measuring this impact. We use the tools in a field experiment in an engineering course (N=45). Responses (timing, amount and types of text changes) were examined using log data and process mining techniques. Two experimental conditions were used: reflective followed by directive feedback (A) and vice-versa (B). We found that both forms of feedback were read multiple times. Students required longer times to respond to reflective, compared to directive, feedback. The type of feedback, however, made little difference to the types of revisions that students performed. Overall, our findings point to the difficulty of encouraging students to reconsider and revise what they have already written.
Introduction
Writing is considered to be a critical form of learning activity at all educational levels. Writing is also a particularly complex activity, and it is generally believed that both novice and experienced writers benefit from feedback provided by others in order to improve their writing. This paper presents a new approach to study the impact of feedback and findings from a field trial in engineering education. This paper examines the impact of different types of writing feedback (directive vs. reflective) on students writing process, including the types of revisions students make to their document. Feedback can be defined as information provided to a person about his/her performance in a task. In educational contexts, the provision of feedback is intended to increase not only a students performance, but also the likelihood of learning from the task. Intuitively, feedback should almost always improve learning and performance. Research has shown that the relationship between feedback, performance and learning is nothing but simple (Hattie & Timperley, 2007; Shute, 2008). A meta-analysis found that while feedback improved performance on average, there was a large variation in the effect sizes, and in a third of the studies feedback had a negative impact (Kluger & DeNisi, 1996). To explore possible moderators, Kluger and DeNisi formulated a theory called Feedback Intervention Theory (FIT). FIT draws from control theory and cybernetics to state that feedback interventions cause a person to compare the feedback with a standard or goal. Perceived discrepancies between the feedback and the standard will motivate the person to reduce the discrepancy. FIT provides a framework to predict the influence of different types of feedback on learning. For instance, feedback with criticism (or praise) towards the learner would divert attention from task relevant processes and can impede learning. Similarly, feedback that highlights ones performance compared to others (normative feedback, such as grades) would also impede learning. In contrast, feedback that directs attention to the task should facilitate learning, especially if it contains information needed to address the problem highlighted in the feedback (Kluger & DeNisi, 1996, pp. 267-268). Feedback that includes cues about the goal or standard of the task outcome (goal-setting interventions) should also increase performance and learning. Writing, the particular task that we are concerned with here, is much more complex than the typical tasks used in feedback research. Nonetheless, there are some findings consistent with FIT. For instance, FIT would predict that without distinguishing different types of feedback, the effect of writing feedback might be negligible. In line with this prediction, a meta-analysis which lumped together feedback of various kinds found that feedback did not significantly increase the effectiveness of writing interventions on learning (BangertDrowns, Hurley, & Wilkinson, 2004, p. 47). But when types of feedback are distinguished, FIT would predict that their impact would differ. Not many studies have examined this issue, but there is some supporting empirical evidence. One study by Nelson and Schunn (2009) examined correlations between features of peer feedback and the likelihood of the feedback being implemented. The writing task was an essay in an undergraduate, introductory course on history. This study found that task-focused feedback (such as those that included specific solutions or specific location of problems) predicted implementation, whereas feedback that focused on the writer (those with affective language such as praise and criticism) did not.
ISLS
416
ICLS2012
To conjecture further about the role of feedback in writing, we draw from a cognitive model of writing proposed by Bereiter and Scardamalia (1987). They proposed that writing could occur in two different modes: knowledge telling and knowledge transforming. In knowledge telling, the composition process begins with the writer picking up topic and genre cues from the task description, and writing down knowledge from memory activated by these cues. Text already produced becomes an additional source of cues to retrieve knowledge from memory. This process of memory retrieval cued by the task description and text already produced is repeated until the writer feels he/she no longer has any relevant knowledge (or until time or space constraints are met). The knowledge transforming mode of writing is more complex. It involves the construction and continual reconstruction of a content problem space (what to say) and a rhetorical problem space (how to say it) (Bereiter & Scardamalia, 1987). The two problem spaces interact, with output from one feeding into the other. For instance, a writer in this mode would think about whether the produced text, in its current form (a rhetorical issue), conveys what they intend to say (a content issue). The writer would also think about whether others, or she herself, believe what the text is saying. This may change the way she thinks about the topic, which in turn may prompt her to find a different way to express her new view. How does writing feedback come into play? Feedback that points to the writing task (as opposed to the writer) has the potential to prompt a writer to reconsider what she/he has written and/or how it was written. In other words, feedback could prompt processes associated with the knowledge-transforming mode of writing, as reflected in more revisions that go beyond cosmetic text changes and also in the greater time lag between feedback and first revision. However, we conjecture that certain types of (task focused) feedback may be more effective than other types at prompting knowledge transforming processes. For instance, feedback that contains specific instructions (what we call here directive feedback) may prompt a writer simply to correct the specific problems, without much consideration. In contrast, feedback that asks students to connect problems in their text with broader content or stylistic issues (what we call here reflective feedback) may prompt more substantial revisions. In this study, we examine two approaches for giving task-focused feedback: directive and reflective. Directive feedback tells the student that there is a problem in the text that needs addressing. Reflective feedback asks the student to consider whether there is a problem.
Method
Research questions and hypotheses
The main question addressed in this study is: What are the different impacts of reflective and directive feedback on students writing process in terms of the timing, amount and types of text changes performed? As mentioned above, directive feedback explicitly informs a student that their text contains a specific problem, and instructs them to address the problem. Reflective feedback merely suggests the possibility of a problem, but asks the student to decide whether there is a problem that needs to be addressed (see examples in Table 2). We conjecture that reflective feedback is more effective than directive feedback in prompting students to reconsider their ideas and revise what they have written (processes consistent with a knowledge-transforming mode of writing). More specifically, we predicted that reflective feedback would prompt more deletion of words. Reflective feedback would not necessarily prompt more addition of words, because adding words (expanding a text) can be performed without much reconsideration of text already produced.
Study design and participants

The participants were (N=45) undergraduate and postgraduate engineering students from The University of Sydney. All participants were enrolled in a project-based course where the main activity was to develop a web application. Students had to individually write a proposal that would become the basis of their software development project. The semester was 13 weeks long; the writing activity occurred in the first half of the semester and spanned 32 days. Table 1. Timeline of the writing assignment Day 0 Day 10 Day 12 Group A Directive Assign1st draft (n=22) feedback sent ment submitt Group B Reflective opened ed (n=23) feedback sent Day 17 2nd draft submitted Day 19 Reflective feedback sent Directive feedback sent Day 27 Peer feedback sent Day 32 Final deadline
In this context, we conducted a crossover field experiment where students were randomly assigned to receive either reflective (n=22) or directive (n=23) tutor feedback in response to the first drafts of their assignment. The first draft was worth 3% of the course mark. Students then had five days to revise before submitting their second draft, which was not assessed. Following this, students who initially received reflective feedback were provided with the directive feedback for their second draft (and vice versa). The crossover setup
ISLS
417
ICLS2012
helped satisfying ethical requirements by ensuring that students were not systematically disadvantaged by the type of tutor feedback. In addition, after the second tutor feedback, the students had to review their peers second drafts. The final submission was worth 5% of the course mark.
Tools and measurement The writing environment

Writing activities were managed through iWrite (Calvo, ORourke, Jones, Yacef, & Reimann, 2011). Students write on Google Docs, a cloud-based application, and revisions of the documents are retrieved using Googles API. Tools and heuristics developed to automatically recognize collaborative writing activities (Southavilay, Yacef, & Rafael A Calvo, 2010) was used here to explore what students did after reading the tutors feedback.
Tutor feedback
The tutor feedback was prepared by the three instructors of the course (who each generated feedback for roughly 1/3 of the students). Feedback was written within the reviewers page in iWrite. In order to better align the feedback provided by the three instructors, feedback samples were kept in a shared document. An email announcing that feedback was available at a certain webpage was sent to all students simultaneously. Students were then able to access the feedback on the same interface. Between 3 and 6 feedback items were provided to each student in each phase (M=4.4, SD=1.12). Reflective feedback items had M=96.52 (SD=26.10) number of words at the first release and M=86.86 (SD=23.44) number of words at the second release. Directive feedback items had M=63.77 (SD=16.78) at the first release and M=58.47 (SD=20.66) at the second one. Table 2. Examples of reflective and directive feedback for the same problem. Reflective feedback Please think whether including a list of users needs, features and design would enhance the clarity of your individual module. Sometimes spelling mistakes can distract readers from the gist of your argument or message. You may want to check any spelling errors in your document. Directive feedback Write a new description of the users needs, they are not clear. Fix the following Spelling problems: [list of problems]
Data analysis Log preparation

While users write, Google Docs (GD) saves the document frequently and stores all its revisions. This feature provides a history of the document, where each version of the document can be retrieved with its timestamp. All major revisions available for the 45 documents (an average of N=7.76, SD=4.33 per document) were used. The content of the revisions was used for detecting the type of text change operations. In process mining terms, the history each document is a process instance (case), In addition, based on the log file of iWrite we used a log file containing all the students interactions with the website, with a timestamp each time they accessed the feedback page. This log revealed when and how often individual students access and read their feedback. The feedback-reading log also has one process for each document (student), in total of 45 processes. The revision and feedback-reading logs were merged for each student. A process mining tool, ProM (ProM., 2010) was used to analyze how students accessed feedback and revised their documents. A Dot Chart Analysis (Song & van der Aalst, 2007) was implemented as a plug-in in ProM to extract a snapshot of student activities of revising their documents and accessing the feedback. The result is shown in Figure 1.
Detecting text change operations

The model developed by Boiarsky (Boiarsky, 1984) was used to analyze semantic changes in the writing process. In particular, the 12 types of text change operations (Southavilay, Yacef, & Rafael A Calvo, 2010) shown in Table 3 were used. Text change operations at both paragraph and document levels were considered. At the paragraph level, beyond surface changes, the other types of text changes are listed in Table 3. A text comparison utility was used to identify text change operations (Southavilay, Yacef, & Rafael A Calvo, 2010). The utility is based on the Unix Diff utility, which takes two revisions of text and produces a difference statement with insertions, deletion, and replacements between the two. The text comparison utility uses both paragraph and word-differencing algorithms to detect text changes operations at paragraph and word levels. For each document each two consecutive revisions were compared and the differences computed. First,
ISLS
418
ICLS2012
the utility uses the paragraph differencing algorithms to discover the addition of new paragraphs, the deletion of existing paragraphs, and the alteration of existing paragraphs. Based on the paragraph-differencing algorithm, we detect text change operations of adding, deleting, moving/reordering, merging, and splitting of existing paragraphs. For each altered paragraph in the later revision, the utility then uses the word differencing algorithm to compare it to the corresponding paragraph in the former revision in order to detect text change operations of moving/reordering, replacing, inserting, deleting, and appending words in the altered paragraphs. Table 3. Types of text changes automatically identified Text structure Content change within individual paragraphs (i.e. word level changes) Moving/reordering Moving or reordering words in a paragraph paragraphs Replacing words in a paragraph Merging paragraphs Inserting words in the middle of paragraphs Splitting paragraphs. Deleting words in a paragraph Adding words at the end of a paragraph Content change at the paragraph level Adding new paragraphs Deleting paragraphs
Findings and discussion

The dotted chart (Figure 1) shows the timing of each students access to the tutor feedback, and also the timing of revisions performed in Google Docs. Each row represents a process of one student revising his or her document and accessing the tutor feedback. One column (in between white vertical lines) represents a period of one week. Directive feedback is represented by triangles, reflective feedback by squares, and revisions by dots. Group A (who received directive feedback first) occupies the upper half of the chart.
Figure 1. Dot chart showing when each student accessed directive feedback (red triangles) and reflective feedback (blue squares), and also instances of revisions (green dots). We refer to the period between first and second tutor feedback as Phase 1, and the period between the second feedback and final submission as Phase 2. The black line represents a date when the first tutor feedback sent and the brown line represents a date when the second feedback sent. Several observations can be made from Figure 1. While the feedbacks were sent at the same time, students accessed them at different times (most accessed the feedback on the same day they received it, but some took one or more days). Also, almost all students accessed the feedback multiple times (except for one student who accessed the feedback only once). On average students in Group A accessed the directive feedback 5.8 times and the reflective feedback 4.6 times. In Group B, on average students accessed the reflective feedback 6.3 times, and the directive feedback 4.9 times. Students in both groups revisited the first tutor feedback during Phase 2 (i.e. after receiving their second feedback), as can be seen from the mix of red triangle and blue squares on the dot chart Hence, students behaviors in Phase 2 need to be seen as potentially influenced by both types of tutor feedback. An examination of the timing the changes shows that students took longer to respond to reflective feedback than to directive feedback (1 day longer in Phase 1, and 2 days longer in Phase 2). While not predicted theoretically, this difference is understandable, as reflective feedback requires students to think for themselves, as opposed to being a direct instruction. In order to see how students revised their documents according to feedback types, we analyzed four types of revision types (adding paragraphs, deleting paragraphs, adding words and deleting words in existing paragraphs) and the corresponding number of words added and deleted. For each phase, we compared the
ISLS
419
ICLS2012
number of words added and deleted in the four types of revision types in the two experimental conditions. In order to distinguish between headings (not considered in the analysis) and content paragraphs, we defined a paragraph as containing at least 7 words. This resulted in several findings. First, not many students performed major revisions, especially in Phase 1. For instance, in Phase 1, only 13 students from Group A and 12 from Group B added new paragraphs, and only 10 from Group A and 9 from Group B added words to existing paragraphs. Even fewer students revised their documents by deleting words: in Phase 1, only 4 students from Group A and 5 from Group B deleted paragraphs, and only 8 from Group A and 3 from Group B deleted words in existing paragraphs. In other words, most of the students in both groups performed only minor revisions in Phase 1. In Phase 2, more students did revise their documents. They mainly did so by adding paragraphs and by adding words to existing paragraphs, rather than deleting paragraphs or deleting words within paragraphs. Over the two phases, only about half of the students (in both groups) performed revisions by deleting words. This indicates that only about half of the students reconsidered and revised what they have already written (which are indicators of knowledge transforming writing processes). A second set of observations are related to the differences, or lack thereof, between Groups A and B. Although in terms of average number words (see Table 4) Group B seemed to have performed more additions and deletions (in both phases), the box plots above show that there is more variation within rather than between the groups. That is, the difference in the average seemed to be inflated by several students who performed much more extensive revisions than most other students. This difference of amount of revision between the groups largely disappeared if we consider the median (as shown in the box plots). Groups A and B are also similar in terms of the number of students performing the three of the four types of revisions shown in the box plots. There seems to be a difference between the groups in Phase 1, in terms of the deletion of words in existing paragraphs: 8 students in Group A, compared to 3 in Group B (although these 3 students made more extensive deletions compared to the 8 students from Group A). Together, these observations suggest that the feedback (both reflective and directive) failed to prompt major revisions for most of the students. This is an important pedagogical point: that students have the liberty to take into account, or not, their tutors' feedback. Most students did not revise extensively, despite the feedback given, the time students had to address the feedback, and also the fact that students accessed the feedback several times. Furthermore, the observations indicate that our theoretical predictions about the impact of different feedback types were not supported by the data. This could have been due to several reasons which are more to do with the methods, rather than the hypotheses themselves. One possible explanation is that not all of the reflective feedback was of the same quality: an initial inspection of the reflective feedback items indicated that some items were more directive (instructing students to do specific things). Another possible reason for the lack of difference between the groups revision behavior is that too few students engaged in extensive revisions during Phase 1, mainly because the students had little incentive to perform those revisions (as indicated above, students behavior in Phase 2 cannot be taken as indicators of the influence of different feedback types).
References
Bangert-Drowns, R. L., Hurley, M. M., & Wilkinson, B. (2004). The effects of school-based writing-to-learn interventions on academic achievement: a meta-analysis. Review of Educational Research, 74(1), 2958. Bereiter, C., & Scardamalia, M. (1987). The psychology of written composition. Mahwah, NJ: Lawrence Erlbaum Publishers. Boiarsky, C. (1984). Model for Analyzing Revision. Journal of Advanced Composition, 5, 65-78. Calvo, R. A., ORourke, S. T., Jones, J., Yacef, K., & Reimann, P. (2011). Collaborative Writing Support Tools on the Cloud. IEEE Transactions on Learning Technologies, 4(1), 88-97. Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81-112. Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: a historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119(2), 254284. Nelson, M. M., & Schunn, C. D. (2009). The nature of feedback: how different types of peer feedback affect writing performance. Instructional Science, 37, 375-401. ProM. (2010). http://prom.win.tue.nl/tools/prom/. Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153-189. Song, M., & van der Aalst, W. M. P. (2007). Supporting Process Mining by Showing Events at a Glance. Paper presented at the 7th Annual Workshop on Information Technologies and Systems.
Acknowledgments
This project has been funded by Australian Research Council DP0986873 and a Google Research Award.
ISLS
420
ICLS2012
Variation in Fifth Grade Students Propensities for Managing Uncertainty during Collaborative Engineering Projects
Michelle E. Jordan, Arizona State University, Mary Lou Fulton Teachers College, Farmer Education Building 1050 S Forest Mall, Tempe, AZ 852871811, michelle.e.jordan@asu.edu Abstract: Uncertainty is ubiquitous to learning, particularly in collaborative learning contexts, which are likely to induce uncertainty related to both social and task issues. The purpose of this qualitative study was to explore how students vary in their individual propensities for managing uncertainty they experience while working with peers. Using observations and interviews I examined how students in one fifth grade class managed uncertainty while collaborating on robotics engineering projects. Relying on techniques of grounded theory and microanalysis of discourse, I addressed the question: How do students vary in their individual propensities for managing uncertainty they experience during collaborative academic tasks? In particular, I examined variation in the size and composition of the set of strategies from which students drew, changes in students use of strategies across projects, and their willingness to entertain or take up uncertainty.
Introduction
Learning in academic contexts involves experiencing psychological uncertainty, and the ways students manage their uncertainty influences their behavior and their learning in collaborative contexts (Huber, Sorrentino, Davidson, Epplier, & Roth, 1992). Although learning is often portrayed as a process of reducing uncertainty, it is long recognized that learning may also require the cultivation of uncertainty (Bereiter & Scardamalia, 1993; Piaget, 1972; Sieber, 1974). Furthermore, when academic tasks entail collaborating with peers, uncertainty may be particularly prevalent as it is likely to stem from diverse social and task issues. If uncertainty is a pervasive experience in collaborative learning contexts, it is important for educators and educational researchers to understand learners responses to this experience in order to develop strategies for helping students increase their skills at effectively managing the uncertainty they experience while working with peers. Educational philosophers the likes of Dewey (1929) and Bruner (1996) have extolled the place of psychological uncertainty in learning; however, the experience of uncertainty has most often been discussed tangentially to concepts of more focal interest to educational researchers. Wishing to focus directly on how students manage uncertainty during collaborative academic tasks, I conducted a qualitative study relying on observations and interviews with fifth grade students collaborating in small groups to design and build robots. Specifically, I addressed the question, how do students vary in their propensities for managing uncertainty during collaborative tasks?
Uncertainty is an individuals psychological experience of doubting, being unsure, or wondering about how the future will unfold, what the present means, or how to interpret the past. Uncertainty can pertain to ones self, other individuals, or other aspects of the environment. Rather than a coldly cognitive phenomenon, uncertainty is a cognitive feeling (Clore, 1992) that can be more or less conscious and more or less tied to emotions. Uncertainty management is behavior students engage in to facilitate action in the face of uncertainty. Although uncertainty management is often conceptualized as entailing tactics to reduce uncertainty, it also pertains to efforts aimed at ignoring, maintaining, or intentionally increasing uncertainty (Jordan, 2010; Babrow & Matthias, 2009). Which strategies might be appropriate in a given situation depends on multiple factors, including the social context and social norms for managing uncertainty that have been established in a particular community of practice (Goldsmith, 2001; Lingard, Garwood, Schryer, & Spafford, 2003). Self-report scales and experimental methods to measure individual orientations to various forms of uncertainty can be traced from the mid-20th century (e.g., Budner, 1962; Debacker & Crowson, 2006). Such approaches may not reflect the complicated nature of uncertainty management as observed by teachers and as experienced by students working with peers. Therefore, I used a methodological approach that allowed me to observe the rich nature of uncertainty as it is experienced and managed by students in collaborative contexts. Even young students can recognize their uncertainty (Metz, 2004), but they often fail to experience uncertainty when uncertainty is warranted (Sieber, 1974). The amount of uncertainty students experience during academic tasks can be influenced by task characteristics such as evaluation systems (Doyle & Carter, 1984), novelty (Herbst, 2003), task framing (Schauble, Klopfer, & Raghavan, 1991) and discourse (Hierbert et al., 1996). Therefore, I chose to study uncertainty in a collaborative design setting, open and ambiguous, illstructured and generative (Jonassen, 2000; Kolodner et al., 2003).
ISLS
421
ICLS2012
Method
The study was initiated at the beginning of instruction in robotics engineering and preceded through the completion of three collaborative projects (14 sessions each); two were more-structured and close-ended and one was an ill-structured task (Spiro, Feltovich, Jacobson, & Coulson, 1991). All projects utilized LEGO Mindstorms materials. Students worked in groups of three or four, changing group membership for each project. Participants were 24 fifth graders in a public suburban classroom. The study concentrated on 15 focal students purposefully selected for diversity of gender (eight female), ethnicity (seven Black, four White, two Hispanic, two Asian), and academic achievement (five received special education services; two gifted-and-talented). I conducted naturalistic observations (1-2 hours each) and interviews. Data sources include expanded field notes, audio and video recordings, transcripts for 12 groups across three projects, and interview transcripts. Data analysis was inductive and interpretive, relying on techniques of grounded theory (Corbin & Strauss, 2008) and influenced by Ericksons (1992) ethnographic microanalysis of discourse and interactional sociolinguistics (Mercer, 2000). Member checking with the teacher and peer debriefing occurred throughout the study. A two-pronged strategy was used to identify individual propensities. First, drawing on knowledge of linguistic, paralinguistic, and gestural markers of uncertainty (e.g., Feldman & Wertsch, 1976; McNeill, 1992), I conducted open coding of uncertainty experiences and management using transcripts of small group sessions and student interviews randomly selected from projects 1 and 3. Using axial coding to refine categories, I identified 20 management strategies and categorized them as tactics for reducing, ignoring, maintaining, or increasing uncertainty (see Table 1). Re-reading the data and focusing on individuals, I determined which strategies dominated each focal students approach to uncertainty. I then organized these data in a strategy-bystudent matrix to identify students whose overall patterns were similar. Table 1: Strategies and Categories for Managing Uncertainty. Reduce Analyze issues Test systematically Trial-and-error Explain clearly Seek consensus Observe others Seek expert other Ask for confirmation Request info. Draw on experience Refer to authority figure Seek information from materials or texts Ignore Avoid Blame Pass off task Keep going (persist, bluff) Dismiss (do not consider introduced uncertainty) Maintain Delay action, decision, or evaluation Acknowledge Express doubt Increase Open the problem space
In a separate analysis, I created mini-portraits of fifteen focal students (Do & Schallert, 2004), for whom I had data from at least two robotics projects (the exception was Nathan, for whom I had data from only one project). Creating case studies in NVIVO, I first selected five to ten episodes from small-group sessions in which the focal student played a major part in group interaction. I conducted microanalysis of transcripts of each episode, examining uncertainty (and certainty) and uncertainty management and comparing with field notes and recordings. I read through all interview transcripts with each focal student multiple times and integrated information across data sources, making extensive memos and tuning the picture of students propensities. I then compared these portraits with the matrix to determine consistency between the two analyses.
Findings
Of the 20 management strategies identified, 16 were used by all 15 focal students at least once. Despite the fact that all students used a majority of the strategies, I was able to identify for each student his/her propensity for responding to uncertainty. There were five propensities in all (see Table 2). Table 2: Characteristic Patterns of Uncertainty Management. 1 Pause for Reflection Demetre Berta 2 Seek a Plausible Explanation Satya Ray Isabel 3 Take Action 4 Can Somebody Help Me? Alexis Sierra Shamitra 5 What, Me Uncertain? Bobby Trevor Luis
Kisha Becky
Derrick Nathan
Pause to reflect. Students in this category accepted uncertainty as a normal and inevitable aspect of academic tasks. They had little reticence about expressing uncertainty, recognizing and acknowledging it openly. They exhibited a desire to collect more information and reflect prior to action, often using strategies
ISLS
422
ICLS2012
categorized as maintaining uncertainty. They asked for explanations of members ideas. Rather than address and resolve an uncertainty only in relation to a present situation, they reflected on its greater meaning. Seek a plausible explanation. These students expected uncertainty and did not seem threatened by it. Their uncertainty management was instrumental, focused on achieving the task. They were proactive in using group members, seeking confirmation for potential actions and comparing perspectives to create plausible explanations. They tended to interpret outcomes of action in a positive light, thus sustaining task focus. They planned for and took tentative action and observed and analyzed the outcome of each step. Take action. Students with this set of characteristics were gung-ho, forward-oriented, and impatient with uncertainty. They oscillated between managing uncertainty by risk-taking/trial-and-error experimentation and dismissing uncertainty by making positive predictions. They rarely consulted about their own or their group members decisions. When pressed for explanations, they often replied, Ms. Katell said... Eager to experiment and see what happened, these students rarely let analysis stand in the way of the next test, either prior to or subsequent to action. Rare reflective actions were usually based on immediate past experience. Can somebody help me? These students preferred to fly under the radar. They were willing to acknowledge uncertainty but tried to avoid it. They frequently turned tasks over to group members, relied on outside expertise, and requested scaffolding from their group members for their project participation. These students requested help with immediate uncertainty, rarely considering the long-range use of that help. What, me uncertain? Unique in their dogged denial of uncertainty, these students minimized or avoided acknowledgment of uncertainty. They were quick to blame their peer collaborators rather than puzzle over a problem. They made confident claims, (e.g., I know what to do) rather than mitigate their claims with uncertainty markers. They prepared for failure by expressing certainty, for instance, Its not going to work. One way to better understand variation is to examine the interaction of students with different propensities. There was always tension between action and reflection as students worked toward joint academic goals, a tension that impacted learning and project success. The ways students experienced and managed uncertainty played a large role in how this tension played out. This was the case with Nathan and Demetre, group members for the first robotics project. Demetres propensity for managing uncertainty was to pause for reflection. He often used analogies in his attempts to reduce uncertainty as represented in his response to my inquiry about his groups problems programming the obstacle course. We need to have a little more figuring, study more, get more knowledge if its possible to do that. Its possible to measure distance on kilometers and all that. The only problem is this is a rotation of how far the robot will move. But it is possible to measure, like, the rotations of the earth. So I started thinking on my own, Is it possible for the wheel to be like the earth, to be spinning around in a rotation? Demetres response to his uncertainty about programming his groups robot was to collect more information and reflect on similar problems before taking action. Contrasted with Demetres need for reflection is Nathans propensity to take action. Nathan operated from a model of risk-taking in the face of uncertainty. We did this new program that actually worked, but at first I was a little bit unsure that it would work. And it ended up coming through for us when it hit the wall... W ell since I wasn't very sure, testing stuff, well, that program was kind of strange. But a scientist needs to take his risks... Sometimes when people do experiments and they're not sure, they take a risk and most of the time it comes through. Nathan managed uncertainty by mimicking one way he saw other scientists manage uncertainty. He attributed his groups success not to knowing what to do and doing the right thing, but to their risk-taking. These contrasting strategies for managing uncertainty caused friction throughout the project, with Nathan wanting to take action and Demetre pushing for reflection. There was little explanation or discussion to create sense together that might have helped these students resolve uncertainty, learn, and progress with the project. I also organized the data in terms of whether students used many/few strategies and whether they did/did not easily acknowledge uncertainty (see Table 3). Table 3: Range of Strategies and Recognition of Uncertainty Many Strategies Satya Berta Ray Isabel Demetre Few Strategies Alexis Sierra Shamitra Derrick Trevor Becky Kisha Bobby Luis Nathan
Easily Acknowledged Uncertainty Did Not Easily Acknowledge Uncertainty
ISLS
423
ICLS2012
The reader will note that Cell 3 is empty. Note also that students with propensities 1 and 2 used many strategies for managing uncertainty. Students with propensities 3, 4, and 5 used few strategies. Those in Cell 2 not only made frequent use of many strategies, but also tended to draw from a diverse range of strategies that included ignoring, reducing, and maintaining uncertainty. Students with propensities 1, 2, and 4 tended not to acknowledge uncertainty, whereas students with propensities 3 and 5 did. For example, Satyas willingness to take up uncertainty was exhibited when she responded to an announcement that Kisha and Becky had dismantled the trailer the group had labored on for days, intending to make a robo-claw instead. Satya was visibly upset by the decision her group members made without her. Yet, even while being thrown off-kilter by the change of design, Satya rather quickly surrendered her commitment to her past work, distracted by the new set of problems. Satya: Kisha: Satya: Kisha: Satya: But is it going to be able to move, go like this and then go like this? I dont know. But if it doesn't do that we just// //Oh, yeah, it has a motor. Yeah, it should, yeah. No, it needs two motors then, right? It needs to be able to move; go like this and then move this way like this. Kisha: It should be. Satya: You're sure, 100% sure? Kisha: I'm 90% sure. Satya: Well then* Satya: I'm going to cry. Wait, how many motors does it need?
Satyas response could have been to be quite certain that the new idea would not work. Instead, she was willing and able to entertain uncertainty about the situation in which she found herself.
Significance
Students brought with them to the collaborative engineering setting tendencies for managing uncertainty that had developed through experience, their habits and histories of participation in prior groups. The students all used a range of strategies, but at the same time they had individually identifiable patterns of uncertainty management. Some students had many ways to manage uncertainty; some students had only a few ways. Some students easily acknowledged uncertainty; others seemed to resist taking up uncertainty when uncertainty seemed warranted to one or more of their group members, to their teacher, or to me as an observer. Although the number of participants from this study is too small to make generalizations, it is interesting that the three students in the category Can somebody help me? were female and that all three student in the category What, me uncertain? were male. Since previous research has noted that the expression of uncertainty can vary by gender and socioeconomic status (Gee, 1996), future studies need to explore the association between these characteristics and uncertainty propensities. Future research should also examine how propensities for managing uncertainty in collaborative environments are related to learning and achievement. Much of the research on uncertainty management has conceived of it as an individual level of analysis; however, some see uncertainty management as a social issue (Gill & Babrow, 2007). A majority of activities students are likely to face as adults are collaborative tasks, and most of the learning they do will be of a social nature (Bruner, 1981). For most individuals most of the time, the primary resource they have is each other, whether in face-to-face or virtual interaction; thus, how one manages uncertainty in a collaborative context is important. Given the central position of relationships in determining how social systems emerge and unfold (Arrow, McGrath & Berdahl, 2000), it is likely that the nature of interdependencies and interactions within a collaborative peer group will influence the ways individuals in that group manage uncertainty (Jordan & McDaniel, under review) and that the propensities of individual group members will likewise influence the quality of interactions in the group. The study described here contributes to understanding of these issues. If uncertainty is required for learning and is a ubiquitous experience in collaborative learning contexts, then it behooves educational researchers to understand how students manage uncertainty and ways in which the interaction among students with different propensities influences the quality of peer collaboration. Doing so could allow us to identify instructional strategies for helping students increase their range of strategies, willingness to acknowledge uncertainty, and abilities to recognize when different strategies are appropriate.
References
Arrow, H., McGrath, J. E., & Berdahl, J. L. (2000). Small groups as complex systems: Formation, coordination, development, and adaptation. Thousand Oaks, CA: Sage Publications.
ISLS
424
ICLS2012
Babrow, A. S., & Matthias, M. S. (2009). Generally unseen challenges in uncertainty management: An application of problematic integration theory. In T. Afifi &. W. Afifi (Eds.), Uncertainty, information management, and disclosure decisions: Theories and applications (pp. 9-25). London: Routledge. Bereiter, C., & Scardamalia, M. (1993). Chapter four: Expertise as process. In Surpassing ourselves: An inquiry into the nature and implications of expertise (pp. 77-120). Chicago: Open Court. Bruner, J. (1996). The culture of education. MA: Harvard University Press. Bruner, J. (1981). The pragmatics of language acquisition. In W. Deutsch (Ed.), The childs construction of language (pp. 39-55). New York: Academic Press. Budner, S. (1962). Intolerance of ambiguity as a personality variable. Journal of Personality, 31, 29-50. Corbin, J. M. & Strauss, A. L. (2008). Basics of qualitative research. UK: Sage. Clore, G. L. (1992). Cognitive phenomenology: The role of feelings in the construction of social judgment. In A. Tesser & L. L. Martin (Eds.), The construction of social judgments (pp. 133-164). Hillsdale, N.J.: Erlbaum. DeBacker, T. K., & Crowson, H. M. (2006). Measuring need for closure in classroom learners, Contemporary Educational Psychology 3(4), 711-732. Dewey, J. (1929). The quest for certainty. New York: Minton, Balch, & Company. Doyle, W., & Carter, K. (1984). Academic tasks in classrooms. Curriculum Inquiry, 14, 129-149. Erickson, F. (1992). Ethnographic microanalysis of interaction. In M. D. LeCompte, W. L. Millroy, and J. Preissle (Eds.), The handbook of qualitative research in education (pp. 202-224). San Diego, CA: Academic Press. Erickson, F. (2004). Talk and social theory. Malden, MA: Polity Press. Feldman, C. F. & Wertsch, J. V., (1976). Context dependent properties of teachers speech. Youth & Society, 7 (3), 227-258. Gill, E. A. & Babrow, A. S. (2007). To hope or to know: Coping with uncertainty and ambivalence in womens magazine breast cancer articles. Journal of Applied Communication Research, 35(2), 133-155. Gee, J. (1996). Social linguistics and literacies (2nd edition). London: Routledge Farmer. Goldsmith, D. J. (2001). A normative approach to the study of uncertainty and communication. Journal of Communication, 51, 514-533. Herbst, P. G. (2003). Using novel tasks in teaching mathematics: Three tensions affecting the work of the teacher. American Educational Research Journal, 40(1), 197-238. Hiebert, J., Carpenter, T. P., Fennema, E., Fuson, K., Human, P., Muray, H., Olivier, A., & Wearne, D. (1996). Problem solving as a basis for reform in curriculum and instruction: The case of mathematics. Educational Researcher, 25(4), 12 -21. Huber, G. L., Sorrentino, R. M., Davidson, M., Eppler, R., & Roth, J. W. H. (1992). Uncertainty orientation and cooperative learning: Individual differences within and across cultures. Learning and Individual Differences, 4, 1-24. Jonassen, D. H. (2000). Towards a design theory of problem solving. Educational Technology, Research and Development, 48(4), 63-85. Jordan, M. E. (2010). Collaborative robotics design projects: Managing uncertainty in multimodal literacy practice. Yearbook of the National Reading Conference, 59, pp. 260-275. Jordan, M. E. & McDaniel, R. (under review). Peer Influence on uncertainty management in collaborative robotics engineering projects: Lessons from a fifth grade class. Kolodner, J. L., Camp, P. J., Crismond, D., Fasse, B., Gray, J., Holbrook, J., et al. (2003). Problem-based learning meets case-based reasoning in the middle-school science classroom: Putting learning-bydesign into practice. Journal of the Learning Sciences, 12(4), 495-547. Lingard, L, Garwood, K, Schryer, C. F., & Spafford, M. M. (2003). A certain art of uncertainty: Case presentation and the development of professional identity. Social Science & Medicine, 56, 603-616. McNeill, D. (1992). Hand and mind: What gestures reveal about thought. Chicago: University of Chicago Press. Metz, K. (2004). Childrens understanding of scientific inquiry: Their conceptualizations of uncertainty in investigations of their own design. Cognition and Instruction, 22(2), 219-290. Mercer, N. (2000). Words and Minds: How we use language to think together. New York: Routledge. Piaget, J. (1972). Intellectual evolution from adolescence to adulthood. Human Development, 15, 1-12. Schauble, L. Klopfer, L. E., & Raghavan, K. (1991). Students transition from an engineering model to a science model of experimentation. Journal of Research in Science Teaching, 28(9), 859-882. Sieber, J. E. (1974). Effects of decision importance on ability to generate warranted subjective uncertainty. Journal of Personality and Social Psychology, 30(5), 688-694. Spiro, R.J., Feltovich, P.J., Jacobson, M.J., & Coulson, R.L. (1991).Cognitive flexibility, constructivism, and hypertext: Random access instruction for advanced knowledge acquisition in ill-structured domains. Educational Technology Research and Development, 31(5), 24-33.
ISLS
425
ICLS2012
Breeding Birds to Learn about Artificial Selection: Two Birds with One Stone?
Aditi Wagh & Uri Wilensky, Northwestern University, 2120 N Campus Drive, Evanston, IL 60208, USA, aditiwagh@u.northwestern.edu, uri@northwestern.edu Abstract: Recent research in Learning Sciences has drawn attention to the affordances of enabling students to learn about scientific phenomena through a complex systems lens. In this study, we adopt a complex systems perspective in helping students to learn about artificial selection by using an agent-based participatory simulation - Bird Breeder. Our goal is to identify the strategies students used when using this simulation, and investigate the kinds of abstractions they made about the underlying agent-level mechanisms and emerging population-level outcomes of the model. We answer these questions using data collected from three high school biology classes that used this simulation. Our findings indicate that Bird Breeder was effective in fostering learning about the agent-level mechanisms that drive artificial selection. In contrast, the effectiveness of Bird Breeder at helping students learn about population-level outcomes of artificial selection is less clear. The findings have provided insight into possible design revisions to this simulation.
Literature review
The study of complex systems is increasingly becoming a new strand of literacy (Jacobson & Wilensky, 2006). We consider a complex system to be an emergent system in which population-level trends emerge from individual-level mechanisms. In the educational context, the study of complex systems as a literacy entails empowering students with the opportunity to understand the complexities of a system at varying levels of the system (Wilensky, 2003; Wilensky & Resnick, 1999). Biology is replete with instances of complex systems in which phenomena can best be understood by grappling with relationships between levels in the system (Wilensky & Reisman, 2006). Evolution, which is central to the study of biological sciences, is a complex emergent phenomenon, often grossly misunderstood by the public at large (Alters & Nelson, 2002). Broadly viewed from a complex systems perspective, evolution is a process of change at the level of populations, such as species, resulting from mechanisms at work at the individual level such as organisms, alleles and so on. Popular representations of evolutionary change such as cladograms and graphic simulations have been widely studied for their impact on student learning (Ainsworth, 2009; Evans et al, 2010; Soderberg & Price, 2003). While these representations depict change over time, they fail to provide insight into the mechanistic underpinnings of evolutionary change. They do not represent how populations change over time. Agent-based representations are more effective at helping students learn about how population-level changes emerge from individual-level interactions (Centola, McKenzie & Wilensky, 2000; Wilensky & Novak, 2010, Xiang & Passmore, 2010). Most of this work has been done using agent-based models in which students explored models of evolutionary processes as an observer of the system. In this paper, we examine the use of agent-based models in which students are participants in the system. This is done using agent-based participatory simulations, which enable students to step into an agentbased model by becoming an agent in the system (Wilensky & Stroup, 1999a). By doing this, students discover strategies that characterize the agent-level mechanistic underpinnings in an emerging system. In this way, learners increasingly appropriate the relationship between emergent aggregate-level change in the system and the agent-level mechanisms that drive the change (Wilensky & Stroup, 1999a). A participatory simulation is likely to be a felicitous environment for a notoriously challenging topic such as evolution because it can leverage students agent-level cognitive resources related to evolution (Wagh & Wilensky, 2010). The goal of this paper is to study this potential in the context of a participatory simulation, the Bird Breeder (Wilensky & Novak, in preparation) which forms one activity in the agent-based modeling curriculum for evolution, BEAGLE (Biological Experiments in Adaptation, Genetics, Learning & Evolution) (Wilensky & Novak, 2010).
Research Questions
In this paper, we investigate two questions in the context of students participating in the Bird Breeder simulation. First, we examine the strategies students spontaneously employed when participating in the environment to breed certain variations of birds. Second, we investigate what students learned about the agentlevel mechanisms and population-level outcomes of artificial selection from participating in this environment.
Methods
ISLS 426
ICLS2012
Data Collection
This study was conducted in the context of a two-week long implementation of BEAGLE in three high-school biology classes in a mid-western town in the United States. Students used a participatory simulation of artificial selection, Bird Breeder developed in the HubNet module (Wilensky & Stroup, 1999b) of NetLogo (Wilensky, 1999). The students spent one class period working on Bird Breeder. On the second day, the teacher led a whole class discussion to synthesize what students had learned from the model. Then she asked students to answer one of two questions; to note down either three mechanisms or three outcomes of artificial selection that they learned from the model. The students worked alone and were given about 7-10 minutes to work on this task. Responses of 38 students were collected and transcribed. Students were also video-recorded as they worked in their groups on this model. Videos of three groups were closely analyzed. Bird-Breeder differs from other participatory simulations in BEAGLE in that the class is divided up into groups of four, with each group engaging in their own enactment of the participatory simulation rather than the entire class participating in a single enactment. Within their groups, each student assumed the role of a bird breeder and was randomly assigned three or four birds at the start of the simulation. These birds differed from each other with respect to four traits: the color of their crest, tail, breast and wings. Through the course of the simulation, students could breed birds in the communal breeding site. Students could also release birds they did not want into the wild. Each groups goal was to breed three pairs of birds with a red tail and wings, a purple breast and a blue crest.
Data Analysis
Two coding schemes were developed to analyze the data. The first coding scheme comprises codes that identify strategies students used when breeding birds. These strategies were identified through student utterances; student utterances were coded as a strategy when students verbalized a move by giving directions to another student or justifying their own move. The following codes emerged from open-coding of student talk in their groups. 1) Elimination: A student move was so coded when a certain bird was released from the population. 2) Selective focusing: A student move was so coded when students focused on one or two traits for breeding at a time. 3) Suitable mate: A student move was coded such when students purposefully picked a certain bird as a good mate for another bird. 4) Purebred: A student move was so coded when students bred 2 goal birds assuming that the offspring would be a goal bird. 5) Survey: A move was so coded when students reviewed the population of birds they collectively had in their group. A second coding scheme was developed to analyze student responses to the question posed by the teacher. The codes are: 1) Model-centric: A response was so coded if it included details about the model without identifying a mechanism or outcome of artificial selection. 2) Agent-level mechanism: A response was so coded if it described one or more of the underlying agent-level rules for artificial selection. 3) Population-level outcome: A response was coded such if it described what was happening to the population over time. 4) Other: A response that could not be categorized in any of the previous categories. Each student response was coded and responses in each category were counted up for analysis.
Results
Question 1: What strategies emerged as students participated in this simulation?
As students engaged in this simulation, they spontaneously enacted mechanisms of artificial selection in the form of strategies used to breed the goal birds. Four mechanisms were most commonly adopted by the groups; first, students eliminated birds that did not have the variations needed in the goal birds; second, students actively strategized about which birds would be suitable mates to breed; third, students selectively focused on one or two traits to breed at a time; and last, once a male and female goal bird was bred, students used those birds for further breeding. Across the three groups, students tended to release birds when they ran out of space in the breeding site. In the beginning of the simulation, students retained birds that had even one of the traits they needed. As the simulation progressed, students tended to retain birds with two or three of the desired variations. The following is an excerpt from Group 3. Group 3 had little space left in their breeding grounds to breed more birds. However, they had not yet bred the goal birds and needed to continue breeding birds. S1: Oh. Um, the problem is that you dont have any space. S2: Oh, I dont have any space. S1: Uhhhh yeah. So get rid of your two, your two, that has no color, and spots, and make them fly away. In this excerpt, S1 asks S2 to release the two birds that do not have any of the variations they need. All groups commonly pursued this strategy of releasing birds. Moreover, once a group managed to breed several of the goal birds, they were willing to release even a goal bird. This excerpt, taken from Group 3, illustrates this strategy.
ISLS 427
ICLS2012
S3: We just keep- get rid of some of your birds, S1. S1: Yeah, I know. S3: Be careful you dont get rid of the good female. (The group laughs) S3: Id rather get rid of a good male because we have extra. In this excerpt, S3 is willing to forego a good male (male goal bird) rather than a good female (female goal bird) because they have extra males. The development of this strategy and the willingness to release even goal birds indicates that students were learning something about how the pool of the breeding population determines the kinds of offspring that would result from it. A second strategy students commonly pursued was to purposefully select which birds to mate rather than randomly mating birds. Male and female birds with the maximum number of desired variations were selected for mating. The following excerpt, taken from Group 1, illustrates the use of this strategy. S3: Well, Ive got a red tail and a red wing. So, what we need to doS2: Alright, S3, youve got purple S1: Oh, hold on S3, right off the top, the one you have selectedS4: I have red, I have .. S1: -The one you have selected is going to be a good one to start with because three of the traits are already there, homozygous recessive so we just need to breed that with, does anybody have S4: Ive got a red wing. S2: Yeah, yeah, lets try to focus on the red wing. In this excerpt, S1 encourages S3 to begin mating with the selected bird because it already had 3 of the 4 variations they wanted in the goal bird. Moreover, this group uses the third common strategy used by groups, selectively focusing on one or more traits at a time. This group decides to focus on a single trait, color of the wing in further breeding. It is important to note that Group 1 was using a feature of the model they had been asked to deactivate, the genotype display. Students had been asked to turn it off. This group accidentally left it on, and was using that information as a scaffold to guide their activity. These two strategies, purposeful mating and selective focusing were seen across all three groups. In this excerpt, one sees students using both these strategies; they select a suitable mate for the bird they want to breed, and make that decision based on the traits they plan to focus on, purple breast and red tail. S1: And then after they make love, and have kids, then were going to have to breed your other one. S2: Dang it. S1: Yessss! S3: Closer. S1: Now we gotta breed that bird with your, uh, other one. S2: Alright take, now take that one, and breed it withS1: Your right oneS2: No, the same one you bred it with earlier. S1: No, it needs a red tail. S2: Oh. YeahS1: And that one has a purple breast too, so just breed that one and you can possibly knock out two birds with one stone. (laughs). Pun. S3: Oho! Two birds with one stone. S1: Now we got the purple breast, now we need a red wing and tail- Finally, the fourth strategy that emerged was that once a group bred the goal bird, they exclusively used it for further breeding. The following excerpt is from Group 3, taken from just before they bred their first goal bird. S2: Grrr. This needs a blue cap. God this stupid cmon, cmon, cmon YES! S1: YES! S2: Now we just breed that with your good one- with your good male- I mean, female- and we just keep doing it over and over again until we have S3: We just keep- get rid of some of your birds, S1. S1: Yeah, I know. S3: Be careful you dont get rid of the good female. (The group laughs.) S3: Id rather get rid of a good male because we have extra. S1: Yes! We got our two. We got two. (Students give each other high fives) S1: We have extra females. S3: And then, at the end, you could just move, uhS2: Cmon, cmon, cmon- dangit. Thats a good female. S3: Alright, now we just do this again. At the start of this excerpt, Group 3 needed only a blue cap to breed the goal bird. Their last breeding
ISLS
428
ICLS2012
gave them a goal bird that they called the good male. S2 suggests breeding the newly bred good male with the good female they already have, over and over again. Their plan is to breed a good female and then breed the two over and over again to get more goal birds. An interesting point to note here is that though the group correctly strategized that breeding two purebreds with the four required variations of traits will result in another goal bird, they are inaccurately assuming that the two goal birds they have are indeed purebred. That is, they are assuming that the two birds are homozygous recessive without testing their assumption. Though this group has discovered a mechanism that is key to artificial selection, it is manifesting levels confusion (Wilensky & Resnick, 1999) by taking the phenotype of a bird as an indicator of being a purebred rather than its genotype. This observation has led us to re-think the design of the activity to enable students to view the already existing feature in the model, viewing the genotype of a bird. In conclusion, the findings suggest that engaging in the Bird Breeders model facilitated discovering agent-level mechanisms that contribute to change in populations through artificial selection.
Question 2: What did students learn about the mechanisms and outcomes of artificial selection?
Of the 38 student responses analyzed, 22 students responded to the question about mechanisms of artificial selection while 16 students responded to the question about outcomes of artificial selection. Of the 22 students who responded to the mechanisms question, 16 students described one or more mechanisms of artificial selection. For instance, one of the students responded, Releasing useless stock. Selectively mixing traits together. Getting a lot of eggs per breeding pairs. Another student responded, Breeding lots of purebreds. Removing specimen with undesirable traits. Mating specimen with some desired traits with specimen with other desired traits. The first student described three mechanisms of artificial selection; removal of individuals with unwanted traits from the breeding population, selectively breeding individuals with required traits, and conducting several matings for each pair. The second student described the mechanisms as; breeding several purebreds, removing individuals with unwanted traits from the breeding population, and mating individuals with one desired trait with individuals with another desired trait. The remaining 6 students responded in model-specific terms without describing a mechanism of artificial selection. Of the 16 students who responded to the question about outcomes of artificial selection, 7 students described the outcomes in model-specific terms, by describing the goal birds in the simulation. For example, one of these students responded Blue head, red tail with red wing, purple chests. Blue head red tail/wing purple chest. Blue head with red wing/tail. Purple chest = same outcome. A normal bird. In other words, these 7 students described the outcome in terms of the goal birds they tried to breed in the simulation. Only 5 students described the outcome in terms of population-level trends. For instance, one of the students responded, loss of genetic information. Less durability in the species. Another student responded, Lowered genetic diversity in the species. These findings indicate that while a majority of students reported agent-level mechanisms of artificial selection, fewer students described a population-level trend as an outcome of intentional selection.
Discussion & Implications

The findings of this paper indicate that Bird Breeder was a felicitous learning environment for students to learn about mechanisms underlying artificial selection. Participation in the environment served to foreground the mechanisms of artificial selection for students. It is important to note that students had not been taught these strategies before using the simulation, or been explicitly asked to use them. By spontaneously generating strategies to breed the goal birds, students learned about mechanisms of artificial selection. Fewer students explicitly expressed their understanding of how populations change through artificial selection. We think there are three possible explanations for this. It is possible that the phrasing of the question did not trigger student thinking about what was happening at the population level and led students to respond in terms of model-specific outcomes (description of individual goal birds). Another possible explanation relates to the nature of representations included in the model. Unlike other HubNet models in BEAGLE, this model did not have a population-level representation that depicted the shifting trend in the population over time. This lack of an explicit aggregate-level representation might have made it difficult for students to synthesize how the population had changed over the course of the simulation run. Another explanation might be that Bird Breeder was a small-scale participatory simulation consisting of 4 students rather than the entire class. This small-scale HubNet model might have made it difficult for students to notice the aggregate-level trends that were emerging from their interactions. This study has opened a few exciting avenues for possible design revisions to the activity and future work. With respect to the design, we would like to include an aggregate-level representation in the model to depict how the population changes over time in the model so students can track shifting trends in the population. We would also like to give students access to the genotype indicator of each bird for a part of the activity. We
ISLS
429
ICLS2012
think that information will give students the opportunity to reason at a third level of complexity, individuals genes, to reason about how change at that level influences change at the level of individual phenotypes, and the population. Finally, we would like to compare the affordances and constraints of exploring an agent-based model as an observer vs. a participant.
References
Ainsworth, S. 2009. Can children read trees? Paper presented at the Understanding the Tree of Life. Harvard Museum of Natural History, Cambridge, MA. Alters, B. & Nelson, C. (2002). Teaching evolution in higher education. Evolution, 56, 1891-901. Centola D., Wilensky U., & McKenzie E. (2000). A hands-on modeling approach to evolution: Learning about the evolution of cooperation and altruism through multi-agent modeling The EACH Project. Proceedings of the Fourth Annual International Conference of the Learning Sciences, Ann Arbor, MI, June 14-17. Evans, E. M., Frazier, B., Hazel, A., Kiss, A., Lane, J. D., Spiegel, A. N., & Diamond, J. (2010). Tree-thinking: Do pictorial representations of evolution help or hinder museum visitors' understanding? Understanding the Tree of Life Conference, Carnegie Museum of Natural History, Pittsburgh. Jacobson, M. & Wilensky, U. (2006). Complex systems in education: Scientific and educational importance and implications for the learning sciences. Journal of the Learning Sciences, 15(1), pp. 11-34. Novak, M. and Wilensky, U. (in preparation). NetLogo Bird Breeder HubNet model. http://ccl.northwestern.edu/netlogo/models/BirdBreeder. Center for Connected Learning and Computer-Based Modeling, Northwestern University, Evanston, IL. Soderberg, P., & Price, F. (2003). An examination of problem-based teaching and learning in population genetics and evolution using EVOLVE, a computer simulation. International Journal of Science Education, 25(1), 35-55. Wilensky, U. 1999. NetLogo. http://ccl.northwestern.edu/netlogo/. Center for Connected Learning and Computer-Based Modeling, Northwestern University. Evanston, IL. Wilensky, U. (2003). Statistical mechanics for secondary school: The GasLab Modeling Toolkit. International Journal of Computers for Mathematical Learning, 8(1), 141 (special issue on agent based modeling). Wilensky, U., & Resnick, M. (1999). Thinking in Levels: A Dynamic Systems Perspective to Making Sense of the World. Journal of Science Education and Technology, 8(1). Wilensky, U. & Stroup, W. (1999a). HubNet. Evanston, IL. Center for Connected Learning and Computer-Based Modeling, Northwestern University. Wilensky, U., & Stroup, W. (2002). Participatory Simulations: Envisioning the networked classroom as a way to support systems learning for all. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA, April 13. Wilensky, U. & Reisman, K. (2006). Thinking Like a Wolf, a Sheep or a Firefly: Learning Biology through Constructing and Testing Computational Theories -- an Embodied Modeling Approach. Cognition & Instruction, 24(2), pp. 171-209. Wilensky, U., & Novak, M. (2010). Understanding evolution as an emergent process: learning with agent-based models of evolutionary dynamics. In R.S. Taylor & M. Ferrari (Eds.), Epistemology and Science Education: Understanding the Evolution vs. Intelligent Design Controversy. New York: Routledge. Wilensky, U. & Novak, M. (2010). BEAGLE. Evanston, IL: Northwestern University, Center for Connected Learning and Computer-based Modeling. Wagh, A. & Wilensky, U. (2011). Context counts: Role of the context in triggering productive and unproductive pieces of knowledge about natural selection. Paper presented at Jean Piaget Society Conference, Berkeley, June 2-4. Xiang, L. & Passmore, C. (2010). The Use of Agent-based Programmable Modeling Tool in 8th Grade Students' Model-Based Inquiry. The Journal of the Research Center for Educational technology, 6(2), 130-147.
ISLS
430
ICLS2012
Tracing Ideas and Participation in an Asynchronous Online Discussion across Individual and Group Levels over Time
Alyssa Friend Wise, Ying-Ting Hsiao, Farshid Marbouti, Yuting Zhao Simon Fraser University, 250-13450 102 Avenue, Surrey BC, Canada, V3T 0A3 Email: afw3@sfu.ca, yha73@sfu.ca, fmarbout@sfu.ca, yza174@sfu.ca Abstract: This paper advances a methodology to support a coordinated multi-level analysis of individual actions and group dynamics in asynchronous online discussions. The approach uses log-file data to examine group and individual participation patterns, and argumentation coding of post contents to probe developmental trajectories of individual and collective understandings. Importantly, these traces of ideas and behaviors are coordinated within and across levels. To illustrate the method, the paper presents an analysis of five undergraduate students taking part in a five-day online discussion to address a business challenge. Findings provided insight into the ways in which phenomena at the group and individual levels interrelated and drove each other, specifically the complex interdependency between group and individual willingness to engage in debate, and how one dominant individuals unfounded presumption of consensus led to early abandonment (rather than conscious rebuttal) of ideas.
Introduction
Asynchronous text-based discussion forums are often employed to facilitate online collaboration among students. Temporal flexibility gives students time to reflect on their ideas and those of others, thus theoretically affording opportunities for the group to collectively construct knowledge and for individuals to develop their personal understanding (Stahl, 2005). However, in practice, studies have found that learners often exhibit low cognitive engagement (Hew et al., 2010) with shallow and unfocused argumentation (Nussbaum, 2005). The problems of superficial discussion and weak argumentation can be considered on both the individual level (students dont engage with or challenge each others ideas) and that of the group (teams engage in consensus building without considering multiple ideas first). These levels are inherently interconnected as the groups process emerges from the accumulation of individual actions while also situating and constraining them (Suthers & Teplovs, 2011). Similarly, efforts to support productive dialogue in online discussions may be targeted at one of these levels, but will necessarily have interdependent effects on both. For example, requiring a discussion group to come to a consensus (group-level intervention) will affect the ways individuals contribute to the conversation; conversely giving students specific responsibilities in a discussion (individual-level intervention) will impact their group dynamics. For this reason, understanding how a pedagogical intervention affects participation and argumentation processes in online discussions requires a coordinated examination of individual- and group-level phenomena as they evolve over time. Attention to the multiple levels at which collaborative learning can be studied is growing (Stahl & Hesse, 2008) as is the use of statistical techniques that account for both individual and group influences on collaboration (Cress, 2008). However little work has attempted to examine how phenomena at these levels mutually depend on and affect each other. In addition, recent research in collaborative learning has emphasized the importance of studying the flow of student interactions over time (Reimann, 2009). This paper advances a methodology to support a coordinated multi-level temporal analysis of individual actions and group dynamics in online discussions. To illustrate the method, we present an analysis of a group of five students taking part in a five-day online discussion to reach consensus on how to address an organizational business challenge.
A Coordinated Multi-level Microanalytic Approach

To understand how conversation flows in an asynchronous online discussion we track trajectories of participation and ideas across group and individual levels and examine how they interrelate over time. The detailed traces of student activity created in online discussions present a rich source of data for such examination. Specifically, log-file data of actions (e.g. how students make posts and access those of others) facilitate analysis of group and individual participation patterns, while the content of posts supports examination of collective and individual understandings. While previous approaches have studied these data sources in aggregate (e.g. Palmer et al., 2008), we have found that unpacking data to provide an on-the-ground view of particular students and groups provides insight not available from collectivized data. This work draws on our previous research developing different analytic techniques to study how groups and individuals participate and develop understanding over time (Wise & Chiu, 2011; Wise et al., 2012). The important addition here is the attempt to coordinate analyses across levels. Such coordination is challenging for several reasons. First, while a groups participation is spread over hours and days, individuals experience participation in discrete episodes lasting minutes; thus differing timescales must be considered (Lemke, 2000). Second, the artifact of a groups
ISLS
431
ICLS2012
complete discussion is not how the living conversation appeared to individual participants at any given point in time. Finally, there is the difficulty of inferring relationships between phenomena that exist at different ontological levels. We address these challenges through three sequential analytical steps.
Log-file data analysis

Log-file data is extracted by participant. Three types of actions are coded: views (opening others posts), posts (creating a post), and reviews (revisiting ones own posts). Durations are calculated by subtracting time between subsequent actions. Views are further categorized as scans or reads using a cut-off maximum reading speed of 6.5 words per second (Hewitt et al., 2007). Adjustments are made for unlogged system exits and actions are grouped into sessions (a series of consecutive actions). To trace the groups timeline of participation, we visualize the log-file data to show patterns of group participation over threads and time (Wise & Padmanabhan, 2009; see Fig. 1); an interpretive narrative is then created. Separately, we create a narrative reconstructing the experience for each individual action-by-action using a dynamic discussion map to provide the context of how a discussion appeared at the time a behavior was taken. The combination of these data sources supports a meaningful reconstruction of individuals behaviors in the forum (Wise et al., 2012).
Post content analysis

Our examination of post contents draws on work from the field of argumentation. A coding scheme adapted from Clark and Sampson (2008) is used to evaluate when students propose, support and rebut ideas using the post as the unit of analysis (multiple codes can be applied to each post). The scheme also captures if students analyze the problem and if they attempt to synthesize the groups position. With this approach we can look at how the group negotiates ideas to move to a common conclusion, how individuals ideas change over time, and how these relate to the ways in which students position themselves with respect to the groups dialogue. Other schemes and models of knowledge construction could certainly be used within our overall methodology as well. Once posts are coded, we create a collective visual map of positions on each idea over time to trace the groups idea trajectory (see Fig. 2a). To examine the evolution of individuals ideas, a second map for each student is drawn (see Fig. 2b). Both diagrams are interpreted to create initial idea development narratives.
Coordination of data sources within and across levels

While log-file and post content data are initially analyzed separately, the data streams are highly interrelated. For example, an individuals new idea might be driven by their actions (e.g. reading a certain post) and highly contested ideas might trigger a collective burst of activity. Thus our first coordination step is to integrate the analyses of actions and ideas at each level (individual and group) into coherent narratives. We then use the narratives at each level to inform each other: contextualizing the participation and ideas of each individual within the groups trajectory, and tracing how patterns of group activity emerged from actions of particular individuals. Thus for a group of N students, the result is N+1 accounts of a discussion, one from the perspective of each student and one for the group as a whole. While the narratives at each level necessarily refer to each other, they are distinct in the primacy given to the perspective of the group or a specific individual.
Sample Analysis
We present a sample analysis using our multi-level microanalytic methodology with data from a group of five students (Arlene, Ben, Celia, David and Evan) solving a collaborative task that was seeded with divergent opinions but required consensus. Each students trajectory of participation and ideas was traced along an individual timeline, as was the collective process of participation and argumentation for the group as a whole. These parallel narratives were then compared and coordinated to examine the ways in which processes and phenomena at the individual and group levels interrelated and drove each other.
Learning Context and Participants

Data was taken from a larger study comparing the impact of different task-types on online discussion in a blended undergraduate business course. Students were randomly assigned to small groups, each responsible for discussing one business challenge online (Friday to Tuesday) and leading a subsequent tutorial discussion. The challenge in the week studied was communication. A junior executive was told that her response to criticism of her presentation by senior executives was too forceful. Two possible solutions were given for discussion: either email the senior executives to explain why she spoke up or accept the feedback and adjust her future behavior; additional solutions could also be proposed. Students were asked to come to consensus on a solution with a rationale using theories from class. Posting guidelines and criteria (worth 5% of course grade) were provided.
Process and Findings

Log-file data from the chosen discussion was extracted and processed as described above. Contents of all posts were coded by two researchers; agreement was 89% or higher for all categories except problem analysis (73%).
ISLS
432
ICLS2012
Group and individual timelines of ideas and participation were visualized in diagrams. Due to space limitations in a short paper, in this presentation we focus on the participation of three students (Arlene, Ben, Celia) over two key days as an illustration of the insight that the multi-level microanalytic approach can provide.
Figure 1. Time-Event Diagram. Left side shows all group posts in threaded structure distributed over time on the vertical axis. Letters are student initials, P = professor. Right side shows each individuals reading actions by day. Column height indicates the total time spent reading; labels are [# of sessions, # of posts read].
The Group
The groups discussion focused primarily on three solutions, two provided in the task (accept the feedback [Accept] and email the executives [Email]) and a variation on the latter of these (talk to the executives in person [Talk]). The participation peak for the discussion (greatest number of posting and reading actions) occurred on Sunday (see Fig. 1); however argumentation in the discussion was greatest one day prior in a back-and-forth sequence between Ben and Arlene debating the merits of the three solutions. All three other students also logged-in on this day and read the debate posts, but none joined in the debate. This appears to have been a pivotal moment for the groups discussion; the choice of other students to not join the debate while it was hot rendered what could have been a central locus of argumentation peripheral in the discussion. On Sunday many posts were made in support of the three solutions, but none rebutted the opposing alternatives. That evening, Arlene made a post presuming the groups consensus was Talk. This was a second pivotal moment for the group as all subsequent comments unquestioningly accepted this as the groups solution and focused on how to implement it or how to present it in the class tutorial; none questioned its status as the solution. Thus the groups argumentation was effectively completed on Sunday evening without clear rebuttal of all unchosen alternatives.
(b)
(a)
Figure 2. Timeline of argumentation (a) for group by idea (b) for individuals Arlene, Ben, and Celia (A, B, C).
Arlene
On Friday, Arlene initiated the first thread analyzing the problem and listing three possible solutions (Accept, Email, Talk) without indicating a preference. On Saturday, she logged in the forum and spent just one minute to read and respond to a post by Ben supporting Email. Her position in this post seems to have developed in reaction to Bens stance; she rebutted his argument by pointing out that the executives could easily misinterpret an email, thus she advocated Talk. That evening, Arlene read and replied back to a new counterargument Ben had made by restating the possibility of miscommunication via Email and her preference for Talk. However, she also suggested that maybe no action was necessary (Accept). On Sunday afternoon, Arlene returned to the
ISLS
433
ICLS2012
discussion, quickly skimmed a few posts and then posted in a thread analyzing the problem in light of upward/downward communication theory. An hour later, she logged-in again and immediately started a new thread that attempted to connect her preferred solution (Talk) to her problem analysis. She posted earlier we agreed there are 3 main problems [but] I just realized that in our solution, we actually only address problem #2The face-to-face meeting with the senior exec might clear up [this] misunderstanding. This post assumed that the rest of the group agreed with her about the problems identified (we agreed) though two of these had only been mentioned by her, and about the solution of Talk (our solution) though a group consensus had not been explicitly discussed and both support and dissent for Talk had been expressed. Her reasons for presuming consensus are unclear; it may be that she took Bens abandonment of the debate as tacit agreement for Talk, or that since she spent very little time reading others ideas she was unaware that other solutions still had support. In this post she also shifted away from her earlier support of Accept as a possible solution without explanation.
Ben
Ben first contributed to the discussion on Saturday, reading all the previously made posts and taking a stand for Email for the employee to clear-up any misunderstandings...and show that she is capable of what she is doing. That evening he spent almost a half hour reading a rebuttal responded by Arlene and composing his own counterclaim in support of Email stating that Talk was unpractical based on the theory of power distance in organizations. He also rebutted another peers post suggesting to Accept the feedback since this would leave a bad impression of the employee. Ben logged in again later that night and read Arlenes subsequent rebuttal of his defense; however, he did not continue the debate with her. On Sunday, Ben was active in reading others posts, but not in making his own. He made only one post, and in it did not advocate for or against any solutions. This might be a result of his minority position in the discussion, a reaction to the general lack of dissent, or his realization (revealed in a later post accepting Talk) that the group should agree on a solution. This final possibility suggests Ben interpreted the need for consensus as a reason not to debate different options.
Celia
Celia made her first post on Saturday evening, after spending considerable time that day and the one before reading the posts of others. Though the debate between Ben and Arlene was in progress, she did not join in, instead starting a new thread analyzing the problem. On Sunday morning, Celia again spent time reading others posts and then made another post emphasizing that the group should identify the problem before trying to solve it. After, she responded to the arguments in the Ben-Arlene debate thread by explaining her preference for Accept, but also indicating support for Arlenes idea of Talk. While she did not support Email, she also did not refute it; instead she expressed understanding about why Ben might take that position. Thus Celias response to the debate dynamic was diplomatic, acknowledging the value of each of the ideas involved. Celia did not log in again till Monday night, when she spent the better part of an hour catching up the discussion before writing a short reply accepting the groups consensus of Talk.
Discussion and Conclusions

The coordination of multiple levels of analysis helps shed light on specific ways in which individual actions influenced the groups process and how the group collectively set the stage for individual actions. Close to the start of discussion one student took a stand for a solution, prompting another to rebut his idea and initiate a debate dynamic at the group level. However the reluctance of others to join-in, perhaps because they valued reaching consensus over persuading others of their opinions (Nussbaum, 2005), created a collective lack of dissent. Left in the minority, the first student abandoned his argument. The following day the group shifted to initiating many new threads. This created a disjointed discussion, making it possible for one dominant students unjustified presumption of consensus to go uncontested. As a result, the group stopped considering other solutions, a deviation from dialogical argumentation (Clark & Sampson, 2008). While one student seemed to equally influence the group and be influenced by it, another was insistent in driving the group, and the third seemed content to follow the groups will. This highlights differences in the reciprocity of influence between the group and individuals. Several effects of timing also seemed to mediate: first, comments of individuals who were active early set the groups focus; second, the richest argumentation occurred in a short time. This raises questions about the value of compressed / distributed participation and the temporal weighting of voice. This paper presented a methodology for coordinated multi-level analysis to examine interrelationships between individual actions and group dynamics in online discussions. The methodology is still in development; testing and revision with additional data is needed. While the approach appears fruitful for generating insight into relationships between individuals trajectories of behavior and understanding, and groups patterns of dialogue and idea-development, its fine-grained, interpretive and time-intensive nature raises important questions about intersubjectivity, generalizability and scalability. Here, we present our current thinking around these issues. First, with respect to intersubjectivity, our initial participation narratives and diagrams are based on log-file data and constructed objectively as neutral, straight-forward reports of the temporal sequence of actions
ISLS
434
ICLS2012
in which an individual engaged or in which posts were contributed by the group. Content analysis of posts involves researcher subjectivity; we address this through the well-established practices of using a clear and theoretically-grounded coding scheme, choosing an unambiguous unit of analysis, and reporting inter-rater reliability (de Wever et al., 2006). The major interpretive steps occur when the initial idea and participation analyses for an individual or the group are integrated into coherent narratives and when these interpretive narratives are used to contextualize and inform each other. Making inferences about how different kinds of entities (ideas and behaviors) affect each other and relationships between phenomena that exist at different ontological levels inherently engages the researcher as meaning-maker and thus necessarily involves attention to intersubjectivity. For this reason, the interpretive narratives are repeatedly negotiated by multiple researchers throughout their creation, review and revision Nonetheless, as with any fine-grained analysis that focuses on a small number of participants, the goal is not to make statistical generalizations or suggest that what was found in one particular case will likely be found in another. Instead, the objective is theoretical generalization (Seale, 1999); that is to use the deep examination of a particular occurrence to develop theoretical understandings that can present a useful lens for interpreting other situations and suggest implications for educational practice. For example, this sample analysis highlighted the importance of how and why ideas stop being considered, the notion of reciprocity between group and individual levels; and the importance of temporal sequence and pace. Finally there is the question of scalability. The multi-level microanalytic approach involves multiple steps of technical data processing, content analysis, and interpretation. The time to conduct the complete analysis of this one discussion by five students over five days is estimated at 150 hours. Thus, use on a large scale is prohibitively expensive in terms of manhours. Instead, this methodology might be most effectively used in combination with existing temporallysensitive large-scale methods (e.g. Chiu & Khoo, 2005) which can identify important moments or phases in collaboration worthy of the investment in in-depth multi-level microanalysis.
References
Chiu, M. M., & Khoo, L. (2005). A new method for analyzing sequential processes: Dynamic multi-level analysis. Small Group Research, 36, 600631. Clark, D. B., & Sampson, V. (2008). Assessing dialogic argumentation in online environments to relate structure, grounds, and conceptual quality. Journal of Research in Science Teaching, 45(3), 293-321. Cress, U. (2008). The need for considering multilevel analysis in CSCL researchAn appeal for the use of more advanced statistical methods, ijCSCL, 3(1), 69-84. De Wever, B., Schellens, T., Valcke, M., & Van Keer, H. (2006). Content analysis schemes to analyze transcripts of online asynchronous discussion groups: A review. Computers & Education, 46(1), 628. Hew, K.F., Cheung, W.S., & Ng, C.S.L. (2010). Student contribution in asynchronous online discussion: a review of the research and empirical exploration. Instructional Science, 38(6), 571-606. Hewitt, J., Brett, C., & Peters, V. (2007). Scan rate: A new metric for the analysis of reading behaviors in asynchronous computer conferencing environments. AJDE, 21(4), 215-231. Nussbaum, E. M. (2005). The effect of goal instructions and need for cognition on interactive argumentation. Contemporary Educational Psychology 30, 286-313. Palmer, S., Holt, D., & Bray, S. (2008). Does the discussion help? The impact of a formally assessed online discussion on final student results. British Journal of Educational Technology, 39(5), 847-858. Reimann, P. (2009). Time is precious: Variable- and event-centred approaches to process analysis in CSCL research. ijCSCL, 4(3), 239-257. Seale, C. (1999). Quality in qualitative research. Qualitative Inquiry, 5(4), 465-478. Stahl, G. (2005). Group cognition in computer-assisted collaborative learning. JCAL, 21, 7990. Stahl, G. & Hesse, F. (2008). The many levels of CSCL, ijCSCL, 3(1), 1-4. Suthers. D. & Teplovs, C. (2011). Connecting levels of learning in networked communities. Workshop conducted at CSCL-2011, July 9, 2011, Hong Kong. Wise, A. F., & Chiu, M. M. (2011). Analyzing temporal patterns of knowledge construction in a role-based online discussion. ijCSCL, 6(3), 445-470. Wise, A. F. & Padmanabhan, P. (2009). Developing structural-temporal visualizations of asynchronous threaded online learning conversations. In T. Bastiaens et al. (Eds.), Proceedings of E-LEARN 2009 (pp. 32533261). Chesapeake, VA: AACE. Wise, A. F., Perera, N., Hsiao, Y. Speer, J., & Marbouti, F. (2012). Microanalytic case studies of individual participation patterns in an asynchronous online discussion in an undergraduate blended course. Internet and Higher Education, 15(2), 108117.
Acknowledgments
This work was supported by the Social Sciences and Humanities Council of Canada.
ISLS
435
ICLS2012
Scaffolding a Knowledge Community for High School Physics

Mike Tissenbaum, James D. Slotta, University of Toronto, 252 Bloor St. W., Toronto, Canada (miketissenbaum@gmail.com, jslotta@gmail.com) Abstract: This paper presents a design study of a collective inquiry model for high school Physics, where student-contributions are captured, aggregated, tagged and represented in a coherent visualization. We have developed a flexible technology layer that supports the aggregation of peer responses, including the collection of student explanations and semantic tags. We investigate collaborative inquiry scripts and discuss how they must comprise both the (macro) scripts that are enacted over a long-term curriculum, and the (micro) scripts that are enacted in class. We outline our rationale for inquiry design in Physics, the role scripting and orchestration play in the successful implementation of this curriculum, the role of the smart classroom in their enactment, and three successive iterations of our curriculum.
Knowledge Communities for 21st Century Learning

As we move further into the knowledge age todays workplace is shaped by new technologies, where activities are increasingly data-driven, collaborative, and predicated on a set of fundamental skills commonly referred to as information literacies (Livingstone, 2008). This shift is particularly pronounced across STEM disciplines, where workplace practices are shifting towards large, data-intensive, multidisciplinary collaborations across ever-widening spatial and temporal scales (e.g., the Human Genome project). A theoretical perspective from the learning sciences that is well suited to learning and instruction in the knowledge age is that of knowledge communities, as exemplified by the Fostering Communities of Learners (FCL) project (Brown & Campione, 1996), and Knowledge Building (Scardamalia & Bereiter, 1996), amongst others. These researchers have advanced an epistemological perspective where students consider learning as a social process, and value the collective knowledge of their peers. Although difficult to enact (Sherin et al, 2004), the knowledge community approach has garnered renewed attention, partly as a result of Web 2.0 capabilities, which can support complex pedagogical constructs (Slotta & Najafi, 2010). Using Web 2.0 technologies, students are active participants in a knowledge community that is engaged in the production, aggregation, and assessment of science topics, with an emphasis on inquiry and collaboration (Peters & Slotta, 2010). The socially-oriented process of adding of meta-data (tags) to usercontributed content can provide opportunities to reveal meaningful connections and flexible real-time representations (Mathes, 2004). However, questions still remain about how such collections of content can best serve student learning and foster knowledge communities. Reflection and scripting, are two important aspects of such research and will be discussed in the present paper
Reflection, Discourse, Scripting and Orchestration in Learning Activities

An important dynamic within most inquiry or knowledge community research is that of reflection, which is typically embedded within student learning activities (Slotta & Linn, 2009). While generally accepted as an essential part of the learning process, reflection takes on particular significance in digitally-mediated learning environments (Johnson & Aragon, 2003), where many interactions take place asynchronously. Students are provided with increased opportunities to reflect on their own understanding, think critically about their peers ideas, construct coherent arguments, and reconcile misconceptions, before adding to the public discourse (Garrison, 2003; Chi, 2000). Another topic of interest to learning scientists is the notion of scripting and orchestration (Dillenbourg, Jarvela & Fischer, 2009), where specified learning and interaction designs (i.e., the script) are enacted (the orchestration) by teachers and students. The script can be seen as a formalism that captures the pedagogical structure of a learning design, including a wide range of interaction patterns among students, their peers, and the teacher. When user-contributed materials are introduced, the script becomes more open-ended (Peters & Slotta, 2010), and any inquiry design must be left somewhat unbounded to allow for emergent themes, directions or content. While teachers are the primary orchestrators of the script, this role is also shared amongst students. In technology-enhanced learning environments teachers receive real-time feedback about student ideas, resulting in opportunities for evidence-based decisions that can influence the script itself (i.e., real-time course corrections), and provide opportunities for teacher professional development (Dillenbourg & Jerman, 2007; Lui, Tissenbaum & Slotta, 2011; Slotta & Linn, 2009).
Research Context
In this paper we present a design study of a collective inquiry model, where student-contributions are captured, aggregated, tagged and represented in a coherent visualization in the context of a high school Physics course. We have developed a flexible technology layer that allows the investigation of collaborative inquiry scripts to
ISLS
436
ICLS2012
support the aggregation of peer responses, including the collection of student explanations and semantic tags. In the sections below, we outline our rationale for inquiry design in Physics, we describe the role scripting and orchestration play in the successful implementation of this curriculum, and we discuss the role of the smart classroom in supporting complex interactions in three successive iterations of our curriculum. The research progressed through three design-based iterative advancements. The first two were formative, providing important information about student collaboration using real-time digital features. In iteration three, we dramatically expanded our designs, moving from single session smart classroom scripts, to a persistent digital layer that supported periodic inquiry and collaboration for the duration of the Physics class, both at home and in the field. We worked closely with the teacher to develop designs, including a repository of user-contributed materials, and social and semantic tags, which facilitated the development of new scripts for teachers and students alike. Our specific research questions are as follows: How can the aggregated products of student inquiry cultivate a knowledge community in high school Physics? What kinds of scripts can best aid students in leveraging user-contributed materials towards creating deeper understandings of Physics? What technology supports can aid teachers in the scripting of curriculum within an emergent knowledge community?
Method
This research employs a design-based method, involving successive cycles of design, enactment, analysis, and redesign within authentic classroom settings (DBRC, 2003). Using a co-design approach (Roschelle, Penuel, & Schectman, 2006) our team of researchers, technologists, and teachers worked together developing technologies, curricular materials, activities, and interaction patterns. The study was set within an urban high school, with all activities occurring as part of students regular homework and school activities. All materials and interactions reported in this paper were delivered using SAIL Smart Space (S3) a technology infrastructure for smart classrooms and knowledge communities (Slotta, Tissebaum & Lui, 2011).
Iteration One Developing a Cross-Context Physics Problem Solving Activity

The aim of the first design was to investigate how the aggregation and representation of peers work generated outside the classroom (i.e., at home) could be leveraged for in-class knowledge building, and how different technologies could aid the teacher in gaining insight into the state of class knowledge in support of different scripting and orchestration moves. Students first individually solved a set of multiple-choice Physics problems as a homework activity. In a follow-up synchronous activity, small groups re-solved these problems using the aggregated responses of their peers from the homework stage. To support the process, we developed a portal allowing the teacher to customize the activity (i.e., the number of questions to be served, and the type of questions presented), a visualization displaying the student-negotiated answers and relationships (tags) between problems, and an aggregated report of students homework responses. By viewing the report and visualization before class, the teacher could adjust the upcoming class script based on his perception of the students understanding. The teacher could also use both tools during live classroom activities to gain insight into student group work in real-time, or as a post-activity discussion tool.
Design:
Our study consisted of two Physics classes, with twenty students (n=20) in the first trial and sixteen (n=16) in the second trial. During the enactment, the teacher logged into the portal and uploaded five homework questions. Upon receiving an email alert, each student Tagged, Answered, and provided a Rationale (TAR) for his or her answer before the start of the next class (two days later). In advance of the in-class session, the teacher logged into the portal to view the aggregated student work to develop a sense of the class understanding of the ideas present in the homework. During the in-class activity, student groups viewed aggregated answers of the whole class ,reached consensus, and decided whether or not to re-TAR the question.
Data Sources:
Data were drawn from three sources: 1) All student and group tags, answers, and rationales were captured by the system; 2) Researchers collected in-class activity field notes; 3) A follow-up debriefing with the teacher. The captured student data was examined to reveal changes in the accuracy of responses between students answering individually versus groups, and in their rationales. The field notes provided us with an understanding of how the students were engaging with the curriculum and their peers. Finally, the follow-up with the teacher provided insight into his perceived effectiveness of the added technology scaffolds in meeting curricular goals.
Findings:
Overall, groups faired significantly better at solving problems (97% correct) than individuals at home (80% correct), with t=2.02, df=41, and p<0.05. One problem, for example, had marked improvement with 45% of students answering incorrectly at home, while 100% answered correctly in groups. A potential confound is that the groups were solving problems they had seen individually as homework. However the addition of the whole class rationales made it worthwhile to re-engage the students with the same set of problems.
ISLS 437
ICLS2012
Throughout the in-class activity, groups read their peers tags and rationales, and attempted to make sense of differences through discussion. Overall, student groups recorded forty-eight rationales with their answers. A comparison of individual answers versus group answers showed that in 24 cases (55%) the groups rationales were unique not identical or nearly identical to any individual answer (with an intercoder agreement of 83%) suggesting that students did not simply reiterate individual ideas during the group activity (although, it is possible they ignored those ideas, which is equally problematic). For rationales that were identical or nearly identical to an individual response, it was unclear if this was due simply to re-stating the original idea, or if they really believed the original answer was best. These outcomes suggest the potential benefits of a script that engages students groups around the aggregated individual contributions of their peers. In the de-briefing, the teacher stated that he found the real-time reports useful in understanding where students were having problems with the content prior to conducting the class. During the first class, the teacher decided to allow the lesson to run without changing the script, preferring to see how students fared in their small groups. However, seeing students struggle on a particular question prompted him to intervene (i.e., adapt his script). Drawing from this insight, he then adapted the class orchestration more readily on the second day to address the issue as it arose, reducing the potential for student frustration. The teacher revealed that although the visualization was useful, the need to refresh the screen made it difficult for him to know what was happening in true real-time. Instead, he relied on the large format displays to inform his in-class orchestration. The results of this preliminary study point to the efficacy of certain tools for gaining insight into the state of student knowledge at different points within the script, and in aiding the teacher in making necessary adjustments. This study also highlights the limitations of certain tools within a live activity particularly the visualization given its inability to provide information at key moments within the script. Future designs need to better coordinate the flow of information based on teacher needs.
Iteration Two Adding student expertise groups and real-time teacher feedback
The second iteration aimed to improve students use of the aggregated work of their peers and to further aid the teacher in orchestrating the script. Building upon the first iteration we wanted to further understand what scripts guided students to in depth reflections, and also the role aggregated information played in forming these reflections. In addition, we designed a teacher report application for a tablet computer that used a colour-coded matrix to display group performance on problem solving in real-time. The teacher could touch the icon of any problem to bring up an individual groups TAR response, helping inform his understanding of the groups progress. We were interested in how this tablet application could provide new opportunities for understanding the state of the class knowledge and how this affected the orchestration of class activities.
Design:
The teacher uploaded thirty-five homework questions, representing five distinct topic areas. Each student was assigned to one topic area, and received five (of seven in that area) problems for homework. During the smart classroom activity, students were placed in groups of five (one student from each topic area), and given five questions (one from each topic area) that no member had seen before as homework. This complex tracking of prior exposure to problems, and selection of suitable items, made possible by the S3 framework, would have been challenging using the traditional approach. On the first day, students were not provided with aggregated homework content; rather, they relied on group negotiation to solve the problems. During the second day, groups were supplied with their peers aggregated answers from both classes. The teacher was also given slightly different conditions: on day one, he only had access to the large-format displays in the smart classroom for information about class activities; on day two, he was provided with the tablet for real-time updates.
Data Sources:
Data collection for this iteration was identical to iteration one. Further, student TAR data was examined for changes in the correct responses between students answering individually compared to in groups without the aggregated work of their peers (Day 1) and in groups with the aggregated work of their peers (Day 2). Individual student and group rationales were also evaluated using a four-point scale, developed conjunction with the teacher, to evaluate the depth of student understanding.
Findings:
Two researchers evaluated all student and group responses using the co-developed scale (intercoder agreement of 83%). Overall, the group on Day 2 (score of 2.0) significantly outscored both the individual students during the homework phase (score of 1.32) (t=4.13, p<0.01, df=51), and the groups from Day 1 (score 1.21) (t=4.19, p<0.01, df=50). In groups, students got more questions correct both days (Day 1=83%, Day 2=84%) versus individually (71%), however both cases were only marginally significant. Taken together, these findings suggest that students in groups perform better than individuals in terms of correct answers, without or without access to the broader class ideas, but that access to this information helps in the depth and quality of their reflections.
ISLS
438
ICLS2012
Similar to the previous iteration, the teacher actively moved throughout the class, interacting with students where necessary. At several points during the activity, the teacher read the rationales being written by the groups (projected on the large format displays) and prompted them to refine their thinking towards focusing on the deeper principles relevant to solving and understanding the problems. As the intervention progressed, the teacher adopted a catch phrase words more than numbers, in response to what he observed in the class. The teachers interactions with the tablet elicited surprising results. Initially, he was very engaged with the tablet, clicking on group responses, reading rationales and watching for wrong answers. Eventually however, he abandoned the tablet, stating that it divided his attention and hampered his ability to monitor the class. He noted that although useful for seeing group errors, the information came too late to intervene at critical moments, and he could more effectively monitor the class by watching the large displays. This underscored the importance of thoughtful design, and cultivating a deep understanding of the interaction patterns that are most relevant or helpful within an inquiry script.
Iteration Three: A persistent, multi-context collaborative inquiry environment

In our most recent design we sought to add a dimension of student-contributed content, while still allowing the teacher to insert new materials, based on emergent patterns within the script. In PLACE.Web (Physics Learning Across Contexts and Environments) students capture examples of Physics in the world around them (through pictures, videos, or open narratives), which they explain, tag, and upload to a shared social space. Within this knowledge community, peers respond, debate, and vote on ideas, towards gaining consensus about the phenomena being shown. Student work was visualized as a complex interconnected web of social and semantic relations, allowing students to filter the information to match their own interests and learning needs. The visualization also became the focal point of real-time smart classroom activities in which students leveraged the products of the class in the creation of challenge problems. S3 pedagogical agents coordinated these activities and informed the teacher of student knowledge at-a-glance. The tablet application was redesigned as a student application. Tools were also developed to allow the teacher the ability to assess student work, review individual student progress, and to record their own reflections on class understanding to aid scripting decisions. Building upon the findings of the earlier studies, we designed six activities for the teacher to enact within the curricular script: (1) student-developed phenomena for peer debate, voting, and discourse; (2) teacher-created homework problems: (3) student-created challenge problems for their peers; (4) smart classroom activities drawn from the collective knowledge base; (5) in-class discussion around artifacts within the knowledge base; and (6) group development of shared narratives within the smart classroom. Although initial scripts were co-designed with the teacher, the resulting activities that emerged were flexible, to allow the teacher to adjust their implementation as he saw fit within the context of the class emerging knowledge.
Design:
This iteration involved two grade 11 physics classes (n=20, n=25), spanning three separate curricular units over a six-moth period: Kinematics; Force and Motion; and Energy, Work, and Power. The units were thematically connected, allowing content to carry over between units. To start, each student was given one concept (tag) from a list of 13. The teacher considered these concepts to be fundamental to the understanding of the grade 11 curriculum. Over the six-month period, students were given more concepts to work with). Students focused on their assigned concept when capturing examples for inclusion in the community knowledge base. To assess how the depth of the negotiated discourse of the knowledge community approached expert descriptions, a selection of student-submitted examples were given to graduate physics students (as experts), who were asked to tag (from the list of fundamental concepts) and describe in words how those concepts were being exhibited. For each of the units, we altered the script design towards formalizing a set of interactions that (1) fostered knowledge community growth, (2) supported the teacher in altering the scripts to address student needs, and (3) helped students to use the knowledge base for their own individual constructivist learning. Across the three units two scripts were enacted that engaged students in six the activities mentioned above. The collective inquiry cycle, (CIC) where (1) Students submitted inquiry items to the knowledge base; (2) Collectively (at home or in class) examine and tag peers work, adding comments to explanations; (3) Teacher reviews the community knowledge, (4) In-class activity engages students with collective knowledge artifacts chosen by teacher; (5) Students reflect individually. The second, Revisit, Reexamine, Reflect (RRR) where: (1) Student revisits previously uploaded examples; (2) Reexamines the example for new insights based on his/her evolving understanding of the curriculum; (3) And submits a reflection on his/her new understandings. In the third unit a culminating smart classroom activity was added where students leveraged the collective knowledge base to collaboratively highlight and correct violations of Physics in Hollywood films. During this activity the S3 agent framework responded students work in real-time, altering group configurations, sending specific and timely content to students and the teacher on their tablets, and to the interactive large-format displays around the room.
Findings:
ISLS
439
ICLS2012
While the third iteration is still in progress, findings are encouraging, as students are submitting content to the community, debating and voting on the work of their peers. Examinations of the discourse taking place suggests an evolving understanding of science content and inquiry processes. Several times the teacher used information from PLACE.Web to inform class activities, including using student-submitted examples as a starting point for discussion. Student responses to questions suggest a deeper level of connection to physics principles.
Conclusion
These studies have begun the process of formalizing a set of scripts that successfully engage students in a knowledge community while providing teachers with tools to adjust the scripts in response to emergent ideas within that community. These scripts must take into account both the longer (macro) scripts that are enacted over a curriculum, and consider how they can support (and be supported by) in-class (micro) scripts. Further we have begun to formalize an understanding of the informational needs of the teacher executing these scripts, and the role technology can play in both helping, and hindering, this execution. We are developing an understanding of the important role that a smart classroom infrastructure plays in supporting the orchestration and coordination of real-time knowledge building activities in ways that were simply not possible with traditional pen and paper approaches. These affordances allow students to coordinate their information seeking practices with a group or class towards a common goal. The results from the three studies also promote the role of community voting, debate, and individual reflection around user-created artifacts, as an effective means for developing deeper understandings of the curriculum. Visualizations of this work can also provide powerful means for filtering, sense-making and the re-application of aggregated student work in structured knowledge building activities.
References
Brown, A. L., & Campione, J. C. (1996). Psychological theory and the design of innovative learning environments. Environments, , 289-325. Chi, M. T. H. (2000). Self-explaining expository texts: The dual processes of generating inferences and repairing mental models. In R. Glaser (Ed.), Advances in instructional psychology (pp. 161-238) EBA. Collective, D. B. R. (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5-8. Dillenbourg, P., Jarvela, S., & Fischer, F. (2009). The evolution of research on computer-supported collaborative learning. In N. Balacheff, S. Ludvigsen, T. Jong, A. Lazonder & S. Barnes (Eds.), Technology-enhanced learning (pp. 3-19). Dordrecht: Springer Netherlands. Dillenbourg, P., & Jermann, P. (2007). Designing integrative scripts. Scripting Computer-Supported Collaborative Learning, 275-301. Garrison, D. R. (2003). Cognitive presence for effective asynchronous online learning: The role of reflective inquiry, self-direction and metacognition. Elements of Quality Online Education: Practice and Direction. Johnson, S. D., & Aragon, S. R. (2003). An instructional strategy framework for online learning environments. New Directions for Adult and Continuing Education, 2003(100), 31-43. Livingstone, S. (2008). Internet literacy: Young people's negotiation of new online opportunities. The John D.and Catherine T.MacArthur Foundation Series on Digital Media and Learning, , 101-122. Lui, M., Tissenbaum, M., & Slotta, J. D. (2011). Scripting collaborative learning in smart classrooms: Towards building knowledge communities. Annual Conference on Computer Supported Collaborative Learning Mathes, A. (2004). Folksonomies - cooperative classification and communication through shared metadata. Computer Mediated Communication,1-13. Peters, V. L., & Slotta, J. D. (2010). Scaffolding knowlege communities in the classroom: New opportunities in the web 2.0 era. In M. J. Jacobson, & P. Reimann (Eds.), Designings for learning enviornments of the future international perspectives from the learning sciences (pp. 205-232) Springer US. Roschelle, J., Penuel, W. R., & Shechtman, N. (2006). Co-design of innovations with teachers : Definition and dynamics. Proceedings of the 7th International Conference on Learning Sciences, 606-612. Scardamalia, M., & Bereiter, C. (1996). Adaptation and understanding: A case for new cultures of schooling. International perspectives on the design of Technology Supported learning environments (pp. 149-163) Lawrence Erlbaum Associates. Sherin, M. G., Mendez, E. P., & Louis, D. (2004). A discipline apart: The challenges of 'fostering a community of learners' in a mathematics classroom. Journal of Curriculum Studies, 36(2), 207-232. Slotta, J. D., & Linn, M. C. (2009). WISE science: Web-based inquiry in the classroom Teachers College Press. Slotta, J. D., & Najafi, H. (2010).Technology and learning supports for student collaboration.Cognition,189-196 Slotta, J. D., Tissenbaum, M., & Lui, M. (2011). SAIL smart space: Orchestrating collective inquiry for knowledge communities. Proceedings of the Ninth International Computer-Supported Collaborative Learning Conference, 1082-1083. Ullrich, C., Borau, K., Luo, H., & Tan, X. (2008). Why web 2.0 is good for learning and for research : Principles and prototypes. International Conference on World Wide Web, 705-714.
ISLS
440
ICLS2012
Examining system dynamics models together: Using variation theory to identify learning opportunities in online collaboration
Anindito Aditomo, CoCo Research Centre, The University of Sydney, Australia, and The University of Surabaya, Indonesia Email: aadi4954@uni.sydney.edu.au Kate Thompson and Peter Reimann, CoCo Research Centre, The University of Sydney, Australia Email: kate.thompson@sydney.edu.au, peter.reimann@sydney.edu.au Abstract: This study applies variation theory to examine the discourse of three online groups learning about the structure and behavior of simple population models. Students discussions were examined to reveal whether the discussions provided opportunities to become aware of possible variations corresponding to those critical dimensions. The findings indicated that only in Group 1 did the discussion open variations corresponding to critical dimensions of the model structure and growth patterns. We propose that variation theory is useful as an analytic lens for researchers interested in collaborative learning particularly in online environments, and also as an instructional design tool for teachers in designing collaborative tasks. The goal of this paper is to examine how to identify what the students may have learned (about a given topic) from their online collaboration. The process of online collaborative learning can be analyzed from many perspectives. With regards to making claims about learning from group interaction data, one common approach is to see the collaborative discourse as reflecting cognitive processes, some of which are more germane for learning than others (Hara, Bonk, & Angeli, 2000; Schellens & Valcke, 2006). Another approach to making claims about learning from collaboration discourse is informed by a more socio-cultural view of learning as participating in social practices. This approach is more concerned with the qualities of the discourse at the group level, rather than with learning at the individual level (Kennedy-Clark & Thompson, 2011; Stahl, 2006). These two approaches to analyzing online collaboration discourse have their own limitations. The cognitive approach allows inferences about individual students relative amounts of learning (or potentially germane cognitive processes). However, it provides little insight into what or how students might have learned about the topic. The second approach is useful to highlight group-level processes and properties, but it shies away from making inferences about individual students possible learning outcomes. We propose that variation theory (Marton & Pang, 2006; Marton, Runesson, & Tsui, 2004) offers an approach to analyzing collaboration discourse that complements the two approaches above. Variation theory has been successfully applied to analyze classroom discourse (Ling, Chik, & Pang, 2006; Pang & Marton, 2003; Runesson, 2005), but has not been widely used to examine collaborative learning, particularly in an online learning environment (for exceptions, see Aditomo & Reimann, 2009; Booth & Hulten, 2003). The theory is rooted in phenomenography (Marton, 1981, 1992), which asserts that awareness always has an object, and hence knowing or learning is always knowing and learning about something. Phenomenographic studies have shown that objects (both material and conceptual) can be understood, experienced, or conceived in several qualitatively different ways (Marton & Booth, 1997). Learning, in this view, is becoming able to see, understand, or experience an object in a new and more powerful way. The central tenet of variation theory is that to discern an aspect of something, a person needs to experience variation corresponding to that aspect (Marton & Pang, 2006). It is the awareness of possible variation in certain aspects of an object that enables one to discern those aspects. In this study, we apply variation theory to examine the discourse of three groups learning about system dynamics modeling.
Method
Participants
Data were obtained using a tool called Snooker (Ullman, Peters, & Reimann, 2005) from a postgraduate class. The students worked in groups: Group 1 consisted of two female students and one male student, Group 2 two males and one female, and Group 3 two males and one female.
Topic and task

Students were learning about the structure and behavior of complex systems. Features that distinguish complex systems include emergence, feedback, time-delays, and non-linear cause and effect relationships (for further explanation of these features, see (Sterman, 2000)). The topic addressed in this session was a population model that included the carrying capacity and the models behavior (which was an S-shaped growth pattern). Upon
ISLS
441
ICLS2012
completing the task, students were expected to understand the relationship between the structure of a system and its behavior: that a model with a carrying capacity produces an S-shaped growth pattern. Implicit, but crucial, in this outcome is the understanding that carrying capacity is an emergent property, one which arises out of the interaction of several components of the model: the birth rate, habitat size, density, and death rate. In the example in Figure 1, the carrying capacity is 200; that is where the population stabilizes. This specific carrying capacity is a result of the specific birth rate, habitat size, density, and death rate values in the model; altering any of these would change the carrying capacity in ways that are difficult to predict. Rather than a fixed value, the death rate is formulated as a function of density (i.e. death rate increases as the habitat becomes more dense or populated). Hence, in the first phase of the systems growth, density was low and the death rate was lower than the birth rate. This produced exponential population growth. However, as the population and density rose, the death rate also rose, which slowed the population growth (the system still grew, but not exponentially). When the density reached a certain point (the carrying capacity), the death rate was equal to the birth rate and that is why the population stabilizes. Phrased differently, this type of growth is the result of a non-linear relationship between the systems positive loop (in this case, the birth cycle) and negative loop (in this case, the death cycle).
Figure 1: The deer population model (left) and the S-shaped population growth (right). Students in our course were given a task with the aim that they would learn about several key features of system dynamics models and complexity; features that they could then apply to other models. These were short tasks undertaken in class time. Students were split into groups and asked to discuss three questions, followed by a larger group discussion, facilitated by the lecturer. They were asked to download a simple population model from an external website. In the chat, the students were asked to discuss three questions: (1) This model includes a carrying capacity. What are the implications of this for the behavior of the model? (2) Change the birth rate and death rate in order to find a combination that will result in a decline in the deer population despite unlimited habitat; (3) In real life, there is a limit to` the size of the available habitat. Choose a size of the habitat. What kind of growth does this illustrate? What is the carrying capacity of your habitat?
Analysis and findings

From a variation theory perspective, analysis of learning opportunities starts with the identification of critical aspects of the intended object of learning (i.e. the aspects of the topic which students need to discern, to be able to see it in a way intended by the instructor). The intended object of learning was to understand that a system dynamics model that includes a carrying capacity produces S-shaped growth. To achieve this, students needed to discern two broad aspects simultaneously: structural components of the model, and the corresponding behavior of the model. The structural components were: (1) the death rate formula, which depends on density and area (this is the part of the model which embody the systems carrying capacity), and (2) the relative strengths of the birth and death rates. Aspects of model behavior relevant to this task were: (3) the shape of the population growth (S-shaped vs. exponential), and (4) the phases in an S-shaped growth (increasing in the first phase, and decreasing in the second). Did the students enact the task in a way which made possible the discernment of critical aspects of the topic? (There are issues related to the establishment of a common ground necessary for online communication to occur, but we have analyzed this elsewhere (Reimann, Aditomo, & Thompson, 2009)).
Enactment of Problem 1
This model includes a carrying capacity. What are the implications of this for the behaviour of the model? In discussing this problem, Group 1 discussed the meaning of carrying capacity in a qualitative sense, rather than the formal sense represented in the model. For example, Christine started the discussion by saying the
ISLS
442
ICLS2012
point of carrying capacity is that the land can only support so much life (Line 461), and so if there are too many deer, there wont be enough food for them all and some will die (Line 47). Judy, also offered her general sense of carrying capacity by saying that it have a limition of carrying how much dear in the same area (Line 50), and if the Habitat Area is wilder, then the dear can live longer (Line 60). In this group, only Christine related carrying capacity to the model, by mentioning death rate and density in Line 51, and again mentioning death rate in Line 62. The remainder of Group 1s discussion of Problem 1 (Lines 63 to 76) centered on whether it was better to have a higher or lower carrying capacity. In Group 2, Problem 1 was discussed in the presence of the lecturer (Lines 47-61). As in Group 1, Group 2 focused on the concept of carrying capacity in a qualitative sense. Geoff (Line 48) asked the lecturer what carrying capacity meant, and the lecturer answered carrying capacity is the amount that the system can cope with (Line 53), and provided an analogy with the computer room they often used that has a carrying capacity of X number of students. Geoff indicated he understood, and said that it was straightforward (Line 57). Another member, Teresa, also indicated that she understood (Lines 58). After the lecturer left, the students began Problem 2, indicating that they felt Problem 1 to have been adequately addressed. In Group 3, discussion of Problem 1 began with each member offering their interpretation of the problem. Luke asked, Are we surpposed to explain the model? (Line 31). Anne, responded the first question means how does the carrying capacity of the area affect the model, i think (Line 32). To this, Paul responded with a qualitative definition of carrying capacity: In farming terms we have a carry capacity of so many sheep per acre (Line 34). Anne also offered her definition: yes, so there must be a limit to the number of deers (Line 35). Subsequently, only Luke referred to the model itself, or what it meant for a model to have a carrying capacity. Luke described the relation between density and death: if density is over the carrying capacity, then death rate will rise (Line 36), which in turn lowers density (Line 47). Anne added that in addition to increasing death rate, carrying capacity also decreases birth rate (Line 45), which indicates a potential misunderstanding because in the model, the concept of carrying capacity was embodied in the variable death rate (it is a function of density and habitat size).
Change the birth rate and death rate in order to find a combination that will result in a decline in the deer population despite unlimited habitat. Group 1s discussion started when Frank said that it fixes the death rate, and hence they can only manipulate the birth rate (Lines 79 to 81). Franks statement indicated he was examining the Stella model, unlike Judy, who was using an online simulation. Judy responded by saying she had already obtained a correct combination: just put 3% for birth rate and 50% for death rate (Line 82). The online simulation, which was introduced in a previous task by the lecturer, allowed users to manipulate the death and birth rate in percentage forms (not decimals, as in the model). Furthermore, the simulation did not represent a model with a carrying capacity. This caused obvious confusion, as Christine and Frank tried to understand what Judy meant by 3%. The group began Problem 3 without resolving this communication problem. Hence, for Christine and Frank, but not Judy, addressing Problem 2 enabled the discernment of the difference between constant converters (the birth rate) and graph function converters (the death rate). Group 2s discussion started with Geoffs comment wouldnt we just have to increase the death rate above the birth rate to have a decliing population? (Line 93), which was approved by the other two members (Lines 94 and 95). Geoff continued: because it doesnt really matter how big the area is...if births are less than deaths...or the ratio is leaning that way...its pretty logical (Line 97). Here, Geoff made sense of Problem 2 in a qualitative way; population size is simply a function of the relative sizes of the birth and death rates. This is qualitatively correct, but it trivialized the problem and diverted attention away from the components of the model that formally represented the systems carrying capacity (i.e. the area, density, and death rate converters). Without attending to these components, students would not have the opportunity to discern that the death rate could be represented in different ways (as a fixed value or as a function of density). Furthermore, without examining the density and death rate components, it would be difficult to guess the birth rate that would decrease the population, as Nathan discovered. Approximately 30 seconds later, Nathan said but my numbers dont seem to change the graph (Line 98). This indicates that Nathan was examining the model. This prompted Geoff to think about how density affects the system, as reflected in his next utterance, 40 seconds later: a higher deer density can mean both less food and more protection in numbers from hunters etc (Line 99). This move could have directed the groups attention to how carrying capacity is represented in the model, however the group decided to end their discussion due to time considerations. In Group 3, discussion of Problem 2 started with Anne expressing her puzzlement about the problems suspicious simplicity: i dont understand the second question? because whenever the birth rate is smaller than death rate, it will decline. there must be a catch that i dont get despite unlimited habitat (Lines 52 and 53).
1
Due to space limitations, the data to which this line number refers are not included in the paper, however the line numbers were kept in the paper to allow the reader to have a sense of the order in which statements were made.
ISLS
443
ICLS2012
The discussion then focused on what model should be examined. When they returned to Problem 2, Luke reported that he didnt succeed in lowering the population, despite having specified a very small birth rate (Line 70). Surprised, Anne reported oh. i just put 25% brith and 30 % death - for example- (Line 71). This statement indicated that Anne was examining the online simulation, instead of the Stella model, without a carrying capacity. After Luke elaborated what he was seeing in his model (Line 75; that death rate is determined by density), Anne realized that she needed to examine the model (Line 79), at which point the discussion of Problem 2 ended due to time constraints. Overall, enacting this task enabled Group 3 (or at least the students who examined the model) to discern the model components that represented carrying capacity.
Specify the habitat size (choose a specific value for the Square miles converter), What kind of growth does this illustrate? What is the carrying capacity of your habitat? Only Group 1 had enough time to discuss this problem. The group agreed to try 600 as the habitat size. Christine then said that the system produced an S-shaped growth (Line 114). Judy, on the other hand, said that the population would increase if the birth rate were more than 10% (Lines 112-113). Christine expressed her confusion at Judys prediction (Line 116). This exchange indicates that Judy was still looking at the online simulation, whereas Christine was examining the Stella model. The group ended their discussion without clearly resolving this issue, and without examining the carrying capacity of their system. Did this brief discussion above open a critical dimension of variation? It is possible that the students who manipulated the model were able to perceive the contrast between a model with unlimited habitat (from Problem 2) and one with a limited habitat (in Problem 3), and potentially with the corresponding growth patterns (exponential vs. S-shaped). The students, however, only tried out one habitat size (set to 600, see Line 110). In other words, the habitat size was not varied. Had the students entered other sizes, they would have had the opportunity to see that changes in habitat size correspond to changes in the slope of the S-shaped growth pattern.
Summary of learning opportunities

Table 1 summarizes the learning opportunities that were opened in the groups enactment of the task. If a student stated a variation of an aspect of the topic, or referred to an outcome or event (e.g. model behavior) that requires the discernment of that aspect, these were taken as evidence of a learning opportunity for that student (but not necessarily for other students in the group). Hence, Table 1 lists the names of each student whose utterance provided evidence for the learning opportunity.
Critical aspects of the intended object of learning a. Death rate formula (fixed vs. varying as a function of density and habitat size) b. Relative strength of birth and death cycles c. Population growth (exponential vs. Sshaped pattern) d. Phases in an S-shaped growth (exponential increase vs. decrease) Learning opportunities (variation which enabled discernment of each critical aspect) Group 1 (Christine, Group 2 (Nathan, Group 3 (Luke, Anne) Frank, Judy) Geoff, Teresa) Christine (in Problems 1 & Luke (in Problems 1 and 2), Frank (in Problem 2) 2) Christine, Frank, and Judy Nathan (in Problem Luke and Anne (in (in Problem 2) 2) Problem 2) Christine and Frank (in Problems 2 & 3) -

The groups enactment of the task did not open the complete set of variations necessary to understand the intended object of learning. In discussing the model, Groups 2 and 3 focused on the models structure (aspects (a) and (b) in Table 1), but did not mention any aspect of the models behavior (aspects (c) and (d) in Table 1). Only members of Group 1 discussed an aspect of the models behavior (exponential and S-shaped growth), but not even this group examined the properties of the growth pattern in more detail. The activity did enable students to discern the meaning of carrying capacity in a general sense. For students who examined the model, the activity presented an opportunity to discern the model component that represented carrying capacity: the type of death rate (constant vs. function of density). For students who examined the online simulation, the activity presented an opportunity to simultaneously discern the relative magnitude of death and birth rates, as well as the resulting exponential growth pattern. Students did not have the opportunity to simultaneously discern the critical or defining structural feature of population models with and without carrying capacity, along with their associated growth patterns (exponential vs. S-shaped). The most obvious challenge in this task was that of communication. Establishing adequate common ground is a well-known problem for online chat groups (Reimann, et al., 2009). In this case, the difficulty was
ISLS
444
ICLS2012
due to one member in both Groups 1 and 3 looking at an online simulation, while other members examined a Stella model. Only the Stella model included carrying capacity. Communication problems do not, however, explain Group 2s enactment of the task, in which the evidence suggests that members were examining the same model. We propose that another important challenge for students was concerned with the dual meaning of carrying capacity. This concept can be understood at a general, commonsense level, as members from all groups expressed in this task. However, the objective of the task was to understand carrying capacity in a formal sense, as it was represented in the model; as a property of the models behavior that emerges from the interaction of the models structural components. Group 2 almost exclusively focused on their discussion on the general meaning of carrying capacity, and did not inspect the Stella model. Members of Groups 1 and 3 also expressed this confusion, although in these groups, some of the members did inspect the model. To conclude, using variation theory allowed us to examine online group interactions to make inferences about individual learning opportunities. This gave us useful insights into the design of the task that were implemented in subsequent years. This paper illustrates the utility of variation theory as both an analytic lens and as a conceptual tool for teachers, who often feel that learning theories are too abstract, too complex, while offering few practical guidelines (Yanchar, South, Williams, Allen, & Wilson, 2010). One feature that makes variation theory a potentially practical tool is that it focuses on the subject matter or topic itself, which is something that teachers are familiar with.
References
Aditomo, A., & Reimann, P. (2009). Identifying learning opportunities in online collaboration: A variation theory approach. In S. C. Kong, H. Ogata, H. C. Arnseth, C. K. K. Chan, T. Hirashima, F. Klett, J. H. M. Lee, C. C. Liu, C. K. Looi, M. Milrad, A. Mitrovic, K. Nakabayashi, S. L. Wong & S. J. H. Yang (Ed.), Proceedings of the 17th International Conference on Computers in Education, Hong Kong. Booth, S., & Hulten, M. (2003). Opening dimensions of variation: An empirical study of learning in a Webbased discussion. Instructional Science, 31, 65-86. Hara, N., Bonk, C. J., & Angeli, C. (2000). Content analysis of online discussion in an applied educational psychology course. Instructional Science, 28, 115-152. Kennedy-Clark, S., & Thompson, K. (2011). Patterns of collaborative convergence in a scenario-based multiuser virtual environment. In H. Spada, G. Stahl, N. Miyake & N. Law (Eds.), CSCL2011 Conference Proceedings, Volume 2 (pp. 878-879). Hong Kong: International Society of the Learning Sciences. Ling, L. M., Chik, P. P. M., & Pang, M. F. (2006). Patterns of variation in teaching the colour of light to Primary 3 students. Instructional Science, 34, 1-19. Marton, F. (1981). Phenomenography - Describing conceptions of the world around us. Instructional Science, 10, 177-200. Marton, F. (1992). Phenomenography and "the art of teaching all things to all men". Qualitative Studies in Education, 5(3), 253-267. Marton, F., & Booth, S. (1997). Learning and Awareness. New Jersey: Mahwah. Marton, F., & Pang, M. F. (2006). On some necessary conditions of learning. The Journal of the Learning Sciences, 15(2), 193-220. Marton, F., Runesson, U., & Tsui, A. B. M. (2004). The space of learning. In F. Marton & A. B. M. Tsui (Eds.), Classroom Discourse and the Space of Learning (pp. 3-42). Mahwah, NJ: Lawrence Erlbaum. Pang, M. F., & Marton, F. (2003). Beyond "lesson study": Comparing two ways of facilitating the grasp of some economic concepts. Instructional Science, 31, 175-194. Reimann, P., Aditomo, A., & Thompson, K. (2009). Students engaged in collaborative modeling. In C. O'Malley, D. Suthers, P. Reimann & A. Dimitracopoulou (Eds.), Proceedings of the 9th International Conference on Computer Supported Collaborative Learning - Volume 1 (pp. 28-32). Rhodes, Greece: International Society of the Learning Sciences. Runesson, U. (2005). Beyond discourse and interaction. Variation: a critical aspect for teaching and learning mathematics. Cambridge Journal of Education, 35(1), 69-87. Schellens, T., & Valcke, M. (2006). Fostering knowledge construction in university students through asynchronous discussion groups. Computers & Education, 46, 349370. Stahl, G. (2006). Sustaining group cognition in a math chat environment. Research and Practice in Technology Enhanced Learning. Sterman, J. D. (2000). Business Dynamics. Systems Thinking and Modeling for a Complex World.: McGrawHill Higher Education. Ullman, A., Peters, D., & Reimann, P. (2005). Developing a research supportive web-based learning system. In (Ed.), ODLAA Conference, Adelaide, Australia. Yanchar, S. C., South, J. B., Williams, D. D., Allen, S., & Wilson, B. G. (2010). Struggling with theory? A qualitative investigation of conceptual tool use in instructional design. Educational Technology Research & Development, 58, 39-60.
ISLS
445
ICLS2012
Effects of Computer-Supported Collaboration Scripts on DomainSpecific and Domain-General Learning Outcomes: A Meta-Analysis
Freydis Vogel, Ingo Kollar, Frank Fischer, Ludwig-Maximilians-Universitt Mnchen, Leopoldstr. 13, 80802 Mnchen, Germany freydis.vogel@psy.lmu.de, ingo.kollar@psy.lmu.de, frank.fischer@psy.lmu.de Abstract: CSCL scripts are an effective approach to structure collaborative learning processes in a beneficial way and herewith to foster learning outcomes in various domains. As the amount of research on CSCL scripts grows, it is the right time for a meta-analytical integration of the results. Using a random effects model, this meta-analysis integrated effect sizes of 19 comparisons derived from 13 studies about CSCL scripts. The average effects of CSCL scripts on domain-specific as well as domain-general learning outcomes were estimated. Additionally the meta-analysis targeted the role of additional content support as a potential moderator. Results indicate that CSCL scripts have moderate effects on learning outcomes while additional content support positively influences the effectivity of CSCL scripts on domain-specific learning outcomes.
Introduction
Collaboration scripts for computer-supported collaborative learning (CSCL scripts) are used to support learners acquisition of domain-specific knowledge and domain-general skills (Kollar, Fischer, & Hesse, 2006). As learners are often not spontaneously able to engage in a deep elaborative dialogue while learning collaboratively (Cohen, 1994), collaboration scripts may be used to guide students through a meaningful and beneficial collaborative learning process which may result in deeper individual learning compared to unstructured collaboration (King, 2007). In CSCL scripts, the guidance usually is implemented by distributing roles and activities among the students as well as by sequencing of activities and role switches (Kobbe, et al. 2007). Over the last years a variety of CSCL scripts have been developed and also to some extent been analyzed in empirical studies. In these studies the distribution of roles and activities is mostly implemented in an explicit way by means of assigning and introducing roles directly to the learners (e.g. Ertl, Reiserer, & Mandl, 2005; Schellens, Van Keer, De Wever, & Valcke, 2007) and requesting the learners to perform specific activities that are attached to these roles (e.g. Weinberger, Stegmann, & Fischer, 2010). Sometimes, roles are also induced in an implicit way by creating resource interdependence through the distribution of complementary material among the collaborators (e.g. Molinari, Sangin, Dillenbourg, & Nussli, 2009). Further, CSCL scripts may group learning partners in a way that takes advantage of differences in their prerequisites, e.g. their previous knowledge or their individual attitudes, as proposed by Dillenbourg and Hong (2008). When considering the variety of empirical studies, it also becomes obvious that CSCL scripts have been used in a broad range of domains which provide the content of the domain-specific knowledge the learners are supposed to acquire. The domains range from biology (e.g. Hron, Hesse, Reinhard, & Picard, 1997) over philosophy (e.g. Pfister, Mhlpfordt, & Mller, 2003) to psychology (e.g., Ertl, Kopp, & Mandl, 2006). Beyond domain-specific knowledge, some studies also aim at learners acquisition of domain-general skills. In this meta-analysis, domain-general skills are understood as the learners capability to handle scriptimmanent mechanisms, such as the construction of arguments when a CSCL script guides learners through the sequence of argumentation (e.g. Weinberger et al., 2010). Since CSCL scripts typically have learners repeatedly carry script-immanent mechanisms out, we assume that these mechanisms will be automatized and internalized by the learners so that internal scripts (Kollar, Fischer & Slotta, 2007) are gradually developed (Schank, 1999). In addition, some studies use multi-factorial designs that compare collaboration scripts with other forms of instructional support, more precisely additional content support like content schemes (e.g. Kopp, Ertl, & Mandl, 2006). By definition, CSCL scripts structure interactions on a content-free level and do not give any support regarding the learning content. Yet, learners might also require support to structure the learning content in a meaningful way in order to benefit from the collaborative learning process. Therefore, it is worthwhile to analyze if additional content support can advance the effectiveness of the use of CSCL scripts. Since research on CSCL scripts has been flourishing over the past few years, the time seems right to provide a statistical meta-analysis that integrates the results of the different studies and analyzes the effects of learning with collaboration scripts in CSCL settings on learners domain-specific knowledge and domaingeneral skills, summarizes the single effects by estimating average effect sizes, examines the impact of additional content support as moderator, and critically discusses the effectiveness of the approach. To provide such a statistical meta-analysis is the main purpose of this paper.
ISLS
446
ICLS2012
Research Questions
This meta-analysis is concerned with three research questions regarding the effects of collaboration scripts in CSCL settings on learning outcomes that have been reported in previous studies: RQ 1: What is the mean effect of CSCL scripts on the acquisition of domain-specific knowledge, compared to unstructured CSCL, and to what extent are the integrated effects homogeneous? CSCL scripts usually aim to induce deeper elaboration of domain-specific learning material and herewith foster the acquisition of domain-specific knowledge. Thus, we hypothesize a positive effect of CSCL scripts on domain-specific knowledge compared to unstructured CSCL. As the realizations of the CSCL scripts differ between the included studies in a broad range, a rather high heterogeneity of the effects of CSCL scripts on domain-specific knowledge is expected. RQ 2: What is the effect of combining CSCL scripts with additional content support on domainspecific knowledge acquisition? Collaboration scripts only provide support for collaborative interactions, i.e. they lack content-specific support. The additional use of content support could pre-structure the content of the learning material, so that learners are better capable to examine the material in the way suggested by the script and herewith benefit more from the use of a script regarding their domain-specific knowledge. Therefore, we hypothesize that the use of additional content support maximizes the effect of CSCL scripts on learners domain-specific knowledge. RQ 3: What is the mean effect of the use of CSCL scripts on the acquisition of domain-general skills compared to unstructured CSCL, and to what extent are the integrated effects homogeneous? As collaboration scripts provide learners with the opportunity to repeatedly practice script-immanent domain-general activities, a positive effect of CSCL scripts on domain-general skills is expected as compared to unstructured CSCL. Also for the effects of CSCL scripts on domain-general skills we expect a rather high degree of heterogeneity.
Method
Selection of Studies Criteria for Inclusion
All studies that were included in this meta-analysis met the following requirements: (1) Method: Only empirical studies investigating the effects of at least one experimental condition that involved the use of a collaboration script in a between-subjects design were included in this meta-analysis. In order to distinguish collaboration scripts from other kinds of instructional support, the term script had to be used by the author(s). (2) Dependent variable: To be included, at least one dependent variable needed to be a measure of a learning outcome (domain-specific and/or domain-general) with corresponding parameters reported. (3) Domain: The study needed to be conducted in a CSCL context. (4) Source: To reach a high quality standard, only studies that were published in peer-reviewed journals were included. (5) Language: Only studies written in English or German were included. (6) Sample: For each sample of data only the effects of one published study could be included. If more than one study reported findings based on identical data from an identical sample, only the study reporting the most precise data was included.
Sample of Included Studies

To attain a substantial sample of studies, two steps of data collection were conducted. In the first step data were collected by searching for relevant studies in three different bibliographical databases (OvidSP, ISI and ERIC, all with the same search terms, Boolean operators and search properties ((collaborat* OR cooperat*) AND (computer* OR CSCL) AND (script* OR scaffold* OR structur*) AND learn* within the search fields Title, Abstract, or Keywords). The database search provided an amount of 242 articles. After scanning title and abstract of each article, 122 articles could be identified as being unrelated to CSCL (e.g. articles about scripts in Theater Arts). By reading the remaining articles we filtered 45 articles which were classified as relevant as they were concerned with collaboration scripts in a computer-supported learning setting. In a second step, additional 14 relevant articles were found by scanning the reference lists of the 45 relevant articles,. After inspection of the 59 relevant articles, 46 of them had to be excluded due to one or more criteria for inclusion being violated. Finally, the sample for this meta-analysis includes 13 articles that met all criteria for inclusion. In summary, 1563 learners participated in the studies that are reported in these 13 articles.
ISLS
447
ICLS2012
Variables
This meta-analysis includes three different variables that were coded from the primary studies. As the focus lies on the effects of collaboration scripts on learning outcomes, each primary study provides at least one measure of domain-specific knowledge or domain-general skills. If more than one measure for the assessment of a variable was given, the most general measure was selected to be included into the meta-analysis in order to achieve the most comparable measures (e.g. if the measure for domain-specific knowledge consisted of one test asking broadly for concepts within the whole domain and a second test asking for knowledge about a very specific topic within the domain, the first test would have been selected). Finally, for each sample that was derived from the primary studies the presence of additional content support was coded. In the following, the single variables are described in more detail. Domain-specific knowledge: As domain-specific knowledge, all outcomes were used which assessed learners domain-specific knowledge in a post-test administered after the intervention. The assessment could have been done in an open or closed format and should aim at the knowledge that students were expected to acquire within the domain the CSCL setting was designed for. Domain-general skills: As indicators for domain-general skills, all variables were selected that assessed learners domain-general skills in a post-test accomplished after the intervention. To count as domain-general skills, the used tests were required to aim at knowledge about mechanisms suggested by the script. For instance. this could be knowledge about aspects of good collaboration or knowledge about the construction of argumentation sequences if the script aimed to structure related activities within the collaborative learning process. Additional content support: Some studies reported a multi-factorial design which analyzed the effect of collaboration scripts in combination with a content support (e.g. content schemes, content-related scaffolds). In this case, the studies delivered two different sub-samples, which could be integrated into the meta-analysis. For each sub-sample included into the meta-analysis, the presence of additional content support was coded.
Statistical Analysis
As estimation for the single effect sizes of the primary studies, the unbiased estimate as proposed by Hedges (1981), also called Hedges g (Borenstein, Hedges, Higgins, & Rothstein, 2009, p. 27) was used. To estimate the average effect size d by integration of the single effects, the random-effects model (Borenstein, et al., 2009, pp. 69ff) was used because the features of the studies and the kind of measures for the variables varied between the studies and a common true effect size for all studies could not be assumed.
Results
Research Question 1: Effects of CSCL Scripts on Domain-Specific Knowledge
To estimate the average effect of CSCL scripts on the acquisition of domain-specific knowledge (compared to unstructured CSCL), 19 single effect sizes derived from the 13 selected primary studies were integrated. The integration yielded a moderate estimated average effect (d = 0.36; SE = 0.09; CI90% = [0.21; 0.51]; p < .01, onetailed). As hypothesized, a positive effect of CSCL scripts on domain-specific knowledge compared to unstructured CSCL occurred. Regarding the analysis of heterogeneity of the true effect sizes of CSCL scripts on domain-specific knowledge, as hypothesized, there was considerable variation between the integrated studies with a moderate proportion of real variation within the observed variance (Q(df = 18) = 31.05, p < .01, T = 0.24, I = 42%). As there is a substantial amount of unexplained variance between the integrated studies, it is worthwhile to investigate which moderators are appropriate to explain this heterogeneity.
Research Question 2: Effects of Combining Content Support with CSCL Scripts

To analyze the impact of additional content support on the effects of CSCL scripts on the acquisition of domainspecific knowledge, two subgroups were formed out of the single effect sizes already reported in RQ 1 representing samples with vs. without additional use of content support.. As hypothesized, the estimated average effect of CSCL scripts in the subgroup with additional content support (d = 0.59; SE = 0.16; CI90% = [0.32; 0.86]; p < .01, one-tailed) outperformed the estimated effect of CSCL scripts in the subgroup without additional content support (d = 0.12; SE = 0.16; CI90% = [-0.15; 0.39]; n.s., one-tailed) significantly with a difference of ddiff = 0.47 (SE = 0.23; ZDiff = 2.01; p < .05, one-tailed).
Research Question 3: Effects of CSCL Scripts on Domain-General Skills

Effects of CSCL scripts on domain-general skills were reported in four out of the 13 selected primary studies. To estimate the average effect on domain-general skills, a total of seven single effect sizes derived from the four studies were integrated. As hypothesized, the effect of CSCL scripts compared to unstructured CSCL is positive with a large estimated average effect (d = 1.07; SE = 0.25; CI90% = [0.66; 1.47]; p < .01, one-tailed). Regarding the analysis of heterogeneity of the true effect sizes of CSCL scripts on domain-general skills, the statistics also
ISLS
448
ICLS2012
point to a considerable amount of variation between the integrated studies with a moderate to high proportion of real variation within the observed variance (Q(df = 6) = 15.26, p < .01, T = 0.50, I = 61%), as hypothesized.
Discussion
Generally, the results of this meta-analysis support our hypotheses that CSCL scripts exert positive effects on specific learning outcomes when compared to unstructured CSCL. Regarding the first research question the results indicate that CSCL scripts on average provide more substantial support for learners domain-specific knowledge acquisition than unstructured CSCL. Thus, it can be presumed that CSCL scripts stimulate collaboration processes which are beneficial for domain-specific learning (Dillenbourg & Hong, 2008; King, 2007). Additionally we found quite a high variance of the true effect sizes between studies. This finding shows that there is still a considerable amount of variance left between the integrated studies that cannot be explained by the CSCL scripts. To investigate which variables might explain the remaining variance and moderate the effects, further analyses like the comparison of subgroups that was addressed by RQ 2 are required. The analysis of the impact of additional content support on the effect of CSCL scripts on domain-specific knowledge revealed a positive effect compared to CSCL scripts that are not combined with content-related support. Therefore, the use of additional content support strengthens the effects of CSCL scripts on learners domainspecific knowledge. It may thus be suggested that learning with CSCL scripts can be enhanced by the supplement of additional content support appropriate to the targeted domain. As hypothesized, the meta-analysis also revealed a large positive average effect of learning with CSCL scripts on learners domain-general skills, compared to unstructured CSCL. The repeated practice of scriptimmanent mechanisms should have led to internalizing and automatizing the script (Schank, 1999). This finding supports the approach of using CSCL scripts to foster the acquisition of skills specified within the script itself. This approach is primarily reasonable when the script triggers skills meant to be internalized (e.g. argumentation). Most studies that reported findings on domain general skills, used CSCL scripts for argumentation (e.g. Kollar, et al. 2007; Stegmann, et al. 2007; Weinberger, et al. 2010). This limits the external validity of the finding to some extent. It would be desirable if further studies investigating effects of CSCL scripts on domain-general skills would focus also on other learning-relevant processes beyond argumentation. High variances of the true effect sizes between the integrated studies were found for the effects on domain-specific knowledge and domain-general skills. Therefore, it can be assumed that the different CSCL scripts used in the integrated studies vary in their impact on learning outcomes due to their distinct features. Consequently, further meta-analytical investigation should focus on the identification of features within the CSCL script studies that might moderate the strength of their impact on learning outcomes. One limitation of this meta-analysis lies in its rather strict criteria for inclusion. Certainly, a less strict set of criteria would lead to a larger amount of studies that could be integrated (e.g., studies investigating instructional interventions similar or equal to CSCL scripts that are not labelled as such; e.g. Repman, 1993; Strijbos, Martens, Jochens, & Broers, 2007). Also, studies that are concerned with CSCL scripts without analyzing domain-specific learning outcomes (De Wever, Van Keer, Schellens, & Valcke, 2010; Rummel, Spada, & Hauser, 2009; Schoonenboom, 2007) could be integrated into an enhanced meta-analysis. In addition, several qualitative studies and case studies could not be integrated into this meta-analysis, although they might help to reach a deeper understanding of how CSCL scripts work (e.g. Hmlinen & Arvaja, 2009). Further research should try to find ways to further integrate research on CSCL scripts. From our perspective, this metaanalysis can serve as a starting point for such efforts.
References
References marked with an asterisk (*) indicate that studies were integrated in the meta-analysis Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to Meta-Analysis. Chichester, UK: Wiley. Cohen, G. C. (1994). Restructuring the classroom: Conditions for productive small groups. Review of Educational Research, 64(1), 1-35. De Wever, B., Van Keer, H., Schellens, T., & Valcke, M. (2010). Structuring asynchronous discussion groups: Comparing scripting by assigning roles with regulation by cross-age peer tutors. Learning and Instruction, 20, 349-360. Dillenbourg, P., & Hong, F. (2008). The mechanics of CSCL macro scripts. International Journal of ComputerSupported Collaborative Learning, 3(1), 5-23. *Ertl, B., Kopp, B., & Mandl, H. (2006). Fostering collaborative knowledge construction in case-based learning in videoconferencing. Journal of Educational Computing Research, 35(4), 377-397. *Ertl, B., Reiserer, M., & Mandl, H. (2005). Fostering collaborative learning in videoconferencing: the influence of content schemes and collaboration scripts on collaboration outcomes and individual learning outcomes. Education, Communication & Information, 5(2), 147-166.
ISLS
449
ICLS2012
Hmlinen, R., & Arvaja, M. (2009). Scripted collaboration and group-based variations in a higher education CSCL context. Scandinavian Journal of Educational Research, 53(1), 1-16. Hedges, L. V. (1981). Distribution theory for Glasss estimator of effect size and related estimators. Journal of Educational Statistics, 6(2), 107-128. *Hron, A., Hesse, F. W., Cress, U., & Giovis, C. (2000). Implicit and explicit dialogue structuring in virtual learning groups. British Journal of Educational Psychology, 70, 53-64. *Hron, A., Hesse, F. W., Reinhard, P., & Picard, E. (1997). Strukturierte Kooperation beim computeruntersttzten kollaborativen Lernen [Structured cooperation in computer-supported collaborative learning]. Unterrichtswissenschaft, 25(1), 56-69. King, A. (2007). Scripting collaborative learning processes: A cognitive perspective. In F. Fischer, I. Kollar, H. Mandl, & J. M. Haake (Eds.), Scripting computer-supported collaborative learning - cognitive, computational, and educational perspectives (pp. 13-37). New York, NY: Springer. Kobbe, L., Weinberger, A., Dillenbourg, P., Harrer, A., Hmlinen, R., Hkkinen, P., & Fischer, F. (2007). Specifying computer-supported collaboration scripts. International Journal of Computer-Supported Collaborative Learning, 2, 211-224. Kollar, I., Fischer, F., & Hesse, F. W. (2006). Collaboration scripts - a conceptual analysis. Educational Psychology Review, 18(2), 159-185. *Kollar, I., Fischer, F., & Slotta, J. D. (2007). Internal and external scripts in computer-supported collaborative inquiry learning. Learning and Instruction, 17(6), 708-721. *Kopp, B., Ertl, B., & Mandl, H. (2006). Wissensschemata und Skript - Frderung der Anwendung von Theoriewissen auf Aufgabenbearbeitung in Videokonferenzen [Content scheme and script Facilitating the application of theory knowledge on work on tasks in videoconferencing]. Zeitschrift fr Entwicklungspsychologie und Pdagogische Psychologie, 38(3), 132-138. *Molinari, G., Sangin, M., Dillenbourg, P., & Nussli, M.-A. (2009). Knowledge interdependence with the partner, accuracy of mutual knowledge model and computer-supported collaborative learning. European Journal of Psychology of Education, 24(2), 129-144. *Pfister, H.-R., Mhlpfordt, M., & Mller, W. (2003). Lernprotokolluntersttztes Lernen ein Vergleich zwischen unstrukturiertem und systemkontrolliertem diskursivem Lernen im Netz [Learning with learning protocols - a comparison between unstructured and system-controlled net-based discoursive learning]. Zeitschrift fr Psychologie, 211, 98-109. Repman, J. (1993). Collaborative, computer-based learning: Cognitive and affective outcomes. Journal of Educational Computing Research, 9(2), 149-163. *Rummel, N., & Spada, H. (2005). Learning to Collaborate: An Instructional Approach to Promoting Collaborative Problem Solving in Computer-Mediated Settings. Journal of the Learning Sciences, 14(2), 201-241. Rummel, N., Spada, H., & Hauser, S. (2009). Learning to collaborate while being scripted or by observing a model. International Journal of Computer-Supported Collaborative Learning, 4(1), 69-92. *Schellens, T., Van Keer, H., De Wever, B., & Valcke, M. (2007). Scripting by assigning roles: Does it improve knowledge construction in asynchronous discussion groups? International Journal of ComputerSupported Collaborative Learning, 2, 225-246. Schank, R. C. (1999). Dynamic memory revisited. Cambridge: Cambridge University Press. Schoonenboom, J. (2008). The effect of a script and a structured interface in grounding discussions. International Journal of Computer-Supported Collaborative Learning, 3, 327-341. *Stegmann, K., Weinberger, A., & Fischer, F. (2007). Facilitating argumentative knowledge construction with computer-supported collaboration scripts. International Journal of Computer-Supported Collaborative Learning, 2, 421-447. Strijbos, J. W., Martens, R. L., Jochems, W. M. G., & Broers, N. J. (2007). The effect of functional roles on perceived group efficiency during computer-supported collaborative learning: A matter of triangulation. Computers in Human Behavior, 23, 353-380. *Weinberger, A., Fischer, F., & Mandl, H. (2003). Collaborative knowledge construction in computer-mediated communication: Effects of cooperation scripts on acquisition of application-oriented knowledge. Zeitschrift fur Psychologie mit Zeitschrift fr angewandte Psychologie und Sprache & Kognition, 211(2), 86-97. *Weinberger, A., Stegmann, K., & Fischer, F. (2010). Learning to argue online: Scripted groups surpass individuals (unscripted groups do not). Computers in Human Behavior, 26, 506-515.
ISLS
450
ICLS2012
Volume 2: Posters
Computer Input Capabilities that Stimulate Diagramming and Improved Inferential Reasoning in Low-performing Students
Sharon Oviatt, Incaa Designs, oviatt@incaadesigns.org Kumi Hodge & Andrea Miller, Stanford University, khodge@stanford.edu; afroasianpoet@gmail.com Abstract: This study explores the question: Compared with a non-digital pen, does a digital pen interface have affordances that elicit more diagramming and also more correct inferences about corresponding domain content? Students solved inferential reasoning problems while using: (1) non-digital pen and paper, (2) digital pen and paper interface. They averaged more diagrams, 14% more correct Venn diagrams, and 9.4% more correct domain inferences when using the digital pen interface. An error analysis further revealed that construction of multiple diagrams effectively suppressed scoping errors (e.g. overgeneralizations), the most common inference error. Implications are discussed for designing educational interfaces that improve students visual fluency skills and related reasoning abilities, especially in low performers.
Introduction and Hypotheses

One important objective for the design of future educational interfaces is to stimulate basic thinking and reasoning, including reducing systematic errors that undermine cognition. Recent research has highlighted that digital pen interfaces can substantially facilitate thinking by supporting the expression of different types of representation (e.g., linguistic, diagram, symbol, number). They elicit 44% more nonlinguistic representations, like diagrams, compared with keyboard-based computer interfaces (Oviatt, in press; Oviatt, Miller & Hodge, in press). Diagramming can make information more visually available to think about, especially if it is well matched with a task domain and makes alternatives visually explicit (Bauer & Johnson-Laird, 1993). Past research has revealed that communicating more nonlinguistic content simultaneously stimulates a 36% higher level of appropriate biology hypotheses. Furthermore, knowledge of students level of nonlinguistic communication predicted 72% of the total variance in their idea production (Oviatt, in press). Related findings demonstrated that students active construction of diagrams and marks also were associated with 25-36% higher problem-solution scores, compared with matched problems in which students did no diagramming or marking (Oviatt, in press). These findings support an Activity theory view that communicative activity directly mediates thought, if well matched with a task domain. They also emphasize the importance of designing educational interfaces that support nonlinguistic representations as an aid in thinking, especially spatial ones like diagrams. The current research specifically explores whether: A digital pen will elicit more diagrams and more correct Venn diagrams, compared with a non-digital pen A digital pen will selectively stimulate a higher percentage of correct inferences when students diagram Multiple diagramming will suppress overgeneralization errors, improving inference accuracy
Methods
Participants were twenty-three low-performing English speakers between 18 and 24 years of age, half male and female. All were expert users of keyboard interfaces, but none had ever used a digital pen interface. They solved three-statement inference problems by producing as many valid conclusions as possible, half using an Anotobased digital pen and paper interface and half using a regular pen and paper. Students learned that the digital pen was a computer that could interpret and transmit their writing, and they were shown its features and given practice using it. In one condition, they were told not to make any diagrams while working, and in another they were given a Venn diagram tutorial and instructed to make one or more Venn diagrams before solving the inference problems. Students worked alone and had 3 mins. per problem (for details, see Oviatt et al., in press). Scoring included the total number of diagrams drawn, total correct Venn diagrams, percentage of correct Venn diagrams, percentage of correct conceptual inferences, percentage of correct verbatim inferences (i.e., correct inferences, given the diagram constructed), and inference errors (e.g., overgeneralizations). Interrater reliability averaged 81-99%. Students averaged 50% correct inferences overall (range 21.1%-78.6%).

Compared with using a non-digital pen, when students used the digital pen interface they drew significantly more Venn diagrams per problem by Wilcoxon signed ranks test [x = 1.17 v. 1.25; z = 1.76, N = 9, p < .04, one tailed]. They also constructed significantly more correct Venn diagrams, Wilcoxon [x = 0.82 v. 1.03; z = 2.16, N = 16, p < .016, one-tailed], and a higher percentage of correct Venn diagrams, Wilcoxon [69% v. 83%, respectively; z = 1.84, N = 15, p < .033]. Figure 1 shows these differences in diagramming due to the tool used. Students percentage of verbatim correct inferences averaged 89.2% versus 95.2% when using the regular and digital pen tools, a significant difference by Wilcoxon, z = 1.73, N = 12, p < .042, one-tailed. As predicted and shown in Figure 2, students percentage of correct conceptual inferences during diagram-based
ISLS
451
ICLS2012
Volume 2: Posters
inference tasks also improved significantly from 38.3% to 47.7% when using the digital pen, paired t = 2.53 (df =21), p < .01, one-tailed. No significant change occurred during verbal only tasks.
Figure 1. Total number of Venn diagrams, correct Venn diagrams, and ratio of correct Venn diagrams per total when using the digital pen interface (DP), compared with non-digital pen and paper (PP).
Figure 2. Selective change in percent correct inferences when using digital pen interface for diagram-based inference tasks, but not verbal ones [Note: Error bars shown as 97.5% confidence intervals for the main planned
contrasts; For two-tailed comparison of DP facilitation of diagram vs. verbal inferences, displayed error bars would double]
Constructing multiple diagrams that represent alternative possible worlds can clarify the conclusions that can and cannot be made about information. Students averaged 45.1% correct inferences when they made multiple diagrams, compared with 35.8% for single diagrams, a 9.3% significant difference by Wilcoxon, z = 1.89 (N = 10), p < .03, one-tailed. A least squares regression also revealed that the number of diagrams made predicted the percentage of scoping errors (e.g., overgeneralizations) of the total, R = .48, R2 = .23, F = 6.39 (1, 21), p < .02. That is, constructing more diagrams was confirmed to suppress scoping errors, and knowing the total number of diagrams drawn predicted 23% of the total variance in students percentage of these errors. Table 1 summarizes the digital pens enhancement of correct Venn diagrams, correct verbatim diagrambased inferences, and correct domain inferences. When students had to construct a diagram and use it to reason correctly about a domain, the magnitude of facilitation due to the pen interface averaged a substantial 9.4%. Table 1. Magnitude of enhancement of different types of representation and reasoning abilities related to inference tasks when using the digital pen interface, compared with non-digital pen and paper tools
Type Representation or Reasoning Correct diagram Correct verbatim diagram-based inference Correct domain inference Pen 69% 89.2% 38.3% Digital Pen Difference 83% 95.2% 47.7% +14% +6% +9.4%
References
Bauer, M. & Johnson-Laird, P. (1993) How diagrams can improve reasoning, Psych. Science, 4 (6), 372-8. Oviatt, S. (in press) The Future of Educational Interfaces, Routledge Press, forthcoming in 2012. Oviatt, S., Cohen, A., Miller, A., Hodge, K. & Mann, A. (in press) The impact of interface affordances on human ideation, problem-solving and inferential reasoning, Trans. on Computer Human Interaction, in press.
ISLS
452
ICLS2012
Volume 2: Posters
Alternative Spaces for Engagement: Performance and Conversation in the Connected Classroom
Dr Rachel Perry, Dr Matthew Kearney, University of Technology, Sydney, PO Box 222, Lindfield Email: rachel.perry@uts.edu.au, matthew.kearney@uts.edu.au Teleconferencing is emerging as a potentially emancipative learning technology in higher education. This poster explores the learning journey of one class of teacher education students as they build, prepare and perform a play for primary school pupils across a collaborative space mediated by videoconference facilities. Emphasis will be on the academic and student perspectives of their journey with a consideration of the affordances of this type of technology-mediated activity in drama education.
Introduction
Drama is a compulsory part of the Primary school curriculum and as the connected classroom (teleconferencing) is a new learning technology, it was considered important that associated pedagogical approaches in contexts such as drama education are investigated. The research outlined in this poster took place with a small group of eighteen students studying Childrens Theatre and Creative Arts within their Bachelor of Education degree at an Australian university. The students worked throughout Semester 1, 2011 to explore, build, rehearse and perform an original work to primary school students located in two different schools via the connected classroom (video-conferencing) technology. Half way through semester, the university students moved from developing the early stages of their work on a traditional stage to then working on the connected classroom stage. It was in this transition that we saw the most significant shift in the way students discussed the process being undertaken. It also highlighted unique experiences leveraged by the use of this technology in a Childrens Theatre course. This study is significant in the contribution it makes to an increasingly important body of research focusing on the medium of video-conference mediated teaching and learning in Higher Education and schools (Andrews, 2002; Ardley, 2002; Coventry, n.d.; DEET, 1993; Groundwater-Smith, 2010). It is unique for its emphasis on drama performance, allowing alternative forms of theatre to be explored.
Methodological approaches
The project was structured in three phases as a qualitative study. Data were collected in multiple ways to allow effective comparison and analysis. Data sources included an online survey, focus group, observation in the form of researcher notes and performance video, self and critical collaborative reflection with a critical friend (Bullough & Gitlin, 1991), tutorial notes and follow-up reflections. Questioning focused on four key areas: the teaching and learning process, changing understanding of staging drama performances, understanding and practical issues with video-conference technology, and future implications/applications for student teaching practice. The application of multiple methods for data collection served two purposes in this project. The first was to incorporate techniques that would allow the underlying thoughts and experiences of the pre-service teacher students to be considered and represented in multiple ways. Individuals who may express themselves confidently in writing, for example, may feel more inhibited in a focus group session and visa versa. Applying multiple methods to gain information assisted in the development of a three-dimensional, and hence more realistic, picture of the issues. Second, multiple methods also enabled methodological triangulation (Mason, 1996; Merriam, 1988; Stake, 1994). Data were analysed using qualitative methods and sorted (Coffey & Atkinson, 1996) in multiple ways to allow themes to emerge, but also for the experiences of individual students to be respected (Connelly & Clandinin, 1988; Perry, 2006). Analysis often begins with the identification of key themes and patterns (Coffey & Atkinson, 1996 p26), with preliminary thematic analysis of online responses guiding the emphasis for the focus group session. Themes were again placed as a lens over the newer focus group responses with additional themes emerging. Knowledge construction in education is seen to occur on a storied landscape (Connelly & Clandinin, 1999). The stories of participants in this study will be presented through words and images demonstrating their shifting understanding, confidence and engagement with drama.
Findings
The data demonstrated the primary benefit of the connected classroom technology was its ability to provide a live, immediate experience for students at all levels. As stated by one student in the project, Live and interactive I think that that is the unique feature of the Connected Classroom. That is what it really brings to teaching and learning (Student 11, Focus Group, 8 June 2011). Guiding the presentation of all themes in the poster will be a consideration of what it means for teachers to construct meaning in such a technology-mediated collaborative learning space. Themes relating to the student learning process in developing drama performance as well as general themes will be addressed.
ISLS 453
ICLS2012
Volume 2: Posters
The first three themes presented grew out of the emergent student learning process and subsequent reflections and were triggered by the transition from the traditional stage for performance development, to that of the connected classroom. Immediate feedback on screen during rehearsal. Benefits to the planning process as well as implications for the development of the performances are highlighted. Learning from watching the rehearsal of other students. The natural development of learninggenerated reflections regarding benefits and limitations of the technology, as well as the importance of the real connection experience, became clear over the time in the connected classroom. Recognition of benefits for students What the technology offered in terms of students own learning about drama performance and student engagement, and how this can be applied to the learning of students in schools was emphasised. Analysis of data revealed three further themes as important for consideration within the integration of this new technology into the teaching and learning process. Access Connection (specifically limitations of technology and role of the teacher) Impact of new technology on student response These three areas, along with suggestions made by students for the application of their knowledge in future engagement in the Higher Education setting and their teaching practice, will be addressed in the poster.
Implications and conclusions

The poster will consider implications for practice in three key inter-related areas as identified within data analysis. Notably, university-based teaching and learning using the Connected Classroom, with a particular focus on pre-service teacher education; drama and performance through the Connected Classroom and understanding and management of Connected Classroom technology and implications for use with schools The connected classroom is a unique environment for structuring performance. The poster will raise important implications in regard to developing student awareness of the nuances for drama performance as experienced on this hybrid virtual stage, and what this meant practically for the performers and performances themselves. The idiosyncrasies of the video-conference technology and what that means for Higher Education and connections with schools is also considered.
References
Andrews, T. (2002). Using Videoconferencing for Teaching and Learning. Brisbane: The University of Queensland. Ardley, J. (2002). Unanticipated Findings: Gains by Cooperating Teachers via Video-Mediated Conferencing. Journal of Computing in Teacher Education, 25(3). Bullough, R & Gitlin, A. (1991). Educative communities and the development of the reflective practitioner. In R. Tabachnich and K. Zeichner (dir.), Issues and practices in inquiry-oriented teacher education, (pp. 35-56). London: The Falmer Press Coffey, A., & Atkinson, P. (1996). Making sense of qualitative data: complementary research strategies. Thousand Oaks, California: Sage Publications. Connelly, F. M., & Clandinin, D. J. (1988). Teachers as curriculum planners: narratives of experience. New York: Teachers College Press. Connelly, F. M., & Clandinin, D. J. (Eds.). (1999). Shaping a professional identity: stories of educational practice. New York, NY: Teachers College Press. Coventry, L. (n.d.). Video Conferencing in Higher Education. Edinburgh: Institute for Computer Based Learning - Heriot Watt University. DEET. (1993). Video-conferencing in Higher Education in Australia: An evaluation of the use and potential of video-conferencing facilities in the higher education sector in Australia. Canberra: DEET Higher Education Division. Groundwater-Smith, S. (2010). The Connected Classrooms Program in action: An analysis and evaluation of the qualitative evidence. Sydney: NSW Department of Education and Training. Mason, J. (1996). Qualitative researching. London: Sage Publications. Merriam, S. B. (1988). Case study research in education: a qualitative approach. San Francisco: Jossey-Bass Publishers. Perry, R. (2006). Enhancing the teaching of drama: school-based, needs-driven professional learning. Unpublished Doctor of Philosophy (PhD), University of Technology, Sydney. Stake, R. E. (1994). Case Studies. In N. K. Denzin & Y. S. Lincoln (Eds.), Handbook of qualitative research. Thousand Oaks, California: Sage Publications.
ISLS
454
ICLS2012
Volume 2: Posters
A Comparison of the Reinvestment of Collaborative Asynchronous Discourse Observed by Two Main Actors of Pre-Service Teacher Education
Stphane Allaire, Universit du Qubec Chicoutimi, Canada, Stephane_Allaire@uqac.ca Abstract: Networked communities are growing and they offer new affordances for reflection on practice in education. Few authors have observed how collective asynchronous discourse can be re-used in the classroom. This is the objective that we pursued in research combining a questionnaire filled out by pre-service teachers and in situ observations from their university supervisor. Analyses showed that both the pre-service teachers and the university supervisor recognized that the former used many ideas elaborated online.
Introduction
Since Donald Schons work about the reflective practitioner, reflective analysis has gained importance in education, in particular in pre-service teaching. Such reflection can be individual, but research has also stressed the relevance of its collective aspect (Darling-Hammond & Bransford, 2005). Networked communities are growing and they offer new affordances for reflection on practice (Laferrire, 2005; Lim & Cheah, 2003; Schellens & Valcke, 2006). However, few authors have examined how collective asynchronous discourse produced by such communities can be re-used in the classroom.
Research Question and Objectives

How could pre-service teachers collective asynchronous discourse be reinvested in the classroom? We adopted a comparative perspective between the university supervisor (U.S.) and pre-service teachers to estimate how they perceived each others reinvestment from their own particular point of view. Objective 1: To identify ideas of intervention elaborated by pre-service teachers in a networked learning community. Objective 2: To identify how ideas elaborated online were subsequently used while teaching. Objective 3: To compare the pre-service teachers and U.S.s point of view about how the former reinvested asynchronous discourse in the classroom.
Framework
To identify ideas of intervention elaborated online and during classroom intervention, we referred to the five dimensions of educational intervention that support students learning (Wang, Heartel & Walberg, 1993).
Methods Context of the Study

Our study was conducted with five pre-service teachers (female) who were doing their final practicum, which lasted 14 consecutive weeks. Among formal requirements, reflection on action was stated. In our context, the U.S. asked the pre-service teachers to participate in a networked learning community supported by Knowledge Forum, in replacement of writing individual reflective journals. All five of them accepted. Learning community milestones and knowledge building principles were the major pedagogical concepts that enlightened the collective online dynamic. Pre-service teachers were asked to contribute about three times a week. The U.S. also participated. There were no prescribed topics and no directives were given concerning the requirement to use any of the ideas elaborated by the networked community.
Data Collection and Analysis

Data came from three main sources. The first source comprised of notes written on Knowledge Forum, including those written by the U.S. For each new note, the idea(s) of intervention that was (were) elaborated was (were) extracted and categorized by author. The second source of data was made up of observations made by the U.S. in the pre-service teachers classroom. Four observations (75 minutes each) were completed for each of the pre-service teachers during the 14 week practicum. Finally, the third source of data came from an anonymous online questionnaire in which pre-service teachers had to indicate, no matter the amount of times, which ideas from the collective asynchronous discourse brought them to intervene in such a way. We used qualitative analysis to identify ideas elaborated online with reference to the dimensions of intervention. Next, we matched these ideas with U.S.s observations using a FileMaker Pro database. Descriptive statistics were used to compute the answers to the questionnaire concerning how pre-service teachers said they had used asynchronous discourse in their classrooms. Similar statistics were also utilized to compare the U.S.s observations and the pre-service teachers point of view about ideas that were used while the latter were teaching students.
ISLS
455
ICLS2012
Volume 2: Posters
Results and Conclusion

The analysis of the 124 notes written by the networked learning community, using the five dimensions of our framework, led to the identification of a total of 127 different ideas. Distribution is detailed in table 1. Table 1: Ideas Elaborated by the Networked Community. Dimensions of Educational Intervention Quality of Teaching Classroom Dynamics Classroom Functioning Classroom Direction Quantity of Teaching Number of Different Ideas Elaborated Online 56 36 17 14 4 Examples of Ideas of Intervention Briefly review contents of preceding course at the beginning of class. Verbally encourage students. Give responsibilities to students. Stop speaking to indicate unhappiness with students behaviour in a non-verbal way. Check to see if students are actually working on the assignment that was given to them.
It is noteworthy that the dimension of quality of teaching was, by far, the one that characterized the most the collective discourse elaborated by the networked community. It constituted approximately 43% of all ideas of intervention. Table 2 now presents the reinvestment of these ideas by pre-service teachers while they were teaching students. Table 2: Ideas from the Networked Community Reinvested in the Classroom. Dimensions of Educational Intervention Quality of Teaching Classroom Dynamic Classroom Functioning Classroom Direction Quantity of Teaching Ideas Reinvested Based on Pre-Service Teachers Point of View 32% (N=18) 43% (N=15) 45% (N=8) 44% (N=6) 15% (N=1) Ideas Reinvested Based on University Supervisors Observations 52% (N=29) 47% (N=17) 41% (N=7) 14% (N=2) 25% (N=1)
Overall, we note many resemblances between both actors point of view about ideas reinvested in the classroom. We remark that, from the pre-service teachers point of view, ideas elaborated online that relate to classroom functioning were the most used. More precisely, for 45% of such ideas, at least one pre-service teacher said it was the networked community who brought her to act this way. In counterpart, from the U.S.s point of view, ideas elaborated online that relate to quality of teaching were the most used. Indeed, 52% of such ideas were observed during classroom intervention of at least one of the pre-service teachers. However, the difference between both actors point of view in terms of the dimension of quality of teaching could be explained, in particular, by the fact that pre-service teachers tried to be at their best for this crucial aspect of pupils achievement when the U.S. went to observe them, as his observations contribute to the formal evaluation of the practicum. To conclude, this exploratory research points out the potential of the networked community as a means to enrich pre-service teachers repertoire of intervention in a deliberative way.
References
Darling-Hammond, L., & Bransford, J. (2005). Preparing teachers for a changing world: What teachers should learn and be able to do. San Francisco: Jossey-Bass. Laferrire, T. (2005). Les communauts dapprenants en rseau au bnfice de lducation. Encounters on Education, 6, 5-21. Lim, C. P., & Cheah, P.T. (2003). The role of the tutor in asynchronous discussion boards: A case study of a pre-service teacher course. Educational Media International, 40(2), 33-47. Schellens, T., & Valcke, M. (2006). Fostering knowledge construction in university students through asynchronous discussion groups. Computers and Education, 46, 349-370. Wang, M.C., Haertel, G.D., & Walberg, H.J. (1993). Toward a knowledge base for school learning. Review of Educational Research, 63(3), 249-294.
Acknowledgments
The author would like to thank Qubec Research Funds on Society and Culture for its support.
ISLS
456
ICLS2012
Volume 2: Posters
Effectiveness of combining worked examples and deliberate practice for high school geometry.
Mariya Pachman, John Sweller, Slava Kalyuga, University of New South Wales, Sydney, NSW 2052, Australia Email: korotenkom@yahoo.com, j.sweller@unsw.edu.au, s.kalyuga@unsw.edu.au Abstract: In this experiment more knowledgeable learners who received a set of worked examples aimed at their weak areas for practice (deliberate practice group) outperformed on a final test a group given a choice of their practice problems. This pattern was reversed for less knowledgeable learners. Implications for deliberate practice research as well as the design of worked examples are discussed.
Purpose
Deliberate practice is often characterized as an individualized training regimen with feedback. It is a conscious effortful activity considered to be the most important factor in attaining expert performance (Ericsson & Charness, 1994). However, an experimental investigation of deliberate practice forms and formats in realistic classroom environments has only recently started in well-structured domains (e.g., Pachman, Sweller & Kalyuga, 2011). No investigation of a combination of advanced instructional techniques (such as worked examples) and advanced practice formats (deliberate practice) has been conducted in the field of education. The purpose of the current study was to contribute to learning science by comparing how different formats of practice can accelerate students knowledge formation in well-structured domains and how a combination of instructional methods will influence this final outcome. We also aimed to extend our knowledge of the applications of deliberate practice formats in realistic classroom environments. Specifically, the researchers examined how a practice format aimed at students weak areas combined with worked examples optimizes students acquisition of expertise in high school geometry.
Study hypotheses and research questions

The effectiveness of using worked examples for well-structured domains has been confirmed in a wide range of studies (Cooper and Sweller, 1987; Paas and van Merrinboer, 1994; Renkl & Atkinson, 2003). Using worked examples reduces working memory load and free cognitive resources that the learners could use for a schema development (Sweller, van Merrinboer, & Paas, 1998). Renkl and Atkinson, (2003) saw the value of this instructional format in allowing leaners to gradually automate the subcomponents of worked examples. However, how exactly this automation occurs is still an open question. By freeing cognitive resources, the use of worked examples may encourage learners to choose to practice on harder problems related to their weak, non-automated areas rather than on easier problems. Thus, our research questions were related to practice choices and practicing in ones weak areas: 1. Will a deliberate practice group (a group practicing in their weak areas) outperform their peers on a final test? 2. Will a free-choice group practice in their weak areas or prefer to work in their stronger areas?
Methodology
Sixty nine 8th graders from two Sydney public high schools were randomly assigned to a free-choice group or a deliberate practice group. The participants had instruction and several practice sessions on geometry subtopics relevant to this study prior to the commencement of the study. Materials for the study included a seven-page diagnostic pre-test; booklet of example-problems pairs; feedback sheets and a five-page final test. Worked example-problem pairs and the final test problems covered the same 6 geometry subtopics (area and properties of rectangles, triangles, Pythagoras theorem, complex shapes) as the diagnostics test. Each worked exampleproblem pair and a final test problem required 4 to 5 steps for solution. All the participants were pre-tested on their knowledge of 6 geometry subtopics to determine their areas of weakness using a rapid diagnostic test (Kalyuga & Sweller, 2004). Next, participants in the deliberate practice group were presented with an individualized sequence of example-problem pairs based on their rapid diagnostics assessment results. Participants in the free-choice group were prompted to choose pairs from a booklet that included all possible example-problem pairs presented to the deliberate practice group (14 pairs in total). Following the procedure described in Cooper and Sweller (1987) participants were instructed to study worked example first, and then solve the problem on the next page without looking back at the worked example. Feedback in the form of a worked example (Latu-Manson, 2009) was provided to each participant after solving each example-problem pair. After the practice session, participants knowledge was assessed using a final test containing 6 problems similar to the ones used in the booklet. In summary, during the practice session, the
ISLS
457
ICLS2012
Volume 2: Posters
deliberate practice group was assigned pairs that would cover all their weak areas and only weak areas, while the choice group was allowed to choose their practice problems.
Results
Practice analysis showed that the free-choice group attempted significantly less difficult 4-steps pairs during the practice, t(67) = - 2.82, p = .006 and more easier 3-steps pairs, t(67)=1.97, p=.054. They were choosing pairs covering some (1-2) of their weak areas and some (1-2) of their strong areas and this pattern was significantly different from the deliberate group participants working on their weaknesses only, 2(1, N=69)=34.68, p < .001. We have used a regression analysis for the final test results: learners prior knowledge (1 to 6 weak areas) was included as a continuous independent variable, and the practice condition (deliberate practice vs. free-choice) was included as categorical independent variable. The number of problems solved served as a dependent variable. The regression model was significant, F(3,65) = 2.81, p < .05 and explained 11.5% of variance in the final result. However, we have observed a significant interaction of the learners prior knowledge and the practice format, t(69) = 2.078, p < .05, = .85. The practice format differentially affected less knowledgeable and more knowledgeable learners. To test this difference was we followed with tests of simple effects. For more knowledgeable students (1-3 weak areas, N = 34) the deliberate practice group outperformed the choice group on the number of problems solved t(32) = -2.167, p < .05. For less knowledgeable students (4-6 weak areas, N = 35) the choice group outperformed the deliberate practice group on the number of problems solved t(33) = 1.992, p = .055.
Discussion
Practicing on learners critical areas has yielded significantly higher results on a final test than free-choice practice in regard to the number of successfully solved problems and the number of the correct steps made, but only in regard to more knowledgeable learners. Deliberate practice formats introduced to real classrooms should be used with relatively advanced learners. Weaker learners benefit more from concentrating on a few (one or two) of their weak areas during practice. Given a choice, both, more knowledgeable and less knowledgeable students concentrated on 1 or 2 rather than all of their weak areas during practice. Interestingly enough, this tendency was beneficial for less knowledgeable learners by helping them reduce the number of their weak areas. It was detrimental for more knowledgeable learners performance and further expertise development because they ended up practicing in areas at which they were already competent. To conclude, practice materials should be aligned with leaners current expertise and the design of worked examples for different populations should include narrower or wider subcomponents depending on the targeted population.
Significance of the study

The present study extends the previous research on worked examples and deliberate practice in the following three ways: by a) experimentally manipulating practice formats in realistic classroom environments in order to explore how deliberate practice can be applied to a secondary school classroom; b) investigating how the combination of the advanced formats, such as deliberate practice and worked examples, influences different types of learners and finally, by c) helping better understand cognitive processes behind everyday expertise acquisition activities.
References
Cooper, G., & Sweller, J. (1987). Effect of schema acquisition and rule automation on mathematical problemsolving transfer. Journal of Educational Psychology, 79(4), 347362. Ericsson, K. A., & Charness, N. (1994). Expert performance: Its structure and acquisition. American Psychologist, 49(8), 725-747. Kalyuga, S., & Sweller, J. (2004). Measuring knowledge to optimize cognitive load factors during instruction. Journal of Educational Psychology, 96, 558-568. Latu-Manson, E. (2009). Effect of feedback on mathematics learning. ACER conference, 16-18 August 2009, Perth, AU. Available online at: http://www.acer.edu.au/documents/RC09_Conference-Proceedings.pdf Paas, F., & van Merrinboer, J. J. G. (1994). Variability of worked examples and transfer of geometrical problem solving skill a cognitive load approach. Journal of Educational Psychology, 86(1), 122133. Pachman, M., Sweller, J., & Kalyuga, S. (2011, April). Comparing deliberate and free-choice practice in the acquisition of expertise in geometry. Paper presented at the 2011 annual meeting of the American Educational Research Association (AERA), New Orleans, LA. Renkl, A. & Atkinson, R. (2003). Structuring the transition from example study to problem solving in cognitive skills acquisition: A cognitive load perspective. Educational Psychologist 38(1), 1522. Sweller, J., van Merrinboer, J. J. G., & Paas, F. (1998).Cognitive architecture and instructional design. Educational Psychology Review, 10(3), 251296.
ISLS
458
ICLS2012
Volume 2: Posters
Can Technology-Based Gaze Replays of Experts Model Diagnostic Performance of Novices? A Test in Medical Education
Marko Seppnen, Turku PET Centre, Turku University Hospital, Finland, marko.seppanen@tyks.fi Andreas Gegenfurtner, TUM School of Education, TU Mnchen, Germany, andreas.gegenfurtner@tum.de Abstract: This study examines whether technology-based gaze replays of experts are effective in modeling attentional resources of novices while diagnosing medical visualizations. Contrary to findings in the literature, results reveal that eye movement modeling examples can be used as technology-enhanced instruction for novices. Diagnostic accuracy, sensitivity, and specificity significantly increased after watching the gaze replay. Learners also had significantly more fixations on task-relevant and significantly fewer fixations on task-redundant information. Implications for medical education are discussed.
Introduction
Experts outperform novices in the interpretation of medical visualizations (Gegenfurtner et al., 2009; Helle et al., 2011). One reason for experts superiority is their efficient allocation of attentional resources toward diagnosis-relevant information. By contrast, novices fixate on task-redundant information and may thus miss diagnostically important areas of the visualization (for a recent meta-analysis on expertise differences in the comprehension of visualizations, see Gegenfurtner et al., 2011). As a remedy, experts can model their efficient visual search and detection procedure to novices, for example through eye movement modeling examples (e.g., Litchfield & Ball, 2011; Van Gog et al., 2009). Here, the gaze of experts is first recorded with an eye-tracking device and then superimposed on the screen. This gaze replay can function as a perceptual procedure cue for novices to direct attentional resources from task-redundant to task-relevant information, which ultimately can help students become more accurate in visualization interpretation. Surprisingly, however, the study by Van Gog and colleagues demonstrated that the technology-based gaze replay of experts had detrimental effects on novices. This finding may have resulted from the rather simple type of visualization used: expert gaze replay may thus have provided redundant information. Van Gog et al. (2009) call for testing the effectiveness of eye movement modeling examples with more complex kinds of visualizations, as they are typically found in medicine(Seppnen, 2008). In a response to this call, the present study aims to answer the question whether technology-based gaze replay of experts can model diagnostic performance of novices in medical education.
Method
In this section, we present the methods used to answer our research question. Particularly, we report information on participants and material; procedure; and the measures taken.
Participants and Material

Medical students diagnostic performance in interpreting PET/CT visualizations was compared before and after exposure to eye movement modeling examples. Participants were 18 undergraduate students of medicine (10 women, 8 men, Mage = 21.78 years, SD = 2.15) with no self-reported prior knowledge in interpreting PET/CT. The eye movement modeling example consisted of a 158-sec. digital video. Figure 1 shows a screenshot. The expert was asked to behave didactically during PET/CT diagnosisthat is, the expert was to model the typical way to go about solving tasks of this kind.
Procedure
Participants diagnosed two patient cases: one before and one after the modeling example. There was no time restriction. Participants diagnosed the patient cases in individual lab sessions. Patient cases were displayed in a DICOM (Digital Imaging and Communications in Medicine) standard used in hospitals worldwide, sized 1920 * 1200 pxl, on a 24 TFT monitor. An example is shown in Figure 2.
Measures
Measures included diagnostic performance and eye movements. First, diagnostic performance was assessed with participants written diagnosis. Second, eye movements were recorded with a Tobii T60XL remote eye tracking system with a temporal resolution of 60 Hz; a fixation was defined with a velocity of 35 msec and a distance threshold of 35 pxl.
Results
In this section, we present the results of our analyses in two steps. First, we present pre-post differences on a performance level. Then, we present differences on an eye movement level.
ISLS
459
ICLS2012
Volume 2: Posters
Figure 1. Eye Movement Modeling Example.
Figure 2. Example of PET/CT Patient Case.
Performance
Results of the performance measures showed considerable improvement in diagnostic accuracy after watching the eye movement modeling example, F (1, 17) = 40.95, p < .001, Cohens d = 2.07. Also sensitivity [F (1, 17) = 32.63, p < .001, Cohens d = 1.66] and specificity [F (1, 17) = 4.42, p < .05, Cohens d = 0.59] improved. A more efficient attentional resource allocation may be the source of these improvements.
Eye movements
Results of the eye movement measures showed more fixations on task-relevant areas [F (1, 17) = 56.30, p < .001, Cohens d = 0.89] and fewer fixations on task-redundant areas [F (1, 17) = 12.52, p < .01, Cohens d = 0.99] after modeling. However, it should be noted that improvement in eye movements was not as drastic as improvement in performance.
Discussion
The aim of the study was to test whether technology-based gaze replay of experts can model diagnostic performance of novices in medical education. This research was inspired by previous findings in the literature, indicating that eye movement modeling examples to be detrimental for novice learning (Van Gog et al., 2009). In the present study, watching a technology-based gaze replay of experts had positive effects on novice performance and attentional resource allocation. The different findings on the effectiveness of gaze replay as a training tool between this and previous studies can be explained by the visual material used: the PET/CT visualizations that were utilized here had been considerably more complex in terms of their dimensionality and dynamics, which may have prevented the occurrence of a redundancy effect (Van Gog et al., 2009). In summary, it seems safe to conclude that expert gaze replay is an important resource in interpretation skills training. Although improvement in fixations was not as strong as expected, performance improvement was still considerable. Therefore, eye movement modeling examples can be used as a tool in medical education and training. Future research may extend the first steps reported here to the examination of gaze replays in other medical and non-medical arenas such as, among others, aviation, meteorology, or sports, as well as in the training of observation skills of classroom interaction in the teaching professions.
References
Gegenfurtner, A., Lehtinen, E., & Slj, R. (2011). Expertise differences in the comprehension of visualizations: A meta-analysis of eye-tracking research in professional domains. Educational Psychology Review, 23, 523-552. Gegenfurtner, A., Nivala, M., Slj, R., & Lehtinen, E. (2009). Capturing individual and institutional change: Exploring horizontal versus vertical transitions in technology-rich environments. In U. Cress, V. Dimitrova, & M. Specht (Eds.), Learning in the synergy of multiple disciplines. Lecture Notes in Computer Science (pp. 676-681). Berlin: Springer. Helle, L., Nivala, M., Kronqvist, P., Gegenfurtner, A., Bjrk, P., & Slj, R. (2011). Traditional microscopy instruction versus process-oriented virtual microscopy instruction: A naturalistic experiment with control group. Diagnostic Pathology, 6, S81-S89. Litchfield, D., & Ball, L. J. (2011). Using anothers gaze as an explicit aid to insight problem solving. Quarterly Journal of Experimental Psychology, 64, 649-656. Seppnen, M. (2008). Modern imaging of multiple myeloma. Acta Radiologica, 5, 487-488. Van Gog, T., Jarodzka, H., Scheiter, K., Gerjets, P., & Paas, F. (2009). Attention guidance during example study via the models eye movements. Computers in Human Behavior, 25, 785-791.
ISLS
460
ICLS2012
Volume 2: Posters
Facilitating Teachers Integrated Technological Pedagogical Content Knowledge

Hsin-Yi Chang National Kaohsiung Normal University, Taiwan Email: hsinyichang@nknucc.nknu.edu.tw Abstract: The study reports on how three science teachers develop integrated knowledge over time to guide their students to learn science concepts using technologies. The professional development supporting the teachers employed a co-design partnership. The results indicate better student learning outcomes and more integrated teachers technological pedagogical content knowledge after the professional development in the first year.
Introduction
Modern technologies offer transformative opportunities for instruction and learning. However, teachers need long-term professional development to support them in developing expertise that leads to effective student learning with technologies (Gerard, Varma, Corliss, & Linn, 2011). The professional development of this study established a long-term, co-design partnership (Slotta & Linn, 2009), including science teachers and university researchers working collaboratively to customize science units in the Web-based Inquiry Science Environment (WISE, Slotta & Linn 2009). During the professional development activities teachers and researchers met on a regular basis to customize WISE units, discuss student learning difficulties and instruction dilemmas. This study analyzed how three science teachers who so far participated all of the professional development meetings developed integrated technological pedagogical content knowledge (I-TPCK) over time. Student learning outcomes between the pilot and first-year cohorts were compared to indicate the impact of the teachers professional development on student learning. While continuing to conduct professional development and collect data in the second year, this study proposes an analysis framework that is initially validated by the firstyear data, with the aim to develop an analysis scheme that may be used to formatively assess teachers development of expertise in science instruction using technology.
Theoretical Background
Researchers propose teachers Technological Pedagogical Content Knowledge (TPCK or TPACK, e.g., Jimoyiannis, 2010) frameworks to discern the types of the knowledge necessary for effective technologyenhanced teaching and learning. In addition to the constructs within the TPCK framework such as assessment, learner, representation and pedagogy (Angeli & Valanides, 2009), how to assess the status and development of teachers TPCK is a major issue. Surveys have been developed to measure teachers TPCK confidence (e.g., Graham, Burgoyne, Cantrell, Smith, St. Clair, & Harris, 2009). Surveys provide quantitative results to indicate teachers self-reported confidence but address little teachers knowledge status based on qualitative data such as classroom observations or interviews. This study applies a Knowledge Integration framework (Lee, Liu & Linn, 2011) that has been used to assess students knowledge integration, and proposes an analysis scheme to identify or formatively assess teachers knowledge integration levels of TPCK (Table 1 shows an example). The study has used the scheme to analyze qualitative data including observations and interviews to reveal how the teachers developed their integrated knowledge over time as a result of the professional development.
Methods
The study focuses on three science teachers and their 231 eighth-grade students at two public junior high schools in South Taiwan. The three teachers attended all of the professional development of this study including two summer three-day workshops (with about 15 teachers in total for each workshop) and six discussion meetings during Year 1 (Table 2). The professional development activities engaged the teachers in working collaboratively with three educational researchers to customize WISE units, discuss predicted and actual student learning difficulties and instruction dilemmas, brainstorm teaching strategies and reflect on the implementations. The activities were explicitly designed to foster teachers knowledge integration that research calls for (Gerard et al., 2011). The three teachers are selected to be the focus of this study based on the fact that first, due to their persistent attendance they have become core members of the professional development; second, they are all highly motivated to use technology in their classrooms; third, they have been working on the same WISE unit on thermodynamics; and forth, they have different teaching experiences (4, 8 and 18 years of teaching science, respectively). It is of interest to examine whether the three motivated teachers with different teaching experiences developed integrated TPCK through similar or different trajectories.
ISLS
461
ICLS2012
Volume 2: Posters
Data collected include transcripts of the discussions during the professional development activities, the pre- and post-unit implementation interviews for the three teachers, teacher implementations of the WISE units, and students pre- and posttest scores. An integrated TPCK analysis scheme is proposed, and will be discussed in detail during the poster session, with regard to how the scheme is used to reveal teachers knowledge integration levels at different time points during the professional development phases (Table 1). NVivo is used to facilitate the coding and triangulating processes. Students pre- and posttest scores between the pilot and first-year cohorts of the teachers were also analyzed to indicate the impact of the first-year professional development. The test items are all identical. A multiple regression model was used, with the pretest score, teacher and cohort variables as the predictors, and the posttest score variable is the outcome variable. Table 1: Data collection and time points to assess teachers integrated TPCK (I-TPCK) and students outcomes
PreInterview I-TPCK analysis Students test-score analysis Event 1 N=3 Year 0 Pilot Unit Implementation Event 2 N=3 Pretests Posttests Ncohort0=124 PostInterview Event3 N=3 Three-day Workshop Event 4 N=3 Six PD Meetings Event 5 N=3 Year 1 PreInterview Event 6 N=3 Unit Implementation Event 7 N=3 Pretests Posttests Ncohort1=107 PostInterview Event 8 N=3 Year 2 Three-day Workshop Event 9 N=3
Results
The multiple regression result indicates that all three variables, including pretest, teacher and cohort, are significant predictors of students posttest scores. The model explains 42.1% of variance in students posttest scores. The mean posttest score of the eighth-grade students right after the pilot implementation (Cohort 0 in Year 0) is 9.22 (S.D.=3.60), and the mean posttest score of another cohort of eighth-grade students right after the Year 1 implementation is 12.01(S.D.=5.43). The difference in posttest score between Cohort 0 and Cohort 1, even after controlling for teacher and pretest score differences, was statistically significant. The result provides evidence for more effective teaching and learning with technology from Cohort 0 to Cohort 1. Initial qualitative analysis of the three teachers integrated TPCK at eight points in Year 0 and Year 1 reveals improved levels of integrated TPCK with various trajectories in different aspects. For example, the teacher with 4-year teaching experience demonstrated most integrated TPCK between learner and pedagogy during the postinterview in Year 1. Future professional development will address the teachers need for developing integrated TPCK in assessment and also complex links among the aspects. Trajectories of how the three teachers developed integrated TPCK will be discussed, providing insight into the role of teaching experience and the impact of the professional development activities to facilitate teachers integrated TPCK.
References
Angeli, C., & Valanides, N. (2009). Epistemological and methodological issues for the conceptualization, development, and assessment of ICTTPCK: Advances in technological pedagogical content knowledge (TPCK). Computers & Education, 52(1), 154168. Gerard, L. F., Varma, K., Corliss, S. B., & Linn, M. C. (2011). Professional development for technologyenhanced inquiry science. Review of Educational Research, 81(3), 408-448. Graham, C. R., Burgoyne, N., Cantrell, P., Smith, L., St Clair, L., & Harris, R. (2009). Measuring the TPACK confidence of inservice science teachers. Tech Trends, 53(5), 70-79. Jimoyiannis, A.(2010). Designing and implementing an integrated technological pedagogical science knowledge framework for science teachers professional development. Computers & Education,55(3), 1259-1269. Lee, H.-S., Liu, O. L., & Linn, M. C. (2011). Validating measurement of knowledge integration in science using multiple-choice and explanation items. Applied Measurement in Education, 24, 115-136. Slotta, J., & Linn, M. C. (2009). WISE science: Web-based inquiry in the classroom. New York, NY: Teachers College Press.
Acknowledgments
This material is based on work supported by the National Science Council, Taiwan, under Grant No. NSC992628-S-017-001-MY2. Any opinions and findings expressed in this material are those of the authors and do not necessarily reflect those of the National Science Council. The author would like to thank Fang-Chin Yeh for her help with data management and analysis, and participating teachers for their review of the report.
ISLS
462
ICLS2012
Volume 2: Posters
Use of a CSCW platform by trainers and trainees Trace analysis: multimodal analysis vs data mining approach
SIMON Jean, RALAMBONDRAINY Henri LIM Universit de la Runion, Avenue Ren Cassin 97400, Saint Denis de la Runion, FRANCE {jean.simon ; henri.ralambondrainy}@univ-reunion.fr Abstract: The purpose of this paper is to show that data mining tools can help refine the understanding of what happens on a platform of collaborative work. For this, we compare a multimodal analysis of traces with an analysis of these same traces performed by data mining tools. The second analysis confirms the results of the first one on some points but also gives more information and reveals some dysfunctions in the use of the platform. So, it seems interesting to use such tools to allow a formative evaluation of the training devices.
Introduction
The Reunion Island teacher training school trains the future teachers. Since 2005, trainers and trainees use a platform for computer supported collaborative work (CSCW) to be formed. Trainees and trainers create there and share folders where they deposit and draw various resources which must help them to make class. The objectives for the trainers are: to deposit documents and to serve as collective memory, to improve lesson plans proposed by the trainees, to facilitate preparation of workshop of practice analysis, to pool and share within the framework of the dissertation, to help online and at distance trainees during the training period when they are in charge of a class, to validate the certificate C2i2e which confirms that the trainee is able to use ICT in education. In this paper we compare two analyzes of the 77 folders shared by 15 trainers and 277 trainees, in 2006-2007.
Multimodal analysis of the folders

In 2009 (Simon, 2009), we proposed a first classification of those folders described by 14 numeric variables: number of members, of trainees, of trainers, total of producers, of trainer producers, of trainee producers, of trainer readers, of trainee readers, total of documents, of documents produced by the trainers, of documents produced by trainee, total of readings, of readings made by the trainers, of readings by trainee. This was done through successive analyses of the data collected: Step 1, volume of exchanges (statistical analysis), Step 2, number of producers (social network analysis), Step 3, type of producer: trainer or trainee (social network analysis), Step 4, analysis of the titles of the folders (text analysis). This multimodal analysis
has proposed 6 categories:

Category 1, 7 folders, corresponds to a set of test or error folders because they dont contain any documents. Category 2, 21 folders, corresponds to documents put at the disposal of the group by one trainee producer. They contain lessons plans or sequences or dissertations put on line and subject to the approval of the trainer. Category 3, 14 folders, (one producer) and Category 4, 13 folders, (many producers) are documents put at the disposal of the trainees most often by the trainer: the titles of the documents refer to the discipline and/or the level but also to the group of trainees at the IUFM. Category 5, 20 folders, is the set of folders that must allow trainees to pass the c2i2e. As mentioned, the c2i2e certifies that the student is able to use ICT in the classroom. Normally, all these folders must have the same activity: more or less the same average number of documents, of readings Class 6, 2 folders, is singular by its intense activity. It corresponds to the folders that have been set up to accompany students during their training course and are intended to answer questions "just in time" and "just enough". It is observed that, from category 1 to category 6, the activity in the folders is increasing. We wanted to compare this approach to a more formal approach that uses data-mining methods.
Analysis of the folders by data-mining methods

In a first step we proceed to a quantitative analysis of the folders described by the numerical variables seen above by using data-mining methods (Correspondence Analysis, Ward and k-means algorithms, with the software SPAD). This first study has revealed the salient values for each variable. Some had already been identified by the multimodal analysis, for example one trainer producer by folder, but others not, in particular
ISLS
463
ICLS2012
Volume 2: Posters
the size of the groups associated to folders. After that, to proceed to a qualitative analysis of the folders we defined for each variable six modalities, interval of values, based on the salient values discovered previously. Then we applied again a data-mining approach on the data recoded in those modalities. By this way we obtain 7 classes: The classes 1 and 7 cover categories 1 and 6 of the multimodal analysis, Class 1 "empty folders", 7 folders, which is characterized by a null value on all variables except members. Class 7 "accompaniment during training course", 2 folders, which is characterized by the higher modality on all variables The classes 2,3,4,5 and 6 cover the categories 2, 3, 4 and 5, but there is no one-to-one correspondence between classes and categories: Class 2 "individualized accompaniment, 11 folders, is characterized by the fact that there are only two members per folder: one trainer and one trainee and one single producer, the trainee. These folders correspond to a work requested by the trainer to the trainee or trainee's request for assistance to the trainer. Class 3 "weak cooperation", 12 folders, is disparate. As the folders have mostly several producers, they do not fall within the dissemination classes below, but if we talk about cooperation it should be mentioned that this is low cooperation. It is characterized only by a homogeneous and relatively small number of documents deposited in folders and, to a lesser extent, a small number of members. It contains some folders of the category 5. Class 4 "dissemination to one group", 22 folders, includes on average one producer. Here folders are used to disseminate information to trainees of the group. In fact, as the average number of readings is relatively small, 18, we may question the effectiveness of this dissemination. Class 5 "dissemination to several groups", 4 folders, contains folders that are characterized by a high number of trainees and concern several groups of them. For 3 folders, there is only one producer. They are used to disseminate information. Class 6 "strong cooperation", 19 folders, consists mainly of ICT folders, but it doesn't cover the entire category 5. These folders are used to validate the trainees c2i2e what implies a certain number of exchanges and in particular certain homogeneity in the number of readings by the trainers, in the number of trainee producers and in the number of documents that they deposit. This is a class where there are really production and reading and therefore a strong cooperation, but, however, definitely less than in Class 7. This shows that the analysis performed data-mining methods provides a relevant vision in terms of training and allows distinguishing the different approaches adopted by trainers on the platform and the various uses they make of it. It has led to highlight interesting patterns of behavior as class 2 and class 5 absent from the categorization obtained by the multimodal analysis. For class 2, we see that a CSCW platform can compete with the email usually used in this case. For class 5 the interest is in its opposition with class 4 (diffusion to several groups vs to one group). The diffusion to one group corresponds to a trainer giving documents to his trainees, documents that he has used or will use in his course face-to-face. The diffusion to several groups is a little bit different: it corresponds to documents deposited by a trainer for the trainees for a possible use. In this case the folders are used like databanks. The use of the data-mining tools has also showed that some folders that should have been in "strong cooperation", class 6, are in "weak cooperation", class 3, and, thus, do not satisfy the expected contract. This is the case for some folders used to validate the c2i2e.
Conclusion
From our point of view, one essential contribution of data mining tools is to "objectify" our observation. While the multi-modal analysis approach is more a top-down approach, the data-mining approach is more a bottom-up approach. Romero & Ventura speak about an approach of discovery driven in the sense that the hypothesis is automatically extracted from the data (Romero & Ventura, 2007). The software proposes classes and the researcher must find what the relevance of these classes in his field is. In doing so, he questioned the discrepancy between what it should be and what it is really. Thus, data-mining can be used as a tool for evaluating the work done on the platform. It can lead to improve this device and so to improve the results. It can serve also as feedback to trainers to improve their practice and administrators to improve the system (Romero & Ventura, 2007).
Bibliography
Romero, C., Ventura, S. (2007) Educational Data Mining: a Survey from 1995 to 2005. Expert Systems with x Applications. Elsevier 1:33 (2007) 135-146. Simon J, (2009) Three years of use of a CSCW platform by the preservice teachers and the trainers of the Reunion Island teacher training school, ICALT 09, Proceedings of the 2009 Ninth IEEE International Conference on Advanced Learning Technologies - Volume 00, pp 637-641, Riga, 2009
ISLS
464
ICLS2012
Volume 2: Posters
Social Network Analysis for Knowledge Building: Establishment of Indicators for Collective Knowledge Advancement
Jun Oshima, Yoshiaki Matsuzawa, Ritsuko Oshima, Shizuoka Univ., 3-5-1 Johoku Naka-ku Hamamatsu-shi JAPAN joshima@inf.shizuoka.ac.jp, matsuzawa@inf.shizuoka.ac.jp, roshima@inf.shizuoka.ac.jp Carol Chan, Jan van Aalst, Univ. of Hong Kong ckkchan@hkucc.hku.hk, vanaalst@hkucc.hku.hk Abstract: This study aims at applying social network analysis (SNA) to establishing indicators for collective knowledge advancement. To do so, a SNA application, KBDeX, was developed and the effectiveness of several indicators was examined by applying them to an evaluation of students discourse. Results indicated that SNA with our proposed indicators might quantitatively capture the collective knowledge advancement.
Theoretical Background and Purpose

In educational research on networked learning and computer-supported collaborative learning, there have been discussions on the advantages of using SNA to investigate collective knowledge advancement and individual learners engagement in this advancement (e.g., De Laat, Lally, Lipponen, & Simons, 2007). A limited number of studies have used SNA, especially in the knowledge-creation metaphor (Paavola, Lipponen, & Hakkarainen, 2004). Over a period of three years, Zhang, Scardamalia, Reeve, and Messina (2009) implemented a complementary approach that used SNA to visualize and compare classroom collaboration among fourth grade elementary school students through a CSCL environment designed to support them in knowledge building. An analysis of the students online participatory patterns and knowledge advancement indicated that this learning process facilitated students knowledge advancement effectively, and that this was the case through critical changes in organizations within the classroom: from fixed small groups in the first year of the study to appropriate collaboration through dynamic formation of small teams based on emergent goals. Oshima, Oshima, and Knowledge Forum Japan Research Group (2007) further extended the potential of SNA as a core assessment technique by describing a different type of social network. Ordinary SNA illustrates the social patterns of learners, namely, the learners social network. As de Laat et al. (2007) suggested, this approach is thus informative when examining developments or changes in the participatory structure of a community. However, Oshima et al. (2007) argued that existing social network models are unable to examine how collective knowledge advances through learners collaboration. Instead, they used a similar procedure to that in ordinary SNA, but proposed a different type of social network, one based on the words learners use in their discourse on a CSCL environment. This social network, with words selected as nodes representing learners knowledge or ideas during discourse on a study topic, was compared with a network of words from the discourse of a group of experts on the same topic. The results showed that there were remarkable differences in the collective knowledge of elementary school students and of experts in terms of the words centered on the networks. Oshima et al. concluded that SNA could provide a new type of representation of knowledge building by learners, enabling researchers to adopt a new complementary assessment technique for investigating knowledge building community models. Although educational studies have proposed the application of SNA to learning analysis as a new assessment technique in the knowledge-creation metaphor, an exact methodology has to be established. Currently, researchers familiar with SNA conduct analysis with whatever software is available to them. For those interested in using SNA to analyze discourse data for examining participation patterns and collective knowledge advancement during collaborative learning, SNA software that can easily explore discourse data is needed. As a consequence, we have developed the Knowledge Building Discourse Explorer (KBDeX) software. In this paper, we introduce KBDeX and explain how three different types of discourse-based social networks are represented. Moreover, SNA conducted with KBDeX is demonstrated by using written discourse data in a CSCL environment.
Knowledge Building Discourse Explorer: An Application for SNA of Discourse in Collaborative Learning
To establish a methodology for discourse analysis in collaborative learning from the perspective of knowledge creation metaphor, we are currently developing an application called Knowledge Building Discourse Explorer (KBDeX) (Matsuzawa et al., 2010). KBDeX visualizes network structures of discourse based on a bipartite graph of words discourse units (e.g., conversation turns, BBS postings, or sentences) (Figure 1). Using discourse data (in .csv format) and a list of target words for bipartite graph creation as its input, KBDeX can create visualizations of three different network structures: (1) learners network structure (top right window
ISLS
465
ICLS2012
Volume 2: Posters
in Figure 1), (2) unit network structure (bottom left), and (3) the network structure of the target words (bottom right).
Figure 1. The Interface of KBDeX. KBDeX automatically calculates typical measures for network structures in SNA, (1) the betweenness centrality coefficient, (2) the degree centrality coefficient, and (3) the closeness centrality coefficient. Betweenness centrality is a measure for estimating how each node mediates pairs of other nodes with the shortest path length. High betweenness centrality suggests that the node is a key mediator linking other nodes. Degree centrality is straightforward, showing the extent to which each node is linked to other nodes in the network. High degree centrality means that the node is positioned at the center of the network or a local cluster. Finally, closeness centrality is a more sophisticated measure of how close the node is to other nodes in the network, based on geodesic distances. Based on these three centralities, we demonstrate how the collective knowledge advancement could be measured by using real data sets in a published article (Chan, & Lam, 2010).
References
Chan, C. K. K. & Lam, I. C. K. (2010). Conceptual Change and Epistemic Growth through Reflective Assessment in Computer-Supported Knowledge Building. ICLS2010 Proceedings, 1071-1078. de Laat, M., Lally, V., Lipponen, L., & Simons, R. -J. (2007). Investigating patterns of interaction in networked learning and computer-supported collaborative learning: A role for social network analysis. International Journal of Computer-Supported Collaborative Learning, 2(1), 87103. Matsuzawa, Y., Oshima, J., Oshima, R., Niihara, Y. & Sakai, S. (2010). KBDeX: A Platform for Exploring Discourse in Collaborative Learning. Proceedings of COINs2010: Collaborative Innovation Networks Conference. Oshima, J., Oshima, R., & Knowledge Forum Japan Research Group. (2007). Complex network theory approach to the assessment on collective knowledge advancement through scientific discourse in CSCL. Proceedings of CSCL2007, 563-565. Paavola, S., Lipponen, L., & Hakkarainen, K. (2004). Models of innovative knowledge communities and three metaphors of learning. Review of Educational Research, 74(4), 557576. Zhang, J., Scardamalia, M., Reeve, R., & Messina, R. (2009). Designs for collective cognitive responsibility in knowledge-building communities. The Journal of the Learning Sciences, 18, 7-44.
Acknowledgments
This study was supported by MEXT Japan Grant-in-Aid for Scientific Research on Innovative Areas No. 21300306 (granted to Jun Oshima, Shizuoka Univ.).
ISLS
466
ICLS2012
Volume 2: Posters
Advancing understanding using Nonakas model of knowledge creation and problem-based learning
Meng Yew Tee, Shuh Shing Lee Faculty of Education, University of Malaya,Malaysia {mytee22, lshuhshing}@yahoo.com Abstract: Nonakas model of knowledge creation can provide guidance in designing learning environments and experiences. However, Bereiter is critical of the model because it does not address whether understanding is deepened in the process of socialization, externalization, combination and internalization. To address this issue of understanding, this study proposed a framework that synthesizes the basic phases of problem-based learning with Nonakas model. This design-based study investigated if a course designed based of this synthesized framework can help stimulate knowledge creation that is based on deepening understanding. Based on analysis of multiple data sources, the findings suggest that the participants demonstrated advancing understanding amidst knowledge creating conditions and processes consistent with Nonakas model and the problem-based learning approach.
Framing of this study

Studies (Tee & Karney, 2010; Tee & Lee, 2011) suggest that Nonakas model of knowledge creation (Nonaka, Toyama & Byosiere, 2001) can provide some guidance in designing learning activities that can facilitate knowledge creation. However, Bereiter (2002, p.158-168) was critical of Nonakas model of knowledge creation. He argued that one of the key weaknesses of Nonakas model is that it offers nothing about understanding and depth of understanding (p.161). This is crucial, Bereiter argued, because fundamental understanding is what differentiates from blind luck or serendipitous imitation. He argued that understanding is the crux of expert knowing, and a model that does not explicitly address the issue of understanding is fundamentally flawed. In an attempt to address this issue of understanding, the basic phases of PBL, or problem-based learning, (Bransford & Stein, 2002; Hmelo-Silver, 2004) are built onto the two layers of ba and SECI. According to Hmelo-Silver (2004), the basic premise of PBL is to situate learning in inquiry-based, collaborative, iterative, reflective and self-directed problem-solving contexts. This creates opportunities for learners to focus on an authentic and complex problem and then seek ways to critically evaluate, choose and use emerging knowledge to address the problem. This, in essence, should lay the foundation for deeper understanding. In this regard, this paper reports on a study attempting to understand better how Nonakas model can be used to stimulate knowledge creation, and, if and how deeper understanding is cultivated. With Nonakas model as a backdrop to stimulate knowledge-creating conditions and processes, problem-based learning is used as the basic approach to drive at understanding (Hmelo-Silver, 2004). The knowledge-base in the context of this study is technology, pedagogy and content knowledge, or TPACK (Mishra & Koehler, 2006), a framework built based on Shulmans (1986) seminal work on pedagogical content knowledge. In essence, a teacher who has cultivated advanced TPACK will exhibit a nuanced capability to critically choose or even design and configure, learn, and apply the technologies that will best meet the teaching and learning needs within their context. This research was conducted using the design-based research process. Three types of data were collected, namely: reflections, student-created artifacts, and online discussions. In this study, credibility was addressed with four techniquestriangulation, prolonged engagement, persistent observation, and referential adequacy. The data were coded based on the Nonaka framework and indicators of understandings that emerged from the PBL process. Coding was done by two coders, independently in the first round and collaboratively in the second until consensus was reached.
Advancing understanding by design

The students of this course comprised of 18 in-service teachers, with their ages ranging from mid-20s to early 40s. They taught different subjects at elementary, secondary and tertiary levels. For reporting and discussion purposes, the focus will be on Group A which consisted of three in-service teachers: A1, A2, and A3 who teach language arts (Chinese) in high school. A1 has been teaching for 16 years, with minimal use of technology. A2 has been teaching for 14 years, and has been exploring different technologies with limited success. A3 has been teaching for 20 years, and has never really used technology to enhance learning. The problem they chose to focus on: their students struggle in writing essays in Chinese, a second language for most of their students. The environment (which is known as ba in Nonakas model) was created to enable conditions of autonomy, fluctuation and creative chaos, redundancy, requisite variety, and trust and commitment (due to limited space, this data is discussed in a separate paper). The following table presents the main data and findings:
ISLS
467
ICLS2012
Volume 2: Posters
Table 1: Salient data in relation to problem-based learning, SECI and ba

PBL SECI Highlight of salient data (#1) A2: I do not know where to start because too many challenges and problems in Chinese language teaching. Same problems (are) always discuss(ed) among us (Chinese Language teachers) and no solution(s)we really feel helpless and powerless. (#2) A1: These three weeks, I try power point to teach studentsI thought I am using technology. But I realize that the teacher like me are still repeating the same way of teach(ing).just (that) the teaching material change from picture to computer... (#3) A3: I always think I know the problem of my students but now I start thinking, am I (part of the problem too)? I always believed that I (have) enough experience in teaching (but) the more I learn, the more I am afraid my personal ego has mislead me (#4) A2: Findings show that 90.61% of the error comes from vocabulary error (46% from typo [the Chinese character is written wrongly or is incomplete], 31% from misuse of words [e.g. wrong character but the right sound] and 13% from missing words [e.g. just didnt know how to write the character]). About the first solution: (#5) A2: I seldom refer to other information as a guideI just do it with instinct. I always refer to my own experience while using technology in T& L About the second solution: (#6) A2 :Before I know about TPCK-GPM, I just design my lesson by instinct. . GPM (giving-prompting-making - terms derived from Hammond and Manfra, 2009) open my mind by given some suggestion about activities so that I can make my lesson more diverse (in nature). For example, the second activity our group propose ~ Idioms online games, at first we thought that GPM model only can be use in one way (in one sequential order) but (there)after I know it can be used in other way(s)Ill surely suggest that we follow the sequence M-P-G-M. Let students play the idioms online games first without teaching. After playing the games a few times at home, if they still cant get the correct answer for some questions, then only teacher prompt them by giving them some tips or direct them to get the correct answer through Facebook. Lastly, teacher teaches the meaning of each idiom in class and the students make sentences with the idioms in groups. M-P-G-M model more on training students self-(directed) learning, they will participate and become active in learning process, not just accept passively.
Analysis
In identifying and defining a problem, the socialization process allowed for sharing of feelings and externalization of issues (#1). This is followed by internalization characterized by action (#2) and reflection (#2, #3). A systematic evaluation of 33 student essays led to more advanced understanding of the problem (combination, #4). Articulating awareness of unmethodical bases for decisionmaking (#5). In designing the second activity, the teachers engaged in a more deliberate or methodical combination and internalization process (#6). The group utilized evidence (#4) and research-based practice to resynthesize and design their solutions (#6).
Conclusion
The findings of this study suggest that a problem-based learning approach designed together with a conducive ba to stimulate socialization, externalization, combination and internalization can help teachers deepen their understanding in context of TPACK. They started with simplistic views of how technology itself can transform learning, but over time, began to demonstrate progressing knowledge and understanding of using pedagogical methods and technologies in ways that give the students the best opportunities to achieve the intended learning outcomes. Further studies need to be done to further explicate this synthesized framework.
References
Bransford, J.D., & Stein, B.S. (2002). Ideal problem solver (2nd ed.). New York: W.H Freeman and Company. Bereiter, C. (2002). Education and mind in the knowledge age. Mahwah, NJ: Lawrence Erlbaum Associates Hmelo-Silver, C. (2004). Problem-based learning: What and how do students learn. Educational Psychology Review, 16(3), 235-266. Mishra, P., & Koehler, M.J. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge. Teachers College Record, 108(6), 1017-1054. Nonaka, I., Toyama, R., & Byosiere, P. (2001). A theory of organizational knowledge creation: Understanding the dynamic process of creating knowledge. In Dierkes, Antal, Child & Nonaka (Eds.), Handbook of Organizational Learning and Knowledge (pp. 491-517). Oxford: Oxford University Press. Shulman, L.S. (1986). Those who understand: Knowledge growth in teaching. Educ. Researcher, 15(2), 414. Tee, M.Y., & Karney, D. (2010). Sharing and cultivating tacit knowledge in an online learning environment. International Journal of Computer-Supported Collaborative Learning (ijCSCL), 5(4), 385-413 Tee, M. Y. & Lee, S.S. (2011). From socialisation to internalisation: Cultivating technological pedagogical content knowledge through problem-based learning. Australasian Journal of Educ. Tech, 27(1), 89-104
ISLS 468
I: Identifying problem D: Defining problem E: Exploring solutions A: Act on solutions
ICLS2012
Volume 2: Posters
The Idea Manager: A tool to scaffold students in documenting, sorting, and distinguishing ideas during science inquiry
Camillia Matuk, Kevin McElhaney, Jennifer King Chen, David Miller, Jonathan Lim-Breitbart, Marcia Linn The University of California, Berkeley cmatuk@berkeley.edu, kevin777@berkeley.edu, jykchen@berkeley.edu, david.isaac.miller@gmail.com, breity@berkeley.edu, mclinn@berkeley.edu Abstract: The Idea Manager is a tool integrated into the Web-based Inquiry Science Environment to help students collect and organize their ideas before writing explanations. Findings from its implementation in high school science classrooms show how the Idea Manager supported students in articulating and documenting their developing ideas, and helped identify impacts of curriculum materials on students learning. We describe our design rationale and our planned refinements to the tool, and discuss areas for further research.
The Challenge of Supporting and Assessing Learning

Students have difficulty writing evidence-based scientific explanations and arguments (e.g., Sandoval & Millwood, 2005). Whereas existing scaffolds typically provide templates of standard argument structure that students must complete (e.g., Quintana & Zhang, 2004), the amount of potentially relevant information students encounter, both in school and from their personal experiences, can be overwhelming. As a result, students are likely to retain their initial, often conflicting ideas about science and to require help distinguishing and integrating these into a coherent explanation. By the Knowledge Integration (KI) perspective, when students are first prompted to articulate their prior understanding, they may more thoughtfully evaluate their own ideas against scientifically normative ones (Linn & Eylon, 20011), and ultimately make better predictions and more reasoned arguments (Gunstone & Champagne, 1990). Curriculum units designed in the Web-based Inquiry Science Environment (WISE) and guided by the KI framework have shown promise for helping students develop integrated understandings of science (Linn, Lee, Tinker, Husic, & Chiu, 2006). However, as with other design-based research, it can be challenging to determine how specific features of instructional materials impact learning outcomes, and thus, to identify areas for improvement.
The Design of the Idea Manager

To address these issues, we designed the Idea Manager, a tool integrated in the WISE interface to scaffold and assess knowledge integration. This tool is comprised of two dynamically linked components: the Idea Basket and the Explanation Builder (McElhaney, et al., 2012). The Idea Basket is a repository of students existing and newly encountered ideas. By clicking an icon in the toolbar of the WISE interface, students may type or edit short descriptions of their ideas in the pop-up window that appears. To encourage students to be deliberative in categorizing and refining their ideas, text fields and menu options allow them to specify the source of each entry, to create ad hoc tags, and to flag entries for later consideration. Over the course of the unit, idea entries accumulate into a sortable list, which students may revisit at any time to make additions, deletions, or revisions. Associated with the Idea Basket is the Explanation Builder, an activity type placed at fixed points throughout a unit. Within the Explanation Builder interface, students can drag and drop ideas from their Idea Baskets onto a sorting space. Curriculum authors can customize this space with text and images according to how they wish to guide students organization of ideas relative to the units driving question (e.g., containers labeled For/Against, or Helpful/Not Helpful/Not Sure). A text field within this same interface prompts students to construct verbal explanations based on the spatial organization of their ideas. Because students interactions with the Idea Manager are continuously logged, data show not only what ideas students add, but also the time and point during a unit at which each idea was submitted, revised, and/or deleted, as well as when and where each idea was positioned within the Explanation Builder spaces. In this manner, the Idea Manager complements typical assessments that measure students progress based on fixed points before and after an intervention. In addition to being a scaffold of the KI process, it also functions as a continuous record of students changing repertoire of ideas. Below, we describe how we have used the Idea Manager to research how students integrate their diverse ideas into coherent understanding.
Classroom Implementation
We incorporated the Idea Manager into 3 different WISE units, each 4-10 hours long, and implemented these in several high school classrooms in the western United States. The units guided pairs of students through inquiry
ISLS
469
ICLS2012
Volume 2: Posters
into topics in chemical bonding and planetary motion. As students made predictions, explored dynamic models, and created and interpreted multiple representations, they encountered embedded prompts. These prompts encouraged them to use their Idea Baskets to document ideas, and to distinguish these ideas from one another during pivotal Explanation Builder activities before writing explanations. Analyses of the Idea Manager data revealed a number of insights. For instance, explicit prompts to update their Baskets appeared to encourage students to better attend to the information they encountered. Meanwhile, different activity formats varied in how successfully they elicited students ideas, and in how likely students were to then articulate them in their explanations. However, the content rather than the number of idea entries predicted the quality of those explanations. By tracking the trajectories of students ideas, we were able to identify particular concepts that were difficult for students to understand. This allowed us to identify where in integrating their ideas students might have benefited from more scaffolding, as well as how to refine the units accordingly (McElhaney, et al., 2012). Informed by these and other findings, we plan to explore the advantages of allowing students to share, critique, and build upon their peers Idea Baskets and Explanation Builder responses. We will also continue to investigate how various visual organizations of the Explanation Builder space might scaffold students in distinguishing their many ideas, and ultimately, in constructing better reasoned arguments and explanations.
References
Gunstone, R. F., & Champagne, A. B. (1990). Promoting Conceptual Change in the Laboratory. In E. HegartyHazel (Ed.), The Student Laboratory and the Science Curriculum. New York: Routledge. Linn, M., & Eylon, B. (2011). Science Learning and Instruction: Taking Advantage of Technology to Promote Knowledge Integration. New York: Routledge. Linn, M. C., Lee, H.-S., Tinker, R., Husic, F., & Chiu, J. L. (2006). Teaching and assessing knowledge integration. Science, 313, 1049-1050. McElhaney, K., Matuk, C., Miller, D., & Linn, M. (2012, July 2-6). Using the Idea Manager to promote coherent understanding of inquiry investigations. In ICLS'12: Proceedings of the 10th international conference for the learning sciences, Sydney, Australia, 2012. International Society of the Learning Sciences. Quintana, C., & Zhang, M. (2004). The Digital IdeaKeeper: Extending Digital Library Services to Scaffold Online Inquiry. Paper presented at the American Education Research Association Annual Meeting, San Diego, CA. Sandoval, W.A., & Millwood, K.A. (2005). The quality of students use of evidence in written scientic explanations. Cognition and Instruction, 23, 2355.
ISLS
470
ICLS2012
Volume 2: Posters
Teacher Education Students Research Training and E-Research: Current Perspectives and Potential for Development
Carlos Gonzlez, P. Universidad Catlica de Chile, Vicua Mackenna 4860. Santiago. Chile, cgonzalu@uc.cl Abstract: Research training is part of most teacher education programs. Teachers are required to be reflective practitioners able to conduct small scale investigations. With the emergence of e-research in education, it is important to investigate how research training should incorporate these tools and approaches. Preliminary results from a study on student teachers experience of research training and e-research are presented. Implications for designing research units of study which incorporate e-research tools and approaches are discussed.
Background
Teachers are expected to be critical practitioners who engage in researching their own teaching practice and who are, at the same time, critical research consumers (Cain, Holmes, Larrett, & Mattock, 2007; Berger, Boles, & Troen, 2005). Because of this expectation, most teacher education programs incorporate research training units of study. These units are mostly carried out following social science research traditions. One important recent development in this disciplinary area is the emergence of e-research tools and approaches. Originally developed within science, e-research in social inquiry has been mapped in three broad areas: research organization (for example: e-mail, instant messaging, video conferencing); research processes (for example: survey instrument design, web-based data collection, processing & analysis, visual presentations, dynamic presentations, web archiving, services for outsourcing data collection, analysis & archiving); and scholarly communication (for example: authoring & referencing, blog construction, monitoring & posting, wiki) (Jankowski, 2009; Halfpenny & Procter, 2010). More specifically in education, Markauskaite (2010; 2011) states that e-research has the potential to overcome traditional educational research problems (such as heterogeneity of research traditions and methods, lack of a collaborative research culture, and narrow research dissemination, which limits its impact in policy and practice) through deploying rich data and computation intensive research methods; developing networks which integrate e-research environments for collaborative distributed inquiry; and developing integrated datasets and resources directed to user centred dissemination platforms. One action for deploying eresearch potential is education for e-research (Markauskaite, 2009). This implies that education students should learn about e-research tools and approaches (data mining, collaboration, visualisation, etc.). However, this author recognises that this hardly happens. Therefore, it is important to investigate how are these tools and approaches, if any, are being incorporated into students research training. An initial exploration of this matter through the experience of a group of student teachers enrolled in a research seminar is the focus of this paper.
Method
This study follows a phenomenographic approach. It allows qualitative descriptions of how people experience phenomena in different ways (Marton & Booth, 1997). The sample was purposefully aimed to recruit students from Research Seminar, which is a compulsory unit of study in a secondary school teacher education program at one Chilean University. 20 students participated in the study as interviewees and, in addition, 50 students answered usable open-ended questionnaires. In both cases the following questions were employed: a) what did you learn about research in this unit of study? And b) how did you approach the use of digital research tools available (bibliographic databases, referencing software, analysis software (SPSS, NVIVO) and collaborative writing tool (wiki))? Three qualitatively different experiences of research training and e-research have preliminarily emerged from this analysis. These are presented in the next section.
Results
Category 1: Fragmented ideas about what research is and surface approaches to eresearch
In this category research is conceived of as a set of isolated procedures and practices. There is not a holistic perception where all stages and associated activities are seen as an integrated process with a clear goal. Furthermore, no connection is made between learning to conduct research and their future teaching practice. Eresearch tools were employed in a superficial manner or not employed at all. Some uses were associated with trying to conduct analysis using statistical or qualitative software or trying to find relevant papers in
ISLS
471
ICLS2012
Volume 2: Posters
bibliographic databases. However, the use of these tools was seen as disintegrated from the whole process and as an external imposition to meet unit requirements. Many students did not use of available digital research tools.
Category 2: Advanced ideas about what research is and surface approaches to eresearch
In this category, research is conceived of as an integrated process related to appropriate approaches and techniques. Research is seen as a process to solve or improve situations in schools. It is also seen as a process for creating new knowledge, which may be useful for them, but also for other teachers. In this sense, they see research outcomes not only focused in their immediate environment but also as knowledge to support others practice. In this category, students see a clear link between research training and professional practice. However, e-research tools were employed in a surface manner or not employed at all. Similar to category 1, students tried to use some of the available tools but did not see them as an integrated part of the research project.
Category 3: Advanced ideas about what research is and deep approaches to eresearch
As in category 2, research is seen as an integrated process related to solve problems or improve school situations, with clear links with professional practice. What is different from previous categories is how students approached available digital research tools. They were employed in an integrated way in all stages of the research process. This means they were used to conduct the literature review (bibliographic databases and referencing software), for analysis (through relevant analysis software), and to support collaborative writing (through wiki). In addition, more traditional uses such as e-mail and instant messaging were also employed.
Discussion and conclusion

One important finding of this study is that most students employed e-research tools in a surface manner. This was the case even for the group of students who achieved advanced ideas about research and its importance for teacher professional practice. Even students presenting deep approaches to digital research tools resembled what Markauskaite (2011) describes as adds ons. This means e-research seen as a complement to different stages of the research process rather than an integrated use with the potential of changing the manner research is carried out and disseminated. These results suggest low development, in the context this investigation was carried out, in incorporating sophisticated e-research tools and approaches in teacher education. It implies more reflection and development in this matter is needed for realising e-research potential. In the future, e-research should be embedded in research training. Students should not only learn the available digital research tools, but much should also be done toward fostering a collaborative culture within schools of education for future teachers to be able to participate in knowledge building processes using e-research. This may, eventually, help to overcome current educational research problems related to lack of relevance for practice and researchers isolation.
References
Berger, J. G., Boles, K. C., & Troen, V. (2005). Teacher research and school change: paradoxes, problems, and possibilities. Teaching and Teacher Education, 21(1), 93-105. Cain, T., Holmes, M., Larrett, A., & Mattock, J. (2007). Literature-informed, one-turn action research: three cases and a commentary. British Educational Research Journal, 33(1), 91-106. Halfpenny, P., & Procter, R. (2010). The e-Social Science research agenda. Philosophical Transactions of the Royal Society a-Mathematical Physical and Engineering Sciences, 368(1925), 3761-3778. Jankowski, N. (2009). The contours and challenges of e-research. In N. Jankowski (Ed.), E-research: Transformation in Scholarly Practice. London & New York: Routledge. Markauskaite, L. (2009). Some propositions about eReserach, education and educational research. Accesed October 31, 2011. URL: http://blogs.usyd.edu.au/eresearch/2009/03/some_propositions_about_eresea.html. Markauskaite, L. (2010). Digital media, technologies and scholarship: Some shapes of eResearch in educational inquiry. Australian Educational Researcher, 37(4), 79-101. Markauskaite, L. (2011). Digital Knowledge and Digital Research: What does eResearch offer Education and Social Policy. In L. Markauskaite, P. Freebody & J. Irwin (Eds.), Methodological Choice and Design (Vol. 9, pp. 235-252): Springer Netherlands. Marton, F., & Booth, S. (1997). Learning and awareness. New Jersey: Lawrence Erlbaum Associates.
Acknowledgments
This research is funded by the FONDECYT 11100280 project.
ISLS
472
ICLS2012
Volume 2: Posters
Promoting teacher candidates awareness for teaching dilemmas: A field experiment

Elisabeth Wegner, Iris Kaufmann & Matthias Nckles, Institute for Educational Science, University of Freiburg, Rempartstr. 11, D-79098 Freiburg E-Mail: elisabeth.wegner@ezw.uni-freiburg, iris.kaufmann@web.de, matthias.nueckles@ezw.uni-freiburg.de Abstract: Teachers need to learn that not all aspects of good teaching can be maximized at once. 32 teacher candidates planned and reflected on a lesson in regard to four dilemmatic aspects of teaching. Either they received dilemmatic aspects related to each other by a coordinate system or isolated without relation. Relational planning and reflection reduced simplistic conceptions about demands in teaching and increased awareness for dilemmatic demands, suggesting ways to foster awareness for teaching dilemmas.
Dilemmas in teaching
Can teachers force students to be motivated? Can they teach abstract principles by concrete examples? Can they adapt instruction to students individual needs and treat them equally at the same time? A number of researchers have pointed out that there are several aspects in teaching that are at conflict with each other (e.g. Lampert, 1985; Berlak & Berlak, 1981; Windschitl, 2002), and that teachers need to continuously decide between equally desirable goals, even though deciding for one goal reduces the possibility of reaching another goal. For example, it is impossible to allow for a maximum of self-regulated learning in a learning environment that is at the same time maximally structured. Teacher candidates have been shown to have difficulties to recognize the dilemmatic nature of demands in teaching and to deal with them appropriately (e.g. Schoen, 2005; Levin, 2002; Harrington, 1995). In order to support teacher candidates in handling dilemmatic demands, we developed a planning and reflection tool, in which goals have to be specified within a coordinate system displaying two poles of a dilemma and thus highlighting the interrelation between two dilemmatic aspects. We expected teacher candidates who used the two-dimensional coordinate system a) to plan more realistically (i.e. not trying to maximize both aspects at the same time) and b) to become more sensitive towards the dilemmatic nature of teaching, as compared to a control group of teacher candidates who planned and reflected on the same dimensions separately.
Methods
A total of 32 teacher candidates in their first year were asked to plan and reflect on a lesson in regard to four dilemmas of teaching. Participants of the relational condition (N=16) received a planning and reflection guideline in which the conflicting dimensions were displayed in coordinate fields. Participants of the isolated condition (N=16) received the same dimensions without references to the interdependence of the dimensions. The following aspects were addressed in the guideline: 1. Dilemma of self-regulation: fostering self-regulation vs. structuring the learning process 2. Heterogeneity dilemma: individualized teaching vs. reaching common goals 3. Dilemma of professional relationship: authority vs. proximity to students 4. Dilemma of didactic structure: problem-centred approach vs. systematic approach After conducting the lesson, participants were asked to indicate how they judged their lesson in regard to these dimensions. Both groups were then given prompts to reflect on what consequences they would draw from the lesson and to reflect on supporting and hindering factors. After this reflection we assessed a) sensitivity to the dilemmatic nature of teaching (measured by a rating of the perceived difficulties of several aspects of teaching), b) conceptions of demands in teaching (measured by a questionnaire consisting of the scales simple demands, subjective demands, complex demands, and contradicting demands), c) subjective learning outcomes (measured by a questionnaire).
Results
We found evidence indicating a raised awareness for the dependence of dilemmatic dimensions in the relational condition: Participants in the relational condition were less likely to plan both aspects of the dilemmas at a maximum. They only planned on average in 0.50 (SD=0.65), while participants in the isolated condition planned on average in 1.44 (SD=1.09) out of four dilemmas both aspects at a maximum. After conducting the lesson, participants in the relational condition indicated in 0.50 (SD=0.76) and in the isolated condition in 1.31 (SD=1.40) of the dilemmas that they had fulfilled both aspects at a maximum. A mixed repeated measures analysis of variance with the number of not dilemma-sensitive plannings vs. number of not dilemma-sensitive judgment as a within-factor and experimental condition as a between-factor (F(1,28) = 6.34, p = .018, 2=0.185) showed a significant main effect of the experimental condition. Also, we correlated scores of contradicting
ISLS 473
ICLS2012
Volume 2: Posters
dimensions of each dilemma. Correlations between the dimensions were negative, and the correlations were significantly higher in the relational condition than in the isolated condition, indicating that participants in the relational condition were prompted to think about contradicting dimensions in relation to each other. Also, we found evidence for a raised awareness for the dilemmatic nature of teaching in general: Participants in the relational condition rated dilemma-related difficulties as more challenging than technical difficulties, whereas the participants in the isolated condition rated technical difficulties higher than dilemmarelated difficulties (see Fig. 1). A mixed repeated measures analysis of variance with the ratings on technical difficulties vs. dilemma related difficulties as a within-factor and experimental condition as a between-factor showed a significant interaction effect (F(1, 30) = 12,11; p = .002; 2 = .288). There were no significant differences between the conditions in regard to the questionnaire on conceptions of the demands in teaching, as well as in regard to perceived learning outcomes. However, a hierarchical regression analysis showed that those in the relational planning condition, who perceived a large learning outcome, stated to a lesser extent that demands in teaching were simple and clear (see Fig. 2). No effects were found in regard to the other scales (subjective demands, complex demands and contradicting demands).
3.7 3.5 3.3 3.1 2.9 2.7 2.5
isolated condition
0.6 0.4 0.2 0.0 -0.2 -0.4 -0.6 -0.8 -1.0
relational
Simple demands
low learning outcome
high learning outcome relational planning isolated planning
technical difficulties
dilemma related difficulties
Figure. 1. Rating of technical difficulties (i.e. time management, preparation of material) and dilemma related difficulties (i.e. dealing with heterogeneity, deciding for goals, dealing with contradictions)
Figure. 2. Moderator effects of perceived learning outcomes on the scale simple demands (high learning outcome = +1.00 SD, low learning outcome = -1.00 SD)
Conclusion and Discussion

We found evidence that planning and reflection of dilemmatic aspects in a coordinate system raised awareness a) for the problem that the two sides of a teaching dilemma cannot be maximized at the same time and b) for difficulties stemming from dilemmatic demands. The effect of this kind of relational planning and reflection was mainly found in regard to measures indicating ideas of simplicity in teaching, suggesting that relational planning and reflection helps to reduce overly simplified ideas of teaching. The reduction of simplistic ideas on demands in teaching is one major prerequisite to deal reflectively with contradicting demands. Therefore, we propose that planning and reflection guidelines for teacher candidates should include prompts that help teacher candidates to think actively about contradicting aspects of teaching in relation to each other. This might be a small, but helpful intervention.
References
Berlak, A., & Berlak, H. (1981). Dilemmas of schooling: Teaching and social change: London: Routledge. Harrington, H. L. (1995). Fostering reasoned decisions: Case-based pedagogy and the professional development of teachers. Teaching and Teacher Education, 11(3), 203-214. Lampert, M. (1985). How do teachers manage to teach? Perspectives on problems in practice. Harvard Educational Review, 55(2), 178-195. Levin, B. B. (2002). Dilemma-based cases written by preservice elementary teacher candidates: An analysis of process and content. Teaching Education, 13(2), 203-218. Schoen, L. (2005). Learning to make sense of the dilemmas of teaching practice: An exploration of pre-service teachers. Online Submission Journal Citation: Ph.D. Dissertation. Boston: Boston College. Windschitl, M. (2002). Framing constructivism in practice as the negotiation of dilemmas: An analysis of the conceptual, pedagogical, cultural, and political challenges facing teachers. Review of Educational Research, 72(2), 131-175.
ISLS
474
ICLS2012
Volume 2: Posters
The Candy Factory Game: An Educational iPad Game for Middle School Algebra-Readiness
Michael A. Evans, Department of Learning Sciences & Technologies, Virginia Tech, mae@vt.edu Anderson Norton, Department of Mathematics, Virginia Tech, norton3@math.vt.edu Kirby Deater-Deckard, Department of Psychology, Virginia Tech, kirbydd@vt.edu Mido Chang, Department of Education Research and Evaluation, Virginia Tech, mchang@vt.edu Abstract: The Candy Factory, an app developed for the iOS platform targeting iPads, is an educational game intended to prepare middle school students for algebra-readiness. The Candy Factory differs from existing educational games along three dimensions: 1) the app is designed following evidence-based theories of cognition and engagement; 2) the app scaffolds learners to approach fractions with deep understanding; and 3) the app leverages game mechanics and device hardware for formative assessment purposes.
Overview
According to the U.S. National Mathematics Advisory Panel (2008), students development of measurement concepts for fractions marks a critical point in their progression toward algebra-readiness. Most grade 6 students in the United States commonly rely on part-whole conceptions alone. Conversely, students who learn to coordinate partitioning and iterating to produce fractional sizes go on to construct splitting operations, developing a distinct advantage for algebra-readiness (Hackenberg. 2010). Thus, a major goal of our efforts is to support the construction of splitting operations by designing educational games that require students to coordinate partitioning and iterating operations in goal-directed activities. In our session we describe the underlying theories of our work, the hypothetical learning trajectories and engagement states that drive the design and development process. Below, we provide an overview of the game mechanics of the Candy Factory Game, an educational game developed to motivate grade 6 students (approximate ages 10-12) to engage in deep understanding of fractions. Next, we share data and results from several rounds of pilot tests using teaching experiment, observational techniques, and interview protocol in a mixed-methods design (Creswell & Plano Clark, 2007). We conclude by suggesting how the Candy Factory Game contributes to the learning sciences knowledge base on educational mathematics games by leveraging theories of cognition and engagement to deliver a simple, yet purposeful game, to help struggling youth learn fractions.
The Candy Factory Game: Supporting Splitting Operation Via Play

We have developed a functional prototype, the Candy Factory Game, to run on iOS devices targeting primarily iPads, but available for iPod Touches and iPhones. The educational game is designed to engage students in goaldirected activity that elicits potentially novel uses of existing mental operations, specifically partitioning and iterating. Students are shown a part of a candy bar (customer order) and asked to reproduce that candy bar by partitioning a whole candy bar into an equal number of parts and then iterating one of those parts the appropriate number of times. At Level 1, the customer order and the whole candy bar contain partitioning marks so that students can employ existing part-whole concepts to make sense of the task. Furthermore, at Levels 2 and 3, those candy bars are truly continuous, requiring students to estimate the relative sizes of the bars, using partitioning and iterating operations with the games drag and drop interface. See Figure 1.
Figure 1: Level 1 screen captures in the Candy Factory Game (l: iterating operations; r: player reflection screen).
Results from Teaching Experiments and Survey Pilot Tests

We began testing the educational game in Fall 2011, using whole-class observations of two sixth-grade classes. Classroom observations involved 30-minute weekly episodes in which students paired up to solve game tasks,
ISLS
475
ICLS2012
Volume 2: Posters
requiring them to produce fractions of a specified size from a given whole. Whereas about half of the students started the semester using only part-whole reasoning with fractions, we found that virtually all students began coordinating partitioning and iterating operations in the intended way within the first five weeks (Authors, in preparation). We also conducted intensive, videotaped interviews with one pair of students after school, to characterize such changes. The two students were chosen because a written assessment indicated that they lacked measurement concepts at the beginning of the semester. Indeed, the first clinical interview provides evidence that neither student could formulate appropriate estimates of fractional sizes. For example, one student guessed 2/5 for a piece (customer order) that was clearly larger than one half. However, the students began to correct such errors when they noticed that they needed more iterations of a 1/5 (or another unit altogether) to fit the customer order. In later interviews, the students even began providing alternate names for the same fractional size and demonstrating their equivalence. We also have developed a protocol for quantifying individual differences in students engagement with the device and game. Working in the same classroom, we observed four pairs and four individuals using the device and game in 10-minute periods. We also collected field notes from observing students working in pairs in the classroom. From these data, we identified variance between students in the following aspects of behavior and emotion that now comprise the dimensions of our new observational measurement tool. For individuals, these include attention, persistence, positive affect, frustration, gross/fine motor, touching device, aggression, verbalization, intrusiveness, responsiveness, and autonomy. For dyads, this includes cooperation, competition, conflict, and reciprocity. In addition, we piloted our math engagement instrument that measures the three domain of mathematics engagement (cognitive, behavioral, and affective). The instrument is comprised of 16 items that ask students to rate their own mathematics engagement level on five-point Likert scale scores (1=Really Disagree; 5=Really Agree). We collected data from 18 fifth-grade students using the hard copy of the instrument from one classroom. The data were analyzed for the item quality and reliability of the instrument using jMetrik (Meyer, 2011). The analysis results show that Cronbach Alpha (item reliability) was defensibly high (0.844), indicating that all the items show high consistency each other. The current version of The Candy Factory Game app prototype focuses on the core mechanics of supporting partitioning and iterating operations. In addition, the application will need to support social elements such as allowing students to share and compare scores. The full version will include data collection and reporting features to support the research plan for proficiency and engagement.
Conclusion
What are the video game elements that youth find engaging as they interact with pre-algebra topics? According to Jones (1998) video game features that attract youths interests include: graphics, music, visual effects, and interesting animations. Nevertheless, one of the most engaging features is a challenging problem to solve with scaffolds that propel players to a solution (Hoffman & Nadelson, 2010). In this way, video games provide sufficient challenge and guidance to be engaging and are a sought-after enhanced learning technology. In this proposal, framed as a mixed-method design, we have documented the theoretical and evidence-based approach to developing and testing an educational game, the Candy Factory Game. Pilot results suggest that the video game prototype is engaging and that core mechanics propel learners along the predicted hierarchical learning trajectory. Nevertheless, these findings come from a limited set of interventions from a distinct sample. Thus, we continue to revise the video game based on pilot tests, end-user feedback, and refined instruments and observation protocols. To that end, we are now in the process of developing the Candy Factory v2.0 (Figure 1), which enhances the game experience intended to heighten engagement while attending to the fundamental requirement to scaffold requisite mental actions required for algebra-readiness in middle school and beyond.
References
Hackenberg, A. J. (2010). Students reversible multiplicative reasoning with fractions. Cognition and instruction, 28(4), 383-432. Hoffman, B. & Nadelson, L. (2010). Motivational engagement and video gaming: a mixed methods study. Educational Technology Research and Development, 58, 245-270. Jones, M. G. (1998, February). Creating electronic learning environments: Games, flow and the user interface. In Proceedings of selected research and development presentations at the national convention of the association for educational communications and technology (AECT), St. Louis, MO. Meyer, J. P. (2011). jMetrik (Version 2.0). Charlottesville, VA: University of Virginia. National Mathematics Advisory Panel. (2008). Foundations for success: The final report of the National Mathematics Advisory Panel. Retrieved from http://www.ed.gov/about/bdscomm/list/mathpanel/report/final-report.pdf
Acknowledgments
Reported research is supported by the National Science Foundation (DRL 1118571).
ISLS
476
ICLS2012
Volume 2: Posters
Evaluating Claims in Popular Science Media: Nature of Science Versus Dynamic Epistemological Knowledge
Pryce Davis, Rosemary Russ, Northwestern University, 2120 Campus Dr., Evanston, IL 60208 Email: pryce@u.northwestern.edu, r-russ@northwestern.edu Abstract: A substantial body of research aims to uncover and advance peoples understanding of the nature of science (NOS). We suggest that an exclusive focus on NOS knowledge, as defined by this research, systematically misrepresents when and how people evaluate science claims. In particular we claim three tacit assumptions that NOS research supports about knowledgedomain specificity, coherence, and decontextualizationlimits its ability to uncover the broad range of ideas people regularly use when reasoning about scientific claims. We draw from interviews with adults as they watch science-related reality TV to illustrate an alternative approach to research that describes how people leverage a set of dynamic epistemological resources for making sense of science claims embedded in popular media. From the side effects of vaccines to the benefits of stem cell therapy, scientific knowledge claims bombard people every day. As a result, developing peoples knowledge evaluation skills is of vital importance. Over the last 50 years, a substantial body of research has argued an understanding of the nature of science (NOS) what science is, how it works, how scientists operate as a social group (McComas, Clough & Almazroa, 1998, p. 4)is essential to knowledge evaluation (e.g. Lederman, 1992). A common theme of this work is that the foundation of many illogical decisions and unreasonable positions are misunderstandings of the character of science (McComas, Almazroa & Clough, 1998, p. 511). In this work we suggest that an exclusive focus on peoples understanding of the nature of science as theoretically and methodologically defined in NOS studies systematically misrepresents when and how people evaluate claims. We suggest three tacit assumptions that NOS research supports about knowledgedomain specificity, coherence, and decontextualizationlimits its ability to uncover the range of knowledge that people use in reasoning about claims. Below we discuss these three assumptions and provide illustrative examples from interviews conducted by the first author with 12 adult participants in which they viewed science-related reality TV shows (Mythbusters or Ghost Hunters). In each of these shows, the actors present themselves as scientists investigating a claim. After watching one of the shows, the interviewer asked the participants to evaluate knowledge claims, categorize the actors as scientists, re-evaluate claims based on categorization and make a personal decision related to the claims presented in the show. These interviews provide a context in which to explore how people evaluate scientific claims in their everyday lives, and allow us to demonstrate an alternative paradigm for examining peoples evaluation of knowledge claims. In particular, we describe how people leverage a set of dynamic epistemological resources for making sense of science claims embedded in popular media.
Assumption 1: Domain Specificity of Knowledge

NOS researchers tacitly assume that people only draw on knowledge that is ostensibly about science when reasoning about scientific claims. This assumes that from a participants point-of-view, science knowledge can be considered distinct from other forms of knowledge. We believe that treating the nature of science as a distinct form of knowledge obscures other relevant knowledge that people draw on to evaluate claims. We favor focusing on personal epistemology the set of beliefs people hold about the nature of knowledge itself. Epistemology is more general than NOS and deals with knowledge as it exists in all domains, not just science, and how it relates to ideas like belief, truth, and justification. As an example, we use the case of Bradford who does not believe the claims made by the Ghost Hunters, because they are untrustworthy: I think its almost just a subjective experience and then things they just try to bend into how they want to fit it into the story They're not using any really objective measures of things. Bradford draws on epistemological beliefs about subjectivity versus objectivity. This mirrors findings from NOS research that novices view science as a search for objective truth and emphasize empiricism over subjective interpretation. However, Bradfords activation of disbelief and untrustworthiness is appropriate and expert-like in this case; the subjectivity comes from a lack of objective measures of things. The Ghost Hunters rely on cold chills and mysterious noises - a subjectivity that even NOS researchers would agree is unscientific. Bradford does not base his evaluation on blanket views of the domain of science; instead he attends to particular aspects of this claim by applying appropriate generic beliefs about the subjectivity and trustworthiness of story.
Assumption 2: Coherence of Knowledge

Researchers assume that NOS is a coherent set of knowledge in and of itself, consisting of a unitary set of stable and internally consistent beliefs that are either more or less correct. We adopt a model of mind that has been
ISLS
477
ICLS2012
Volume 2: Posters
described as conceptual ecology, (Demastes, et al., 1995) knowledge in pieces, (diSessa, 1993) and a systems perspective (Smith, diSessa, & Roschelle, 1993). In this account, knowledge consists of a fairly large number of sub-elements that may be more or less appropriate in given contexts but not correct in and of themselves. We apply this theory of mind to personal epistemologies by drawing on Hammer & Elbys (2002) notion of epistemological resources, which are small, general, and dynamic, and can accommodate contextual dependence and provide an account of productive resources (p. 176). One participant, Ashley, demonstrates the type of fine-grained resources we believe people typically use when reasoning about everyday science claims. Rather than having a coherent theory about NOS, Ashley describes it in a variety of ways. For example, she initially states what seems to be a correct theory of science: science involves collecting evidence for a claim. However, she also says that despite the fact that Ghost Hunters do those things, they still are not scientists because the data must be collected in an extremely methodological manner involving standardized procedures. Later, she describes the Ghost Hunters methods as scientific because they used no personal inferences and were not making stuff up. Ashleys varied description would likely be called mixed or developing in many NOS studies, a classification from which we can understand very little about her reasoning in specific cases. However, if we understand her has holding more dynamic, finer-grained resources about the need to accumulate data and not create stories, we can gain more insight into her claim evaluation in this context.
Assumption 3: Decontextualization of Knowledge

One final assumption inherent in NOS research study design is that people draw on the same knowledge in decontextualized experiments as they do when reasoning about knowledge claims in their everyday lives. NOS beliefs are often assessed outside of contexts in which people are also evaluating claims, and the assessments themselves are often devoid of contextual detail that would allow people to use other knowledge to assist in their claim evaluation. Studies that do provide a rich context often do so by using controversial socio-scientific issues, which raises the concern that participants might have already made up their minds about these issues before the study began. By focusing on popular TV, our research allows us to explore how people reason in the moment as they evaluate novel scientific claims in a familiar medium. For example, Jeanie draws heavily on knowledge about media sources to accept the claims presented in Mythbusters. In particular, she describes how things get shortened for TV and then assumes that the explanation presented in the show is just a shortened version of a more thorough experimental process that was filmed and edited out. She goes on to explain, I tend to blindly trust the Mythbusters and later, the Discovery Channel would not have aired something that was incorrect. Jeanie has judged that the Mythbusters and the Discovery Channel are reliable sources of knowledge that she can accept even if specific evidence is not provided. It is Jeanies prior belief in the authority of the Mythbusters program that causes her to accept the claims presented in the show and fill the edits of the show with what she considers to be appropriate evidence for the claim. Here, her knowledge of television as a context with particular norms of construction, rather than her beliefs about what constitutes science or scientific evidence, drive her evaluation of knowledge claims.
Conclusion
In this paper, we have suggested that an exclusive focus on NOS knowledge, as defined by NOS research, systematically misrepresents how people evaluate science claims. We have described how NOS research problematically assumes that people draw on domain-specific, coherent, decontextualized knowledge when evaluating claims. Through peoples statements about popular TV science claims, weve illustrated an alternative approach that describes how people leverage a set of dynamic epistemological resources for making sense of science claims embedded in popular media. We believe this approach allows us to identify a range of productive ways that people evaluate claims that may have been obscured by previous research.
References
Demastes, S., Good, R., & Peebles, P. (1995). Students conceptual ecologies and the process of conceptual change in evolution. Science Education, 79(6), 637-666. diSessa, A. (1993). Toward an epistemology of physics. Cognition and Instruction, 10(2/3), 105-225. Hammer, D., & Elby, A. (2002). On the form of a personal epistemology. In B.K. Hofer & P.R. Pintrich (Eds.), Personal Epistemology (pp 169-190). Mahwah, NJ: Erlbaum. Lederman, N. (1992). Students and teachers conceptions of the nature of science. JRST, 29(4), 331-359. McComas, W., Almazroa, H., & Clough, M. (1998). The nature of science in science education: An introduction. Science and Education., 7, 511-532. McComas, W., Clough, M. & Almazroa, H. (1998). The role & character of the nature of science in science education, In W. McComas (Ed.) The Nature of Science in Science Education (pp 3-39). NLD: Kluwer. Smith III, J., diSessa, A., & Roschelle, J. (1994). Misconceptions Reconceived: A Constructivist Analysis of Knowledge in Transition. The Journal of the Learning Sciences, 3(2), 115-163.
ISLS
478
ICLS2012
Volume 2: Posters
Adolescent profiles of knowledge and epistemic beliefs in the context of reading multiple texts
L.E. Ferguson, I. Brten, H.I. Strms, & . Anmarkrud. Department of Educational Research, University of Oslo, P.O. Box 1092 Blindern, N-0317 Oslo, Norway. Email: l.e.ferguson@ped.uio.no, ivar.braten@ped.uio.no, h.i.stromso@ped.uio.no, oistein.anmarkrud@ped.uio.no Abstract: Adopting a person-centered approach, we set out to identify profiles of prior knowledge and epistemic beliefs in Norwegian 10th-graders. We identified two clusters that were labeled low prior knowledge/moderate personal justification and moderate prior knowledge/low personal justification. Clusters differed significantly regarding multiple-text comprehension, with low prior knowledge/moderate personal justification students displaying poorer comprehension of texts on an unsettled scientific issue.
Major issues addressed

It is generally accepted that prior knowledge is a predictor of multiple-text comprehension. Recently, epistemic beliefs have also been shown to play a role, with those investigating relations between epistemic beliefs and multiple-text comprehension typically adopting a variable-centered approach, using multiple regression analysis to control for prior knowledge while examining the unique contribution of distinct beliefs about knowledge and knowing (Brten et al., 2011). However, the relation between knowledge and epistemic beliefs is complex and the two constructs may interact so that separating them in analyses may yield misleading results (Bromme et al., 2008). At the same time, reliance on aggregate data instead of attending more to individuals may mask patterns existing at a personal level (Alexander et al., 1995). This study set out to identify profiles of prior knowledge and epistemic beliefs in 10th-graders, and to investigate implications for adolescents holding such combinations of knowledge and beliefs on multiple-text reading in the domain of science.
Multiple-text comprehension, using diverse sources of information to construct and communicate a meaningful representation of a particular issue, is essential in and out of school. Literacy tasks involving multiple texts may depend on readers ability to judge the validity of knowledge claims, with the term epistemic cognition being used to describe such judgements (Brten et al., 2011). Much of the current research on epistemic cognition and text-based learning assumes a linear relationship between advanced epistemic beliefs and multiple-text reading (Bromme et al., 2008), but empirical evidence to support this view is equivocal, and there is no agreement as to what advanced beliefs entail. One plausible suggestion is that the relation between epistemic beliefs and knowledge about the matter in question may be non-linear and that complex, context-sensitive interactions may exist (Bromme et al., 2008). We think this suggestion has merit and try to contribute to its investigation in the context of multiple-text reading. Another debated issue in epistemic belief research is the dimensionality of beliefs. We continue a line of research focusing on justification for knowing beliefs (Greene et al., 2008).
Methods
Participants were 64 Norwegian 10th-graders (mean age = 14.9). Before reading, students knowledge about the issue discussed in the texts was assessed using a 20-item multiple choice test. An 18-item measure of epistemic beliefs was administered, focusing on beliefs about justification of knowledge claims in science. As indicated by previous exploratory and confirmatory factor analyses (Ferguson et al., 2012), this measure captures three dimensions pertaining to ways in which students believe knowledge claims should be validated, specifically, by personal justification (sample item: If I think that something is correct in science, then no one can prove that I am wrong), justification by authority (sample item: If a science teacher says something is correct, then I believe it), and justification by multiple sources (sample item: To be able to trust knowledge claims in science texts, I have to check various knowledge sources). Each item was rated on a scale of one to ten. Students then read five authentic texts containing conflicting information on the health-effects of sun exposure, for the purpose of giving a presentation. One text was a neutral, informative text, whereas two texts presented the view that sunrays protect against cancer and two texts put forward the opposing view that sunrays cause cancer. After reading, participants were administered a test of multiple-text comprehension in the form of short-essay questions, designed to capture the integration of ideas across texts.
Results and implications

Acceptable levels of reliability were achieved for all measures (see Ferguson et al., 2012). Wards minimumvariance hierarchical clustering technique was used to identify patterns in participants prior knowledge and
ISLS
479
ICLS2012
Volume 2: Posters
epistemic beliefs. Using a dendogram, we identified two meaningful clusters, with this solution also verified in a comparable group of 10th graders (n = 57). Means and standard deviations for clustering variables and multipletext comprehension are presented in Table 1. We labeled these clusters low prior knowledge/moderate personal justification and moderate prior knowledge/low personal justification. Table 1: Means and standard deviations of scores on measures for the two clusters Prior Knowledge (max = 20) M SD 7.11 1.75 12.28 2.49 Personal Justification (max = 10) M SD 5.36 1.47 3.30 1.58 Justification by Authority (max = 10) M SD 6.86 1.66 7.15 1.58 Justification by Multiple Sources (max = 10) M SD 6.48 1.47 6.01 1.86 Multiple-text Comprehension (max = 14 ) M SD 6.46 3.05 8.14 3.15
Cluster 1 Cluster 2
The students in Cluster 1 (n = 28) achieved relatively low prior knowledge scores (M = 7.11) and moderate personal justification scores (M = 5.36), tending to believe that knowledge claims in science should, to some degree, be assessed by relying on personal opinion. In comparison, students in Cluster 2 (n = 36) achieved moderately high prior knowledge scores (M = 12.28) and low personal justification scores (M = 3.30), shying from the idea that own opinion is an important way to evaluate knowledge claims. A multivariate analysis of variance (MANOVA) with cluster group as the independent variable and knowledge, personal justification, justification by authority, and justification by multiple sources as dependent variables indicated a statistically significant overall difference between clusters, Wilks = .36, F(4, 59) = 26.12, p = .000, partial = .64, and follow-up analyses of variance (ANOVAs) showed statistically significant univariate effects for knowledge, F(1, 62) = 87.05, p = .000, partial = .58, and personal justification, F(1, 62) = 28.59, p = .000, partial = .32, but not for justification by authority or justification by multiple sources, Fs(1, 62) < 1.19, p > .27. Regarding multiple text comprehension, a one-way ANOVA with cluster group as the independent and multiple-text comprehension as the dependent variable indicated a statistically significant difference between the two groups, F(1, 61) = 4.54, p = .037, partial = .069, with Cluster 2 outperforming Cluster 1. This study represents a unique contribution to the field by including students prior knowledge and different dimensions of epistemic beliefs in a person-centered approach. Findings are consistent with Bromme et al.s (2008) suggestion that relations between knowledge and epistemic beliefs may be non-linear and interactive. Specifically, the configuration of students prior knowledge and beliefs about justification of knowledge claims by personal means seems to play a role in their comprehension of a complex scientific issue presented in multiple texts. Possibly, moderate prior knowledge may bulwark students against too much trust in own opinion, making students pay more attention to what authors of texts actually say (Brten et al., 2008). In contrast, students with little prior knowledge may be more inclined to resort to personal opinion when evaluating knowledge claims, the danger being that subjective interpretations then overshadow perspectives contained in texts. Both groups of students that we identified were similar with respect to beliefs about justification by authority and justification by multiple sources. Apparently relatively high levels of those dimensions could not compensate for the detrimental combination of low prior knowledge/moderate personal justification. Hopefully this study may initiate further research on the multidimensional influence of epistemic beliefs and other variables on performance. A better understanding of subgroups that exist in classrooms may also help teachers target less adaptive patterns of knowledge and epistemic beliefs in their instruction.
References
Alexander, P. A., Jetton, T.L., & Kulikowich, J.M. (1995). Interrelationship of knowledge, interest, and recall: Assessing a model of domain learning. Journal of Educational Psychology, 87, 559-575. Bromme, R., Kienhues, D., & Stahl, E. (2008). Knowledge and epistemological beliefs: An intimate but complicate relationship. In M.S. Khine (Ed.). Knowing, knowledge and beliefs: Epistemological studies across diverse cultures (pp. 423-441). Perth, Australia: Springer. Brten, I., Britt, M.A., Strms, H.I., & Rouet, J.-F. (2011). The role of epistemic beliefs in the comprehension of multiple expository texts: Towards an integrated model. Educational Psychologist, 46, 48-70. Brten, I., Strms, H.I., & Samuelstuen, M.S. (2008). Are sophisticated students always better? The role of topic-specific personal epistemology in the understanding of multiple expository texts. Contemporary Educational Psychology, 33, 814-840. Ferguson, L.E., Brten, I., Strms, H.I., & Anmarkrud, . (2012). Dimensionality and change in epistemic beliefs when adolescents read conflicting information presented in multiple documents. Submitted. Greene, J. A., Azevedo, R., & Torney-Purta, J. (2008). Modeling epistemic and ontological cognition: Philosophical perspectives and methodological directions. Educational Psychologist, 43, 142-160.
ISLS
480
ICLS2012
Volume 2: Posters
Improving Americans Modest Global Warming Knowledge in the Light of RTMD (Reinforced Theistic Manifest Destiny) Theory
Michael Andrew Ranney, Graduate School of Education, University of California, Berkeley, ranney@berkeley.edu Dav Clark, Psychology Department, University of California, Berkeley, davclark@berkeley.edu Daniel Reinholz, Science & Mathematics Education, University of California, Berkeley, reinholz@berkeley.edu Sarah Cohen, Environmental Sciences, University of California, Berkeley, sarahc51990@berkeley.edu Abstract: Study 1 (N=270) reveals U.S. knowledge about global warmings mechanism to be poor, but Study 2s experiment (N=149) shows that a brief, 400-word text yields both large knowledge gains and more climate change acceptance. The results cohere with RTMD theory. Global warming (GW; climate change) is generally occurring much faster than species adapted to in past eras (see Harte & Harte, 2008), threatening myriad species (e.g., humans) futures. Unfortunately, U.S. residents are less concerned about GW than those of similarly developed peer nations (Leiserowitz, 2007)one of many dimensions by which Americans are marked outliers (Ranney, in press). Ranney explains a variety of such divergences with Reinforced Theistic Manifest Destiny (RTMD) theory, addressing how various (largely peer) nations collective, theistically-related beliefs have developeddue to military, economic, etc., feedback. Beliefs generally follow understanding; for instance, biologists largely accept evolution more than do Biology 1 students (cf. Shtulman & Calabi, 2008). In two studies, we assess the hypotheses that (1) proper GW understanding is rare, but (2) fostering GW understanding has desirable effects, such as increasing GW acceptance. One rarely finds a concise explanation of GWs basic physical/chemical mechanism without unnecessary details, so Study 2 assessed our prediction that such a mechanistic explanation can markedly help people appreciate the soundness of GWs scienceincreasing acceptance, concern, and imperative action. Please take a minute to answer this: How would you explain global warmings mechanism? (This is a difficult question; even most professors answer inaccurately.) Now, read this 400-word answer, which we used as Study 2s intervention (written by Ranney, Reinholz, and Lloyd Goldwasser, with Ronald Cohens counsel): How does climate change (global warming) work? The mechanism of the greenhouse effect [Or: Why do some gases concern scientistslike carbon dioxide (CO2)but not others, like oxygen?] Scientists tell us that human activities are changing Earths atmosphere and increasing Earths average temperature. What causes these climate changes? { = intervention paragraph break} First, lets understand Earths normal temperature: When Earth absorbs sunlight, which is mostly visible light, it heats up. Like the sun, Earth emits energybut because it is cooler than the sun, Earth emits lower-energy infrared wavelengths. Greenhouse gases in the atmosphere (methane, carbon dioxide, etc.) let visible light pass through, but absorb infrared lightcausing the atmosphere to heat up. The warmer atmosphere emits more infrared light, which tends to be re-absorbedperhaps many timesbefore the energy eventually returns to space. The extra time this energy hangs around has helped keep Earth warm enough to support life as we know it. (In contrast, the moon has no atmosphere, and it is colder than Earth, on average.) Since the industrial age began around the year 1750, atmospheric carbon dioxide has increased by 40% and methane has increased by 150%. Such increases cause extra infrared light absorption, further heating Earth above its typical temperature range (even as energy from the sun stays basically the same). In other words, energy that gets to Earth has an even harder time leaving it, causing Earths average temperature to increase producing global climate change. [In molecular detail, greenhouse gases absorb infrared light because their molecules can vibrate to produce asymmetric distributions of electric charge, which match the energy levels of various infrared wavelengths. In contrast, non-greenhouse gases (such as oxygen and nitrogen that is, O2 and N2) don't absorb infrared light, because they have symmetric charge distributions even when vibrating.] Summary: (a) Earth absorbs most of the sunlight it receives; (b) Earth then emits the absorbed lights energy as infrared light; (c) greenhouse gases absorb a lot of the infrared light before it can leave our atmosphere; (d) being absorbed slows the rate at which energy escapes to space; and (e) the slower passage of energy heats up the atmosphere, water, and ground. By increasing the amount of greenhouse gases in the atmosphere, humans are increasing the atmospheres absorption of infrared light, thereby warming Earth and disrupting global climate patterns. Shorter summary: Earth transforms sunlights visible light energy into infrared light energy, which leaves Earth slowly because it is absorbed by greenhouse gases. When people produce greenhouse gases, energy leaves Earth even more slowlyraising Earths temperature.
Study 1: A Paucity of American Mechanistic GW Knowledge is Evidenced

To assess GW attitudes and mechanistic understandings, we administered 270 surveys to San Diego park visitors (n = 201) and community college students (n = 69). Given space limits, we report here only on some of the surveys items about: (1) GW opinions, (2) short-answer GW knowledge, and (3) other RTMD constructs.
ISLS
481
ICLS2012
Volume 2: Posters
Our hypothesis that Americans rarely understand GWs mechanism was supported. In trying to explain the GW mechanism, only 32 (12%) of our 270 participants referenced any gases (CO2, pollution, emissions, etc.) as raising heat retention (which represents a modest, partial understanding). Only four of these 32 (1%) tried to differentiate types of energy, and not one (0%) mentioned either correct absorption(s) or the input/output light asymmetry (e.g., visible/infrared)which are jointly the crux of GW understanding. Just 8 people (3%) named the greenhouse effect. Misconceptions were abundant; for instance, 42 people (16%) claimed that GW occurs because more heat is entering the atmosphere due to itsor the ozone layersdestruction. Importantly, participants true mechanistic knowledge (scored with high reliability) was correlated with their willingness to accept GW as both real (r = .22, p = .0002) and anthropogenic (r = .17, p = .005). Finally, all 15 correlations among RTMDs six constructs were again found to be in the RTMD-predicted directions, replicating previous findings (see Ranney, in press); thirteen of the 15 were significantly different from zero at p < .01.
Study 2: People Quickly Learn the Mechanism and Then Accept GW More
Our intervention was the 400-word text above (see the prior page). University of California, Berkeley (n=103) and University of Texas, Brownsville (n=46) undergraduates were randomly assigned to one of two conditions (N=149): Sandwich students: (1) provided GW mechanism explanations and completed a knowledge-andattitude survey, (2) read the 400-word GW explanation and rated their experienced surprise, and (3) performed (1) again. No pre-test students only did (2) and (3)thus, no (1). Design-wise, (1) and/or (3) can be thought of as bread slices and (2)the explanationas jam. (The no pre-test group provides a between-subjects contrast via their post-test, obviating anchoring or experimenter-demand concerns in the sandwich group.) We largely report here data from the 85 Berkeley students who had been U.S. residents for ten or more yearsto ensure sufficient American identity. (All students filled out a demographic sheet at experiments end.) We analyzed 43 no pre-test surveys and the pre-test part of 42 sandwich surveys. Due to (expected) time constraints, only 30 sandwich post-tests were completed/obtained. As in Study 1, textual responses were rated to yield true GW knowledge scores. Self-reports of knowledge were reported on a 9-point Likert scale. Study 2 replicated Study 1s findings of prevalent misconceptions. Across all pre-tests, not a single student mentioned different light/radiation types or atmospheric retention time, despite an explicit prompt to explain any differences between the energy moving toward and away from Earth. After reading the 400-word description, though, 61% of Berkeley students (and 55% of Brownsville students) across both conditions correctly answered that Earth emitted infrared light. We also found dramatic increases in true (scored) GW knowledge. Improvements within-subjects for the sandwich condition, as well as between-subjects (the no pre-test post-test vs. the sandwich pre-test), were significantboth overall (roughly doubling from 31% to 64%), and for all three subscales (i.e., knowledge measures about light, energy, and greenhouse gases; p < .05 for all six improvement possibilities). Remarkably, students GW acceptance also increased dramatically after our brief intervention, as predicted. Using all 73 post-test ratings in a paired t-test, and employing imputation for pre-tests scores for the no pre-test group, we found a significant gain in GW acceptance on the post-test, compared to the pre-test (t(72) = 2.28, p = .01). Brownsville surveys clearly replicated this result (t(39) = 4.24, p < .0001). The knowledge-attitude link was also reflected in nave pre-test scores, as students self-perceived ratings of their own GW knowledge significantly correlated with their GW attitudes (r = .39, p = .01).
Summary: Many More People Will Accept GW After They Learn Its Mechanism
None of Study 1s 270 people explained the basic greenhouse effect (which is possible in < 35 words, without using technical words such as absorb or infrared). If our samples even vaguely represent the U.S. public, then Americans rarely understand global warmings mechanism. Just as mechanistic knowledge of reproduction likely supports evolution acceptance (cf. Shtulman & Calabi, 2008), we have shown that mechanistic GW understandings support GW acceptance. In Study 2, we found that increasing GW knowledge correspondingly increases students acceptance; a mere 400 words of instruction not only dramatically increased undergraduates GW understandingsit also increased their mean acceptance that (anthropogenic) global warming is occurring.
References
Harte, J., & Harte, M. E. (2008). Cool the Earth, save the economy: Solving the climate crisis is EASY. Retrieved from http://www.cooltheearth.us/ Leiserowitz, A. (2007). International public opinion, perception, and understanding of global climate change (Human Development Report 2007/2008): UNDP. [Online]. Available via: leiserowitz_anthony6.pdf Ranney, M. A. (in press). Why don't Americans accept evolution as much as people in peer nations do? A theory (Reinforced Theistic Manifest Destiny) and some pertinent evidence. In K. Rosengren, M. Evans, G. Sinatra & S. Brem (Eds.), Evolution Challenges. Oxford: University Press. Shtulman, A., & Calabi, P. (2008). Learning, understanding, and acceptance: The case of evolution. In B. C. Love, K. McRae, & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 235-240). Austin, TX: Cognitive Science Society.
ISLS 482
ICLS2012
Volume 2: Posters
Improving Students Scientific Reasoning Skills via Virtual Experiments and Worked Examples
Shiyu Liu and Keisha Varma University of Minnesota, 56 East River Road, Minneapolis, MN 55455 Email: 1iux0631@umn.edu, keisha@umn.edu Abstract: This work looks at how to improve students scientific reasoning. The findings suggest that neither worked examples alone nor interactive visualization experiences alone are as helpful as the combination. While worked examples are effective instructional devices to assist learning, due to students limited self-explanation skills, virtual experiments are needed to elicit more metacognitive efforts when studying the examples. At the same time, worked examples provide direct guidance that can prevent potential frustration with the visualization.
Introduction
Successful learning in science classrooms requires scientific reasoning skills involved in experimentation, evidence evaluation, theory and evidence coordination and so on (Zimmerman, 2005). National science standards mandate that they are taught to students beginning in early elementary grades and continuing through high school (NRC, 1996). While interactive visualizations may facilitate the learning of these skills, additional support is needed so that students are able to apply them appropriately (Varma & Linn, 2011). Worked examples are effective instructional device to assist learning, as they demonstrate expert problem solving strategies in a step-by-step fashion to avoid overtaxing learners cognitive resources (Renkl, Stark, Gruber, & Mandl, 1998). The cognitive load theory (Sweller, 1988) suggests that, learning from worked examples can more efficiently prevent cognitive overload during initial knowledge acquisition than problem solving. Accumulating research has been studying this worked example effect (e.g., van Gog, Kester, & Paas, 2011), but mainly focused on conceptual learning in well structured domains such as algebra and physics (e.g., Caroll, 1994). The present work transitions the focus of worked examples research to explore the learning of scientific reasoning in authentic classroom contexts. In particular, we investigate the following research questions: (1) How does learning from worked examples influence students understanding of scientific experimentation? (2) What is the role of inquiry activities with interactive visualizations in the learning process? and (3) How will the combination of worked examples and inquiry activities impact students learning?
Methodology
Over a span of six weeks, 69 seventh-grade students (42 females and 27 males; mean age=12) were randomly assigned to three conditions: only studying worked examples (Example/Practice Only Group), only interacting with the visualization (Visualization Only Group), or learning via both worked examples and visualization activities (Combination Group) (see Table 1). A pre-and posttest (Cronbachs reliability is 0.717) was administered to examine students knowledge of scientific experimentation. Worked-example learning sessions occurred during regular class time (two sessions each week and approximately 15 minutes each session), when students were asked to study a worked example (see Figure 1) and solve a practice problem. Besides, the Combination and Visualization Only Groups participated in three virtual experiments, where they investigated the role of a particular variable in the greenhouse effect by manipulating the interactive visualization (see Figure 2).
Results and Conclusions

Results show that students who both reflected on worked examples and interacted with the computer visualization significantly outperformed others in designing scientific experiments [F(2,66)=5.166, p<.01]. Across the six weeks, they had notable improvement in applying the control-of-variable strategy when asked to evaluate and refine experimental designs [F(1,16)=22.07, p<.001]. In contrast, only learning from worked examples or interacting with the visualization resulted in much less progress in students performance, and there was no significant difference between these two approaches [F(1,42)=.731, p=.397]. Rasch Model is used to measure the practice problems item difficulty and preliminary results indicate that the questions in general tap well into the distribution of students scientific reasoning abilities. Our findings indicate that combining worked examples with visualization inquiry activities is a promising approach to facilitate the learning of scientific reasoning. Although computer visualizations are helpful tools to enhance learning, students are found often tempted to manipulate variables randomly when direct guidance is not provided, and fail to recognize the importance of controlling variables in a systematic manner. Worked examples can serve to encourage their reflection on the specific knowledge and motivate them to devote more metacognitive efforts when interacting with the visualization. On the other hand, without
ISLS 483
ICLS2012
Volume 2: Posters
additional exploratory experience, learning from worked examples is less effective than would be expected. Due to limited self-explanation skills, students may only focus on the superficial features of the examples and pay little attention to possible deficiencies in their learning (Chi & Lewis, 1989). This lack of sufficient mental effort thus makes it difficult for them to transfer what have been learned from the examples to new scenarios. In this sense, interacting with the visualization provides valuable opportunity for students to try out the reasoning strategies they have learned from the worked examples, which in turn will greatly elicit their deep understanding of them. Although we cannot ignore the possibility that the different learning time across groups may have impacted students performance, our follow-up study will adopt relevant tasks to make sure that participants in all the conditions spend equal time learning the topic so as to eliminate the confounding effect there may exist.
References
Caroll, W. M. (1994). Using worked out examples as an instructional support in the algebra classroom. Journal of Educational Psychology, 86, 360-367. Chi, M. & Lewis, M. (1989). Self-explanations: How students study and use examples in learning to solve problems. Cognitive Science, 13, 145-182. National Research Council. (1996). National science education standards. Washington, DC: National Academy Press. Renkl, A., Stark, R., Gruber, H., & Mandl, H. (1998). Learning from worked-out examples: The effects of example variability and elicited self-explanations. Contemporary Educational Psychology, 23 (1), 90108. Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12, 257-285. van Gog, T., Kester, L., & Paas, F. (2011). Effects of worked examples, example-problem, and problemexample pairs on novices learning. Contemporary Educational Psychology, 36, 212-218. Varma, K., & Linn, M. (2011). Using Interactive Technology to Support Students Understanding of the Greenhouse Effect and Global Warming. Journal of Science Education and Technology, 20(1). Zimmerman, C. (2005). The development of scientific reasoning skills: What psychologists contribute to an understanding of elementary science learning. Final draft of a report to the National Research Council committee on science learning kindergarten through eighth grade. Table 1: Group Assignment and Schedule. Group Combination Visualization Only Example/Practice Pre-test Week 1 X X X Vis. Week 1 X X WE Week 2 & 3 X X Vis. Week 3 X X WE Week 4 & 5 X X Vis. Week 5 X X PostWeek 6 X X X
Note: 1. WE: worked-example learning; Vis.: activities with the visualization 2. Shaded boxes indicate that the group did not participate in the particular activity.
Figure 1. Screenshot of a sample worked example.
Figure 2. Screenshot of the visualization.
ISLS
484
ICLS2012
Volume 2: Posters
Synthesizing Quantitative Predictors for Interaction in an Asynchronous Online Course

Daniel Zingaro, Murat Oztok, Jim Hewitt OISE University of Toronto, 252 Bloor St.W. Toronto, Canada Email: [daniel.zingaro,murat.oztok, jim.hewitt]@utoronto.ca Abstract: The effectiveness and potential of asynchronous online courses hinge on sustained, purposeful collaboration. And while many factors affecting interaction have been uncovered by prior literature, there are few accounts of the relative importance of these factors when studied in the same online course. In this paper, we develop a literature-informed model of six predictors on the likelihood that a note receives a reply. We corroborate earlier findings (such as the impact of the date that the note was posted) but also obtain one contradictory result (that reading ease does not appear to be a significant predictor). We offer hypotheses for our findings and suggest future directions for this type of research.
Introduction
From a social constructivist perspective, online learning is seen to be effective when it permits the exchange of views and allows students to learn through social communication (Swan, 2005). Lengthy discussions do not inherently suggest deep processing or collaborative meaning-making, but it is unlikely that such processes could occur in the absence of sustained discourse. Many authors, therefore, have offered hypotheses and implemented supports toward the growth of online discussion. For example, the specific discussion tool used by students (Guzdial, 1997) and habitual reading and writing practices (Hewitt, 2003) dramatically influence thread structures and lengthy communication. In the current paper, we add to this body of knowledge through a study of note-level features related to interaction. Why do some notes get replies but others do not? We are interested in a binary (yes/no) classification model, the level of prediction accuracy we can reach with such a model, and the importance of individual predictors in determining whether a reply is predicted. When a note does not receive any replies, the thread of discussion of which that note is a part also necessarily terminates. We believe that understanding which notes are likely to receive replies is a step toward understanding which lines of discussion are likely to be elaborated upon by students and which are likely to falter.
Measured Variables
We searched the literature for measurable aspects of notes that might contribute to whether they would receive a reply. We arrived at six literature-informed hypotheses and used a logistic regression model to assess the significance of these possible predictors in an online education graduate course taking place at a large Canadian research university. Our six independent variables are as follows: Posting date. The course we analyzed was structured into weekly discussion modules, and such a structure is not amenable to discussions that extend past their designated week (Hewitt, 2003). We predicted a reply advantage for those notes written early in the week. The variable name in the model is date, suffixed with 2 (mid-week), 3 (weekend), or 4 (beginning of next week) to indicate its value compared to the baseline of 1. Active participant. Research suggests that, in terms of percentage of notes posted, no single student dominates the communication, and that most notes are produced by the collectively low-posting students (Guzdial, 1997). In spite of this, we wondered if high-posting students may have a more indirect effect on the discourse, in the sense that their messages might be more likely to receive replies. We therefore coded notes into two categories depending on whether they were written by a frequent poster (top half of the class in terms of post count, and the baseline value of the variable) or infrequent poster. The variable name is inactive, and compares inactive participants to the activeparticipant baseline. Reading ease. In a study of 37 graduate courses, Hewitt and Peters (2007) found that courses with high reading ease were more interactive (i.e. contained more notes written as replies) than courses with low reading ease. We have added Flesch reading ease to our model (variable name ease, mean 101.9, sd 14.4) in order to determine whether it is a robust predictor of replies, or whether it is measurable only as a broad, course-wide effect. Word count. Prior literature suggests that students are more likely to quickly skim longer notes as compared to shorter notes. As notes reach 500 words, for example, half of their readings occur at a rate indicative of skim-reading (Hewitt, Brett, & Peters, 2007). We hypothesized that word count (variable wordcount below) would be a negative predictor of reply-likelihood, based on a guess that students would be less likely to reply to a note that they quickly skimmed. The mean word count was 180.8, with sd 142.3.
ISLS
485
ICLS2012
Volume 2: Posters
Student or teacher. In the course analyzed here, the instructor quantitatively had only moderate presence: 61 notes were written by the instructor, compared to 1095 by students. We wondered whether instructor-authored notes, even in minority, would shape dialogue relatively more forcefully than the bulk of the notes written by students. This teacher-student variable is referred to as prof; the baseline is that the note was written by a student. Contains a question. We suspected that notes containing at least one question would garner more replies than notes containing no questions. Question-asking is a common and important discussionfacilitation technique (Hew & Cheung, 2008). Lengthy threads often contain at least one note that asks students for clarifications or personal opinions on a matter, and it is hypothesized that notes with questions obligate students to reply (Hew & Cheung, 2008). Below, the variable question is coded as no (no question in the note, the baseline value) or yes, containing 660 and 496 notes, respectively.
Results
The overall model was significant (chi-square = 123.00, df = 8, p = 0); Nagelkerkes pseudo R-squared = 0.128. The coefficients, standard errors of the coefficients, Wald Z-statistics, p-values, and odds change in reply probability for a one unit increase in each predictor are given in table 1. (wordcount was log-transformed in order to satisfy the logit-linearity assumption.) Table 1: Logistic regression model Predictor intercept date2 date3 date4 wordcount prof ease inactive question B -3.01 -0.40 -1.07 -0.98 0.47 -0.35 0.01 -0.02 0.56 SE 0.86 0.19 0.20 0.27 0.08 0.29 0.005 0.15 0.13 Wald Z -3.48 -2.06 -5.37 -3.70 5.51 -1.22 1.83 -0.12 4.24 p 0 .039 0 .0002 0 0.22 0.07 0.91 0 Exp(B) 0.05 0.67 0.34 0.37 1.60 0.70 1.01 0.98 1.76
Word count and reading ease are the two results discordant with our hypotheses; the latter also contradicts prior literature. Perhaps the fact that longer notes are more likely to receive a reply is due to these notes containing more information and hence more opportunities for students to find something of interest. Even though such notes are more often scanned, it is possible that students search such notes for isolated aspects to which they can reply. On reading ease, we suggest that it is the social nature of the course at large, rather than reading ease arising from that sociality, that impacts resultant interaction.
Conclusion
Taken together, our findings largely concur with the literature on which our model was based. Long notes, notes written early in the week, and notes containing at least one question all lead to higher probabilities that a note receives a reply. Reading ease of notes did not have a significant effect on the likelihood that a note would receive a reply. Overall, our binary classification model predicted 64.4% of the notes correctly.
References
Guzdial, M. (1997). Information ecology of collaborations in educational settings: Influence of a tool. In Proceedings of the 1997 conference on computer support for collaborative learning (pp. 8390). Toronto, Ontario, Canada: International Society of the Learning Sciences. Hew, K. F., & Cheung, W. S. (2008). Attracting student participation in asynchronous online discussions: A case study of peer facilitation. Computers & Education, 51(3), 11111124. Hewitt, J. (2003). How habitual online practices affect the development of asynchronous discussion threads. Journal of Educational Computing Research, 28(1), 3145. Hewitt, J., Brett, C., & Peters, V. L. (2007). Scan rate: A new metric for the analysis of reading behaviors in asynchronous computer conferencing environments. American Journal of Distance Education, 21(4), 215231. Hewitt, J., & Peters, V. (2007). The relationship between student interaction and message readability in asynchronous online discussions. In Proceedings of the 2007 international conference on computer supported collaborative learning (pp. 292294). New Brunswick, New Jersey, USA: International Society of the Learning Sciences. Swan, K. (2005). A constructivist model for thinking about learning online. In J. Bourne & J. Moore (Eds.), Elements of quality online education: Engaging communities (pp. 1330). Needham, MA: Sloan-C.
ISLS
486
ICLS2012
Volume 2: Posters
Engines of Representation: Processing Raw Student Data into Useable Information

Suzanne Rhodes, University of Wisconsin Madison, Educational Sciences, 1025 W Johnson St, Madison, WI 53706, serhodes@wisc.edu Richard Halverson, University of Wisconsin Madison, 270-A Education Building, 1000 Bascom Mall, Madison, WI 53706, halverson@education.wisc.edu Abstract: This paper first examines the relationship between student data and information representation from a socio-technical systems perspective. We then present a Design-Based Research solution to support educators as engines of formative representations at the source of teaching and learning. The KidGrid iOS apps standardized and customizable features complement and support teachers existing real-time assessment tools and practices structurally and functionally, and integrate with existing classroom and district data systems.
Introduction
Aaron Koblin (2011) builds systems that convert real-world data such as airplane altitudes and time-based locations into interactive information visualizations to help people reflect on socio-technical trends. In his art, both technology and artist act as engines of representation by conceptualizing and then processing raw data into beautifully rendered dynamic information formats. These formats function to help general audiences construe new meaning about the world we live in. Current data collection methods and information outputs for education lack the visual sophistication of Koblins art. However, by exploring the relationship between student data and external representations of student knowledge from a socio-technical systems design approach, learning scientists can identify system weaknesses and leverage opportunities through current technologies and extant teacher practices in the classroom.
Processing Student Data Representations into Useful Information, and Back Again
We define data, at the simplest form of abstraction, as unprocessed qualitative or quantitative output from a measurement device or sensing organism as according to Ackoffs (1989) conceptual systems framework of data, information, and knowledge. Information is data that is given meaning when processed and contextualized and knowledge is the internalized collection of useful information used to guide action. The purpose of the use of summative data in education is to report out to meet the requirements of external accountability initiatives such as the No Child Left Behind Act (NCLB, 2001). Summative student data is often static information about a students performance and is usually collected in a quantitative format given the large amount of data required to meet reliability and validity standards. The processing of this data and subsequent representations of the information is increasingly computerized; districts and states manage the flow and form of representations via student information systems and data warehouses. Therefore, technology acts as the primary engine of representation. Furthermore, because summative data is collected asynchronously, decontextualized from the teaching and learning environment, the data are symbolic rather than indexical in nature and must be reprocessed by educators for use within the classroom. Formative student assessment data, on the other hand, helps educators to adjust the teaching and learning process as it unfolds in practice (Halverson, Grigg, Prichett, & Thomas, 2007). Formative data can be both quantitative and richly qualitative. Teachers, for example, use and also design their own paper-based data collection instruments. These locally created or modified artifacts such as rubrics, checklists, and spreadsheets are in turn frequently augmented. Teachers write memos on student work, add sticky notes, and annotate grade sheets; they conduct observations and record conversations with students. Students create learning portfolios of their work with teacher guidance. The external forms of teachers representations of student knowledge consist of highly contextualized but often decentralized forms of both data and information created in or soon after authentic assessment activity. Formative student data is, then, usually collected, rapidly processed, and represented in robust ways at the local level by both teachers and students via real-time interaction within a dynamic learning and assessment activity. Teachers, therefore, using low-tech instruments and manual representational means, act as engines of formative representations. Teachers have difficulty reconciling the roles of summative and formative assessments (Black & Wiliam, 1998). This problem is twofold. First, teachers must process indexical, formative information into symbolic data in order to make their knowledge of student learning backwards compatible (structurally) with district data systems. Second, teachers must make sense of district level data and then reprocess and repurpose that data for use in the classroom as data-driven instructional systems studies demonstrate (Halverson, Prichett,
ISLS
487
ICLS2012
Volume 2: Posters
Grigg, & Thomas, 2005; Halverson, Prichett, & Watson, 2007; Wayman, Midgley, & Stringfield, 2006). Therefore, our design challenge was to address the dual knowledge-information-data (and vice versa) transformation and transfer problem. Our research goals were to 1) connect teachers formative systems with district summative systems and 2) support, reveal, and hopefully stretch, teachers formative assessment data collection and information processing and representation practices.
Integrative Solution
We applied a Design-Based Research methodology (Design-Based Research Collective, 2003) to the challenge with detailed documentation, collaboration, and frequent design/implementation iterations. Our initial pilot study focused on a data tool design. We worked closely with an intermediate school selected because of its reputation for effective data use and established record of improving student test scores. Via purposive sampling, we recruited 11 experienced educators across subjects and grade levels who were knowledgeable about district- and school-wide and classroom assessments to help us design the tool. We collected 40 teacherlevel assessment artifacts (grade sheets, rubrics, student self-assessments, observations checklists, etc.), observed assessment activities, and recorded notes at multiple design meetings. Guided by Normans (1990) recommendation that tool designs follow real-world conventions and our document analysis findings, we created KidGrid 1.0, an iOS app: http://tinyurl.com/kidgrid. KidGrids standardized and customizable features map to teachers current assessment document designs and connect to district systems.
Opportunity Space for Socio-Technical Approaches

The practice of formative assessments positively influences student achievement (Black & Wiliam, 1998) and the use of data promotes professional learning (Light, Wexler, & Henize, 2004). However, to know, and to use, the data, particularly summative data, is not enough to foster reflection and agency. Advances in technology make it possible to evoke evidence of knowledge more broadly conceived, and capture more complex performances. Making sense of complex data that results is the bottleneck (Mislevy, Steinberg, & Almond, 1999, p. 2). If the theoretical, conceptual, and technological models are in place for knowledge capture, the opportunity to widen the bottleneck between teachers and districts data and information representation and sense making is here as well. While some standardization of the tools and artifacts of student data collection may be currently required, future information representational tools can support educators, and students, as engines of representations in activity at the source of teaching and learning. With sustained effort, tools and design-based activities like this research and development project, which also focus on representing viable realtime information about students in the classroom, can foster this goal creatively and practically.
References
Ackoff, R. L. (1989). From data to wisdom. Journal of Applied Systems Analysis, 16, 3-9. Black, P., & Wiliam, D. (1998). Inside the black box: Raising standards through classroom assessment. Phi Delta Kappan, 80, 139148. Design-Based Research Collective. (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5-8. Halverson, R., Grigg, J., Prichett, R., & Thomas, C. (2005). The new instructional leadership: Creating datadriven instructional systems in schools (WCER Working Paper No. 2005-9). Madison: University of WisconsinMadison, Wisconsin Center for Education Research. Retrieved August 1, 2008, from http://www.wcer.wisc.edu/publications/workingPapers/Working_Paper_No_2005_9.php Halverson, R., Prichett, R., & Watson, J., (2007). Formative feedback systems and the new instructional leadership (WCER Working Paper No. 2007-7). Madison: University of WisconsinMadison, Wisconsin Center for Education Research. Retrieved August 1, 2008, from http://www.wcer.wisc.edu/publications/workingPapers/Working_Paper_No_2007_3.php Light, D., Wexler, D., & Henize, J. (2004). How practitioners interpret and link data to instruction: Research findings on New York City Schools' implementation of the Grow Network. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA. Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (1999). Evidence-Centered Assessment Design. Retrieved from http://www.education.umd.edu/EDMS/mislevy/papers/ECD_overview.html Norman, D. (1993). Things that make us smart. New York: Double Day. TED: Ideas Worth Spreading (2011). Aaron Koblin: Artfully visualizing our humanity. Retrieved from http://www.ted.com/talks/aaron_koblin.html?utm_source=newsletter_weekly_2011-0524&utm_campaign=newsletter_weekly&utm_medium=email Wayman, J., Midgley, S., & Stringfield, S. (2006). Leadership for data-based decision-making: Collaborative data teams. In Danzig, A., Borman, K., Jones, B., & Wright, B. (Eds.), New models of professional development for learner centered leadership. Erlbaum.
ISLS
488
ICLS2012
Volume 2: Posters
The Role of Technologies in Facilitating Collaborative Engagement

Suparna Sinha, Karlyn Adams, Toni Kempler Rogat, Cindy E. Hmelo-Silver Rutgers University, 10 Seminary Place, New Brunswick, NJ suparna.sinha@gse.rutgers.edu,k.raylene86@gmail.com, toni.kempler.rogat@gse.rutgers.edu,cindy.hmelo-silver@gse.rutgers.edu Abstract: Computer-supported inquiry learning has the potential to foster productive engagement with the task and also enhance students' motivation. This may occur because students have the opportunity to collaborate around on authentic problems, often situated in media-rich environments. However, we have limited understanding of the quality of engagement fostered in these contexts and how technology might support high quality engagement. This study explores technological affordances on influencing students engagement technology-rich curriculum unit on aquatic ecosystems. Previous research has identified design features of technologies that foster self-regulation and high quality engagement (Azevedo, 2005; Gresalfi, et al., 2009). Current research suggests that students can be engaged if given opportunities to work in computer-supported, inquiry based learning environments (Jrvela & Salovaara, 2004; Veermans & Jrvela, 2004). However, we have limited understanding of the specific affordances for students engagement in these technology-enhanced settings, as well as the range in quality of engagement fostered in these contexts. Our primary research interest here is to compare the extent to which computer technologies with varying features promote two forms of engagement- task and conceptual-to-consequential (Gresalfi, et al., 2009), in middle school science students. Facilitating high quality engagement is critical given benefits of engagement for learning outcomes. We define high quality task engagement (TE) as attempts to plan, monitor the plans enactment, and move beyond focusing on superficial features. Planning is more than superficial when it moves toward the task solution and problem solving. Conceptual-to-consequential (CC) engagement characterizes attempts at content connections on a continuum that range from simple knowledge telling (low), to content connections (moderate), to connections to prior knowledge, everyday experiences or the larger problem (i.e., consequential engagement).
Defining engagement
Instructional context
Over the course of a 6-week 7th grade science unit, two four-student groups were videotaped as they investigated possible causes of fish death in a pond. Students used Net Logo simulations (Wilensky & Reisman, 2006) to explore aquatic ecosystem processes (see Figure 1) to uncover the cause of lack of oxygen for the fish, and the Ecological Modeling Toolkit (EMT; Vattam et al, 2011) to model their evolving understanding of the problem (see Figure 2). We predicted that the simulations would foster moderate CC engagement because we anticipated students would engage in interpret the simulations without making connections to the larger problem. In comparison, we predicted EMT would foster high quality CC engagement given the opportunities to make sense of data gathered from multiple sources that they would then integrate into their evolving explanatory models. Two full-lesson observations per group were selected for analysis; as students worked with simulations and then as they revised their models using EMT. Videos were segmented at five-minute intervals and coded as high, medium or low engagement based on our definitions of TE and CC engagement; codes were accompanied with justification.
Figure 1. Aquatic ecosystem simulation

ISLS 489
ICLS2012
Volume 2: Posters
Missing conceptual connection
Figure 2. EMT model
Results
In contrast to our predictions, groups demonstrated higher quality TE and CC engagement using the simulations. For example, high levels of CC engagement were observed when students set up parameters (i.e., high sunlight, low nutrients) to recreate the problem scenario for understanding fish death. Students discussed relationships among components (content connections) and considered how these factors may have led to the conditions that caused fish death (CC engagement). Equally surprising, groups focused on specific, but individual, aspects of the larger problem when working with the EMT software. This included defining the components they considered in their model (e.g. algae, sunlight). However, groups did not go beyond to make connections among these concepts (Figure 2). For instance, both groups discussed that high temperature is responsible for the algal bloom, but neither connected it to lack of oxygen. Moreover, observed TE was moderate in quality, given that planning during model creation focused on the physical layout, rather than on solving the larger problem.
Discussion
There is a general concern that schools do not give students opportunities to engage with curricular content in conceptually and consequentially meaningful ways (Gresalfi et al., 2009). If students have not been prepared to think about conceptual connections between varying contexts, it is not likely that they will transfer what they have learned beyond the school setting. This study is a step towards developing models of how high quality collaborative engagement is mediated by specific technological affordances and how that affects learning and transfer. Future work needs to generalize these findings beyond this case study and examine the relation between conceptual and consequential engagement on one hand, and learning and transfer on the other hand.
References
Azevedo, R. (2005). Using hypermedia as a metacognitive tool for enhancing student learning? The role of selfregulated learning. Educational Psychologist, 4, 199- 209. Blumenfeld, P. C., Kempler, T. M., & Krajcik, J. S. (2006). Motivation and Cognitive Engagement in Learning Environments. In K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (pp. 475-488). New York: Cambridge University Press. Gresalfi, M., Barab, S., Siyahhan, S., & Christensen, T. (2009). Virtual worlds, conceptual understanding, and me: designing for consequential engagement. On the Horizon, 17, 21-34. Jrvela, S., & Volet, S. (2004). Motivation in real-life, dynamic and interactive learning environments: Stretching constructs and methodologies. European Psychologist, 9, 193-197. Vattam, S., Goel, A., Rugaber, S., Hmelo-Silver, C., Jordan, R., Gray, S., et al. (2011). Understanding complex natural systems by articulating structure-behavior-function models. Educational Technology & Society, 14, 66-81. Veermans, M. & Jrvel, S. (2004) Generalized achievement goals and situational coping in inquiry learning. Instructional Science, 32, 269291. Wilensky, U. & Reisman, K. (2006). Thinking like a wolf, a sheep or firefly: Learning biology through constructing and testing computational theories an embodied modeling approach. Cognition and Instruction, 24, 171-209.
ISLS
490
ICLS2012
Volume 2: Posters
Integrating Insights from Critical Race and Queer Theories with Cultural-Historical Learning Theory
Indigo Esmonde, University of Toronto, 252 Bloor St. W., Toronto, ON, indigo.esmonde@utoronto.ca Miwa Takeuchi, University of Toronto, 252 Bloor St. W., Toronto, ON, miwa.takeuchi@utoronto.ca Lesley Dookie, University of Toronto, 252 Bloor St. W., Toronto, ON, lesley.dookie@utoronto.ca Abstract: Within cultural-historical learning theories, recent research has focused on the practice-linked (i.e., contextual) nature of identity. In this paper, we argue that while the concept of practice-linked identities represents a significant advance, this understanding of identity is limited because it cannot account for broad systemic issues of power and privilege. We integrate insights from Critical Race Theory and Queer Theory to develop a more nuanced understanding of identity development. In recent years, learning scientists have begun to theorize the close connection between identity development and learning. Contemporary cultural-historical learning theories have replaced static conceptions of identity with a conception of identity as fluid, context-based, and linked to the practices in which people participate. (We use the term cultural-historical learning theories and the acronym, CHL, to refer to a broad set of theories, rooted in the work of Vygotsky (e.g., 1978), that consider learning in its social, cultural and historical context.) One major challenge in understanding identity development is the influence of broad social identity categories like race, gender, nationality, and so on. While CHL has much to offer, these theories do not adequately deal with broader social categories (Nasir & Hand, 2006) nor do they fully appreciate the way communities of practice shape identities as a result of the power that is omnipresent in such communities. Critical race theory (CRT) and queer theories (QT) offer powerful insights into the construction of race and gender that can help CHL theorists better understand local, contextual practice-linked identities. This poster presents a theoretical literature review. Key texts both historical and contemporary have been considered and distilled to find the major themes and set of assumptions underlying the three different theoretical frameworks. In future work, we will also include texts from other critical theories that consider the intersectionality of race, sexuality, gender and class (Anzalda & Keating, 2002; Hill Collins, 2000).
Theoretical Framework: Cultural-Historical Learning Theories of Identification

CHL theories consider all learning to be cultural and social. These theories have several assumptions in common, including: learning and development are inextricably linked to the contexts in which people participate; learning and development are spread across the people, artifacts, and norms that make up the context; and learning and development happen at several interconnected levels. Drawing from these assumptions, CHL considers identity development to be an aspect of participation in practices. The benefit of considering identities as practice-linked is that identities are no longer confined to the person but also distributed across people, artifacts, and norms. A single person will participate in many different practices, and thus will develop many practice-linked identities that may be more or less aligned. The central metaphor that we will use in this poster is drawn from Varenne and McDermott (1998) to describe culture: culture can be conceptualized as the houses we inhabit rather than as the habits we acquire (p.14). The metaphor captures how culture pre-exists any one persons engagement with it, but is always modified, to some degree, by human intervention. One can rearrange furniture, paint walls, and invite people in. One can make small renovations oneself, and larger ones if one has enough people and resources, but some renovations might be impossible. There are boundaries to how much one can change the house itself, or how one uses it. For example, in most North American homes, it is quite difficult to cook a full meal in a room without a stove or microwave. We have to cope with the house as it is, or seek to change it. Learning, from a cultural-historical viewpoint, can take many forms. Learning occurs when people enter new buildings and learn the ways to interact with others and with the space appropriately. It occurs when they innovate to create different ways of living or using the resources in the house, or when they build new structures, often by repurposing or replicating pieces of other structures (this is analogous to Gutirrez and Rogoffs (2003) repertoires of practice). As described above, this learning is distributed across people, and the artifacts and norms that are made available in the houses we inhabit. Identities cannot be disconnected from the houses or rooms in which we live.
Critical Race Theory in Education

The key assumptions of CRT, as they relate to the study of education, are: 1) racism is endemic to life in the U.S., 2) a contextual/historical analysis is necessary, 3) experiential knowledge of people of color is critical to understanding society, 4) ending racial oppression and broader forms of oppression is the ultimate goal of this
ISLS
491
ICLS2012
Volume 2: Posters
work (Dixson & Rousseau, 2006). CRT and CHL have important points of connection in that both insist on historical analyses; however, CRT is not a theory of learning. It is a theory of the influence of race on social structures (including the law and education). Therefore, CRT offers some tools for understanding power and privilege in studies of learning in context. Further, CRT insists that race is central and ever-present in contexts of learning. This contradicts CHLs emphasis on understanding each activity system on its own terms. CRT suggests that CHL should consider how race is built in to the fabric of activity systems. CRT can contribute to the house metaphor in several ways. First, CRT suggests the need to consider broader structural forces, such as legislation, and how these structures influence daily life. Second, these laws and norms are racialized. With respect to the house metaphor, we can consider the ways North American homes are constructed with a particular lifestyle in mind: one in which, for example, each member of a family has an individual bedroom, the sleeping spaces are separate from the living spaces, and so forth. People of various races, economic statuses, ethnicities, may live within the same structure quite differently. CRT suggests the need to focus attention on the way the houses we inhabit render non-normative living habits more difficult. Finally, CRT suggests the need to consider the ways different people within a building interact differently. In a home, each of the people who live there might have separate spaces; others who use the space might include housecleaners, guests, burglars, and more. Each of these people learns to use the space differently.
Queer Theory in Education

Contemporary queer theory builds heavily on the pioneering work of Judith Butler (see, e.g., 1990). Butlers intention was to highlight how gender is always performative, rather than a reflection of a stably gendered inner self. Instead, gender is seen as a context-sensitive self-referential system, in which people perform gendered ways of being in order to be recognizable to others. Just as CRT aims to challenge racial oppression, QT highlights the cost of performing gender in non-normative ways, and demonstrates that all people sometimes step out of the bounds of these gender norms, with varying costs. QT offers CHL some methods for considering the role of emotion and desire in learning, and offers the notion of performativity as a key conceptual tool for understanding issues of identity. Considering all identities as performative acts, and studying the costs associated with performing unrecognizable or stigmatized identities, can help CHL theorists tackle gender in non-essentializing ways. Additionally, CHL can be enriched by considering the powerful emotions associated with these performative acts (e.g., the fear and pain as well as the joy and comfort of performing gendered identities, and of having these identities recognized). In the house metaphor, QT suggests an emphasis on the importance of ones actions being legible. New forms and structures can be created, but only with the cooperation and understanding of others. For example, one person in a house cannot decide that the living room is actually the kitchen without other peoples active agreement. Additionally, QT resonates with CHL because both state that identity shifts in relation to the context; ones gendered way of being varies in different rooms in a house or in different types of buildings.
Significance of the Study

CHL theories, and empirical work conducted with these theories, often lacks a focus on the broad social categories that are associated with systems of oppression, and yet these categories have an impact on learning. By integrating insights from CRT and QT, the theoretical framework can consider how identities are local and contextual, as well as broadly formed and influenced. These new insights enable critical work that examines the micro-structure of how learning happens in specific contexts, while also considering the racialized and gendered nature of these contexts and therefore of the learning that takes place within them.
References
Anzalda, G., & Keating, A. (Eds.). (2002). This bridge we call home: Radical visions for transformation. New York, NY: Routledge. Butler, J. (1990). Gender trouble: Feminism and the subversion of identity. London: Routledge. Dixson, A. D., & Rousseau, C. K. (Eds.) (2006). Critical race theory in education: All God's children got a song. New York: Routledge. Gutirrez, K. D., & Rogoff, B. (2003). Cultural ways of learning: Individual traits or repertoires of practice. Educational Researcher, 32(5), 19-25. Hill Collins, P. (2000). Black feminist thought: Knowledge, consciousness, and the politics of empowerment (2nd ed.). London: Routledge. Nasir, N. S., & Hand, V. (2006). Exploring sociocultural perspectives on race, culture, and learning. Review of Educational Research, 76(4), 449-475. Varenne, H., & McDermott, R. (1998). Successful failure: The school America builds. Boulder, CO: Westview Press. Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Cambridge, Massachusetts: Harvard University Press.
ISLS
492
ICLS2012
Volume 2: Posters
Iterative Technology-based Design with Deaf/Hard of Hearing Populations: Working with Teachers to Build a Better Educational Game
Brett E. Shelton1, Mary Ann Parlin1, Jon Scoresby1, Vonda Jump1, Claudia Pagliaro2 1 Utah State University 2830 Old Main St., Logan, UT 84322 2 Michigan State University 349A Erickson Hall, East Lansing, MI 48824 brett.shelton@usu.edu, maryann.parlin@usu.edu, jscores@gmail.com, vonda.jump@usu.edu, pagliaro@msu.edu Abstract: Leveraging the use of mobile devices for educational games has been an area of increasing interest for targeted subpopulations of students, such as those who are deaf/hard of hearing. This poster outlines the involvement of Deaf Education teachers in directing the design and development of a mobile game for their students, and the ways their participation reinforced the goals of the educational objectives as embedded in unique gaming experiences.
Objectives
With this paper, we offer a rich description of one project aimed at building a global positioning GPS mobile game to help students who are deaf and hard of hearing (D/HH) learn mathematics. By working with Deaf Education teachers in an iterative design and development cycle, critical aspects of instructional design were identified and monitored as manifested in the prototype technology. Researchers created a prototype mathematics game for mobile devices equipped with GPS technology called GeePerS*Math: The Logic Machine Rescue. This paper presents the results of the initial design and implementation decisions of creating the GPS game with specific requirements for the student D/HH population. It is part of a larger project investigating GPS-enabled mobile learning games with D/HH students.
Theoretical Perspective
Successful communication and language proficiency are critical to learning and classroom performance, however D/HH students struggle with English, its vocabulary and structure, in both oral and written modes (Barham & Bishop, 1991; Kelly & Mousely, 1989). Particularly in mathematics, the frequent use of conditionals, comparisons, negation, and inferences, as well as multiple meaning words often impede the D/HH student from understanding mathematical concepts and problems (Kelly, Lang, Mousely, & Davis; 2003; Kidd & Lamb, 1993; Kidd, Madsen, & Lamb, 1993). In addition, signing these concepts (whether in ASL or an English-based sign system) without awareness or knowledge of mathematics may lead to a mis-representation of the concepts/problems causing further misunderstanding (Ansell & Pagliaro, 2001). Increases in achievement among hearing students can be associated with instructional approaches that engage them in challenging mathematics problems as recommended by the National Council of Teachers of Mathematics (NCTM, 2000). However, studies show that mathematics instruction within Deaf Education relies heavily on traditional practices of rote learning and procedural understanding with little emphasis on higher order thinking skills and true problem solving (Kelly, Lang, & Pagliaro, 2003; Pagliaro, 1998; Pagliaro & Kritzer, 2005).
Methods
Project GeePerS*Math is being implemented in schools for children who are deaf and hard of hearing as well as those who are being mainstreamed in the regular public school system in order for project developers to better understand the needs and challenges of implementation in each type of school. One project school (oral) has 420 children from K-grade 6, with 40 children who are deaf or hard of hearing in inclusive settings within the school. The second school (ASL) serves approximately 50 children, all of whom are deaf or hard of hearing. To add meaning to data collected and analyzed quantitatively, a multiple-case qualitative research study design will be followed throughout the development of this project. Small focus groups were held with participating teachers in the fall of 2010 and spring of 2011 to engage teachers in the development and evaluation of the gaming technology. Questions focused on the development of the gaming technology, as the project staff needed to understand the context in which the children learned, their level of sophistication, their attention spans, and how they might perceive the games. In the fall, teachers were asked questions that elicited the types of mathematics problems their children would need
ISLS
493
ICLS2012
Volume 2: Posters
to practice in the gaming technology, the types of games and characters necessary to engage children (cartoons, realistic), the game setup, and how the games should proceed
Findings
The focus group responses were categorized by type of question and frequency of response. The comments were used in the design process of the first game prototype (see Table 1). Table 1: Examples of Most Mentioned Design Considerations by teachers of students of D/HH Design Consideration 1. Use a superhero Game Implementation Welcome to the X-Fraction Hero Training Program. There are five stages to your training. In each part you must do a task key to progressing the narrative to get an item and ruin the plans of evil Dr. Ickles and his monsters. The storyline of the game centers on superhero training. The child using the game is designated as a superhero in training. In the story, Dr. Ickles has stolen the logic machine from the research lab and the trainee (the child playing the game) must solve a number of challenges (and mathematics problems) in order to progress through the game and rescue the logic machine. The river is 26 feet wide. The blue rope is 12 feet long. If you tie the blue and red ropes together, how much more of the red rope do you need to be able to get across the river? A: 14; B: 2; C: 12; D: 10
2.
3.
Make the mathematics fun; not just mathematics problems Make the mathematics problems straightforward
Significance
Using teachers to help create an iterative design for the production of a mobile game to learn mathematics is challenging due to scheduling, curriculum integration, training, and other resource constraints. In order to integrate audience participation in the design and development of games, we involved the teachers of students who are D/HH. This involvement facilitated communications about enjoyable elements infused and aligned with relevant mathematics standards. An exciting aspect of this project is its potential contribution to knowledge in the field as described in the previous section, but perhaps especially related to practice. By using the strategies and technology to be developed, it is expected that more D/HH children will successfully achieve mathematics milestones, and in turn, improve their chances to reach competency in mathematics.
References
Ansell, E., & Pagliaro, C. M. (2001). Effects of a signed translation on the type and difficulty of arithmetic story problems. Focus on Learning Problems in Mathematics, 23(2 & 3), 41- 69. Barham J., & Bishop, A. (1991). Mathematics and the deaf child. In K. Durkin & B. Shire (Eds.), Language in mathematical education: Research and practice (pp. 179-187). UK: Open University Press. Kelly,R., & Mousley, K. (2001). Solving word problems: More than reading issues for deaf students. American Annals of the Deaf, 146(3), 251-262. National Council of Teachers of Mathematics (2000). Principles and standards for school mathematics. Reston, VA: NCTM. Pagliaro, C. M. (1998). Mathematics reform in the education of deaf and hard of hearing students. American Annals of the Deaf, 143(1), 2228. Pagliaro, C. M., & Kritzer, K. (2005). Discrete mathematics in deaf education: A survey of teachers knowledge and use. American Annals of the Deaf, 150, 251259.
Acknowledgements
This work has been supported by the Department of Education, Office of Special Education Programs grant number H327A100038.
ISLS
494
ICLS2012
Volume 2: Posters
Enhancing Students Understanding of What Ideas Are for Knowledge Building

Chieh-Hsin Chiu & Huang-Yao Hong, National Chengchi University, Taiwan, hyhong@nccu.edu.tw Abstract: This study investigated the effects of knowledge building activities on college students understanding of the concept of idea. Participants were 41 undergraduates. Data came from a pre-post survey with open-ended questions that asked students to define what idea is and why it is important, etc. Findings indicate that engaging in knowledge building was likely to enhance students understanding of ideas as what Bereiter (1994) called conceptual artifacts or what Popper (1972) called World 3 objects that have a social life of their own and their continual improvement leads to knowledge advancement. The need for new knowledge in our societies is ever increasing. Accordingly, how to cultivate students with the capacity for creating new ideas for solving pressing societal problems has become ever more important. Ideas are essential for creating new knowledge and humans are by nature capable of generating ideas. The challenge for most learners, however, is to maintain sustained effort of improving ideas (Scardamalia & Bereiter, 2006). Traditional education tends to highlight the importance of acquiring existing knowledge (e.g., textbook knowledge), while neglecting the importance of providing students with opportunities to generate and keep improving ideas. Papert (2000) argue that most learning environments are anti-ideas. As such, students do not have a chance to work with ideas, let along developing strong sense of the importance of ideas for constructing knowledge. To change this, in his study, we directly engaged students in Knowledge Forum (KF), a CSCL environment designed based on knowledge building (KB) theory. KB is defined as a social process focused on the production and continual improvement of ideas of value to a community (Scardamalia & Bereiter, 2003). A theoretical basis underlying KB pedagogy is Poppers (1972) construct of World 3. Other than World 1 (the physical world) and World 2 (the world inside the mind), Popper argues that World 3 is comprised of ideas that are critical for KB. Bereiter (1994) argue ideas should be viewed as conceptual objects that, once created, has a social life of their own in that they can be continually improved. The purpose of this study is to figure out whether engaging students in KB would help them cultivate a World 3 view of ideas.
Method
Participants were 41 college students who took a course titled "Introduction to Living Technology" (for 18 weeks). Students in this course were guided to design innovative living technology products. Students were engaged in KB by working with ideas online by contributing, sharing, and improving ideas in KF. The course process was as follows: In the beginning, students tried to address authentic technology-related problems/issues that interested them (e.g., designing energy-saving devices for solving environmental problems); then they produced real ideas to address their problems. Next, students formed groups based on their interests and tried to solve the issues they identified by continuously working with ideas. Finally, group members tried to collectively design a new technology product. To examine students understanding of what idea is, a survey with 5 open-ended questions (e.g., what is idea?) was designed. A coding scheme was developed based on Popper's conceptual distinction (1972) between World 2 vs. World 3 view of ideas (see Table 1 for detail) in order to analyze the questions (Strauss & Corbin, 1990). Inter-rater reliability was computed to be 0.94. Table 1. Coding scheme based on Poppers conceptualization of World 2 vs. World 3 views of ideas Code World 2 view of ideas World 3 view of ideas Description Considers ideas as abstract thoughts existing in ones mind; or sees ideas as ones viewpoints, or subjective opinions; or views ideas as something derived from individual thinking. Defines ideas as an accumulation of experiences and external stimuli to improve the quality of life; or sees ideas as means for people to solve problems; or sees ideas as collectives generated from group discussion, communication and/or sharing for co-constructing community knowledge; or views ideas as public (rather than personal) entities that can be exchanged and improved, just like a real-life objects. Example Idea is a kind of abstract thought produced while thinking (S19); Ideas are ones point-of-views about something (S10); and Ideas can help improve ones intelligences (S23). Idea can make our lives better and more convenient (S02); Ideas originate from our demands, and our needs for solving problems (S06); Through sharing ideas of others, we can deduce, think, and integrate these ideas to become a better idea (S16); and Idea is a product through the thinking process; but it can become a concrete object via group discussion (S39).
Results
Online KF activities. Pre-post comparisons were made between the two KB phases (using midterm as a separating point, 9 weeks for each phase) for online KB activities, including # of notes contributed (M=13.61,
ISLS 495
ICLS2012
Volume 2: Posters
SD=11.28 vs. M=13.88, SD=12.18, t=-0.122, p>.05), # of problem worked on (M=7.93, SD=6.11 vs. M=5.98, SD=4.99, t=1.969, p>.05), # of notes built-on (M=11.07, SD=10.98 vs. M=9.76, SD=12.28, t=0.610, p>.05), # of scaffolds used (M=10.07, SD=9.52 vs. M=12.34, SD=14.88, t=-1.085, p>.05), # of notes read (M=225.44, SD=138.44 vs. M=319.07, SD=204.69, t=-3.283, p<.01), and # of notes with keywords (M=10.83, SD=9.94 vs. M=7.39, SD=7.17, t=2.564, p<.05). Overall, the KF activities increased in Phase 2, but except for two measures (# of notes read and # of notes with keywords), no significant differences were found. As baseline information, the findings suggest that the time and effort spent on online learning and discussion in KF were quite equally distributed over the two phases (although students were slightly more active in phase 2). Moreover, all above KF measures were found significantly correlated with one another (Pearson r between .59 and .99, p<.01 for all 15 relationships). This suggests that the more active the participants were in one KB activity (i.e. KF measure), the more likely they would be active in other activities. Further, comparison between pre and post assessment has found that there was no significant difference in terms of use of lower-level thinking scaffolds in KF (M=3.26, SD=3.04 vs. M=2.31, SD=3.15, t=1.94, p>.05) but there was a significant difference in the use of higher-level thinking scaffolds (M=0.11, SD=0.28 vs. M=1.68, SD=2.15, t=-4.73, p<.001). So, while the activity amount in KF is similar between the two phases, the quality of thinking activity in KF in phase 2 was enhanced. Change in students concept of idea. There was significant decrease in student rating in terms of their World 2 view of ideas (M=0.62, SD=0.30 vs. M=0.46, SD=0.31, t=2.39, p<.05). Further, there was significant increase in student rating in terms of their World 3 view of ideas (M=0.94, SD=0.45 vs. M=2.04, SD=0.91, t=-7.86, p<.001). The findings suggest that students started to consider ideas more as tangible, conceptual objects for collective knowledge advancement (i.e. Poppers Word 3 view of ideas). In addition, in terms of the process of how students work with ideas, we traced the development of ideas that students worked with their final technology products over time. In the present study, students in total generated 11 products. As an example to demonstrate the process of idea improvement, we analyzed students discourse in KF that contained ideas related to one product about shoes. Table 2 shows 5 key diversified ideas related to the main idea about shoe improvement: (1) Replaceability--e.g., The bottom of the shoes always wears off quickly, if the sole of shoes can be changeable, it can save money (S15); (2) Comfort--e.g., Shoes are often made by plastic and air can not get through so its not comfortable wearing them (S20); (3) Waterproof--e.g., the shoes will not be wet easily if we invent some waterproof spray that can form a thin layer on shoes (S35); (4) Nano tech--e.g., [Using] nano-technology to make shoe surface to be waterproof! (S35); and (5) Aesthetics--e.g., Is it possible to invent shoes that are both beautiful and practical (especially for wearing in rainy days? (S30). Figure 1 further shows the evolving process of idea improvement over time(each line shows when, and how frequent, a key idea was referenced, built-on, elaborated, or improved over a duration of about 10 weeks). Table 2. 5 key ideas related to shoes improvement # of Duration # of notes # of notes in weeks read discussants Idea1: Replaceability 19 10 35.84 14 Idea2: Comfort 6 7 39.17 6 Idea3: Waterproof 9 8 24.63 6 Idea4: Nano tech 8 6 31.63 5 Idea5: Aesthetics 3 8 35.33 3 Figure1. Duration of idea improvement about shoes
Conclusion
The challenge in all knowledge-based organizations or schools is sustained creativity: working with and developing ideas into powerful and useful solutions, products, or theories (Scardamalia & Bereiter, 2003). The result of this study reveals that engaging students in a KB environment is helpful to help students develop a more informed, Poppers World 3 view of ideas, making them more likely to treat ideas as concrete, public and social knowledge objects. Further analysis on idea improvement will be conducted to triangulate the findings.
References
Bereiter, C. (1994). Constructivism, Socioculturalism, & Popper's World 3. Educational Researcher, 23(7), 21-23. Papert, S. (2000). Whats the big idea: Towards a pedagogy of idea power. IBM Systems Journal, 39, 3-4 Popper, K. R. (1972). Objective knowledge: An evolutionary approach. Oxford: Clarendon Press. Scardamalia, M., & Bereiter, C. (2003). Beyond brainstorming: Sustained creative work with ideas. Education Canada, 43(4), 4--7, 44. Scardamalia, M., Bereiter, C. (2006). Knowledge building: Theory, pedagogy, and technology. In K. Sawyer(Ed.), Cambridge Handbook of the Learning Sciences, 97--118. NY: Cambridge Univ. Press. Strauss, A. L., & Corbin, J. (1990). Basics of qualitative research: Grounded theory procedures and techniques. Newbury Park, CA: Sage Publications.
ISLS 496
ICLS2012
Volume 2: Posters
Teacher-Education Students Beliefs in Science Teaching in a Knowledge Building Environment

Chih-Hsuan Chang & Huang-Yao Hong, National Chengchi University, Taiwan, hyhong@nccu.edu.tw Abstract: Teaching beliefs have been regarded as an important factor that influences teaching practices; therefore teacher-education programs should pay attention to their students teaching beliefs. The purpose of this study was to help teacher-education students develop more informed science teaching beliefs. Participants were 28 prospective science teachers who took a course about science teaching. Knowledge building pedagogy was employed to engage student in constructing and sharing knowledge in this course. Data came from online activities and a belief survey with open-ended questions. The results showed that after engaging in knowledge building for a semester, the teacher-education students developed more student-centered teaching beliefsthey tended to see teaching practices as facilitating students to explore science rather than accumulating knowledge from science textbooks. A key to effective education is quality teaching. How teachers teach however is deeply influenced by their beliefs about education, teaching, learning, and students (Pajares, 1992). Studies suggested that teaching beliefs not only impact teachers' instructional plans, but their actual teaching practices (Richardson, Anders, Tidwell & Lloyd, 1991; Stuart & Thurlow, 2000; Wilson, 1990). Teaching beliefs, however, was usually developed over a long period of time, and they are influenced by teachers past learning experiences as well when they were students (Nespor, 1987; Rath, 2001; Stuart & Thurlow, 2000). Thus, during the stage of teacher education, it is essential to examine teacher-education students teaching beliefs, and more importantly to help them build more meaningful and productive teaching beliefs. To this end, this study employed a deep constructivist approach, i.e. knowledge building, as an instructional approach. Knowledge building is a social process focused on sustained production and improvement of ideas of value in a community (Scardamalia & Bereiter, 2003). In a knowledge building community, members collectively construct and innovate knowledge through idea production and improvement. It is posited that learning in a more constructivist manner would provide more opportunities for teacher-education students to reflect on their teaching beliefs during the course, and eventually construct more informed teaching beliefs.
Method
Participants were 28 teacher-education students who attended a course about science teaching at a national university in Taiwan. The course lasted for one semester (18 weeks). Instructional design was based on knowledge building pedagogy, with Knowledge Forum (KF) being used to complement students working with ideas and knowledge. All students ideas and discussion were recorded in KF. Throughout the course, students were required to practice their teaching twice. During their practices, they were guided by knowledge building pedagogy to discuss and reflect their teaching both in class and online. Further, the instructor and students used open-ended questions for discussion about how to improve their teaching. Both the instructor and students worked as community members. Everyone could freely propose questions for discussion and deeper reflection. Data came from online performance in KF and a pre-post survey containing six open-ended questions: (1) What do you think is an ideal way to teaching science, and why? (2) What do you think are the key factors to ensure successful science teaching, and why? (3) What constitutes an ideal science teacher, and why? (4) What do you think is an ideal way to learn science, and why? (5) What do you think are the key factors to ensure successful science learning, and why? (6) What does an ideal science learning environment meant to you, and why? The survey data were content analyzed based on the coding scheme (teacher-centered vs. student-centered teaching) emerged during the coding process (see Table 1). The inter-rater reliability, using 50% of this data and Spearman correlation, were computed to be 0.92 (p<.01). Table 1. Coding Scheme Category Teachercentered teaching beliefs Code, description, and example Teacher as providers: Teachers should provide students with comprehensive teaching aids and laboratory equipment. Example: [Teachers should provide] complete laboratory materials and various equipments (s11). Teacher as knowledgeable expert: Teachers have the expertise to impart knowledge. Example: [Teachers should] transfer basic knowledge, by giving lectures and helping students establish knowledge base (s11).
ISLS
497
ICLS2012
Volume 2: Posters
Studentcentered teaching beliefs
Teacher as model/demonstrator: To teach abstract knowledge by demonstration Example: Teaching through demonstration and experiment is important. (s08). Teacher as enabler for creating an open learning environment: Teachers make good use of open learning environment to give students space to perform. Example: Open and comfortable environment. (s04). Teacher as promoter for adaptiveness and creativity: Help students develop ability to be diverse and innovative thinkers, and learn from life and use knowledge more flexibly. Example: Respond to students differences and needs by using suitable teaching pedagogy (s21). Teacher as motivator for inspiring students to be initiative and to reflect and explore sciences: Stimulate curiosity and interest in science and to encourage students to explore science problems and think reflectively. Example: Stimulate learning motivation and curiosity (s09).
Result
Online KF activities. Pre-post comparisons were made between two KB phases (using midterm as a separating point, 9 weeks for each phase) for three key online KB activities/measures, including # of notes contributed (M=11.37, SD=4.78 vs. M=13.78, SD=5.22, t=-2.28, p<.05), # of notes read (M=124.74, SD=70.86 vs. M=189.00, SD=139.60, t=-2.72, p<.05), and # of notes built-on (M=9.78, SD=4.46 vs. M=12.89, SD=4.69, t=3.11, , p<.01). Overall, the findings suggest that the time and effort spent on collaborative learning and discussion in KF were progressively increasing over time. Belief change. Using the above coding scheme for analysis, it was found that the participating students teachercentered beliefs have changed significantly after the semester; there was a significant difference between pretest and post-test (M=1.08, SD=1.35 vs. M=0.46, SD=0.76, t=2.54, p<.05). In contrast, it was found that the participants student-centered beliefs also significantly changed after the semester; there was a significantly positive increase in ratings from the pre-test to the post-test (M=5.15, SD=2.49 vs. M=8.00, SD=3.26, t= -4.58, p<.01). Further, when looking into the specific codes, it was found that the main factor that was significantly changed (decreased in ratings) in teacher-centered beliefs was Teacher as demonstrator/model (M=0.31, SD=0.55 vs. M=0.04, SD=0.20, t=2.27, p<.05). On the other hand, the key factor for students to increase ratings in their student-centered beliefs came from Teacher as motivator (M=2.92, SD=1.83 vs. M=5.23, SD=2.23, t=4.30, p<.01).
Conclusion
To summarize, after knowledge building for a semester, students became less likely to see teachers as a demonstrator who should directly tell students what to do or learn. Instead, students were more likely to emphasize self-initiated and self-directed inquiry learning and exploration. In other words, it is more important, from the participants perspective, for teachers to play a role as a motivator than a knowledge teller. We are entering into a knowledge age, during which knowledge is created rapidly; accordingly, sustained knowledge sharing and innovation is essential. Traditional instruction, however, tends to highlight knowledge assimilation, rather than knowledge creation. Arguably, learning should not be limited to only textbook knowledge; likewise, teaching should not be merely based on pre-determined scripts (Sawyer, 2004). On the contrary, in a knowledge age, important qualities that students should acquire are such as creative and collaborative capacity (Sawyer, 2004), and to this end, teachers need to develop more knowledge-creationoriented beliefs so as to help students develop these desirable capacities. Finally, the study is still work-inprogress and more discourse data will be analyzed in order to better understand the process of belief change for teacher-education students.
References
Nespor, J. K. (1987). The role of beliefs in the practice of teaching. Journal of Curriculum Studies, 19(4), 317328. Pajares, M. F. (1992). Teachers' beliefs and educational research: Cleaning up a messy construct. Review of educational research, 62(3), 307. Raths, J. (2001). Teachers beliefs and teaching beliefs. Early childhood research and practice, 3(1), 1-10. Richardson, V., Anders, P., Tidwell, D., & Lloyd, C. (1991). The relationship between teachers' beliefs and practices in reading comprehension instruction. American Educational Research Journal, 28(3), 559-586. Sawyer, R. K. (2004). Creative teaching: Collaborative discussion as disciplined improvisation. Educational Researcher, 33(2), 12-20. Scardamalia, M., & Bereiter, C. (2003). Knowledge building. In Encyclopedia of Education (2nd ed., pp. 13701373). New York: Macmillan Reference, USA. Stuart, C. & Thurlow, D. (2000). Making Ittheir own: Preservice teachers experiences, beliefs, and classroom practices. Journal of Teacher Education,51(2),113-121. Wilson, S. M. (1990). The secret garden of teacher education. Phi Delta Kappan, 72, 204-209.
ISLS
498
ICLS2012
Volume 2: Posters
Investigating Teacher Change in Creating Classroom Knowledge Building Communities

Katerine Bielaczyc, Learning Sciences Lab, NIE Singapore, kateb369@gmail.com Sunhee Paik, Learning Sciences Lab, NIE Singapore, sunhee.paik@nie.edu.sg John Ow, Innova Primary School, Singapore, john_ow_eu_gene@mac.com Abstract: Our focus is on how teachers navigate the pedagogical shifts in moving from traditional classrooms to knowledge building community classrooms (Scardamalia & Bereiter, 2006). We ground our inquiry in Ideas First, a design-based research program within a Singapore primary school. Based on data drawn across 5 years of implementation, we examine three areas of teacher change: (1) understanding student understanding, (2) engaging in pedagogical moves supporting students inquiry, and (3) developing representational practices for making knowledge work visible for reflection. A major emphasis in contemporary educational models is on investigating ways to create classroom cultures that mirror disciplinary cultures, thus socializing students into working with knowledge in ways consistent with disciplinary norms and practices (e.g., Gresalfi & Cobb, 2006; Herrenkohl, et. al., 1999; Hogan & Corey, 2001). Through participating in student-driven inquiry, learners are meant to develop the agency and know-how necessary for creatively working with disciplinary knowledge. However, such approaches require teachers and students to engage in new modes of teaching and learning that tend to be quite different from those in more traditional classrooms. This need gives rise to our central research question: What pedagogical changes are needed to support new approaches to learning and teaching in the Knowledge Age? For the past several years we have been engaged in a design-based research program in a Singapore primary school. Ideas First is a two-year science program co-designed with primary school teachers that has been operating in fifteen grade 3 and grade 4 classrooms since 2006 (Bielaczyc & Ow, 2007, 2010). The program is based on the vision of a knowledge building community where students work to advance the science understanding of the classroom community through engaging in collectively building knowledge in response to problems of understanding (Scardamalia & Bereiter, 2006). Students are supported in their work by Knowledge Forum, a computer-based learning environment that allows learners to construct a communal multimedia knowledge base that visually traces the community inquiry (Scardamalia, 2004). Creating knowledge building communities in classrooms is challenging. When the teachers are themselves new to such a culture, classroom change becomes even more challenging. The P3 and P4 Science teachers involved in the Ideas First program work together as a local community in the school, supporting each others development through shared engagement in a common P3 and P4 Ideas First curriculum. The program also engages teachers with the existing international community among teachers, researchers and other educational stakeholders who are working to implement the model in their local contexts (http://ikit.org). Professional development (PD) sessions are held throughout the year. The PD sessions focus on teacher inquiry into knowledge building principles, classroom practices, and student artifacts. What does it mean to teach in classrooms where student-driven inquiry, diversity of ideas, and collective products are the norm? We believe that describing teachers as facilitators or a guide on the side, while accurate, does not provide the necessary level of detail needed to understand teacher change. Our goal in the poster presentation is to provide insights into the ways in which teachers come to understand new educational models and shift the ways in which they support students in their learning.
Investigating Teacher Change in Ideas First

We have been working closely with teachers as they work to create classroom knowledge building communities. Teacher discussions during professional development and interview data indicate that teachers face many challenges in teaching in this way, including such difficulties as: Remaining open to students ideas, even when they differ greatly from the norm Allowing students to explore (and even fail to find solutions) without intervening with the answer Feeling comfortable enough to say I dont know to students Listening and trying to understand where the students are coming from Working at the collective level and attending to what the community knows
We investigate the ways that teachers have worked to address such challenges, specifically examining teacher learning and pedagogical change in three key areas: Working to understand student understanding In order to work with the diversity in student work, teachers need to understand the variations in student understanding underlying such richness. The focus
ISLS
499
ICLS2012
Volume 2: Posters
moves away from matching student responses to a model answer to instead determining the potential within all students ideas (even if they appear wrong). Engaging in pedagogical moves that support students inquiry In order to support student-driven inquiry, teachers need to develop a rich repertoire of pedagogical moves that permit flexible response to student work and interactions. Developing representational practices for making visible knowledge and learning processes For students to learn how to take control of the learning process, it is critical to make visible to students the elements of the process and their progress in understanding.
Over the first 5 years of the project we have collected teacher data from interviews, classroom videos, and PD sessions (involving 12+ teachers each year). The data includes case studies of 3 particular teachers during their first 2 years with the project, and continued study with one of these teachers for his first 5 years. This data corpus permits us to look both across a broad range of participants as well as deeply into particular cases. In the poster we will discuss how teachers worked to unpack student meaning-making and to use the diversity in student work to enrich classroom dialogue and create linkages to disciplinary understanding and practices. In addition, we document the broad array of features that contributed to and inhibited teacher change, including teacher knowledge and beliefs, school-based practices, and MOE policies. In using a design-based research approach, we are also able to detail interventions used to support teacher change processes.
Significance
A broad research and policy literature has evolved over the past two decades on the needed changes in learning and teaching for education in the Knowledge Age (e.g., Darling-Hammond & Sykes, 1999; Fullan, 2007; Lagemann & Shulman, 1999). One common theme is the centrality of teachers and, in turn, teacher learning, in classroom reforms. In describing the course that educational research will need to take, Linda DarlingHammond (1996) argues that the problem of the 21st century is the advancement of teaching, and its resolution will depend on our ability to develop knowledge for a very different kind of teaching than what has been the norm for most of this century (p. 7). The poster presentation contributes to advancing our understanding of teacher change in enacting new educational models, supporting the development of knowledge for a very different kind of teaching needed in the 21st century.
References
Bielaczyc, K. & Ow, J. (2007) Shifting the social infrastructure: Investigating transition mechanisms for creating knowledge building communities in classrooms. In Proceedings of the International Conference for Computers in Education. Bielaczyc, K. & Ow, J. (2010). Making knowledge building moves: Toward cultivating knowledge building communities in classrooms. In Proceedings of the International Conference of the Learning Sciences. Gresalfi, M., & Cobb, P. (2006). Cultivating students discipline-specific dispositions as a critical goal for pedagogy and equity. Pedagogies, 1, 49-58. Herrenkohl, L.R., Palincsar, A.S., DeWater, L.S., and Kawasaki, K. (1999). Developing scientific communities in classrooms: A sociocognitive approach. Journal of the Learning Sciences, 8, 451-493. Hogan, K., & Corey, C. (2001). Viewing classrooms as cultural contexts for fostering scientific literacy. Anthropology & Education Quarterly, 32(2), 214243. Darling-Hammond, L. (1996, March). The quiet revolution: Rethinking teacher development. Educational Leadership, 53(6), 4-10. Darling-Hammond, L. & Sykes, G. (Eds.) (1999) Teaching as the learning profession: A handbook of policy and practice. San Francisco, CA: Jossey-Bass. Fullan, M. (2007). The new meaning of educational change 4th edition. New York: Teachers College Press. Lagemann, E.C. & Shulman, L.S. (Eds.) (1999) Issues in Educational Research: Problems and Possibilities. Jossey-Bass: San Francisco Scardamalia, M. (2004). CSILE/Knowledge Forum. In Education and technology: An encyclopedia. (pp. 183192) Santa Barbara: ABC-CLIO. Scardamalia, M. & Bereiter, C. (2006). Knowledge building: Theory, pedagogy and technology. In R. K. Sawyer (Ed.), Cambridge handbook of the learning sciences (pp. 97-115).Cambridge University Press.
Acknowledgements
We would like to thank the Ideas First Team --- both the researchers and teachers who made this work possible. This work was funded by two Singapore MOE grants to the first author through the National Institute of Educations Learning Sciences Laboratory (R8019.735.PM07 and OER 18/10 KB).
ISLS
500
ICLS2012
Volume 2: Posters
Transformative Experiences and Biological Evolution: Facilitating Deep Engagement

Benjamin C. Heddy & Gale M. Sinatra, University of Southern California heddy@usc.edu Abstract: The goal of this study was to explore the effectiveness of the Teaching for Transformative Experiences in Science (TTES) model for facilitating deep learning as compared to a typical teaching method implemented in science class when learning about evolution. We then wanted to explore what these experiences look like. Both quantitative and qualitative methods were used to explore participant experiences. This study seeks to improve teaching by exploring how to facilitate out-of-school engagement.
Dewey believed that the educational experience should expand the perception of the students experiences above and beyond the classroom (Dewey, 1938). So how can we create a learning environment that can expand a students perception of experiences outside the classroom? Pugh (2002) developed the concept of transformative experience which occurs when students use concepts to experience their world in a meaningful way. Transformative experience can be defined using three qualities: 1) active use of the concept, 2) an expansion of perception, and 3) an expansion of value with the concept (Pugh, 2002) and is considered a continuum from in-school engagement to high out-of-school engagement. Teaching for Transformative Experience in Science model or the TTES model has three general methods: (a) framing content in terms of its experiential value, (b) scaffolding re-seeing, and (c) modeling transformative experiences (Pugh & Linnenbrink, 2007). TTES was found to be an effective teaching method for facilitating conceptual change and positive affect in previous work (Author & Author, 2011; Pugh, Linnenbrink-Garcia, Koskey, Stewart, and Manzey, 2010). The goal of the current research was to provide a closer examination of the experiences that students have with biological evolution concepts when implementing the TTES model. Evolution is typically seen by students as something not relevant to their everyday lives. TTES should encourage students to engage with biological evolution ideas out-of-school. But what do these experiences look like? How do these experiences influence learning about evolution? These are the questions that drive this research.
Method Participants
Participants were recruited from an educational psychology participant pool at a large southwestern university in the United States. Participants ranged from age 18 to 50, (mean = 27). Participants were 70.9% female. The ethnicities were 58.2% Caucasian. There were 28 participants in the treatment and 27 in the comparison.
Procedures
The treatment group was taught about evolution using the TTES model. The comparison group was taught by implementing a typical college biology lecture, reading, and discussion method. On Day 1, both conditions received the same lecture about evolution, with an added emphasis on the instructors own transformative experiences with the concepts and the value of the material for the treatment condition. On Day 2 the comparison group read a text about evolution and then had a whole group discussion, whereas the instructor for treatment condition modeled transformative experiences, scaffolded student experiences, and discussed value. On Day 3, both groups received instruction similar to Day 2 before completing the Transformative Experience Scale after instruction. The transformative experience scale contained 20 Likert scale items and three open-ended questions. For the presentation, there will be a focus on the open-ended questions. The first question addressed active use and asked students to give an example of how they used the evolution ideas they learned. The second question assessed expansion of perception and asked participants to give an example of how their experiences changed due to learning the evolution ideas. The final question assessed experiential value and asked participants to give an example of how they valued the evolution ideas they learned. Answers were coded and analyzed for degrees of engagements. The coding scheme was as follows: a 0 was given for no answer, an incoherent answer, or statements that participants did not use or value the concepts. A 1 was given for stating the concepts were used only in class. A 2 was given when students claimed that they used the concepts outside of class but did not elaborate. Finally, a 3 was given when participants stated they used the concepts outside of class and
ISLS
501
ICLS2012
Volume 2: Posters
provided an elaborated example. The scores were analyzed for all three dimensions and summed for an overall degree of engagement score. Conceptual knowledge of evolution was assessed using the Evolutionary Reasoning Scale (Shtulman, 2006). This survey contained 14 closed-ended and open-ended questions and was implemented as pre-test and post-test measure.
Results
The treatment condition scored significantly higher in degree of engagement when the coded open-ended scores where subjected to an ANOVA, F (1, 53) = 21.85, p < .0005, 2 = .29 (these findings are consistent with those for the Likert-item analysis). These finding suggests that the treatment condition engaged in significantly higher levels of transformative experience and deeper out-of-school engagement than the comparison condition. The treatment condition also scored significantly higher on the active use dimension F (1, 53) = 19.49, p < .0005, 2 = .27,the expansion of perception dimension F (1, 53) = 8.66, p = .005, 2 = .14 and the experiential value dimension F (1, 53) = 7.847, p = .007, 2 = .13. Examples of students out-of-school engagement with evolution concepts will be shared. As an example of active use, one participant stated, When I was at the zoo, I thought about variation when I looked at elephants. African elephants are taller and have larger ears. Expansion of perception is seen in this statement, I think I will pay more attention to a species and see why it looks different and what are the chances of survival. Also, I never thought I was related to a plant! Experiential value, is evident in this statement, It (learning about evolution) was an excellent reminder to stop and reflect on what is going on around me. It makes my life more meaningful and answers questions about existence while creating new ones. Engagement and learning data were submitted to a multivariate repeated measures ANOVA. Results showed a significant interaction between time and condition, F(1, 53) = 13.47, p =.001, 2 = .20. Follow-up univariate analysis showed a significant gain in learning at at posttest for the treatment condition F(1, 53) = 21.27, p < .0005, 2 = .29.
Discussion
The TTES model was effective for facilitating transformative experience with evolution concepts and also predicted higher learning. This may be due to the high levels of engagement and personal relevance facilitated by the TTES model. These findings have implications for teaching science and evolution. Science concepts and specifically evolution are typically not viewed as influencing a students everyday experience in a personally meaningful manner and being deeply engaging manner. The TTES model may foster engagement in evolution ideas which could facilitate deep learning. Engaging in transformative experiences allows students to realize the personal relevance of evolution ideas. Innovative teaching methods such as TTES may facilitate deep understanding and engagement with evolution concepts.
References
Dewey, John (1938). Experience and education. New York: Touchstone. Pugh, K. J. (2002). Teaching for transformative experiences in science: An investigation of the effectiveness of two instructional elements. Teachers College Record, 104, 1101-1137. Pugh, K. J., Linnenbrink-Garcia, L., Kelly, K. L., Manzey, C., & Stewart, V. C. (2007). Learning to teach for conceptual change and transformative experiences. Paper at the American Educational Research Association conference, Chicago, IL. Pugh, K. J., Linnenbrink-Garcia, L., Koskey, K. L. K., Stewart, V. C. and Manzey, C. (2010), Motivation, learning, and transformative experience: A study of deep engagement in science. Science Education, 94: 128. Shtulman, A. (2006) Quantitative differences between nave and scientific theories of evolution. Cognitive Psychology, 52, 170-194.
ISLS
502
ICLS2012
Volume 2: Posters
Argument-Based Inquiry and Students with Disabilities: Improving Critical Thinking Skills and Science Understanding
Jonte C. Taylor The University of Iowa jonte-taylor@uiowa.com
Abstract: The author will provide an overview of the Science Writing Heuristic (SWH); an argument-based inquiry approach to science instruction. This approach shown promise in improving science scores and critical thinking assessment scores for students with disabilities. The SWH approach incorporates the use of argument and hands-on experiences to increase science achievement by improving critical thinking skills. Instructional components of the SWH approach and data supporting its use for students with disabilities will be presented. Students with disabilities have long lagged behind their general education peers in all academic areas. As debate persists regarding whether or not general science education in is in a state of crisis, there is little disagreement in that when it comes to students with disabilities and science education there is a significant deficit in achievement. Since the late 1980s, advocates for students with disabilities have recommended greater emphasis be placed on teaching science to students with disabilities (Mastroprieri & Scruggs, 1992; Patton & Andre, 1989). However, even with increased accountability standards for students with disabilities (NCLB, 2001) and the push for better general education access, students with disabilities have not made significant gains in the areas of science education. In general, students with mild intellectual disabilities are rarely considered in the teaching and learning of science. Mastropieri, Scruggs, and Boone (1998) conducted a review of the research to that point of students with disabilities and science education. They reported a number of issues as possible causes of poor science achievement for students with disabilities. Some of these issues included: teachers manuals not addressing students with disabilities perspectives, literacy demands, and teaching approaches (Mastropieri, Scruggs, & Boone, 1998). As inclusive and collaborative teaching of students with special needs increases in general education classrooms, a number of additional issues have emerged as concerns. Many general education teachers feel ill prepared to teach students with disabilities (Soodak, Podell, & Lehman, 1998). Patton, Polloway, and Cronin (1990) reported a number of disturbing revelations including: over two-fifths of special education teachers receive no training in science education, over one-third of students with disabilities in selfcontained classes receive no science instruction, nearly two-thirds of special education teachers use the general science education textbook to teach.
The Science Writing Heuristic (SWH)

The Science Writing Heuristic (SWH) is a teaching approached designed to involve students in inquiry, argumentation, and experimentation as a means of learning. Based on the theories that include writing-to-learn strategies as a means of teaching and learning science (Yore, Bisanz, & Hand, 2003), the SWH approach was developed as a means of providing students a conceptual framework for science knowledge through constructive debate and experimentation while simultaneously allowing for multiple means of expression to display robust understanding of science themes and concepts. Using teacher directed, individual and small group instruction, students learn how to generate testable questions, conduct investigations, generate claims based on their investigations, and back their claim with evidence.
Significance
The (SWH) is a structured inquiry based approach that includes the instructional components reported by Scruggs, Mastropieri, and Boon (1998) and Swanson (2001) to be essential. Specifically the approach provides teachers and students with writing templates to guide science inquiry. Lessons focus on big ideas (i.e., the major concepts that students should fully understand at the completion of the unit) instead of isolated facts. Explicit instruction, extended practice and formative feedback are provided to students as they learn science argument. Using teacher directed, individual and small group instruction, students learn how to generate testable questions, conduct investigations, generate claims based on their investigations, and back their claim with evidence. While based in inquiry-teaching theory, the SWH approach incorporates teaching and learning structure that allows access to better educational opportunities more students including those with mild intellectual disabilities. The use of structured components in tandem with inquiry-based instruction has been recommended by researchers as a more viable means of including all students and maximizing science learning opposed to one dominant teaching methodology (Mayer, 2004; Scruggs & Mastropieri, 2007). The components
ISLS
503
ICLS2012
Volume 2: Posters
of the SWH approach allow students with disabilities to actively engage in science instruction and scientific practices through the improvement of critical thinking skills.
The aims of the authors were to tests the efficacy of the SWH approach with an experimental design using a random assignment of buildings to treatment and control groups. The project seeks to examine the efficacy of the SWH approach for students with disabilities by tracking the following: 1. Grade 3 6 students with disabilities performance comparing treatment and control scores on the Iowa Test of Basic Skills (ITBS) science subscale. 2. Grade 5 6 students with disabilities performance comparing treatment and control scores on the Cornell Critical Thinking Test (CCT) and subscales. This random control trial study of the SWH approach included 48 elementary schools (24 treatment and 24 control) in the state of Iowa. The study was designed to examine impact of the SWH approach on students in grades third to sixth with disabilities scores in science achievement (ITBS-Science subscale, grades 3 - 6) and critical thinking skills (CCT, grades 5 6).
Findings, Conclusions, and Implications

Students ITBS scores, CCT test results (grade 5 6 only) were used to measure student outcomes and the effectiveness of the SWH approach. To date, grades 3 6 students ITBS data from 2009 and 2010 have been collected and a database has been developed for analysis. The CCT pre-test was administered at the beginning of the school year and the post-test at the end of the school year (2010-2011) is currently in progress. Preliminary data suggest that students with disabilities in the treatment group scored higher than students in control group on ITBS science scores and CCT pre and post-test difference scores. A comprehensive analysis and discussion of the ITBS, Critical Thinking Skills on results for Year 1 will be available after the completion of the data collection. The data generated and findings presented will be used to inform the recommendations and conclusions of the study.
Acknowledgement
The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305B10005 to The University of Iowa. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.
ISLS
504
ICLS2012
Volume 2: Posters
Fostering Teachers Use of Talk Moves to Promote Productive Participation in Scientific Practices
William R. Penuel, University of Colorado Boulder, UCB 249, Boulder CO 80305, william.penuel@colorado.edu Yves Beauvineau, Farrell B. Howell School, 14250 E. Albrook Drive, Denver, CO 80239, Yves_Beauvineau@dpsk12.org Angela Haydel DeBarger, Savitha Moorthy, SRI International, 333 Ravenswood, Menlo Park, CA 94025 angela.debarger@sri.com, savitha.moorthy@sri.com Abstract: This paper explores variation in how teachers use talk moves intended to help students engage productively in science discourse. Participants in the study were 13 middle school Earth science teachers from a large, diverse urban school district and their students. Teachers implemented tools intended to help students orient to other students contributions in class, particularly their reasoning about weathering and erosion and about plate tectonics.
Introduction
Strengthening students competency with scientific practices related to specific science content is a key goal of the framework for the next generation of science standards in the United States (National Research Council, 2011). One line of inquiry into how to facilitate students productive participation in scientific practices focuses on talk moves as tools for teachers. Talk or conversational moves are discursive practices teachers can use to elicit student thinking, promote scientific reasoning, encourage students to explain their thinking so others can understand, and build knowledge within classroom communities (O'Connor & Michaels, 2011). Some talk moves, such as revoicing, position students differently vis--vis one another and scientific knowledge in ways that support these goals (O'Connor & Michaels, 1993). At present, learning sciences researchers are only beginning to investigate how to support teachers effective use of talk moves as tools for promoting science learning (OConnor & Michaels, 2011).. Although there is abundant research on how the processes by which eliciting student thinking, promoting productive participation in practice, and adapting teaching to students facilitate learning (National Research Council, 1999), professional development programs focused on helping teachers make productive use of talk moves in science classrooms have emerged only in recent years.
The Current Study

In this paper, we analyze variation in implementation of a program of professional development designed to help teachers make effective use of talk tools to support student learning in an investigation-based middle school Earth science program. A total of 13 teachers and their students from a single large urban school district in the US participated in the study. The study of implementation variation is useful in helping teams prepare to expand the reach of learning sciences innovations that have been developed and tested in a few classrooms. Here we focus on variation associated with teachers uses of tools to support students developing proficiency with explanation in Earth science. We present vignettes from videotaped observations of practice to illustrate variations in teachers use of talk tools to help students formulate compare their claims and reasoning to peers.
Variation in Generativity of Tool Uses for Helping Students Think with Others
In this paper, we focus on variation in use in one type of move, the weighing perspective move. In this move, teachers encourage students to consider their ideas in relation to other ideas. This move is a key strategy for creating a culture of argumentation within the science class, because peers are invited to agree or disagree and to argue using evidence to support different views. A strong example of using the weighing perspectives move occurs in Ms. Thomass class, at the end of an investigation on weathering. The teacher is checking students understanding of core ideas of the investigation by posing a series of three clicker questions. For each question, students first write their thinking individually on a handout to explain the reason for their choice while voting with the response systems. After collecting all students clicker votes, Ms. Thomas displays the distribution of responses for each answer choice on the screen and engages the whole class in discussing their thinking for each answer choice. In discussing the problematic answer choice Live plants release carbon dioxide to break down rocks to the clicker question How might plants break rocks down through physical weathering? Ms. Thomas first elicits students reasoning when asking Why might someone have chosen that? This initial move invites all students to offer an explanation even if they had not selected that answer, as illustrated in the following excerpt: T: We are gonna start with B. Why might someone have chosen that? [pauses] Ok
ISLS
505
ICLS2012
Volume 2: Posters
S1:
I think someone might have chose that, chosen that, because I think when we saw the picture of the big rock that had broken I think someone said it was plants that could have broken it, broken it down, and also because carbon dioxide has, I think, doesn't it have acid in it that dissolves it?
We see here that the first move successfully engages a student who offers a reason why a student might have chosen one of the responses. After revoicing and linking the thinking shared to a previous learning experience, Ms. Thomas uses the weighing perspectives move to engage students in arguing for their ideas: T: Carbon dioxide can mix with water and oxygen in the atmosphere to form a weak acid, that's absolutely true. Very good, very good, and S1 said someone may have remembered our conversation from that original elicitation question when we saw those big boulders and that grassy field, do you guys remember that? Cool. Does anybody want to disagree with or agree with S1? I want to disagree with myself. [laugh] Oh cool! One second. S2? Eh, I disagree with S1 because of the plant, well, it's really lichen, and lichen is an organism that sprays chemicals on rock.
S1: T: S2:
This particular weighing perspective move succeeds by eliciting a different response, one that closer approximates (but does not yet meet) the goal or target understanding and by inviting a self-correction from S1. A lesson led by Mr. Smith reveals counter-productive uses of weighing perspectives moves. In this example, which took place at the beginning of an investigation during the study of the Dynamic Planet unit, Mr. Smith follows an elicitation pattern in order to bring forth students initial ideas on forces that cause movements within the Earth. The clicker question The cookie dough above has just come out of the freezer. When it gets warmer, it will become easier to mold into cookies. How is the cookie dough like Earth's mantle? is displayed side by side with a photograph of a ball of cookie dough resting on a baking sheet. Each of the four answer choices reflect some aspects of the goal facet and provide an entry point for discussion. After reading over the question and the answer choices, Mr. Smith asks his students to vote using the response systems. After displaying the distribution of responses, he instructs them to turn and talk to a partner to discuss the reasons for their choices and to try to make a convincing argument when their choices differ. To spark the discussion, Mr. Smith uses a form of a weighing perspective move when saying If the person next to you clicked on the same one, I want you to say why, but if you clicked D and your neighbor clicked C, I want you to prove your point and try to change their minds. After students got a chance to share their ideas in pairs, they engage in a whole class discussion about the answer choices. Few students, however, engage with the content in the productive ways Ms. Thomas students do.
Conclusion
Fostering productive participation in argumentation in science class is not an easy task for teachers. The current paper offers hope and examples for ways that talk moves can support teachers in doing so. At the same time, a conclusion we draw from this study is that the distinction between generative and less generative uses of moves is often subtle, easier to see in retrospect, yet consequential within the flow of classroom activity.
References
National Research Council. (1999). How people learn: Brain, mind, experience, and school. Washington, DC: National Academy Press. National Research Council. (2011). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas. Washington, DC: National Research Council. O'Connor, M. C., & Michaels, S. (1993). Aligning academic talk and participation status through revoicing: Analysis of a classroom discourse strategy. Anthropology and Education Quarterly, 24(4), 318-355. O'Connor, M. C., & Michaels, S. (2011). Problematizing dialogic and authoritative discourse, their coding in classroom transcripts, and realization in the classroom. Paper presented at the ISCAR Congress, Rome, Italy.
Acknowledgments
This material is based upon work supported by the National Science Foundation under Grant Number DRL0822314. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. The authors are grateful to teachers and administrators in our partner district for their support and contributions to this research and development effort.
ISLS
506
ICLS2012
Volume 2: Posters
Inquiry in the kindergarten science: Helping kindergarten teachers to implement inquiry-based teaching
Louca, T. L., Tzialli, D., Department of Education Sciences, European University-Cyprus, 6, Diogenous Str., Engomi, 1516 Lefkosia, Cyprus, Louca.L@cytanet.com.cy, D.Tzialli@euc.ac.cy Constantinou, P. C., Department of Educational Sciences, University of Cyprus, P.O. Box 20537, Lefkosia 1678, Cyprus, c.p.constantinou@ucy.ac.cy Abstract: Twenty kindergarten in-service teachers participated in a 25hr professional development program (PDP) supporting the development of abilities for identifying and responding to students in-class inquiry. Each teacher taught a science lesson at the beginning and at the end of the PDP. These were videotaped and analyzed in terms of the student inquiry the teachers responded to. Findings suggest that average student talk duration increased and students leads were increasingly used to guide lesson flow. Teachers increased responses to students reasoning and logic, offered more clarifications to students ideas and reasoning, and decreased evaluations of students ideas. However, teacher responses to knowledge claims increased and those to everyday experiences decreased.
Scientific literacy includes more than just understanding the concepts of science. Current science education reform efforts, both in Europe and USA, emphasize promoting inquiry in science education (EC, 2007; NRC, 2007). Several studies (e.g., Boulter & Gilbert, 1996; Chin, 2006; Louca, Tzialli & Zacharia, in press), have emphasized the teachers role in promoting inquiry in the classroom when considering that inquiry-based teaching involves a complex process of adjusting teacher questioning based on the evaluation of the classroom discourse (e.g., van Zee & Minstrell, 1997). The ability to adapt instruction during teaching requires that teachers are able to notice and respond to aspects of classroom-based student inquiry (van Es & Sherin, 2002). To do so, teachers need to be able to listen to their students (van Zee & Minstrell, 1997), indentify elements of inquiry, and decide how best to proceed based on an evaluation of what they have identified (Louca, Tzialli & Zacharia, in press). We refer to these abilities as teacher noticing and responding (TNR). By noticing we refer to teachers abilities to attend to and reason about student inquiry as this takes place during instruction. Responding refers to teachers abilities to evaluate specific classroom exchanges and decide which teaching strategy is most appropriate for supporting each case of student inquiry (van Zee & Minstrel, 1997; Louca, Tzialli & Zacharia, in press). Developing TNR abilities depends largely on teachers' initial preparation and subsequent professional development in teaching science, although, TNR abilities have not traditionally been part of teacher preparation programs for science teaching (e.g., Fleer & Hardy, 2001; Freese, 2006). To address this need we developed a professional development program (PDP) aiming to help kindergarten teachers develop a range of teaching strategies to support students scientific inquiry. Our PDP had three distinctive parts. The first part introduced teachers to the aspects of scientific inquiry and inquiry-based teaching and learning in kindergarten science education. The second part focused on helping teachers develop and implement inquiry-based lesson plans in kindergarten science. The third part involved reflecting on video clips of each others lessons illustrating student inquiry and teachers responses. More information about our PDP can be found at http://prescience.co.uk/s-team/course/view.php?id=37. Our purpose in this paper was to investigate the possible impact of the PDP on the teaching of 20 in-service kindergarten teachers in Cyprus.
Methodology
Each of the 20 participating teachers taught two science lessons: one at the beginning of the seminar and another one at the end of the seminar. The first lesson was on a topic that none of the teachers had taught before, the phases of the moon. The topic of the second lesson was chosen by individual teachers and was part of the national curriculum for preschool science. Teachers collaborated with the first author to revise lessons, where necessary, to reflect more authentic inquiry practices for young learners. Transcripts of videotaped lessons served as the primary data source. We analyzed whole-classroom conversations in terms of (a) the time spent on teacher and student talking, (b) the student inquiry to which the teachers responded, and (c) how they responded to those elements. Analyses (b) and (c) were based on a coding scheme that we have developed elsewhere (Louca, Tzialli & Zacharia, in press).
Findings
During their first lesson teachers talked for a slightly larger portion of the lesson (average talk: 51.8%; average utterance duration 16 sec) compared to students (average talk: 48.2%; average utterance duration 5 sec). On
ISLS
507
ICLS2012
Volume 2: Posters
average teachers responded to (i) knowledge claims (67.9% average of the total utterances coded), (ii) scientific reasoning and logic (8%), (iii) everyday experiences (22.6%), (iv) epistemologies (0%), and (v) the direction of the conversation (1.5%). Teachers responded to their students ideas by prompts (35.3% of the total utterances coded), clarifications (36.9%), evaluations (12.4%) and restatements of students ideas (15.4%). In an average, 59.8% of the utterances coded, teachers seemed to follow the sequence of activities they had planned for their lessons, paying little attention to students responses which might guide their lesson in a different direction. During the second lessons teachers spent a similar portion of the lesson talking (teacher average talk: 52.4%; student average talk: 47.6%) as in the first lesson. However, average teacher utterance duration increased (12s) and average student utterance duration increased (8 sec). Teachers responded to (i) knowledge claims (81.9%), (ii) scientific reasoning and logic (12.4%), (iii) everyday experiences (4.5%), (iv) epistemologies (0%), and (v) the direction of the conversation (1.2%). Moreover, in their second lesson teachers responded to their students inquiry using prompts (37.8%), restatements of student ideas (16.7%), clarifications (38%) and evaluations of students ideas (7.5%). In an average 42.2% of the utterances coded, teachers followed the sequence of activities they had planned for, paying little attention to students responses, whereas for 57.2% of the utterances coded, teachers allowed their students responses to guide their activities sequence.
Conclusions and implications

Overall, our findings suggest that by the end of the PDP teachers refrained from evaluating their students ideas and reasoning; rather they clarified student reasoning and ideas, possibly in an effort to make those ideas more accessible to the rest of the class, providing their students with more space and time to talk science. Also, the increase in teachers attention to knowledge claims in the second lessons (from 67,9% to 81,9%), and the subsequence decrease in their attention to students everyday experiences (from 22.6% to 4.5%), possibly seemed to be related to the fact that for the first lessons topic, teachers experienced the process of collecting, assessing and interpreting observations of the moon for themselves during the PDP, thus indicating that there is probably a need for the local teaching community to directly experience more authentic inquiry. Additionally, the fact that lesson 2 was part of the official curriculum and was chosen by teachers themselves, possibly suggests that they felt comfortable teaching the topic, probably due to the fact that they had taught it numerous times in the past. This might have guided teachers in placing more emphasis on the content and knowledge claims in terms of the canonical body of established knowledge in science. Lastly, findings suggest a wide need for detailed studies regarding teacher cognition, in particular for the development of TNR abilities. Research has shown that experienced teachers may engage in these practices already, while TNR abilities in classroom interactions is something that is perceived as developing over time. In order to better support this development, further studies are needed for investigating how teachers develop TNR abilities.
References
Boulter, C., & Gilbert, J. (1996). Texts and contexts: Framing modeling in the primary science classroom. In G. Welford, J. Osborne, P. Scott (Eds), Research in Science Education in Europe (pp.177-188). London: Falmer Press. Chin, C. (2006). Classroom Interaction in Science: Teacher questioning and feedback to students responses. International Journal of Science Education, 28(11), 1315-1346. European Commission [EC] (2007). Science Education Now: A renewed Pedagogy for the Future of Europe. A Report of the High-Level Group on Science Education Brussels. Fleer, M. & Hardy, T. (2001). Science for children. Sydney, Australia: Prentice Hall. Freese, A. R. (2006). Reframing ones teaching: Discovering our teacher selves through reflection and inquiry. Teaching and Teacher Education, 22, 100-119. Louca, T. L., Tzialli, D., & Zacharia, Z. (in press). Identification, Interpretation - Evaluation, Response: A framework for analyzing classroom-based teacher discourse in science. Manuscript accepted for publication in International Journal of Science Education. National Research Council [NRC] (2007). Taking science to school: Learning and teaching science in grades K-8. Washington, DC: Committee on Science Learning, Kindergarten Through Eighth Grade. van Es, E. A., & Sherin, M. G. (2002). Learning to notice: Scaffolding new teachers interpretations of classroom interactions. Journal of Technology and Teacher Education, 10(4). van Zee, E. & Minstrell, J. (1997). Using Questioning to Guide Student Thinking. The Journal of the Learning Sciences, 6(2), 227-269.
Acknowledgments
The work reported in this paper was supported by a European Commission Coordination and Support Action under FP7, #SIS-CT-2009-234870
ISLS
508
ICLS2012
Volume 2: Posters
Metacognizing across Self and Socio Dialectics

Azilawati Jamaludin, David Hung Nanyang Technological University, National Institute of Education 1 Nanyang Walk Singapore 637616 azilawati.j@nie.edu.sg, david.hung@nie.edu.sg Abstract: In this paper, we discuss metacognition against a backdrop of 21st century traversals, where learners are constantly moving and interacting across different contexts. We describe how learners traversals are underpinned by triadic coupling relationships between self, social others, and cultural resources. Drawing our observations from contemporary contextual spaces of online games, we articulate how a situated and embodied form of metacognition pivots the dialectics of the aforementioned coupling relationships.
Introduction
The work on metacognition has historically been situated within individual, cognitive-constructive activity (Winne, 1997) that places focus on individual characteristics such as self-efficacy, goal setting, and achievement. In a globalized learning landscape afforded by new technologies, the ability to engage in metacognition and reflective dispositions remains much desired skills for learners as they traverse across information laden and socially rich environmentslearners are constantly engaged in co-appropriation of knowledge, as their individual actions impact the collective learning of their social communities. This intertwining relationship between individual performances and the emergence and regulation of social communities represents the central focus of our research. At the individual level of analysis, we are concerned with learners metacognition as they make meaning and self-regulate learning in the context of social communities. At the collective level, we posit that the role of metacognition is central within the emergence of social communities. Within this flux of interactivity, concepts of play and learning become increasingly intertwined as new genres of interactive spaces, such as the massively multi participant worlds of World of Warcraft (WoW), afford players with novel and unique opportunities for learning. Implicit to this interplay, we posit that the role of metacognition is critical for the successful enactment of self- and socio-regulation. We argue for a pedagogical shift from individual constructivist perspectives of metacognition to a contextuallycognized framing of metacognition within the context of 21st century literacies.
Metacognition within interactive spaces of learning

Situating our research within the contextual space of WoW, we conducted a case study on four youth game players between the ages of 14-18 studying in local educational institutions. We focused on unpacking the ways by which these game players create, negotiate, and reproduce the social realities of their multispatial experiences within the contexts of their respective social spaces through constructs of cognition and metacognition. In-game chat logs were captured using a third-party game add-on called ChatLogger while enactment frames and videos were captured via Fraps. Textual data was then analyzed in terms of the level of discourse as evidenced by sets of chat utterances on specific topics, for example, focusing on strategy talk regarding best approaches to in-game problems or talk as evidence of critical reflection. In-depth interviews were supplemented with field notes from the first authors participant-observer sessions to corroborate informants logged texts and game enactments.
Metacognizing in context
Arising from our study, findings indicate that situatedness of metacognition is key within the process of finding an affordance fit in managing or regulating ones fluency of experience and performance-in-action (thus embodied). For it is when one's intramental epistemic structures and knowledge encounters a breakdown within novel social contextual nuances that metacognition or the ability to engage in situated higher order thinking is critical in managing the tension between individual performances and the collective. In the course of problem solving processes, metacognition comes into fore when ones current thinking and strategies fail. At such a breakdown stage, learners may have to unlearn and relearn new strategies and skills in order to cope with new contexts. In this sense, a new, situative view of metacognition pervades traditionally construal of thinking about thinking in its mentalistic, Cartesian form. Instead, such a perspective has a strong coupling relationship with the process of coming to recognize ones shortcomings in a particular context both in terms of thinking and embodied performance and regulating (including perseverance) towards fluency in that new and novel situation. Figure 1 explicates this relational perspective, of metacognition as achieving fits between self, others, and cultural resources.
ISLS
509
ICLS2012
Volume 2: Posters
Figure 1. Metacognition and three coupling relationships Metacognition and self The construct of metacognition involves both the concept of self and the emergence of conscious control over ones actions. For example, we observed that the fantasy constructions of identity in WoW offer an exploration of alternative subjectivities and playful representation of self. Cognizant of an audience, performative spaces such as WoW creates the context for player to enter into a dialectical relationship with his character role At the crux of such enactments, is the construct of self-belief and self-efficacy that, according to Goffman (1959), spans a continuum with the tendency to accept ones self as performed. This is to say that on a more quotidian level, a persons sense of self is very much tied to his ability to believe in the roles he played. The matter is complicated because, as we acknowledge, character roles are not played by a single, stable self. Rather, the extent to how well a person plays a social role usually depends on how much the person believes in the role. Metacognition and socio others Within the flux of interactions in WoW, establishing intersubjectivity stems from not only a players sense of self but also a collective sense of shared identity. This is evidenced by the strong commitments informants placed on the tasks, errands, and obligations they ascribed to within the game space. We noted that not only do the game interactivity interweaves a deeply meaningful and experiential space that entails immense investments of energy, emotion, and time, but so too do they seed strong feeling of affinities, underpinned by a common passion where players are further incited to participate and perform, to identity ones self as another. Metacognition and cultural resources The WoW environment represents a complex constellation of resources that must be managed by players as they level up, gearing towards their goals. Players are afforded with not only design and technical add-ons that aid in the leveling process but so too social support arising from the critical mass of players. For example, the add-on Questhelper, provides cartographic waypoints support as players engage in quests in a bid to move up to the next level. Such tools exist as structures in the game environment that seed metacognition tracking positioning of players and offering indicators of game play simplification relate to how well players monitor their progress, reflect on their strengths and weaknesses, and leverage on cultural resources to achieve their game objectives.
Discussion
In our case study, we see how the structurally coupled relation between player and game environment brings to fore a focus not only on representations of self but so too socio and cultural affordances. As opposed to traditional construal of metacognition, that still dichotomizes mind and body, our informants have shown how their engagement in higher order thinking are not without refractions from the embodied subjectivities they experience within the WoW contextual space. A situational awareness of the shared social space within WoW impinges on informants action and performance as a dialectical relationship to ones metacognition. Actions and performance may be reifications of ones thinking but these are not without contextual and situational refractions of ones embodied experience within that space. We see how as players re-aligned and reappropriated in anticipation of their next raid, their resolute to learn, unlearn, and relearn as a group reflects the nuances of their collective metacognitive and regulative endeavors. In this sense a situated and embodied notion of metacognition, at both self and socio regulatory levels, does not only interfaces with an individual/groups respective appropriation of a communitys way of seeing things and making meaning, but it also contextualizes an individual/groups self-socio-cultural enactments in terms of establishing fluency and structural between one-self and the context.
References
Goffman E. (1959).The presentation of self in everyday life. New York: Doubleday, Anchor Books. Winne, P. H. (1997). Experimenting to bootstrap self-regulated learning. Journal of Educational Psychology, 89, 397-410.
ISLS 510
ICLS2012
Volume 2: Posters
Combining knowledge and interaction perspectives to decipher learning during a clinical interview
Shulamit Kapon, School of Education, Tel Aviv University, Tel Aviv 69978, ISRAEL, kaponsh@post.tau.ac.il Abstract: The mystery of how instruction leads to learning of disciplinary-based knowledge is examined through a fine grained analysis of a learning episode taken from a clinical interview in physics. The analysis employs two distinct theoretical perspectives, the Knowledge in Pieces perspective on conceptual change (diSessa, 1993), and a distributed perspective of learning (Stevens & Hall, 1998), linking the students knowledge in-use and indevelopment to the interaction with the interviewer/tutor and the learning artifacts.
Introduction
How exactly does instruction lead to learning of disciplinary-based knowledge? Based on interactions between participants in disciplinary practices and related visual inscriptions, Stevens and Hall (1998) suggested that practicing a discipline involves the development of a disciplined perception. For instance, the development of visual practices unique to the discipline of architecture includes a set of specific forms of embodied actions, such as looking at the world in terms of layout, terrain, etc. Thus, instruction in its very essence is an asymmetric action which involves disciplining students perception. Stevens and Hall argued that the relevant unit of analysis should be the materially mediated, distributed character of real world perception rather than internal cognitive processes (p. 110). A focus on internal cognitive process led diSessa and colleagues (diSessa & Sherin, 1998; diSessa & Wagner, 2005) to hypothesize the construct of expert-like concepts termed coordination classes. A coordination class includes schemes of perceiving the concept in the environment, and thus seems intimately related to the notion of disciplinary perception, though the focus of analysis is very different. The fault lines between these two approaches are whether the relevant unit of analysis is the practice or the concept, and whether the interesting story is in the interaction level or in the knowledge level. In my view these two approaches complement rather than contradict one another, particularly if the aim of analysis is to understand the subtle relations between instruction and learning. After all, expert knowledge is developed during interactions with more capable others while using artifacts and objects unique to the discipline, and expert interaction is shaped by experts knowledge. In fact, there was a call for more theoretical work associating the more functional conceptualizations of interaction analysis and the more structural constructs of knowledge analysis in a recent AERA funded conference entitled "Integrative Approaches to the Analysis of Cognition and Learning". This poster presents an analysis of a short learning event in an episode drawn from a clinical interview with a high school student about the existence of the normal force in physics. The analysis is aimed at linking the students knowledge in-use and in-development to the interaction with the interviewer/tutor and the learning artifacts.
Context of inquiry
The episode discussed here is drawn from a data corpus that was collected as part of a project that examined the role of prior knowledge in analogical reasoning (Kapon & diSessa, 2010, in press). As part of this project I conducted six clinical interviews with high school students around a tutoring session that used a sequence of bridging analogies on the existence of the normal force from classical Newtonian mechanics. During the interviews the interviewees could handle all the objects used in the analogies (books, springs, boards, etc.). A fine grained comparative knowledge analysis (diSessa, 1993; diSessa & Sherin, 1998) was employed in these interviews, and showed that differences in individual responses to an instructional sequence can be explained by differences in the particular knowledge (termed explanatory primitives) that is activated, and differences in the relative reliability (confidence) the individual assigns to a particular explanatory primitive. However, the fine grained knowledge analysis raised additional detailed questions about the reasoning prompted by instructional analogies, such as which aspects of the interaction with the interviewer/tutor and objects in the learning environment activated a particular explanatory primitive, convinced the learner to shift preferences from one primitive to another, and later use the primitive in the construction of new understanding. This poster does not attempt to answer these questions but rather present insights from a preliminary analysis that tries to integrate the knowledge and interaction perspectives. The poster discusses an episode that illustrates the nature of this inquiry exemplifying an instance of a development of a disciplined perception in the context of Newtonian mechanics, namely perceiving motion in terms of the forces involved. The analysis specifically attends to the interviewer's instructional moves and the interviewees responses. Activation of prior knowledge constructs and their relative considerations were based on the interviewees choice of words, focus of attention, gestures, and tone, and triangulated with other episodes from the interview (triangulation of expression) and the literature on intuitive knowledge in early Newtonian
ISLS
511
ICLS2012
Volume 2: Posters
mechanics (triangulation of form). The knowledge analysis was guided by the Knowledge in Pieces perspective on conceptual change (diSessa, 1993). The interaction analysis was guided by distributed perspectives of learning (Stevens, 2010; Stevens & Hall, 1998) and conversation analysis (Sacks, Schegloff, & Jefferson, 1974).
Preliminary analysis
The episode starts when the interviewer reacts to a student's account for the fact that the book she placed on the table did not fall to the floor. The student's explanation was that: the particles in the table [] are compacted too tightly for the book to get through. He claimed that it does not make sense that the table exerts a force while preventing the book from moving through it. At this point the interviewer asked the student to hold out his hand, and she placed a book on it. She asked him whether his hand exerted a force on the book. He was not sure, and he played with the book for a while, as if weighing the book with his hand, thinking silently. The interviewer started piling more books on his hand. At this point the student said Id say its the amount of force for [stops an think] my hand to compensate for the attraction of gravity. The knowledge analysis suggests that the student activated the supporting p-prim when initially making sense of the book on the table scenario. The supporting p-prim (diSessa, 1993) stipulates that a strong or stable underlying object merely keeps overlaying and touching objects from falling. The interaction with the book on the hand activated a competing p-prim, dynamic balance. Dynamic balance (diSessa, 1993) explains situations of balance when forces are recognized, namely when a pair of forces or directed influences are in conflict and happen to balance each other. Note particularly the student's compensating language, and his explicit mention of gravity as a force (attraction). The activation of dynamic balance is mediated here by the activation of two other primitives: The first is that effort entails force. Effort is a common co-occurring element of situations where force applies (diSessa, 1993); the second is that gravity pulls things downwards. Now let us attend to the interaction with the interviewer during this exchange. The interviewer piling increasingly more books on top of the hand of a student who was not certain whether his hand exerted a force on a book that is placed on it could be interpreted as a form of recipient design (Sacks et al., 1974) aimed at disciplining the student's perception by directing his attention to the effort made by his muscles. The next turns reveal that though the student activated dynamic balance he was not certain that it was more appropriate than supporting in the case of the book on the hand. This preference was gradually solidified during the interaction with the interviewer who constantly disciplined the student's perception towards the forces involved. For instance she reiterated his description of the force of gravity while explicitly adding the direction of this force (downwards) to the sentence; she asked him if he felt the force, which he immediately answered by: Yeah. To maintain it at this level, it requires a certain amount of force to keep it level. Note again how effort entails force mediated dynamic balance. This example illustrates how the student's specific activation of p-prims and his consideration of their relative appropriateness as explanatory means were influenced by the particular conversational moves of the interviewer.
References
diSessa, A. A. (1993). Toward an Epistemology of Physics. Cognition and Instruction, 10(2&3), 105-225. diSessa, A. A., & Sherin, B. L. (1998). What changes in conceptual change? International Journal of Science Education, 20(10), 1155-1191. diSessa, A. A., & Wagner, J. F. (2005). What coordination has to say about transfer. In J. Mestre (Ed.), Transfer of learning from a modern multi-disciplinary perspective (pp. 121-154). Greenwich, CT: Information Age Publishing. Kapon, S., & diSessa, A. A. (2010). Instructional explanations as an interface - the role of explanatory primitives. In M. Sabella, C. Singh & S. Rebello (Eds.), Physics Education Research Conference Proceedings (Vol. 1289, pp. 189-192): American Institute of Physics. Kapon, S., & diSessa, A. A. (in press). Reasoning through instructional analogies. Cognition and Instruction. Sacks, H., Schegloff, E. A., & Jefferson, G. (1974). A simplest systematics for the organization of turn-taking for conversation. Language, 696-735. Stevens, R. (2010). Learning as a members phenomenon: Toward an ethnographically adequate science of learning. NSSE Yearbook, 109(1). Stevens, R., & Hall, R. (1998). Disciplined perception: Learning to see in technoscience. In M. Lampert & M. L. Blunk (Eds.), Talking mathematics in school: Studies of teaching and learning (pp. 107-149). New York: Cambridge University Press.
Acknowledgments
This research was supported by a Marie Curie International Outgoing Fellowship within the 7th European Community Framework Program.
ISLS
512
ICLS2012
Volume 2: Posters
Asking for too much too early? Promoting mechanistic reasoning in early childhood science and mathematics education
Louca, T. L., Papademetri-Kachrimani, C., Department of Education Sciences, European University-Cyprus, 6, Diogenous Str., Engomi, 1516 Lefkosia, Cyprus, Louca.L@cytanet.com.cy, C.Papademetri@euc.ac.cy Abstract: Following research attention to childrens use and understanding of causation, in this paper we contend that young children as early as kindergarten are able to engage in an effort to develop understanding about the causal mechanism underlying or explaining physical and mathematical phenomena. We draw on a two examples from early childhood education settings to suggest that children are able to use (novice) abilities of mechanistic reasoning both in early science and mathematics education. We discuss implications for the role of the teacher and activity design.
During the second half of the twentieth century, good science teaching and learning has become increasingly associated with inquiry (Anderson, 2002). Working from a variety of perspectives and intellectual traditions, research regarding childrens abilities for scientific inquiry shows a general consensus with respect to the sorts of things we should value and try to promote in childrens inquiry. That consensus, however, does not extend to the definition of what scientific inquiry looks like in the science classroom. One area of research is drawing particular attention to the scientific discourse that involves causal mechanisms (Russ, Scherr, Hammer and Mikeska, 2008), following a number of studies that partly focused on childrens use of causation in science. This suggests that assessing when and how children seek causal mechanisms in their understanding should be part of assessing their reasoning as inquiry (Russ, et al., 2008, p.1). Using a framework derived from the philosophy of science, Russ et al. (2008) developed a coding scheme of 7 major components of mechanistic reasoning that can be used to identify and assess childrens use of such reasoning. Those components include (i) descriptions of the target phenomenon (what we see happening), (ii) identification of the set-up conditions that are necessary for the phenomenon to happen, (iii) identification of entities (conceptual or real objects) that play a particular role in the phenomenon, (iv) identification of the entities activities that cause changes in the surrounding entities, (v) the entities properties, (vi) the entity organization (how entities are located, structured or oriented within the phenomenon), and (vii) chaining; that is using knowledge about causal structure to make claims about what has happened prior to a phenomenon and what will happen. In this paper, we analyze a number of videotaped incidences from authentic early childhood education settings. We aim to provide evidence that searching for opportunities to promote or to design for, mechanistic reasoning is a tangible goal for early childhood education. Our stance is that children come to class already having abilities to engage in mechanistic reasoning. Teachers need to help them refine these abilities and more importantly, they need to help children develop reliable access to these abilities, for their use in the appropriate context and time.
Methodology
This interpretive case study illustrates young childrens nascent abilities for mechanistic reasoning. Data originate from two activity sequences carried out with children (ages range from 3,5-5 years old). The activity sequences were implemented by two different senior student-teachers, one in science and the other in mathematics. Prior to these activity sequences, the children had no previous formal instruction of, or about mechanistic reasoning. From the videotapes, we identified episodes of student inquiry related to mechanistic reasoning in both activity sequences, and coded the transcripts using Russ et als (2008) coding scheme to determine which aspects of mechanistic reasoning were used by the children. In doing this we sought to describe the variability in childrens mechanistic reasoning, as well as the contextual possibilities that might have led to different uses of mechanistic reasoning in these pre-school science and mathematics activities. Below, we briefly present two short analyzed excerpts that are representative of the findings in support of our claims.
Findings
The science activity sequence was carried out in a class of 3,5 and 4 year olds. The topic was floating-sinking, and childrens goal was to help an ant to build a boat to cross a river. From the outset, children realized that the ants suggested materials could not be used for the boat, as they sank. Thus, children decided that they needed to investigate which alternative materials might float and subsequently be used for building a boat. Among the materials the teacher had prepared, were two identical pieces of aluminum foil, one left as a thin sheet and the other crumpled into a ball. During the experiment, all the student groups discovered that the two pieces of aluminum foil behaved differently in water the one crushed into a ball sank, while the sheet floated. After the
ISLS 513
ICLS2012
Volume 2: Posters
groups presented their findings to the whole class, the teacher wanted to draw childrens attention to the different behavior of the two foil objects. During that conversation, the types of student responses varied. For some time, children focused on the differences in the two objects characteristics that the foil that sinks is folded, small, like a pie etc. Then, a child started a very different line of thought, in an effort to explain how the different behavior of this aluminum foil piece is caused. His contribution approaches a representation of the mechanism that causes the phenomenon. Having already described the target phenomenon and its set-up conditions, the child identified a possible physical entity (force) that may account for how the phenomenon took place, indicating that it was this force that may have made the difference and resulted in the foil floating. The mathematics activity was carried out in a class of 4 and 5 year olds. The children were given the following problem: How many different shapes can you make by putting together two congruent scalene, rightangled triangles so that one pair of congruent sides is always shared? After the children worked in pairs they concluded that they had found 5 different solutions to the problem. Then, they reproduced the 5 solutions by using sets of triangles which were marked using blue, red and green to show each set of congruent sides. At this point the children noticed that there were two different solutions with the blue sides joined. The teacher asked the children how come there were two solutions with the same sides joined. One child gestured using her hands that if you flip over one of the triangles you get a different solution. After discussing the 5 solutions the children concluded that since there were two solutions for the blue sides, two for the red sides but only one for the green there was probably one solution missing. Based on Russ et als (2008) coding scheme we can identify the components of mechanistic reasoning. The children described what they saw happening there are two solutions for the same set of congruent sides. In order to do that, the children identified the conditions and described the entities which played an important role in the phenomenon this phenomenon arose while trying to construct shapes by putting together two congruent scalene triangles so that one pair of congruent sides is completely shared. Finally, the children became involved in a chaining procedure where they tried to interpret their observation which allowed them to formulate a hypothesis in relation to the missing solutions to the original problem, namely that there are two solutions for each set of congruent sides which result when flipping over one of the two triangles, thus the total number of solutions to the problem must be 6.

Although we do not claim that our findings cover the complete spectrum of mechanistic reasoning in early childhood education, findings reveal insights with respect to the challenge of defining how mechanistic reasoning might look like in the classroom and what teachers should expect to see, regarding what productive inquiry entails in young children. Overall, our findings provide evidence that young children come to class already having abilities to engage in mechanistic reasoning, and we describe the contexts in which mechanistic reasoning might take place. In this context, our findings also highlight some ways in which the teacher can grasp childrens spontaneous reasoning, be aware of the manner in which an activity might develop so as to engage children in mechanistic reasoning and design activities to create opportunities for this. Lastly, we would like to underline that the children in the videotaped incidences described exhibited the ability for mechanistic reasoning with no prior formal instruction in this type of sophisticated thinking. We do not suggest that young children are experts in scientific inquiry, but rather that they have the beginnings of abilities for mechanistic reasoning. In this, we feel that the literature over-emphasizes the need to teach students how to construct, evaluate and respond to causal mechanisms, analogies and arguments. Rather, our findings question whether we should refocus teaching efforts towards identifying the beginnings of young childrens abilities for mechanistic reasoning, and describing the learning conditions under which mechanistic reasoning might take place.
References
Anderson, R.D. (2002). Reforming science teaching: What research says about inquiry. Journal of Science Teacher Education, 13, 1-12. Russ, R., Scherr, E., R., Hammer, D., & Mikeska, J. (2008) Recognizing mechanistic reasoning in scientific inquiry. Science Education, 92(3), 499-525.
Acknowledgments
The work reported in this paper was supported by a Cyprus Research Promotion Foundation Grand, # //0609()/4
ISLS
514
ICLS2012
Volume 2: Posters
SMILE smartphones in a university learning environment: a classroom response system

Celia Kaendler, Institute of Psychology, University of Freiburg, Engelbergerstrae 41, 79085 Freiburg, Germany, kaendler@psychologie.uni-freiburg.de Linus Feiten, Katrin Weber, Chair for Computer Architecture, University of Freiburg, Georges-Koehler-Allee 51, 79110 Freiburg, Germany Email: feiten@informatik.uni-freiburg.de, weber@informatik.uni-freiburg.de Michael Wiedmann, Institute of Psychology, University of Freiburg, Engelbergerstrae 41, 79085 Freiburg, Germany, wiedmann@psychologie.uni-freiburg.de, Manuel Buehrer, Sebastian Sester, Bernd Becker, Chair for Computer Architecture, University of Freiburg, Georges-Koehler-Allee 51, 79110 Freiburg, Germany Email: buehrerm@informatik.uni-freiburg.de, sesters@informatik.uni-freiburg.de, becker@informatik.unifreiburg.de Abstract: We describe and evaluate a tool using smartphones in a university learning environment. Mass lectures often lack in interactivity. Due to missing feedback, the lectures content may not be adapted to the students understanding. The tool consists of two modules, a comprehension level visualization and multiple choice questions, that are hypothesized to increase interactivity and adaptation to students understanding. We report data from a formative evaluation of the tools first implementation in a university lecture. Teaching lectures with large class sizes provides some unique challenges regarding interactivity and adaptation to students understanding. A constrained timeframe and class size make answering questions difficult, so lecturers may avoid interacting with students altogether. Furthermore, students avoid posing questions in front of a large and rather anonymous audience (Ratto, Shapiro, Truong, & Griswold, 2003). Finally, gathering feedback on the students understanding is often not possible in a large lecture hall containing about a few hundred of students, resulting in low adaptation of the lecture to the comprehension level of students (Scheele, Wessels, Effelsberg, Hofer, & Fries, 2005). In the past, classroom response systems such as i<clicker have been used to improve interactivity in and adaptability of lectures. However, these systems are expensive to purchase and maintain. Mobile wireless technologies may offer an alternative because of their reduced size, portability, and low costs (Ogata, Wada, Gan, & Yano, 2008). Instead of distributing a classroom response system, students personal mobile devices may be used. Indeed, Ratto and colleagues (2003) suggest that mobile devices may improve the quality of university teaching by encouraging active participation and by adjusting the lectures using students feedback. A classroom interaction tool called SMILE (SMartphones In LEctures) was developed by students as part of a programming class at the chair of Computer Architecture at the University of Freiburg, Germany. Using free software and online resources (Feiten & Becker, 2012), an Android app was programmed as a native smartphone client. In addition, a web browser variant was written for maximum compatibility with other platforms. The tool consists of two main modules that allow for interaction between students and lecturer. The feedback module enables the visualization of feedback to the lecturer. Students can indicate their understanding of the lectures content by using a slide control ranging from cant follow [bin abgehaengt] to got it [alles klar] (see Figure 1, left panel). The result is visualized as a bar diagram on a screen for the audience and the lecturer. In addition, the lecturer is provided with a function graph presenting mean scores and the development of the audiences comprehension over time.
Figure 1. Modules in the Android client: Feedback slider (left panel), multiple choice questions (middle panel), and evaluation questions (right panel).
ISLS 515
ICLS2012
Volume 2: Posters
The multiple choice module allows the lecturer to pose multiple choice questions that students can answer on their mobile devices (see Figure 1, middle panel). The results are immediately presented on the screen in front, giving lecturer and students the percentages of the different answers. We conducted a formative evaluation of SMILE during its first implementation in a Computing Science lecture over the course of one semester. We hypothesized that a visualization of the audiences general comprehension may, when showing that the majority of students are not able to follow the lecture, encourage students to raise their hands and ask questions. In addition, the multiple choice questions module may help lecturers detect students misconceptions and lack of understanding better compared to using the feedback module alone. Information from both modules may be used to adapt current and future lectures. Lastly, we assumed that students would feel less frustrated if they perceived that the lecturer was aware of their comprehension problems. To investigate these hypotheses, at regular intervals throughout the semester students completed questionnaires that were integrated directly into SMILE as an evaluation module (see Figure 1, right panel). A paper-based questionnaire was administered mid-term to all students with the goal of reaching participants who did not use SMILE. Finally, we conducted interviews with the lecturer and teaching assistants at the end of term. The questionnaires and interview guides contained items regarding the perception of the lectures adaptation and interactivity, the usability of the SMILE interface, and the acceptance of its implementation in the lectures. The formative evaluation also served to continually refine the technical features of the application throughout the term. Results revealed a low participation rate of about a third of students actively using the tool during the lecture. This might be due to initial technical problems concerning overloads of the wireless local area network and usability features such as automatic system logouts during the lecture frustrating the students. Also, students and lecturer reported that the feedback module created distractions because feedback results and slides were displayed on different screens. As a positive outcome, the feedback visualization stressed students comprehension level to the lecturer or corroborated on the lecturers estimation of difficult learning content during the lecture. However, the feedback visualization did not provoke more question-asking during the lecture. Nevertheless, the feedback sometimes induced the lecturer to additionally explain difficult content on the black board. In comparison to the feedback module, the multiple choice module was more accepted by both, students and lecturer. It was seen as a good preparation for exams and led to more exchange amongst the students in cooperatively finding the correct answer. Further, the lecturer-student-interaction increased by discussing and explaining the correct answer. The multiple choice format served to uncover students misconceptions and to summarize the learning content. Against concerns that SMILE may interrupt the flow of the lecture, the lecturer did not report more noise during the lecture but rather improved students attention. Additional results will be presented on the poster and this evaluation will be used to further improve SMILE. For example, we will modify the feedback module by integrating the feedback visualization into the lecture slides. The lecture slides will also be modified to better set up multiple-choice questions. We plan to test the revised SMILE tool during the next winter term.
References
Feiten, L., & Becker, B. (2012, April). SMILE smartphones in lectures: Initiating a smartphone-based audience response system as a student project. Paper presented at the 4th International Conference on Computer Supported Education (CSEDU 2012), Porto, Portugal. Ogata, H., Wada, M., Gan, L. H., & Yano, Y. (2008, October). Supporting a decision making for task assignments in language learning outside classroom with handhelds. Paper presented at the 16th International Conference on Computers in Education, Taipei, Taiwan. Ratto, M., Shapiro, R. B., Truong, T. M., & Griswold, W. G. (2003). The ActiveClass project: Experiments in encouraging classroom participation. In B. Wasson, S. Ludvigsen, & U. Hoppe (Eds.), Proceedings of Computer Supported Collaborative Learning 2003: Designing for Change in Networked Learning Environments (pp. 477486). Dordrecht, Netherlands: Kluwer Academic Publishers. Scheele, N., Wessels, A., Effelsberg, W., Hofer, M., & Fries, S. (2005). Experiences with interactive lectures: Considerations from the perspective of educational psychology and computer science. In T. Koschmann, D. Suthers, & T. W. Chan (Eds.), Proceedings of Computer Supported Collaborative Learning 2005 (pp. 547-556). Mahwah, NJ: Lawrence Erlbaum Associates.
Acknowledgments
The initial programming team of SMILE included the students Peter Berli, Fabian Fischer, Johannes Mller, Benedikt Neumann, Markus Schwenk, and Markus Thal. We also thank Peter Winterer for his technical support and the teaching assistants, Sven Reimer and Paolo Marin, for their essential support in this study.
ISLS
516
ICLS2012
Volume 2: Posters
Stability and Change in Achievement Goals and Transfer

Erkka Laine, Centre for Learning Research, University of Turku, Finland, etlain@utu.fi Andreas Gegenfurtner, TUM School of Education, TU Mnchen, Germany, andreas.gegenfurtner@tum.de Abstract: This meta-analytic study aimed to examine stability and change in the relationship between achievement goals and transfer of training, using a 2 X 2 goal framework (mastery and performance orientation, approach and avoidance motivation). Data consisted of cumulated correlation estimates extracted from k = 28 studies (N = 4,394) from research of the past 25 years. The results provide evidence for the presence of temporal dynamics. Implications for theory and practice are addressed.
Introduction
Achievement goal orientations and transfer of training represent two important themes in contemporary training research. Typically, the literature conceptualizes achievement goals in 2 X 2 dimensions (Elliot, 1999; Fryer & Elliot, 2007), consisting of mastery-approach goals (when we have the goal to attain task-based or intrapersonal competence), mastery-avoidance goals (when we have the goal to avoid task-based or intrapersonal incompetence), performance-approach goals (when we have the goal to attain normative competence), and performance-avoidance goals (when we have the goal to avoid normative incompetence). Figure 1 offers a schematic overview of this conceptualization (Elliot, 1999). Research suggests that achievement goals are important predictors of the transfer of trainingthat is, whether or not we are using what we learn in training classrooms in real life (Gegenfurtner, 2011; Gegenfurtner et al., 2009; Gegenfurtner & Vauras, 2012). However, to date, it is not clear whether the influences of achievement goals on transfer are stable over time or whether they change dynamically as time after training unfolds. There are theoretical reasons to expect both stability and change. On one hand, reasons to expect stability in achievement goal endorsement lie in the hierarchical nature of achievement motivation and in the nature of continued goal pursuit (Elliot, 1999; Fryer & Elliot, 2007; Tuominen-Soini, Salmela-Aro, & Niemivirta, 2011). On the other hand, reasons to expect change in achievement goal endorsement lie in the multiple types of change that may take place in our goal commitments, goal intensifications, and regulative striving (Fryer & Elliot, 2007). Previous studies have attempted to address the question of stability and change in achievement goals (Fryer & Elliot, 2007; Tuominen-Soini et al., 2011), with heterogeneous results. In an attempt to address this heterogeneity, the present study uses meta-analytic techniques to inquire whether achievement goals, after controlling for sampling error and error of measurement, exhibit a variant or invariant influence on transfer as time after training unfolds. For this purpose, achievement goals were conceptualized in four dimensions: mastery-approach goals, mastery-avoidance goals, performance-approach goals, and performance-avoidance goals (see Elliot, 1999; Figure 1). Although there are important alternative conceptualizations of goal orientations (e.g., Ke, 2008; Ng & Bereiter, 1991), we chose the 2 X 2 framework because it had been used in numerous studies before and so afforded cumulating a larger number of individual study correlation estimates than would have been available otherwise.
Method
The literature was searched in two ways. First, the PsycINFO, ERIC, and Web of Science databases were searched using the keywords achievement goals, goal orientations, training application, training use, and transfer of training. In addition, a manual search of journal issues covering a 25-year period (from January 1986 through December 2010) was conducted. Studies that reported correlations between achievement goal orientations and transfer of training were located. To be included in the database, a study had to report an effect size r or other effect sizes that could be converted to r ( coefficient; Cohens d; F, t, or Z statistics). A total of k = 28 independent data sources from articles, book chapters, conference papers, and dissertations that contributed at least one effect size to the meta-analysis were included in the database. A full list of all included studies is available from the first author. Total sample size was N = 4,394 participants.
Achievement goal orientations
Approach motivation
Avoidance motivation
Mastery approach
Performance approach
Mastery avoidance
Performance avoidance
Figure 1. Four dimensions of achievement goal orientations (adapted from Elliot, 1999).
ISLS 517
ICLS2012
Volume 2: Posters
Different characteristics were tabulated from this selected research literature. Specifically, each study was coded for effect size estimates and for time after training (in days) when transfer had been assessed. Effect size estimates included Pearson product-moment correlation r of the goaltransfer relationship, Cronbachs reliability estimate of the independent variables (achievement goals), and Cronbachs reliability estimate of the dependent variable (transfer). We also coded the first author, publication year, the number of participants, their age (in years), and gender (percentage of females). Cohens = .96. Analysis occurred in two stages. A primary meta-analysis aimed to estimate the true score population correlation of the relationship between the four dimensions of goal orientations and transfer of training. Metaanalytic moderator estimation then aimed to identify stability or change by estimating the effect of time after training on those relationships. (1) The primary meta-analysis was done using the methods of artifact distribution meta-analysis of correlations (Hunter & Schmidt, 2004). These methods provide an improvement from earlier statistical formulae when information such as reliability estimates is only sporadically reported in the original studies. First, study information was compiled on three distributions: the distribution of the observed Pearsons r, the distribution of Cronbachs of the independent variable (achievement goals), and the distribution of Cronbachs of the dependent variable (transfer). Next, the distribution of Pearsons r was corrected for sampling error. The distribution corrected for sampling error was then further corrected for error of measurement using the compiled Cronbachs reliability estimates. This last step provided the final estimate of the true score population correlations between the four dimensions of achievement goals and transfer. (2) The meta-analytic moderator estimation followed the primary meta-analysis. The effect of time after training on the relationship between the four dimensions of achievement goal orientations and transfer of training was assessed using weighted least squares (WLS) multiple regression. This method was chosen because, when estimating continuous moderators, WLS tends to be largely unaffected by multicollinearity and converges toward the true moderator effect size, despite variations in heteroscedasticity.
Preliminary Results and Discussion

The primary meta-analysis identified stronger correlations of transfer with mastery-approach orientation (k = 26, N = 4,154, = .27) rather than with performance approach orientation (k = 24, N = 3,421, = .04) and performance-avoidance orientation (k = 10, N = 1,813, = -.11).The two dimensions of approach goals (mastery-approach goals and performance-approach goals) showed rising population correlation estimates when time after training increasedthat is, the longer the time between training end and the transfer measure, the higher the effect sizes. By contrast, performance-avoidance goals showed falling population correlation estimates when time after training increasedthat is, the longer the time between training end and the transfer measure, the lower the effect sizes. These preliminary findings tend to highlight the temporal dynamics of how achievement goal orientations influence the transfer of training. It follows that our meta-analytic synthesis of 25 years of research suggests change to be more dominant than stability in achievement goal endorsement. This is more in line with the findings of Fryer and Elliot (2007) than with Tuominen-Soini and colleagues (2011). The results tend to indicate the need to re-consider the stability of goal orientations. Implications of these findings for educational practice are associated with the crucial role of measurement time when evaluating training effectiveness.
References
Elliot, A. J. (1999). Approach and avoidance motivation and achievement goals. Educational Psychologist, 34, 169-189. Fryer, J. W., & Elliot, A. J. (2007). Stability and change in achievement goals. Journal of Educational Psychology, 99, 700-714. Gegenfurtner, A. (2011). Motivation and transfer in professional training: A meta-analysis of the moderating effects of knowledge type, instruction, and assessment conditions. Educational Research Review, 6, 153-168. Gegenfurtner, A., & Vauras, M. (2012). Age-related differences in the relation between motivation to learn and transfer of training in adult continuing education. Contemporary Educational Psychology, 37, 33-46. Gegenfurtner, A., Veermans, K., Festner, D., & Gruber, H. (2009). Motivation to transfer training: An integrative literature review. Human Resource Development Review, 8, 403-423. Ke, F. (2008). Alternative goal structures for computer-based learning. International Journal of ComputerSupported Collaborative Learning, 3, 429-445. Ng, E., & Bereiter, E. (1991). Three levels of goal orientation in learning. Journal of the Learning Sciences, 1, 243-271. Tuominen-Soini, H., Salmela-Aro, K., & Niemivirta, M. (2011). Stability and change in achievement goal orientations: A person-centered approach. Contemporary Educational Psychology, 36, 82-100.
ISLS
518
ICLS2012
Volume 2: Posters
Commercial Development of e-Learning Materials for Science and Mathematics Subjects in Hong Kong: Preliminary Evaluation
Yau-yuen YEUNG and Irene Chung-man LAM, The Hong Kong Institute of Education, 10 Lo Ping Road, Tai Po, N.T., HONG KONG Email: yyyeung@ied.edu.hk and icmlam@ied.edu.hk
Abstract: The recent education reform in Hong Kong has led to a greater demand for elearning materials in schools and so it is important to identify an appropriate approach for commercial development of effective e-Learning materials. Working with an e-Learning provider company for evaluation of its pilot implementation in 3 schools, our preliminary findings reveal various kinds of difficulties to be overcome before e-Learning can be seamlessly integrated with formal school curriculum.
Background
The provision of quality and sufficient resources for interactive self-learning is one of the main objectives on the strategy on Information and Communications Technology (ICT) in education in Hong Kong. In 2008, the Hong Kong Government has launched The Third Strategy on IT in Education: Right Technology at the Right Time for the Right Task so as to further integrate ICT and implement e-Learning (Education Bureau, 2008). This strategy built upon prior strategies that empowered learners and teachers with IT, enhanced educational leaders to integrate IT in schools and provide necessary IT infrastructure and training to teachers etc. As the New Senior Secondary Curriculum (NSSC) was being put into practice in Hong Kong in September 2009 with the aim of re-aligning the senior secondary and university education system with Mainland China and USA, the Science and Mathematics Key Learning Areas, curriculum structures and the assessment requirements have been substantially revised (Lee, Lam and Yeung, 2010). Since then e-Learning has become a popular alternative learning platform, more textbook publishers have started to provide electronic resources in addition to printed textbooks. Concurrently, there is an emerging number of commercial companies that concentrate on providing e-Learning resources. One of them requested for our professional service to objectively assess the quality of their e-Learning materials and so this provides a good opportunity for us to conduct research on the effectiveness of students e-Learning, the feasibility and difficulties of implementation, and the attitude and receptivity of teachers and students towards the use of e-Learning materials (see, e.g. Ardito et al, 2006 and Ozkan, & Koseler, 2009) in addition to regular teaching in school.
Methodology
The e-Learning provider company involved in the present study has been developing a bank of self-learning materials in science and Mathematics subjects, i.e. Physics, Chemistry, Biology and Mathematics, in an online learning system with an aim to not only help student to develop the basic key concepts of the topics that will be taught in the next lessons but also enable them learn the whole subject syllabus by themselves. There are around 100 learning objects (LOs) to be developed for each subject in accordance with the official curriculum and assessment guide of the NSSC. A variety of multimedia content such as movie, animation and simulations, guided exercises and assessments are employed to assist students in achieving the learning outcomes of various topics. Our project team which consists of both subject experts and science/maths teacher educators was consigned with a research contract to carry out the relevant evaluation. We continually provide advice on the evaluation of the learning system, as well as the practice on school-based test-bed study. The theoretical framework for the present study is based on the self regulated learning theory which consists of three components: metacognitive awareness, strategies used and motivational control (Bruning & Bruning, 2004). Many studies evidenced that student who engages in effective planning generally do better (Boekaerts et al., 2000). For development of the evaluation instrument, experienced secondary school teachers are invited to design multiple-choice (MC) questions for evaluation of students learning effectiveness as based on the public examination papers and project team members serve as consultants to monitor the quality and standards of those MC questions. Other research tools (in form of questionnaire for pre and post surveys and interview guidelines) are specifically developed for collecting both quantitative and qualitative data from students and teachers who participated in the school-based test-bed study. Three secondary schools voluntarily participated in the study. After negotiation with the company, the effectiveness of the learning system is evaluated by the students scores after undergoing up to 3 close-loop learning processes. There are 10 MC questions for each of the 20 LOs selected for the pilot test and students are required to attain a score of 80% or above in each LO. Those who fail to achieve 80% will be asked to study the LOs again and then re-take another MC test (of similar questions with different answers). Each student will have a total of 3 attempts to study each LO. If less than 80% of the students achieve scores of 80%, the relevant LO will be revised by the e-Learning developer.
ISLS
519
ICLS2012
Volume 2: Posters
The qualitative data obtained from the semi-structured interview is used for triangulation with the quantitative data, as well as to provide further insights into the self-regulated learning strategies used. Also, some selected test-bed cases are video-recorded for analysis of instruction and students attitudes towards different self-regulated strategies. At the final stage of the project, both the school teachers and our project team members will be interviewed (using a pre-designed interview instrument) to collect their view and feedback on the feasibility, difficulties, receptivity and attitudes towards this kind of teaching and learning approach in their respective subjects.
Preliminary Findings and Discussion

This study is being implemented in the classroom or computer room with a total of about 400 students (at S4 or S5, i.e. Grade 10 or 11) from three different secondary schools. During August to October 2011, around 200 students participated in the first round test and a total of 24 LOs were developed for testing in the four subjects. A pre-test questionnaire survey was administered to the participating students before their e-learning lesson. The specifically developed questionnaire included 5 parts, to wit (1) personal particulars and ICT facilities at home, (2) students past experience of e-learning, (3) interest towards science/maths subject and metacognitive abilities, (4) students general learning strategies, and (5) students subject-specific learning strategies. From Part (1), our initial findings indicate that over 90% of students possess desktop PC at home (plus around 40% with notebook and 10% with netbook computers) while 3 quarters of them have broadband Internet access and around a half with wireless Internet access at home. This provides a very favorable context and environment for widespread implementation of e-learning in Hong Kong. For subsequent parts, there are around 50 questions using the 5-point Likert scale (with 1 for strongly disagree and 5 for strongly agree). For example, on asking the students perception of e-Learning in the three, most of them have not developed a regular practice to use eLearning material while many of them know about the meaning of e-Learning. Qualitative results are based on the observation as conducted by the student helpers, project team members and subject teachers in schools. It was reported that rather few students read LOs before completing the LOs test. Instead, most of them tended to read the MC questions and then search for the answers from the LO content. Some students even took less than 4 minutes to finish a LO test which consists of 10 MC questions, the results were below average with scores of 20% to 30% for all 3 available attempts. The helpers also noticed that students who score below average often do other things such as surfing the net and playing online games while studying LOs. Subject teachers argued that textbook maybe more interesting than e-Learning materials. They pointed out that the instruction of the systems is unclear and the students learning diversity should require different time allocation for their test. It is also important to provide feedback to students after attempting the MC test. Some students found many LO of certain subjects like Mathematics boring as they were even less attractive than traditional textbooks and so the LOs and e-Learning system should better be revamped to include more innovative and interactive content. Moreover they found that the extent of LOs were not consistent in length as some required students half an hour to study whereas some took them an hour. Hence they suggested the developer to take into account of the amount content for each LO during the design consideration. The result of first round test revealed a rather disappointing evaluation result for the learning effectiveness of those LOs because most students could not get a satisfactory score, especially for the Mathematics subject. However, those preliminary findings could hopefully help the developer make a critical reflection on their pedagogical design of the LOs and substantially refine them for the second and third rounds of testing before they could be put into the commercial market for adoption/purchase by teachers and students.
References
Ardito, C., Costabile, M. F., De Marsico, M., Lanzilotti, R., Levialdi, S., Roselli, T., & Rossano, V. (2006). An approach to usability evaluation of e-learning applications. Universal Access in the Information Society, 4(3), 270-283. Boekaerts, M., Pintrich, P. R., & Zeider, M., (2000). Handbook of self-regulation. San Diego: Academic Press. Bruning, R. H., & Bruning, R. H. (2004). Cognitive psychology and instruction (4th ed.). Upper Saddle River, N.J.: Pearson/Merrill/Prentice Hall. Education Bureau. (2008). The Third Strategy on IT in Education: Right Technology at the Right Time for the Right Task. Retrieved 15 March 2011 from http://edbsdited.fwg.hk/3ITED/ Lee Yeung-chung, Lam Chung-man and Yeung Yau-yuen. (2010). From changes in education system to curriculum reform: A critique of the new Hong Kong senior secondary science curriculum. Asia-Pacific Forum on Science Learning and Teaching, 11(1), Article 13. Ozkan, S. & Koseler, R. (2009). Multi-dimensional students evaluation of e-learning systems in the higher education context: An empirical investigation. Computers & Education, 53(4), 1285-1296
Acknowledgements
Mr. & Mrs. S. H. Wong Foundation Limited and HKIEd are thanked for their financial support.
ISLS 520
ICLS2012
Volume 2: Posters
Relationships between Representational Characteristics, Students Education Levels, and Beliefs of Models
Silvia Wen-Yu Lee, National Changhua University of Education, Taiwan, silviawyl@cc.ncue.edu.tw Hsin-Yi Chang, National Kaohsiung Normal University, Taiwan Hsin-Kai Wu, National Taiwan Normal University Abstract: We developed a set of context-based, multimedia questionnaires for surveying students beliefs of scientific model and modeling. The questionnaire constructs included: nature of model, purpose of model, nature and process of modeling, and evaluation of modeling (Schwarz et al.,2009). We investigated the extent to which the representational features of the models (i.e., modality, dimensionality, and dynamics), and students education levels influence students beliefs of models.
Major Issues Addressed

Models are seen as inseparable part to science and both models and modeling are essential to science learning (Gilbert, 2008, 1993; Harrison & Treagust, 1998; Lehrer & Schauble, 2003, 2006; Schwarz, 2009; Stewart, Cartier, & Passmore, 2005). For instance, National Research Council (1996) enlisted students understanding of models and modeling as one of the major goals of science teaching. Researchers suggested teaching models and modeling should not only focuses on science concepts but also should promote sophisticated beliefs of scientific models (Gobert, et al., 2010; Prins, Bulte, & Pilot, 2010; Raghaven & Glaser, 1995). Beliefs of scientific models can be seen as part of epistemology of science (Justi & Gillbert, 2003; Prins, et al., 2010). Among studies investigating the students beliefs or views of models, however, few studies investigated the relationship between the type of representations of the models and students beliefs of models. Therefore, in this study, we took advantage of online multimedia and presented models with various representational characteristics. In this study, we aimed to answer the following research questions: 1.How do students beliefs of models relate to different representational characteristics of the models? 2.To what extent do high school students beliefs of models differ from middle school students?
Significance of This Study

In addition to addressing the potential impact of representational characteristics on beliefs of models, this study also attempts to understand the potential impact of the subject domain by contextualizing survey questions. Past research used interviews or text-based questionnaire to understand students general beliefs of models and modeling; however, those questions are usually context-free or decontextualized. For instance, Gorsslight et. al. (1991) asked students what comes to mind when you hear the word model? We believed that students beliefs of models are similar to epistemological beliefs that are domain specific. Therefore, we anchored the survey questions in middle-school level biology content. This will give the survey questions richer context. In the future, we will further design questionnaires based on different subject domain.
Theoretical and Methodological Approach

In this study, we designed an online survey with multimedia representations. The online survey included four different constructs: 1) nature of models 2) purpose of models, 3) nature or process of modeling, and 4) model evaluation (Grosslight, Unger, Jay & Smith, 1991; Schwarz et al.,2009). The three representational dimensions are modality, dimensionality, and dynamics (Ainsworth, 2006). For modality, we refer to the difference between text-based model and graphical or chart models. The dimensionality refers to comparison between 2D models and 3D models. Finally, we compare students beliefs of dynamic and static models. This is a study in progress and we are currently in the process of collecting data. In this study, we will survey about 40 8th graders and 60 11th graders. The questionnaire is semi-structured with multiple choices and open-ended questions with regards to why students choose the answer. We will also interview 8 middle school students as well as 8 high school students to better understand the reasons behind their beliefs of models. We used the same questionnaire as an interview protocol. The answers to the multiple choices will be tallied and analyzed statistically. The interview data will be transcribed verbatim. Opened coding will apply to the answers to the open-ended questions and then to create categories of students reasoning.
ISLS
521
ICLS2012
Volume 2: Posters
Predicted Results and Implications

We hypothesized that students epistemological beliefs of models can be influenced to some extent by the representational characteristics. If it is true, then models with different representational characteristics may also influence the modeling process. Also, we believed that there are differences in students beliefs between middle school and high school students, as high school students may have more knowledge in biology and may have had more opportunities to explore science models. Finally, the results from the open-ended questions can serve as the foundation for creating a two-tier epistemological survey. The first tier would be selecting their beliefs and the second tier would be choosing the reasons behind.
Selected References
Gilbert, J. K. (Ed.). (1993). Models & Modelling in Science Education. Hatfield, UK: The Association for Science Education. Gilbert, J. K. (2008). Visualization: An emergent field of practice and enquiry in science education. In J. K. Gilbert, M. Reiner & M. Nakhleh (Eds.), Visualization: Theory and Practice in Science Education (Vol. 3): Springer. Gobert, J. D., O'Dwyer, L., Horwitz, P., Buckley, B. C., Levy, S. T., & Wilensky, U. (2010). Examining the relationship between students' understanding of the nature of models and conceptual learning in biology,physics,and chemistry. International Journal of Science Education, 1-32. doi: 10.1080/09500691003720671 Grosslight, L., Unger, C., Jay, E., & Smith, C. (1991). Understanding models and their use in science : conceptions of middle and high school students and experts. Journal of Research in Science Teaching, 28(9), 799-822. Harrison, A. G., & Treagust, D. F. (1998). Modelling in science lessons : Are there better ways to learn with models? . School Science and Mathematics, 98(8), 420-429. Lehrer, R., & Schauble, L. (2003). Origins and evolution of model-based reasoning in mathematics and science. In R. Lesh & H. M. Doerr (Eds.), Beyond Constructivism: Models and Modeling Persepctives on Mathematics Problems Solving, Learning and Teaching. Mahwah, NJ: Lawrence Erlbaum Associates. Lehrer, R., & Schauble, L. (2006). Cultivating model-based reasoning in science education. In R. K. Sawyer (Ed.), The Cambridge Handbook of The Learning Sciences (pp. 371-388). New York, NY: Cambridge University Press. Prins, G. T., Bulte, A. M. W., & Pilot, A. (2010). Evaluation of a design principle for fostering students' epistemological views on models and modelling using authentic practices as contexts for learning in chemistry education. International Journal of Science Education, 1-31. Raghaven, K., & Glaser, R. (1995). Model-base analysis and reasoning in science: The MARS curriculum. Science Education, 79, 37-62. Schwarz, C. V. (2009). Developing preservice elementary teachers' knowledge and practices through modeling-centered scientific inquiry. Science Education, 93, 720-744. Stewart, J., Cartier, J. L., & Passmore, C. M. (2005). Developing understanding through model-based inquiry. In M. S. Donovan & J. D. Bransford (Eds.), How Students Learn: Science in the Classroom (pp. 515-565). Washington DC: National Academies Press.
ISLS
522
ICLS2012
Volume 2: Posters
Matchballs a Casual Game for Learning and Domain Ontology Enrichment

Sabrina Ziebarth, Nils Malzahn, H. Ulrich Hoppe, University Duisburg-Essen, Lotharstr. 63, 47048 Duisburg {ziebarth, malzahn, hoppe}@collide.info Abstract: Training and continuing education in and for the food industry has to face the challenge of reaching and motivating workers without or with only a low level of formal qualification. We have developed and evaluated a casual game for training knowledge on food safety and hazardous material regulations that provides - in addition to the fun of playing it several incentives for motivating different types of users. The food industry in Germany is characterized by a high amount of workers without or with only a low level of formal qualification. While these workers are easily found and taught to perform the simple and often physically exhaustive tasks, there is a lack of employees with a higher qualification (e.g. skilled workers), who are able to use and control the complex machines and processes of the food production industry. Thus, the human resource managers try to train some of the lowly qualified to a higher qualification level to close the gap. This is not an easy task since these people often have a migration background and therefore language problems and/or they are not very motivated to learn because of various reasons, e.g. education is not an asset for them or their work is too exhaustive. The project Foodweb2.0 (funded by the German Ministry of Research and Education) aims at training the employees of the German food industry using two basic strategies: motivating employees for vocational training and performing education in collaborative, blended learning using Web2.0 technologies. One of the applications combining both strategies is the use of a collaborative, serious and casual game: Matchballs. Casual games are known to be small games with a high potential for frequent gaming by people of various social and educational background. These games are characterized by simple and easy to learn rules (Kuittinen, Kultima, Niemel & Paavilainen, 2007) and either by a slowly increasing difficulty or a time limit combined with a high score list. Serious games are games that have a serious context. Serious games are often used to virtually train situational behavior like conflict resolution or firemen training or to implicitly transport some knowledge that would not be transferred easily otherwise, because it is too abstract (e.g. nutritional education for young diabetes patients) (Michael & Chen, 2006). They have been established in vocational and advanced training over the last years and have a big potential for informal further vocational training. To be flexible concerning the learning domain we decided to use an ontology-based approach. The ontology may be easily exchanged to adapt the game to another domain. Furthermore, the ontology principally offers the opportunity to encode specific feedback for common misconceptions like it is often done in intelligent tutoring systems. Ontologies are usually incomplete, since it is nearly impossible to represent even a limited domain in full detail. Accordingly, relations created by the players that do not occur in the knowledge base are not necessarily wrong, but possibly just missing, especially if a significant amount of players creates them. Therefore, frequently occurring relations are very interesting, because either they can be used by knowledge engineers to enhance the ontology or they are typically misconceptions to be resolved by the teachers. Thus, apart from teachers being interested in getting information about common misconceptions to correct these, it is possible to use the so called wisdom of the crowd of the game players to enrich a pre-built ontology. In this way the game presented here is also a game with a purpose (cf. von Ahn, 2006). Games with a purpose (gwap) are often used for human computing, i.e. by exploiting the human intelligence to facilitate tasks that are otherwise difficult or impossible to reach by computational means. There already are gwap approaches for building, maintaining and aligning semantic web vocabularies (e.g. Siorpaes & Hepp, 2008). Matchballs is designed as a simple allocation game, in which the player creates statements by linking (matching) concepts displayed as balls (see Figure 1). A statement consists of two concepts linked by a relation. The game can be played either as two player game or as single player game with a bot. Each pair of players sees the same game field (concepts) and the goal is to agree with the teammate on as many relations as possible in a given time. To agree on a relation both players have to create it. If they agree on a relation, they score points and get time bonuses. Players may see the relations of their teammates, but not the relation types. As knowledge domain we use the domain of food safety and hazardous material regulations, which is an important topic of further education in the German food industry. The considered concepts are specific situations, actions, dangerous substances and edibles, which can be linked by using the four semantic relations is similar to, is more general than, results in and then you may not.
Introduction
Matchballs
ISLS
523
ICLS2012
Volume 2: Posters
Figure 1. Matchballs user interface.
Figure 2. Display of achievements.
The initial ontology was hand-crafted by our knowledge engineers. Feasible statements are for example machine overheats results in fire danger or if oil starts burning then you may not extinguish the fire with water. There are several incentives for playing the game considering different types of players: For competitive players there are high scores and time bonuses, which are a well-known and often used incentive since early arcade games. Furthermore, players can collect achievements, which are trophies for solving certain predefined tasks, e.g. for playing a given number of games with another player or with the bot (see Figure 2). Achievements are a more recent kind of incentive often used in modern console games. They not only address competitive players, but also people with a collectors passion, who want to unlock the full set of obtainable awards. While competitive players will tend to play against the bot to be not dependent on the teammate, for team players the possibility to play together with another human is an incentive of its own.
Discussion
The Matchballs game has been evaluated with a class of 18 students at the Academy of Sweets in Solingen, Germany. The evaluation results show that the Matchballs game was perceived as a casual game that is addictive enough to encourage the learners to try another game to improve their score. Based on the data generated during the game sessions we were able to identify 17 relevant associations that were not represented in the initial ontology. These associations were integrated into the ontology by our knowledge engineers. Thus, we could also prove that our approach of using a learning game also as game with a purpose is feasible. In this way the game may be seen as self-extending with respect to closing gaps in the ontology. In a way the game may be viewed as a concept map creation game when used as a multiplayer game, where the players create a shared concept map. Concept maps are successfully used as learning tool for linking existing and new knowledge as well as for evaluation and identifying valid and invalid ideas of students (Novak & Cans, 2006). If the game is played in single player mode, the game may still be used as an advanced vocabulary trainer. In spite of playing the game individually the students still collaborate indirectly. Accordingly, teachers can apply the game to get an overview of typical misconceptions of the group but also of single students. In the context of the Foodweb2.0 project there have already been several requests by teachers and students for transferring the game to further knowledge domains. We will try to incorporate these domains and enhance these ontologies with specific feedback on the relations made by students. The feedback will be given at the end of the game. For the multi-player scenario there will be a feedback about the existence of their relation in the ontology. In single-player scenarios the feedback hints at possible misconceptions automatically based on information directly represented in the ontology for particular error types and exploiting the semantic ontology structure for the generation of generic feedbacks following an intelligent tutoring approach. Kuittinen, J.; Kultima, A.; Niemel, J., & Paavilainen, J. (2007). Casual games discussion. In Proceedings of the 2007 conference on Future Play (pp. 105-112), New York: ACM. Michael, D., & Chen, S. (2006). Serious Games: Games that educate, train, and inform. Boston: Thomson Course Technology PTR. Novak, J. D., & Cans, A. J. (2006). The theory underlying concept maps and how to construct them. Technical Report IHMC CmapTools 2006-01 Rev 01-2008, Florida Institute for Human and Machine Cognition, available at: http://cmap.ihmc.us/Publications/ResearchPapers/TheoryUnderlyingConceptMaps.pdf. Siorpaes, K., & Hepp, M. (2008). Games with a Purpose for the Semantic Web. Journal IEEE Intelligent Systems, 23(3), 50-60. von Ahn, L. (2006). Games with a Purpose. Journal Computer, 39(6), 96-98.
ISLS 524
References
ICLS2012
Volume 2: Posters
Becoming a Writer: Examining Preschoolers Interactions, Modes of Participation, and Use of Resources at a Science Writing Center
Amy Gillespie, Deborah W. Rowe, Vanderbilt University, 230 Appleton Place Nashville, TN 37203 Email: amy.gillespie@vanderbilt.edu, deborah.rowe@vanderbilt.edu Abstract: In this paper, I describe the talk, activities, and nonverbal behaviors of 2 preschoolers at a science writing center. Both students became writers by producing multidimensional texts (e.g., letters, drawings) about frogs. Although group interactions and writing tools were important, access to the instructor was the critical resource both students required to create written products. The findings have implications for creating productive learning environments for young learners to practice writing skills across the content areas.
Introduction
From a sociocultural perspective, as young learners begin to develop writing skills, they rely on their knowledge of other symbolic forms of expression (e.g., talk, drawings, dramatic play). Therefore, early writing is typically multimodal, as children combine talk, drawings, objects, and gestures to convey their intentions in written form (Leland & Harste, 1994; Rowe, 2008). Further, from this perspective, writing is considered a socially situated act, mutually constituted by the writer, his use of and interaction with resources (human and material) in his environment, and his social and cultural experiences (Meier, 2000; Rowe, 2008). Therefore, to better understand how students take on the role of a writer during their time at the writing center, I find it important to study how they participate in writing activities and use the available resources in their immediate surroundings. Interactions with peers, interactions with the instructor, and use of writing tools (e.g., paper, pencil) at the writing center may scaffold students learning of writing skills, help them reach their potential as writers (Vygotsky, 1978), and help them demonstrate their understanding of subject matter. Understanding which scaffolds prove most useful to students and how these scaffolds appear to facilitate their learning could prove powerful for structuring effective content area writing activities for preschool learners. Therefore, the research question guiding my analysis was: How does a student use the available resources (e.g., tools, interactions, talk, gestures, body positioning) to become a writer at the science writing center?
Participants and Setting

Two African American male preschool students, Joe and Bryce (both age 5), were the focal participants in this study. Video was taken in the students preschool classroom. The preschool was located in a public housing project in an urban area of the southeastern United States.
Data Analysis
The first author watched and transcribed a 7 min video segment of Joe and Bryce working at the science writing center with a teacher/researcher (i.e., DR) directly after a science lesson on frogs. At the center, students had access to writing tools, books about frogs, and a live tree frog in a small container on the table. Students were expected to draw a frog and write a sentence describing the frog. The particular video segment was chosen because it captured two different student approaches to completing the same writing task. In addition to talk, the author noted nonverbal behaviors (e.g., gestures, body positioning) in the transcript. To the right of the transcribed speech, an activity bar for each participant indicated what he or she was doing during each turn at talk. Eight distinct activities were coded throughout the session: (a) finding the frog, (b) drawing the frog, (c) using the audio recorder, (d) writing, (e) finding and describing the frog, (f) storytelling about frogs, (g) imitating a frog, and (h) looking at books. Portions of the transcript and images of participants interactions and positioning are presented on the poster.
Findings
Throughout the video segment, the importance of shared interactions and positioning within the group for becoming a writer was evident. At the start of the session, Joe and DR established an F-formation (p. 243), an interactional stance characterized by deliberate positioning, both physical positioning as well as the positioning of talk and behavior, to create and sustain an interaction (Ciolek and Kendon, 1980). DR and Joe achieved this F-formation by aligning their talk, body positioning, and gaze with one another as they attempted to find the tree frog in the container. When Bryce approached the table, he disrupted this formation by positioning himself between DR and Joe. The disruption caused a reordering, as each member of the group had to negotiate his or her position within it and attempt to preserve the group order (McDermott, Gospodinoff, & Aron, 1978). During this reorganization, Joe lost access to the groups shared formation and interactions. Bryce positioned his body closer to the writing tools and further between Joe and DR. Additionally, Joe stopped making relevant
ISLS 525
ICLS2012
Volume 2: Posters
contributions to the group discussion and no longer aligned his gaze or body positioning with the other group members, further positioning himself at the margins of the group activities (Goodwin, 2007). Joes pattern of misaligned speech and behaviors continued throughout the first half of the session. During much of this time, he appeared one step, or one activity, behind what DR and Bryce were doing. Bryce began the task of becoming a writer early in the session. His ease into the writing role appeared to be facilitated by his access to important resources, including writing tools and a shared interactional space with DR. The need for writing resources, and most importantly, access to DR, to become a writer was also apparent with Joe. Halfway through the session, Joe demonstrated an attempt to obtain congruent alignment (p. 357) with the others by loudly pronouncing the word hiding four times and synchronizing his first utterance with DR and Bryce (Goodwin, 2007). Following Joes overlapping talk with the others, his apparent bid for reentry into the participation framework, Bryce moved his body away from the table. This permitted Joe to join the formation and to have greater access to DR. After this, Joe became assertive about maintaining his positioning within the group and more focused on accessing the necessary resources for writing. For example, Joe took a pencil from Bryce and pushed Bryce away when he leaned over to look at Joes writing. Further, Joe aligned his talk, gaze, and behaviors more closely to those of DR for the remainder of the session. Yet, at the same time, Joes shift into the role of a writer resulted in another reordering of group roles and organization. This time, Bryce faced the challenge of redefining how he would proceed in the writing center activities after the shift in group dynamics (McDermott et al., 1978). Bryce made bids for DRs undivided attention during the last 2 min of the session, showing his reliance on her as a resource for becoming a writer. He interrupted DR when she spoke to Joe. At one point, he even wrapped his arms around DRs shoulders, laying his head on her. Yet, without DRs sole focus on him, Bryce seemed to lack what was necessary for him to continue in his role as a writer at the writing center. Like Joe, Bryce made bids for re-entry and alignment with the others, but his efforts were unsuccessful, leaving him on the periphery of the group formation and group activities. Therefore, for the remainder of the session, Bryce reverted back to an earlier activity, drawing the frog (instead of writing about it). DRs status as a valuable resource, who students sought out and seemed to require for becoming a writer, became overwhelmingly evident in the last 10 seconds of the video. Seconds prior to DRs departure, both students were talking over one another, vying for her attention. When DR left the table, Joe and Bryce stopped speaking immediately. Joe continued to look at a book silently, while Bryce looked around the classroom, appearing at a loss for what he should do for several seconds. In DRs absence, the entire participation framework seemed to fall apart. Similarly, without access to and interactions with DR, both students were uncertain, unable, or unwilling to become a writer at the writing center or to continue producing written products.
Conclusions and Potential Significance of the Findings

Although Joe and Bryce followed different paths toward becoming writers, ultimately each student produced a drawing of a frog and a corresponding attempt at writing about what the frog was doing in the drawing. At different points during the session, each student lacked access to the group formation and the shared interactions of the other group members, and thus, he ceased writing. Bryce created an F-formation (Ciolek & Kendon, 1980) with DR at the beginning of the session, from which Joe was excluded. This participation framework with DR assisted Bryce in becoming a writer early in the session. When Joe gained access to DR halfway through the session, he too created a shared interactional space with her and then was able to produce a written product. Based on my analysis, becoming a writer at the science writing center could be accomplished in multiple ways and through different activity sequences, yet the process was chiefly dependent on each students shared interactions and joint activities with their instructor. Overall, these findings lack generalizability due to the small number of participants in my analysis and the limited time I observed their behaviors. I provide suggestions for designing effective learning centers for preschoolers with these limitations in mind. First, based on my analysis, writing tasks embedded in content area subject matter can be successful and productive learning activities, even for very young students. The students in this analysis were not proficient writers. Yet, even at the emerging writer stage, they were able to produce written products related to science concepts. Second, content-based writing centers should be designed to include opportunities for students to collaboratively construct written products with their instructor, or perhaps, another more advanced writer. Third, and most importantly, the structuring of effective content-based writing centers should take into account each students readiness for writing and the scaffolds, both material and human, he or she needs to be successful with the assigned tasks. Future researchers could expand on my analysis by observing students paired with more advanced writing peers, to understand if these scaffolds prove helpful at preschool learning centers. Additionally, researchers could alter group configurations at content-based writing centers to examine if larger groups of students appear to benefit from this type of instructional arrangement.
ISLS
526
ICLS2012
Volume 2: Posters
Fifth and Seventh Graders Patterns of Understanding About Cells and Heredity in a Technology-Enhanced Curriculum
Dante Cisterna, Michelle Williams, and Joi Merritt, Michigan State University, 301-E Erickson Hall, East Lansing, MI 48823, USA. Email: cisterna@msu.edu, mwilliam@msu.edu, jmerritt@msu.edu Abstract: In this study, we explored fifth and seventh grade students ideas about heredityrelated concepts and described patterns of understanding for these topics in the context of technology-enhanced curricula. Analysis of embedded assessments shows students progressed to more sophisticated levels of understanding, especially by reviewing non-normative ideas and integrating new content to their previous understandings. Students tended to struggle in distinguishing genes, chromosomes, and DNA and in connecting cell division with traits inheritance.
Introduction
To develop scientific literacy as it relates to genetics, students need to learn about cells and inheritance (American Association for the Advancement of Science, 2001; Tsui & Treagust, 2007). Research on students understanding of genetics and heredity shows that these topics are difficult for students to learn because they are complex and abstract (e.g., Lewis & Wood-Robinson, 2000; Tsui & Treagust, 2007). Students also have problems understanding complex genetic phenomena such as the relationship between genetic material and proteins (Duncan & Reiser, 2007). Since most research on students understanding of genetics has focused on the secondary level (Venville, Gribble, & Donovan, 2005), there is a need for research on students thinking and learning at late elementary and middle school levels (Duncan, Rogat, & Yarden, 2009). Our study explores upper-elementary and early middle school students ideas about cells, reproduction, and heredity and suggests patterns of understanding for these topics over time, through the implementation of a technology-enhanced curriculum.
Methods
Our cross-sectional, qualitative study was conducted in a socially and economically diverse, Midwestern, suburban school district. Fifth (ages 10-11) and seventh grade (ages 12-13) students worked with heredity curricula developed in the Web-based Inquiry Science Environment (WISE), a technology-enhanced learning environment. The fifth grade unit focused on helping students to distinguish inherited and acquired traits in organisms; understanding reproduction through plants; and learning that cells are building blocks of all living things. The seventh grade unit builds on what students learned in fifth grade to learn about cell molecules, structures, and functions related to inheritance DNA, alleles, genes, and chromosomes, as well as the relationship between each of these concepts. The units included several online embedded assessments whose purpose was to capture students progress in learning the particular content of each activity, so that students could integrate their ideas into more sophisticated levels of understanding. Student responses to online embedded assessments from both units served as our main data source. The responses of 90 fifth-graders and 54 seventh-graders were selected and analyzed using an open-coding procedure (Bohm, 2004). The iterative analysis of student responses resulted in definite categories of student understanding. Patterns of understanding were analyzed according to the knowledge integration (KI) framework, which defines student learning as the continuous addition of new ideas and the resulting reorganization of their personal knowledge (Linn, 2006). So, student responses were organized into two groups: cells and reproduction, and traits and inheritance.
Results
Along the fifth grade unit, students systematized their understanding of cells as the basic units that make up multicellular organisms and understood cells as units with some degree of specialization. They also recognized basic aspects of trait inheritance, namely that traits are equally inherited from both parents. Seventh-graders integrated the new content with their previous knowledge and provided more sophisticated and detailed explanations. Responses show that students were able to characterize cells by describing their genetic material, to explain the process of cell division and some of its implications, to distinguish between sexual and asexual reproduction, and to explain how traits are inherited at the cellular level. Students, however, tended to struggle in distinguishing genes, chromosomes, and DNA and in connecting cell division with traits inheritance. We also analyzed embedded assessments designed to provide evidence of how students were reframing their ideas when they added new content to their existing knowledge. One of these assessments had students explain whether it is true that girls inherit most of their features from their mothers (or boys from their fathers).
ISLS 527
ICLS2012
Volume 2: Posters
Students in fifth grade used more varied explanations to respond to the embedded assessment (See Table 1). A group of students (13%) considered that this statement to be true, and their arguments were based on the common traits that they shared with their own same gender parent. Other students responded that the statement was false and their explanations consisted of giving examples of traits they shared with their opposite gender parent (32%) or simply paraphrasing the question statement (44%). Since the seventh grade unit introduced new topics of cell division and trait inheritance at the cellular level, students at this grade level tended to respond correctly (84%), providing more sophisticated and detailed explanations. In some cases, students included new concepts in their responses, for example, characteristics of sex cells, genes, alleles, and chromosomes (39%). Table 1. Examples of student responses in 5th and 7th grade in a cross-grade embedded assessment Embedded assessment: Is it true or false that boys inherit more traits from their fathers than from their mothers? Please explain your answer. 5th grade students 7th grade students True, because guys are taught everything by their It is false. All [offspring] inherit half of their dads like sports and stuff like that chromosomes from their mother and half from their father, so there is an equal chance for boys to inherit more from their mother than from their parent. I look like both of my parents but I think I look more False both parents put in 50 percent of the traits. The like my dad than my mom. My face structure is more female holds the egg and the male holds the sperm like my dads but my eyes look more like my moms which makes up 50 percent for each parent. eyes.
Implications
When upper elementary and early middle school students experience coherent technology-enhanced, instructional materials over time, they are more able to explain and concepts that require the use of complex concepts of cells, reproduction, and heredity and did not maintain non-normative ideas described by previous research such as the belief that girls inherit only their mothers traits and boys inherit only their fathers (Kargbo et al., 1980). However, students still struggled to distinguish among types of genetic material (e.g., Banet & Ayuso, 2002; Lewis & Wood-Robinson, 2000), especially when making connections among topics of cells and heredity in order to solve problems to explain complex phenomena, for example, some students responses were more fragmented, lacking adequate sophistication necessary to explain the implications of the meiosis process on trait inheritance. These findings can inform researchers and curriculum developers about areas where teachers need to scaffold student understanding, the range of student ideas, and the need to support students and teachers in connecting the different pieces of learning into sophisticated explanations.
References
American Association for the Advancement of Science. (1993). Benchmarks for science literacy. New York: Oxford University Press. Banet, E., & Ayuso, E. (2002). Alternativas a la enseanza de la gentica en educacin secundaria [Alternatives to genetics teaching in secondary school]. Enseanza de las Ciencias. 20(1), 133-157 Bohm, A. (2004). Theoretical Coding: Text analysis in grounded theory. In: U. Flick, E. v. Kardorff, & I. Steinke (Eds), A Companion to Qualitative Research (pp. 270275) London: Sage. Duncan, R. G. & Reiser, B.J. (2007). Reasoning across ontologically distinct levels: Students understandings of molecular genetics. Journal of Research in Science Teaching, 44(7), 938-959. Duncan, R. G., Rogat, A. D., & Yarden, A. (2009). A learning progression for deepening students understandings of modern genetics. Journal of Research in Science Teaching, 46(6), 655-674. Kargbo, D. B., Hobbs, D., & Erickson, G. L. (1980). Childrens beliefs about inherited characteristics. Journal of Biological Education, 14(2), 137-146. Lewis, J., & Wood-Robinson, C. (2000). Genes, chromosomes, cell division and inheritance-do students see any relationship? International Journal of Science Education, 22(2), 177-195. Linn, M. C. (2006). The knowledge integration perspective on learning and instruction. In R. K. Sawyer (Ed.), The Cambridge Handbook of the Learning Sciences (pp. 243-264). New York: Cambridge University Press. Tsui, C-Y., & Treagust, D. F. (2007). Understanding genetics: Analysis of secondary students conceptual status. Journal of Research in Science Teaching, 44(2), 205-235. Venville, G., Gribble, S. J. & Donovan, J. (2005). An exploration of young childrens understandings of genetics concepts from ontological and epistemological perspectives. Science Education, 89(4), 614 633.
ISLS
528
ICLS2012
Volume 2: Posters
Ontological Stances and Systems Thinking in the Favela and Asfalto

Izabel Duarte Olson, Northwestern University, 2120 Campus Drive, Evanston, IL 60028, izabel@u.northwestern.edu Abstract: Favela residents in Brazil have been marginalized throughout their history. Nonetheless, the favela environment seems to create conditions conducive to an ontological stance that could be advantageous for engaging in complex systems. This study is a preliminary examination of people from different social backgrounds in Rio de Janeiro, both favela (working-class) and asfalto (middle-class) residents, and evaluates whether their ontological stances might present any markers for complex systems thinking potential.
Introduction
Favelas are low-income communities in Brazil that exist in close proximity to middle- and upper-class neighborhoods (asfaltos) in major cities such as Rio de Janeiro. Economists Bourguignon, Ferreira, and Menndez (2007) found that favela residents encounter both marginalization and a public education system that discourages intergenerational educational mobility, producing chronic underachievement in these communities. Nonetheless, based on my observations over the past five years, I believe that life in favelas encourages residents to leverage an ontological stance that could aid their understanding of complex systems. This study is a first step in exploring this hypothesis.
Literature Review
According to Resnick and Wilensky (1998), learning about complex phenomena requires thinking in levels. Multi-level thinking is the simultaneous consideration of (1) the micro level by taking the perspective of the individual agent, (2) the macro level by taking a step back and appreciating a third-person perspective, and (3) the connections between levels by attending to the relationships among individual agents of the system (Resnick & Wilensky, 1998). Ontological stances that are geared toward understanding systems thinking would thus present the following markers: (1) the ability to see multiple perspectives (Resnick & Wilensky, 1998; Wilensky & Reisman, 2006), and (2) a focus on relationships (Resnick & Wilensky, 1998). My central hypothesis is that the socialization of favela residents causes them to develop a propensity for attending to relationships and taking multiple perspectives of familiar situations. This propensity, in turn, could produce an advantageous psychological milieu for engaging in complex systems thinking.
Method
There were 89 participants in this study (67 adults [mean age = 41.5] and 22 children [mean age = 9.6]). Fortytwo participants had a middle-class background (37 adults and 5 children); the overall mean age of middle-class participants was 36.3 years (SD = 18.4) and their mean years of schooling was 14.3 years (SD = 4.9). Fortyseven participants had a favela background (30 adults and 17 children); their overall mean age was 31 years (SD = 19.8). The favela sample came from eight different favelas around the state of Rio de Janeiro. The mean years of schooling among favela adults was around 7.3 years (SD = 5.4). The interview consisted of a drawing task and 12 open-ended questions that were intended to uncover participants differences in reasoning on familiar issues. The present focus is on participants drawings and reports on what they drew. Initially, participants were prompted to draw the places where they lived with the following question: Could you please draw the place where you live? Once participants had finished drawing, a follow-up question was asked to prompt them to describe their drawings: Tell me about what you drew? The first interviewees contacted were students, parents, and employees from a community center in the favela of Chumbada. A modified snowball sampling procedure was used where informants were asked to recommend another person for the interview who was as different from him or her as possible. Kappa was established by having a blind rater code 20% of all the drawings and reports.
Results
My results revealed that favela residents were more likely to talk about relationships and to use multiple visual perspectives (including both standard and aerial views) when drawing where they lived.
Aerial perspective.
ISLS
529
ICLS2012
Volume 2: Posters
Drawings were coded as having either an aerial perspective (birds eye view) or a standard perspective . Favela residents were more likely than middle-class residents to adopt an aerial perspective when drawing (2(1, N = 87) = 9.43a, p = .002, phi = .33, Kappa = 0.658, (p = .005). Since residents dwellings could have influenced their visual perspectives (e.g., living on a hill might encourage drawing from an aerial perspective), I controlled for the type of home (high-rise or house) and the elevation of its location (hill or plain). There was no correlation between those variables and the adoption of an aerial perspective.
Use of multiple perspectives.

Rather than depicting a solely standard or aerial perspective, some participants drew from mixed spatial perspectives; for example, illustrating both the inside and outside of a house, or using both an aerial view and standard view in one drawing. The favela participants were more likely to use multiple visual perspectives. The results were statistically significant (2 (1, N = 87) = 15.237a, p = .000, phi = .418, Kappa = 0.54, (p <0.000).
Use of relationships in the drawing.

I also counted the number of relationships that participants reported when describing their drawings. I coded for relationships when participants (1) used prepositional phrases relating components of the drawings to each other (e.g., This house is behind X street) or (2) related the components of the drawing to a personal episode (e.g., This house is where my aunt lived; she liked to invite the whole family over every year). Overall, the favela participants referred to more relationships among the components of their drawings. The favela sample talked about relationships an average of three times more than the middle-class sample did (t (66) = 4.58, p < .000, degrees of freedom adjusted for unequal variance, eta squared = 0.198, Kappa = 0.73, (p <0.001). I controlled for the number of words people used and the difference was still statistically significant.
Findings
My main finding is that the favela residents drawings and descriptions revealed multiple perspectives and relationships. This finding is significant because it suggests favela residents potential for appealing to these ontological stances when thinking about complex phenomena in familiar environments. These stances could be leveraged in educational settings by starting instruction with familiar systems to the favela dweller.
Conclusion, Implications, and Future Work

Despite the heterogeneity among the favela populations I studied, there was a consistent pattern of results that revealed what Wilensky and colleagues have long advocated as a base for thinking in levels: the use of multiple perspectives and a focus on relationships. These markers signal the ability to develop and build on an understanding of aggregate complex phenomena. Favela residents from diverse backgrounds seem to exhibit these abilities spontaneously on issues that are familiar to them. Because many systems have similar archetypes, favela residents quotidian knowledge could be leveraged in thinking about other systems in fields as diverse as science and sociology. While this study was focused on the favelas of Brazil, these findings could potentially translate to other low-income communities around the world, creating visibility for the alternative perspectives adopted in differing sociocultural and economic environments. It is also important to highlight that these findings do not imply that the middle class population cannot or does not show potential for thinking about complex systems. In the future, I would like to examine whether these markers of multi-level thinking translate into better comprehension of and reasoning about complex systems. In that vein, I next plan to examine participants responses to questions about inequality and social mobility to see what types of resources participants appeal to when speaking about emergent phenomena in their society.
References
Bourguignon, F., Ferreira, F. H. G., & Menndez, M. (2007). Inequality of opportunity in Brazil. Review of Income and Wealth, 53, 585618. doi:10.1111/j.1475-4991.2007.00247.x Resnick, M., & Wilensky, U. (1998). Diving into complexity: Developing probabilistic decentralized thinking through role-playing activities. The Journal of the Learning Sciences, 7(2), 153172. Wilensky, U., & Reisman, K. (2006). Thinking like a wolf, a sheep, or a firefly: Learning biology through constructing and testing computational theoriesan embodied modeling approach. Cognition and Instruction, 24, 171209. doi:10.1207/s1532690xci2402_1
Acknowledgments
I am indebted to Dr. Medin, Dr. Horn, and Dr. Iliev for all their support with this research.
ISLS
530
ICLS2012
Volume 2: Posters
Challenges of teaching through web-based inquiry: A longitudinal case study of a veteran high school teacher
Eleni A. Kyza, Iolie Nicolaidou, Department of Communication and Internet Studies, Cyprus University of Technology, P.O. Box 50329, 3603, Limassol, CYPRUS Eleni.Kyza@cut.ac.cy, Iolie.Nicolaidou@gmail.com Abstract: This case study follows a veteran biology teacher, as she pursued the integration of technology-supported inquiry over the period of three years. The teacher was involved in the design and enactment of a web-based inquiry learning environment. Multiple data collected included videotapes of all the enactments, teacher interviews and videotapes of all design meetings. Teacher challenges relating to inquiry and technology during the design and enactment of web-based inquiry are presented and discussed.
Introduction
Research and policy documents have long called for changes in the role of teachers to support inquiry learning and technology use; however, both issues still present teachers with several challenges (Cuban, Kirkpatrick, & Peck, 2001). Even experienced science teachers appear to need support to shift from inquiry rhetoric to inquiry practice (Luft, 2001) while teachers who are novices to technology-enhanced inquiry learning may, in addition, face many logistical and technical problems during their attempts to integrate technology in science learning and teaching (Gerard, Varma, Corliss, & Linn, 2011). This work outlines the findings of a case study of a secondary school teachers efforts to integrate technology-supported inquiry in her teaching over a period of three years. The analysis focused on two issues: First, the challenges that this teacher faced and how they were resolved and second, the development of the teachers technological pedagogical content knowledge (TPCK) over time. This work can contribute to the situated understanding of teacher cognition, as researchers have argued that there is a need to examine teacher learning from a learning sciences perspective, situating teacher research in the context of classroom practice (Fishman & Davis, 2006).
Methodology
Data sources
This study adopts an interpretive methodology, choosing to describe a case study of one teacher. The analysis follows Anna, a biology teacher, who at the onset of this work had 17 years of teaching practice. Anna volunteered as a participant in a researcher-teacher partnership with the goal of designing a web-based inquiry learning environment. Anna was keen on employing technology-enhanced inquiry learning with her students; even though she valued technology, she never received any professional development on how to integrate ICT in her class. Through a combination of iterative data collection measures, which included multiple interviews, design meetings, informal conversations, observations of classroom practice and reflections on own practice, we sought to examine the challenges that an experienced science teacher faced as she integrated a web-based inquiry learning environment in her practice. The teacher was interviewed four times, with each interview lasting about 2 hours. Qualitative and quantitative data were obtained from three classroom enactments, while videotapes from three preparatory meetings between the researchers and the teacher during the third enactment, and 31 design meetings helped identify episodes when the teacher discussed the design and enactment process of the web-based inquiry learning environments.
Data analysis
The Technological Pedagogical Content Knowledge (TPCK) framework (Mishra & Koehler, 2006) guided the analysis of the data. TPCK draws from Shulmans (1986) concept of Pedagogical Content Knowledge (PCK), and focuses attention on the interactions between technology, pedagogy, and content. Following Koehler, Mishra, and Yahya (2007) we coded the teachers discourse during design meetings, interviews and enactment preparatory meetings, characterizing the talk according to three categories: Content, Pedagogy, Technology. The evolution of the teachers talk was qualitatively analyzed over time and phase (e.g. design phase, enactment preparatory phase, reflection phase) with the goal of identifying and triangulating teacher challenges, and gaining a better understanding of the factors that contributed to these challenges.
Findings and Discussion

ISLS 531
ICLS2012
Volume 2: Posters
As with many novice teachers, evidence from Annas interviews, classroom enactments and insights extracted from design meetings, points to the fact that integrating technology into her teaching was challenging. Even though the teacher faced several technical challenges during enactments 1 and 2 which she attributed to her lack of knowledge of how computers work (Interview 3, Year 3), it was the interaction between pedagogy and technology that the teacher identified as most problematic. Indeed, Anna even described the use of technology in this context as frightening, stating among others in Interview 2 that Inquiry was not frightening. Computers were frightening. Not as a learning tool but the isolation they caused. I feel that I am not close to my students. That's why I talk/intervene to explain things! My only problem is the lack of immediacy with my students. They work independently. Reviewing their work is impersonal. You cannot see the logic behind their work. I want to be close to them and I can't be with all of them at the same time. This finding, which was re-iterated by the teacher in design meetings and interviews, and triangulated by the examination of her classroom behavior, can be explained through examining the interaction between technology, pedagogy and content talk. It is the development of this form of integrated knowledge about the interaction between the technology and pedagogy that troubled Anna over the years. Annas reflection above is implicitly a realization of her struggle to reconcile existing forms of Pedagogical Content Knowledge (e.g. it is important to have a deep understanding of students evidence-based reasoning) with the new technological affordances (e.g. allowing students to work independent of the teacher) and constraints (e.g. computer-supported collaborative inquiry is not immediately visible to the teacher). Over time, Anna developed strategies to combat her initial feelings of self-identified inadequacy about how to best guide student inquiry thinking. However, Annas case study suggests that the development of TPCK is not a linear function depending on time and accrued experience. Findings from Enactment 3 indicate that, without the opportunity to adapt the unit to her developing TPCK, the teacher expressed insecurity and disappointment in teaching with technology in this context, a fact verified by our analyses of her classroom practices during this enactment. Her confidence in her understanding of the interaction between technology, pedagogy and content recessed when, during Enactment 3, she taught a web-based unit, using the same technology but on a different topic and without the opportunity to tailor content, technology and pedagogy to her own epistemology of teaching. The combined examination of Annas enactments and reflections after these enactments suggests that Anna became more confident in her teaching with technology over time, not simply because of continued exposure to teaching with technology but, in addition, because of the opportunity to engage in the iterative design of the technology-enhanced curriculum. The latter allowed her to technology to address her pedagogical concerns. In this sense, her design involvement proved to be an emancipating factor allowing her to function within her own zone of proximal development for the specific enactment. Such findings have implications for teachers professional development for technology-enhanced learning.
References
Cuban, L., Kirkpatrick, H., & Peck, C. (2001). High Access and Low Use of Technologies in High School Classrooms: Explaining an Apparent Paradox. American Educational Research Journal, 38(4), 813-834. doi: 10.3102/00028312038004813 Fishman, B., & Davis, E. A. (2006). Teacher Learning Research and the Learning Sciences. In K. Sawyer (Ed.), The Cambridge Handbook of the Learning Sciences, (pp. 475-588). New York: Cambridge University Press. Gerard, L. F., Varma, K., Corliss, S. B., & Linn, M. C. (2011). Professional Development for TechnologyEnhanced Inquiry Science. Review of Educational Research, 81(3), 408-448. doi: 10.3102/0034654311415121 Koehler, M. J., Mishra, P., & Yahya, K. (2007). Tracing the development of teacher knowledge in a design seminar: Integrating content, pedagogy and technology. Computers & Education, 49(3), 740-762. Luft, J. A. (2001). Changing inquiry practices and beliefs: The impact of an inquiry-based professional development programme on beginning and experienced secondary science teachers. International Journal of Science Education, 23(5), 517-534. doi: 10.1080/09500690121307 Mishra, P., & Koehler, M. J. (2006). Technological Pedagogical Content Knowledge: A Framework for Teacher Knowledge. Teachers College Record, 108(6), 1017-1054. Shulman, L. S. (1986). Those Who Understand: Knowledge Growth in Teaching. Educational Researcher, 15(2), 4-14.
Acknowledgments
This work was funded by the Science in Society Initiative of the Seventh Framework Research Programme (FP7) of the European Community, under the CoREFLECT grant (217792). The opinions expressed as solely of the authors.
ISLS 532
ICLS2012
Volume 2: Posters
The role of online communities in science teachers continuous professional development

Yiannis Georgiou, Eleni A. Kyza, Department of Communication and Internet Studies Andri Ioannou, Department of Multimedia and Graphic Arts Cyprus University of Technology, P.O. Box 50329, 3603, Limassol, CYPRUS Ioannis.Georgiou@cut.ac.cy, Eleni.Kyza@cut.ac.cy, Andri.I.Ioannou@cut.ac.cy Abstract: We report on the creation of online teacher communities within the framework of the PROFILES project, aiming at teachers professional development in inquiry-based science education. It was anticipated that teachers would form active communities of practice to support the integration of inquiry and new technologies into their lessons. An exploratory approach was used to trace the local needs of the teachers and to identify the role of the online communities to support continuous professional development.
Aims
In many nations around the world, science curricula are currently undergoing reforms, focusing on the adoption of inquiry-based methods to increase students understanding of science. Such curriculum reforms require teachers to be well-informed about inquiry, therefore rendering the professional development of in-service science teachers as an issue of paramount importance. New technologies can enable the flourishing of communities of practice by creating online environments, where teachers can synchronously and asynchronously exchange ideas and reflect on their professional practice (Riverin & Stacey, 2008). Researchers (e.g. Guldberg & Mackness, 2009) caution that the greatest challenges with these online communities relate to the establishment and sustainment of the communities. This study reports on the initial stages of establishing online communities to engage science teachers in what Desimone (2009) has called embedded professional development; the latter can be briefly described as linking professional development discussions and reflections to an authentic task, closely related to teachers practice. In this case, the authentic task was to co-design and enact a technology-enhanced learning unit following the PROFILES three key components (student motivation, inquiry learning, and decision-making). At the same time, we examined ways to increase teacher ownership of the professional development process. This poster presents some preliminary results and discusses challenges and implications of this effort.
Methodology
Context, Participants, and Tools
The work reported in this paper is situated within a four year European project (PROFILES). The aim of the project is to support science teachers in adopting inquiry-based learning by being involved in active teacher networks. In the local implementation of the effort teachers were asked to collaboratively design an inquirybased teaching module that integrated new technologies. Thirty-eight (n=38) in-service science teachers at one of the PROFILES countries were involved in this study, forming five online communities organized around their specialization (Primary Education, Chemistry, Upper & Lower Physics, Biology). The Moodle online platform was used to support the online activity of these five communities and was customized to scaffold teachers as designers and learners. Taking into account that inquiry is a complex teaching approach that is difficult to be framed in specific learning and teaching sequences, a scaffolding scheme, presented in the form of design questions, was incorporated into the platform to guide the teachers design process. Following a hybrid model for the delivery of professional development, several face to face meetings were also scheduled which focused on discussing the content of teachers design, and the progress of the subsequent classroom enactments of the designed curricula.
Data collection
Data were collected via a questionnaire, semi-structured telephone interviews from the 38 participating teachers, and an analysis of teachers participation in the online environment created on Moodle. A Teachers Needs Questionnaire, administered at the beginning of the project, aimed to assess (a) teachers confidence in science teaching and (b) teachers professional development (PD) needs on ten different dimensions (namely: Nature of Science, Scientific and Technological Literacy, Inquiry-based Science Education, Classroom Learning Environment, Student Motivation, Assessment, Learning Theories, Technology Integration, New Curricula, Self-Reflection). The internal consistency of the questionnaire items was assessed using Cronbachs alpha. Telephone interviews with all participants two weeks after the project started focused on
ISLS
533
ICLS2012
Volume 2: Posters
reported obstacles for online participation; teachers participation on Moodle and in the local teacher network was also examined.
Results
The questionnaire subscale mean scores for teacher confidence and teacher professional development needs were above the midpoint of the 5-point response scale. This suggests that while, overall, the participants were confident in their abilities to teach science they also indicated that they could benefit from professional development based on the 10 dimensions assessed by the instrument. Compared to their confidence levels for each dimension, professional development was desired in all dimensions, with the exception of the Nature of Science dimension for which teachers felt they needed less support. Results pointed to inquiry-based science education, reformed curricula, and technology integration as the top three areas of need for additional professional development. These results contributed directly to the design of the continuous professional development program, and confirmed the initial ideas behind the offering of these professional development opportunities. The analysis of teachers contributions to the online community indicated a slow and reluctant acceptance of the virtual space as a space for collaborative design. Not all teachers equally engaged in online discussions about the authentic design task. The phone interviews queried the extent of teachers involvement in the online communities created to foster the development of a community of practice. The analysis of the phone interviews (n=38) provided insights into the obstacles individual teachers faced and guided the next steps in organizing appropriate support for professional development, with the goal of promoting teacher ownership of the professional development process. More specifically, according to the phone interviews, four aspects of challenges to participating in the online community were identified: lack of basic familiarity with the technology used, teachers busy professional, after-school schedule, insufficient leadership within each of the groups formed, and insecurity about inquiry skills and knowledge which influenced their motivation to participate in the online community. These results concur with results reported elsewhere in the literature (e.g. Guldberg & Mackness, 2009). These problems were addressed through targeted actions to overcome the self-identified barriers for each individual teacher. Progress was made, and teacher teams gradually begun to use the online community for exchanging design suggestions and discussing their experiences from classroom enactment. However, the online community was mostly used as a peripheral but supportive mechanism that helped hold together the design process, spread over the busy lives of the participating teachers. Building on existing local networks, which were primarily reinforced through synchronous, face to face meetings, some of which through a web-based video conferencing system, the online teacher communities served a role that was identified by Schlager and Fusco (2003), that of supporting a developing local community of practice. These results reinforce the understanding of professional development as a process of developing local capacity, and suggest that tools should be used to provide targeted, individualized support for teachers to participate in the developing communities of practice at both the online and offline level (Lock, 2006).
References
Desimone, L. M. (2009). Improving impact studies of teachers' professional development: Toward better conceptualizations and measures. Educational Researcher, 38(3), 181-199. Guldberg, K. & Mackness, J. (2009). Foundations of communities of practice: enablers and barriers to participation. Journal of Computer Assisted Learning, 25(6), 528-538. Lock, J. V. (2006). A new image: online communities to facilitate teacher professional development. Journal of Technology and Teacher Education, 14(4), 663-678. Riverin, S. & Stacey, E. (2008). Sustaining an online community of practice: A case study. Journal of Distance Education, 22(2), 43-58. Schlager, M. S., & Fusco, J. (2003). Teacher Professional Development, Technology, and Communities of Practice: Are We Putting the Cart Before the Horse? The Information Society, 19(3), 203-220. doi: 10.1080/01972240309464
Acknowledgements
This work was funded by the Science in Society Initiative of the Seventh Framework Research Programme (FP7) of the European Community, through a grant to project PROFILES (266589). The opinions expressed as solely of the authors.
ISLS
534
ICLS2012
Volume 2: Posters
Reasoning About Mechanism: Childrens Explanations of Pop-ups

Mayumi Shinohara, Rob Rouse, Vanderbilt University, 230 Appleton Place, Nashville, TN 37203-5721 Email: mayumi.shinohara@vanderbilt.edu Abstract: In order to investigate how children reason about mechanism, we presented 31 children from two third-grade classrooms with the task of constructing a pop-up book. Here we report the results of a qualitative analysis of 29 childrens semi-structured interviews in which they answered questions about their pop-up books post-construction. We found that nearly all children exhibited seeds of mechanistic reasoning, and that their attention to pop-up mechanisms varied.
Significance
Mechanistic reasoning is a core disciplinary practice of both science and engineering. It involves both relating causes with effects and describing the processes responsible for those relations. By focusing on the processes that underlie cause-and-effect relationships, mechanistic explanations take into account (a) the components of mechanisms, (b) the properties and organization of those components, and (c) the activities through which those components affect one another, producing causal, explanatory chains that span from input to output (Machamer, Darden, & Craver, 2000; Russ, Scherr, Hammer, & Mikeska, 2008). This study uses Russ et al.s framework for mechanistic reasoning to investigate mechanistic elements of student thinking in a new system: student-designed pop-ups. Children are often highly motivated to learn about pop-up mechanisms. In addition, the act of designing, rather than simply operating, pop-ups may provide important supports for the development of mechanistic reasoning.
Research Questions
We set out to answer three questions: (a) Do children engage in mechanistic reasoning about pop-ups? (b) If so, in what ways and under what circumstances? and (c) What aspects of mechanism do they attend to and associate in their mechanistic accounts?
Method
Participants in our study were 31 children (12 boys and 19 girls) from two third-grade classes in a suburban elementary school in the Midwestern United States. Twelve of the 31 children were classified as ELLs.
Instructional Design
We worked with 3 teachers (two classroom teachers and a student teacher) to plan pop-up lessons, based on a 12-part pop-up curriculum. In order to connect pop-ups to their required curriculum, the teachers combined popups with two other units: researching animals of the world and writing poems. This resulted in a hybrid instructional design in which children first researched animals (e.g., lions, tigers, anacondas, snowy owls), then learned about and wrote different types of poems (e.g., acrostic, diamante, haiku, persona), and, finally, learned to make several types of pop-up mechanisms. Each component of the instructional design was linked to a central project, in which children created their own animals of the world pop-up poetry books.
Data Collection
After instruction, 29 of the 31 children were interviewed individually. The interviews (all videotaped) were semi-structured and designed to elicit information about how children constructed their pop-up book (e.g., Tell me about your pop-up, How does your pop-up work? and How did you make that pop-up?). In addition, pictures were taken of each page of every pop-up book the children made. On average, each book had five pages, each featuring a different animal and comprised of a poem, pop-up, and illustration.
Analysis
We began by watching and transcribing each childs interview. We identified instances of mechanistic talk within each interview (in total there were 133). Mechanistic talk was defined as any place where children noticed, described, or explained how their pop-up worked. To develop an analytic framework to characterize childrens mechanistic reasoning about pop-ups, we performed an inductive analysis (Thomas, 2006); emerging themes were developed by studying transcripts and video of instances of mechanistic talk iteratively and in detail. We classified childrens responses along two dimensions (aspect of mechanism and type of explanation), each with multiple thematic categories. Aspects of mechanism included components (e.g., center fold, attached piece of the pop-up), activities (e.g., opening or closing the book and horizontal, vertical, or
ISLS
535
ICLS2012
Volume 2: Posters
diagonal motion of the pop-up), or properties of the pop-up system (e.g., symmetry and height of the opened pop-up). Types of explanation included mechanistic attention without connection (child attends to, but does not associate, mechanistic aspects of the pop-up system); empirical regularities (child attends to mechanistic aspects and explains that two mechanistic aspects co-occurred); and mechanistic connections (child attends to mechanistic aspects and explains how one aspect of a mechanism causes another). To investigate the frequency and circumstances under which children engaged in different types of mechanistic explanation, as well as differences in attention to aspects of mechanism across types of explanation, we looked for patterns in coded responses across dimensions and categories.
Findings
Nearly every student explanation (92%) exhibited seeds of mechanistic reasoning. Moreover, many children associated the output motion of their pop-up with structural properties of the pop-up system (i.e., points of attachment or distance from the central fold), not just with the input activity of opening or closing the book. However, not all explanations were equally mechanistic. Children used multiple forms of mechanistic reasoning, which differed in degree and type of connection (i.e., no mechanistic connections, empirical regularities, or mechanistic connections) as well as attention to aspects of mechanism (i.e., particular components, activities, or properties). (See Table 1.) Explanation type varied by child, but not by type of pop-up or animal being depicted. Table 1: Children used multiple forms of mechanistic reasoning. Type of reasoning No attention to mechanism Mechanistic attention without connection Empirical regularities Mechanistic connections Description Did not attend to mechanistic aspects of pop-up Attended to, but did not associate, aspects of mechanism Explained that two mechanistic aspects cooccurred Explained how one mechanistic aspect caused another Example I made this one [snake] like in the grass and I didn't just want the grass really green so I added a little brown. This is my jaguar. It's head is popping out and I like it because it looks like it's running after it's food. When you close it, it folds over to the other side and when you open it, it comes back to the spot where its at. If you put it out farther, it doesnt go out as much. % of student explanations 8%
22%
17%
53%
Moreover, attention to mechanistic aspects varied with type of explanation. Student attention to mechanistic activities narrowed across types of explanation, whereas attention to properties of components and activities grew, and student attention to mechanistic components shifted in a more nonlinear fashion across types of explanation.
Implications
Student-designed pop-ups may be a fruitful context for developing the seeds of mechanistic reasoning. At the same time, it may be important to: (a) systematically build from design activities and discussion to the creation and testing of mechanistic conjectures and verbal and written systems for representing those conjectures; (b) support attention to less visible aspects of mechanism (e.g., position and shape of the attachment and two or three-dimensional motion of the pop-up) and the association of inputs, outputs, and intermediate components, activities, and properties of mechanisms; and (c) focus initial discussions on noticing pop-up activities, components, and properties, then begin to ask questions that call for consideration of longer and longer chains of association among those noticings.
References
Machamer, P., Darden, L., & Craver, C. (2000). Thinking about mechanism. Philosophy of Science, 67(1), 1-25. Russ, R.S., Scherr, R.E., Hammer, D, & Mikeska, J. (2008). Recognizing mechanistic reasoning in student scientific inquiry: A framework for discourse analysis developed from philosophy of science. Science Education, 92(3), 499-524. Thomas, D. R. (2006). A general inductive approach for qualitative data analysis. American Journal of Evaluation, 27(2), 237-246.
ISLS
536
ICLS2012
Volume 2: Posters
Resources for Reasoning: Use of Video and Text in a CSCL Environment

Cindy E. Hmelo-Silver, Carolyn A. Maher, Marjory F. Palius, Robert Sigley Rutgers University, 10 Seminary Place, New Brunswick, NJ USA cindy.hmelo-sllver@gse.rutgers.edu,carolyn.maher@gse.rutgers.edu, marjory.palius@gse.rutgers.edu, robert.sigley@gse.rutgers.edu Abstract: Video Mosaic Collaborative is a multifaceted research and development project, for which we have developed a repository containing research video featuring a longitudinal study of development of mathematical thinking. We developed a hybrid course using video resources for graduate students to study childrens development of mathematical reasoning, which enables us to examine how groups of students used video and text in their online discussions. We examine here how video and text were used in collaborative discussions. To trace how use of Video Mosaic resources affects how graduate students attend to childrens learning, we analyzed online discourse among groups of students. The graduate mathematics education hybrid course was conducted during a 15-week semester and used the eCollege course management system. There was minimal instructor participation in the online discussions. For the online component, questions were posed to trigger initial discussion and direct students to assigned video. There were 25 students enrolled in the introductory graduate course. Four online groups were formed of 6-7 students. Data for this study consist of postings from online threaded discussions collected in a two and a half week course unit. The unit consisted of face-to-face and online activities that had three components: (1) in class problem solving of the same task that students in the video were solving; (2) access to video; and (3) related readings. The problem-solving task is called Ankurs Challenge (named after the 10th grade boy who came up with the problem during his participation in the Rutgers-Kenilworth longitudinal study). The statement of the problem is: How many different block towers can be built, four tall, selecting from three colors of blocks such that the towers have at least one block of each color? After collaborative problem solving in the classroom, students engaged in asynchronous discussion where the only instructional guidance came from the introductory text to the course unit that gave students their assignment and questions to prompt conversation. They were assigned one video (http://hdl.rutgers.edu/1782.1/rucore00000001201.Video.000062055) and two readings (Maher & Muter, 2010; Yackel & Hanna, 2003). The video showed two groups of 10th graders working on the problem. It primarily focuses on the group with Romina, who shares their solution on the board and later in writing. One of the readings (Maher & Muter, 2010) included a figure with Rominas final written solution to the Ankurs Challenge task, which is focal data for the first discussion question in the graduate students posted assignment that asked them to consider Rominas solution, consider whether it was a convincing argument or not, and to relate that to constructivist theories of mathematics learning.
Methods
To study how students used video resources in the hybrid course, the following questions guided the analysis: (1) To what extent do videos and readings promote online discussion within and across groups? (2) How, if at all, do learners relate practice to online discussion? (3) What evidence is there that that we have challenged teachers beliefs about what children are capable of doing regarding math? and, (4) To what extent do students relate videos to readings in their online discussions? All posts were coded for references made to video and readings. Comments coded for Video (V) were direct references, such as direct quotes and summaries, or how the graduate students reacted to what they saw in the video. Comments coded for Readings (R) were references, such as a direct quote or summarized portion, to specific authors or the whole reading. Within these two categories, comments were coded for sub-categories relating videos and/or reading to classroom problem solving or to one another. These include comparison of their own problem solving to the video (PV), comparison of their problem solving to what they read (PR); comparison of others students problem with the video (OV), later combined with comparison of their own problem solving to what they read (OR); comparison of earlier events by relating an activity to what they saw in the video (EV), relating an earlier event to what they read (ER); expression of positive affect (e.g., being impressed, surprised) about childrens ability to do and learn mathematics by what they saw in a video (AV), expression of positive affect about what they read (AR); and relating specific teaching and learning experiences
ISLS
537
ICLS2012
Volume 2: Posters
from their classroom practice to what they saw on the video (TV), relating their own classroom practice to what they read (TR). The coding for affect about childrens ability to learn and do mathematics is noteworthy because the students made comments that were explicit or readily interpreted as challenging previously held beliefs.
Results
Our analysis focuses on how students connected participation in course activities and problem solving to the video and text resources, as well as directly relating video to texts. Table 1 shows that there were substantial percentages of postings in which students related the video and text resources to their own and others problems solving, with somewhat more of the former than the latter. As Table 1 shows, the students used a range of resources in their reasoning, much of which connected to the video. While some students made reference to class activities earlier in the semester, this was the lowest percentage of commentary in discussion thread postings. Groups A and D made fewer references to earlier interventions and to practice than the other groups. Table 1: Connection to video and reading Students Relate Videos and Readings To: Own problem solving (PV/PR) Others Problem Solving (OV/OR) Earlier interventions in the class (EV/ER) Positive affective reaction (AV/AR) Teaching Practice (TV/TR) Total No. Posts Percent of Student Postings by Group A B C D Percent: Total - All Groups 36.1 18.0 14.8 31.1 39.3 61 95% CI: Percent - All Groups 24.0 48.1 8.4 27.7 5.8 23.7 19.5 42.8 27.1 51.6
27.8 5.6 11.1 38.9 33.3 18
44.4 5.6 11.1 27.8 38.9 18
27.3 27.3 18.2 18.2 36.4 11
42.9 42.9 21.4 35.7 50.0 14
Additional qualitative analysis showed several themes organized around making connections to one or more of the resources, as well as to personal experiences with the activities of problem solving and teaching practice. They demonstrate that the video afforded ways for students to engage in reflection.
Discussion
The study demonstrates that video is an effective catalyst to spur threaded discussions in an online graduate course environment. With opportunity to draw from a combination of problem solving, video, and course reading activities, learners were able to engage in productive discussions utilizing a range of resources. The results reported here are suggestive of the richness of online group discussions when carefully selected multimedia resources are used in conjunction with face-to-face mathematical problem solving; however, there is much about the process still to understand. We need to more deeply analyze the affordances of different media for productive discussions and how these discussions unfold over time. There was also considerable variability among the groups. Further analysis will need to examine those differences and factors related to more or less productive discussions.
References
Maher, C. A. & Muter, E. (2010). Responding to Ankurs Challenge: Co-construction of Argument Leading to Proof. In C. A. Maher, A. B. Powell, & E. B. Uptegrove (Eds.), Combinatorics and Reasoning: Representing, Justifying, and Building Isomorphisms (pp. 89-96). Springer: New York. Yackel, E. & Hanna, G. (2003). Reasoning and proof. In J. Kilpatrick, G. W. Martin, and D. Schifter, (Eds.), A Research Companion to Principles and Standards for School Mathematics (pp. 227-236). Reston, VA: National Council of Teachers of Mathematics.
Acknowledgments
We gratefully acknowledge the support from the National Science Foundation (award DRL-0822204) and note that the views expressed here are those of the authors are not necessarily those of the NSF.
ISLS
538
ICLS2012
Volume 2: Posters
Weaving Together Parts to Achieve a Whole: Gestural Activity for the Coordination of Information in the Teaching and Learning of Chemistry
Stephanie Scopelitis, University of Washington, 1019 Harms Ave, Libertyville, IL 60048, scope@u.washington.edu Mike Stieff, University of Illinois-Chicago, 845 W. Taylor St. (M/C 111) Chicago, IL 60607, mstieff@uic.edu Abstract: In this study we examine the effects of how instructors and students use gestures to coordinate concept information for science learning. The findings reveal that as gesture use can be an effective instructional method, gestures as teaching and learning resources are only as valuable as they are visible. We conclude our study by encouraging new instructional designs that make use of the affordances of gestures in coordinating representations and orienting visual perspectives.
Introduction
We see learning as a dynamic process constructed through actions and interwoven relations among resources (Hutchins, 1995; Stevens & Hall, 1998). These resources include participants, settings, objects, sequences of activities, and multiple communicative modalities that go beyond talk. In our analysis we employ the idea that it is not only essential to cross resources, but it is also necessary that resources be coordinated, ordered, and highlighted with gestural activity (Goodwin, 2000; Stevens & Hall, 1998). We take the focus that gestural activity can assist in the construction and communication of scientific insights and add to this that gestural activity used to coordinate science knowledge is a fundamental practice of teaching and learning in STEM disciplines. Recently, researchers have focused on the role of gestural activity in teaching and learning across STEM disciplines (e.g., Becvar, et al., 2005; Roth & Bowen, 1999; Scopelitis, Mehus, & Stevens, 2010) to suggest that gestural activity supports concept explanation and understanding. We expand on this by examining in detail how gestures are used in chemistry teaching. Prior studies of student and expert problem solving in chemistry (e.g., Stieff, 2011) suggest that gesture plays an important role in scaffolding spatial reasoning in the domain for the individual, but little is known about how higher education chemistry instructors and students use gestures interactively in this domain. Through our study we first show how an instructor employs gestural activity in university small group tutoring sessions to coordinate and sequence layers of disciplinary information and how students use gestures for achieving understanding interactionally. We further examine how and when critical gestures are attended to by all participants.
Methods
We employ microanalysis of videotaped interactions, which involves fine-grained, qualitative analysis of sequences of talk, gesture, and action. (cf., Erikson, 1995). The recorded interactions were logged and transcribed, and the verbal and gestural techniques used to communicate concepts were identified and analyzed. Using a constant comparative approach, we engaged in repeated cycles of analysis to produce emergent themes that characterize the functional and pedagogical role of gestures in chemistry.
Results & Discussion

Our results show three distinct uses of gesture by the instructor for coordinating representations and discipline information. Our analysis of the instructor revealed how gestures are employed with the intent (1) to connect present information, (2) to bridge previously acquired information with present information, and (3) to orient a learners spatial perspective to external representations. Importantly, these uses of gesture offer critical actions that weave together layers of information and link and sequence ideas to achieve the communication of concepts. Our data further show that students also use gestures (1) to coordinate discipline information to communicate understanding, (2) to make requests for clarification, and (3) to negotiate transitioning knowledge to achieve understanding. Finally, the data demonstrate that while in some instances students are able to make use of gestural activity to coordinate and communicate critical content information, in other instances they are not. Students are challenged as their attention is pulled in multiple directions. Students simultaneously need to attend to information communicated in both instructor diagrams and gesture and information in their notebooks. This alternation of attentional orientation renders it difficult to capture all the necessary coordinating gestures performed in the process of teaching and learning.
ISLS
539
ICLS2012
Volume 2: Posters
Scholarly Significance & Implications

Gesturing for the coordination of information is especially critical in the teaching and learning of chemistry. The discipline is populated with diverse and multiple representations that individually fail to communicate an idea completely. Students need to learn how representations are related to one another and how a single representation links to others to make a whole. As a result, gesturing for chemistry understanding is not only to animate talk or enrich information. Rather, gesture can often be the crux of the information itself. Examples from the data show that an instructors gesturing is means for coordinating present information, invoking passed knowledge, and demonstrating a perceptual stance. We also show how gesturing to coordinate information is used by the students in interaction with the instructor and peers to extend understanding and think through problems together. The data show that the diverse gestural activity as means for coordinating information can be part of a teachers repertoire of intentional pedagogical practices necessary to orchestrate the myriad representations and multiple layers of information that flood the chemistry classroom. The data, however, further reflect that the critical information these coordinating gestures provide is only as valuable as it is visible. This suggests new designs for chemistry teaching and learning environments that not only make use of an instructors gesturing to coordinate information, but also gives opportunity for instructors gestural demonstrations for delivering information to be continuously accessible to learners. Certainly, by the very nature of the lecture-format classroom arrangement prominent for chemistry teaching, the available uses and functions of gestures are different compared to those present in these small group arrangements. However, answers for new designs might be found through further investigation into how teaching and learning unfolds in other arrangements such as small group sessions that provide different opportunities to engage material resources such as gestures.
References
Becvar, L. A., Hollan, J., & Hutchins, E. (2005). Hands as molecules: Representational gestures used for developing theory in a scientific laboratory. Semiotica, 156, 89-112. Erikson, F. (1995). Ethnographic microanaylsis. In N. Hornberger & S. McKay (Eds.), Sociolinguistic and language teaching (pp.283-306). Goodwin, C. (2000). Action and embodiment within situated human interaction. Journal of Pragmatics, 32 (10), 1489-1522. Hutchins, E. (1995 ). Cognition in the wild. Cambridge: MIT Press. Kendon, A. (2004). Gesture: Visible action as utterance. Cambridge: Cambridge University Press. Roth, W. M., & Bowen, G. M. (1999). Decalages in talk and gesture: Visual and verbal semiotics of ecology lectures. Linguistics & Education, 10, 335-358. Scopelitis, S., Mehus, S., & Stevens, R. (2010.) Made by Hand: Gestural Practices for the building of complex concepts in face-to-face, one-on-one learning arrangements. Paper presented at the International Conference of the Learning Science 2010, Chicago, IL. Stevens, R. & Hall, R. (1998). Disciplined perception: Learning to see in technoscience. In Magdalene, L. & Blunk, M. L., (Eds.), Talking mathematics in school: Studies of teaching and learning. Cambridge, MA: Cambridge University Press. Stieff, M. (2011). When is a molecule three-dimensional? A task-specific role for imagistic reasoning in advanced chemistry. Science Education, 95(2), 310-336. Streeck, J. (1994). Gesture as communication II: The audience as co-author. Research on Language and Social Interaction, 27(3), 239267.
Acknowledgments
The research reported here was supported by National Science Foundation (REESE #0723312). The opinions expressed are those of the authors and do not represent the views of these agencies.
ISLS
540
ICLS2012
Volume 2: Posters
Participatory Learning and Assessment in e-Learning Contexts

Daniel T. Hickey, Andrea M. Strackeljahn, & Rebecca C. Itow, Learning Sciences Program, Indiana University dthickey@indiana.edu, amstrack@indiana.edu, rcitow@indiana.edu, Abstract: Expository e-learning efficiently supports coverage and accountability, but is shallow and unsuited for inquiry & participation; constructivist e-learning emphasizes choice and projects but is uneven and exhausting. This presentation introduces a comprehensive design model that combines the best of both approaches while exploiting and preparing for new digitally networked knowledge contexts. Four general e-learning design principles are presented, along with specific features and evidence from an online graduate-level education course on cognition and learning. The explosion of e-learning has highlighted enduring tensions between expository and constructivist methods. Expository instruction relies on highly structured content and frequent tests (Clark & Mayer, 2007). Tight structure, broad coverage, and regular tests mean that expository courses are easy to teach and well-suited to traditional assessment and evaluation. But many argue that expository instruction is too mechanistic and focused on basic skills (Hmelo-Silver, et al. 2007). Conversely, constructivist approaches generally view knowledge as higher order schema (Bransford et al., 2001) and assume that meaningful learning involves individual problem solving and personalized inquiry, along with social-constructivist features such as discussion forums and wikis. But they require instructors who are knowledgeable about the domain, teaching the domain, and e-learning technology. The idiosyncratic nature of constructivist learning calls for open-ended and portfolio assessment methods, which clash with institutional accountability goals. While efforts are ongoing to compare these two methods, the underlying philosophical and epistemological differences confound direct comparison. More insidiously, combining them leads to paradoxical practices like high-stakes traditional exams in constructivist courses and mandated discussion forums for factual content. More importantly, neither of these approaches is particularly well suited to the way knowledge is likely to be used and learned in digitally networked environments (Brown & Adler, 2008).
Participatory Assessment
Interest-driven social networking inspires a very different model of e-learning (Ito, 2009). Low barriers to expression & engagement, strong support for creating & sharing, informal mentorship, and strong social connections foster participatory cultures (Jenkins, 2009). Connectivism (Siemens, 2005) captures this educational potential by emphasizing diversity of opinions, connections across networks of networks, the value of learning over knowledge, and the value of current knowledge. Connectivism is consistent with situative theories of learning (e.g., Greeno, 1998) that focus on the social and technological contexts of learning. Situative theories characterize learning as interactive participation in social and technological practices (rather than individual change). This allows them to treat individual learning as special cases of social change. Situative theories of assessment (Hickey & Anderson, 2007) push very hard on this distinction to uncover new strategies for obtaining broad and otherwise antithetical learning outcomes. The four general design principles that currently define Participatory Assessment have emerged across a decade of design-based studies in a range of innovative instructional contexts. These include the Concord Consortiums GenScope genetics software, three inquiry-oriented multimedia programs in NASAs Classroom of the Future, Indiana Universitys Quest Atlantis multi-user virtual environment, Project New Media Literacies Teacher Strategy Guide, and two graduate level e-learning courses. This presentation introduces the four general design principles that make up Participatory Assessment, along with the specific principles and specific e-learning features that emerged after four semester-long design cycles of Learning and Cognition in Education. This course serves a challenging mix of working educators, future administrators, and experienced doctoral students, ranging from anxious e-learning newcomers to techsavvy students in online degree programs. This course was completed using the Sakai open-source course management system. In addition to evidence of shared participation, individual understanding, and aggregated achievement, this presentation shows how this approach accomplished two ambitious goals that had previously been tackled in a more advanced doctoral-level course. The first challenge was helping students appreciate the difference between assumptions about knowing & learning and corresponding methods for teaching, motivating, and assessing. Most students initially conflate learning and teaching, often referencing very different methods without recognizing underlying conflicts. The second challenge concerned the nuances of scholarly referencing.
ISLS
541
ICLS2012
Volume 2: Posters
General and Specific Design Principles

The primary results from design-based research are the design principles and features that emerged from iterative refinements. Following are the four general principles that make up Participatory Assessment; the presentation shows examples and evidence associated with each principle as it was transformed into a more specific principle and specific e-learning features.
1. Focus Primarily on Contextual Knowledge

This means transform domain-specific skills and procedures into procedural and conceptual tools so that learning can be reframed as practicing using those tools appropriately in particular contexts. This principle builds on the assumption that this contextual knowledge is (a) easily conceptualized and shared, (b) essential for helping struggling (i.e., less experienced) learners engage in discourse and practice using these tools, and (c) crucial to ensure that the residual knowledge readily transfers to new contexts. In this e-learning context, each student first defines a personally-relevant instructional goal. The class is then organized into professional networking groups associated with one of the five primary domains in education (literacy, comprehension, writing, math, or science). Learning is organized around weekly wikifolios and groupwikis.
2. Scaffold Participation in Context x Concept Discourse

This means structuring assignments so that students discuss how the targeted concepts and skills take on different meaning in different contexts. Most of the assignments involve articulating and discussing the relative relevance of the main ideas in each chapter from the perspective of each students instructional goal and their educational role (teacher, administrator, or scholar). Most discourse occurs via threaded comments posted directly to the wikis within and across networking groups. Instructor efficiency is established by providing extensive feedback on early posts, and then directing other students to those interactions as appropriate.
3. Assess Reflections Rather than Artifacts

This principle is intended to keep agency with the learner and avoid students expecting overly specific guidelines or unsustainable levels of individualized feedback. In this e-learning context, each student includes three reflections directly on their weekly wikifolios (or posts individual reflections on the groupwiki): consequential engagement concerns the consequences of the ideas presented that week for practice; critical engagement concerns the way each students context gave meaning to the course concepts; collaborative engagement concerns how comments from ones classmates and the instructor shaped understanding.
4. Assess Understanding Prudently and Test Achievement Discreetly

This means using assessments and tests primarily to assess the effectiveness of the prior instruction and not to directly drive learning and instruction. Overall understanding of targeted concepts is summatively assessed with essay items on an online midterm and final exam. Overall knowledge of the domain is prudently evaluated with a timed online exam using multiple-choice items randomly selected from the textbook item bank. But the exams only count for 20% of the grade and do not drive the curriculum or instruction. Bransford, J. D., Brown, A., & Cocking, R. (1999). How people learn: Mind, brain, experience and school. Washington, DC: National Research Council. Brown, J. S., & Adler, R. P. (2008). Minds on fire: Open education, the long tail, and learning 2.0. Educause Review, 43(1), 16. Clark, R. C., & Mayer, R. E. (2007). E-learning and the science of instruction: Proven guidelines for consumers and designers of multimedia learning. San Francisco, CA: Pfeiffer & Co. Greeno, J. G. (1998). The situativity of knowing, learning, and research. American Psychologist, 53(1), 5-26. Hickey, D. T., & Anderson, K. T. (2007). Situative approaches to student assessment: Contextualizing evidence to support practice. Yearbook of the National Society for the Study of Education: Evidence and Decision Making, 106, 264287. Hmelo-Silver, C. E., Duncan, R. G., & Chinn, C. A. (2007). Scaffolding and achievement in problem-based and inquiry learning: A response to Kirschner, Sweller, and Clark (2006). Educational Psychologist, 42(2), 99107. Ito, M. (2008). Hanging out, messing around and geeking out: Living and learning with new media. Cambridge, MA: MIT Press. Jenkins, H. (2009). Confronting the challenges of participatory culture: Media education for the 21st century. Cambridge, MA: MIT Press. Siemens, G. (2005). Connectivism: A learning theory for the digital age. International Journal of Instructional Technology and Distance Learning, 2(1), 310
ISLS
542
ICLS2012
Volume 2: Posters
Irreducible Complexity: How Do Causal Bayes Nets Theories of Human Causal Inference Inform the Design of a Virtual Ecosystem?
M. Shane Tutwiler, Tina Grotzer, Amy Kamarainen, Shari Metcalf, Chris Dede, Graduate School of Education, Harvard University, Cambridge MA Email: Michael_Tutwiler@mail.harvard.edu, tina_grotzer@pz.harvard.edu, kamaraam@gmail.com, Shari_Metcalf@harvard.edu, Chris_Dede@harvard.edu Abstract: Recent computational theories on causal inference, developed by machine learning theorists and co-opted by psychologists and cognitive sciences, predict specific patterns of behavior when humans infer causal connections in simple systems. However, these theories may not be scalable to model complex causal systems, such as ecosystems. Said theories are reviewed herein, and future strands of research are suggested.
Introduction
Hannah, a middle school earth science student, was asked to set up an experiment to see if lawn fertilizer impacts algal growth, and was recording her final observations to take back to her teammates, who were each performing independent experiments of their own. She had carefully controlled for fertilizer amounts in different samples, and had a control sample with no fertilizer for comparison. Confident of her results, Hannah returned to her group to compare her findings to theirs. She became confused when her teammate, Angela, found that dissolved oxygen levels also impacted algae growth. Angelas results were not as consistent, however, so Hannah and her teammates disregarded them in favor of the fertilizer-based theory. The above vignette illustrates a common scenario: students integrating information about causal relationships in a complex setting, in this case an ecosystem. Results of different theory-based experiments are seemingly at odds, and the students eventually choose one as a dominant model. While parsimonious, this may not reflect the actual complex causal relationship within the ecosystem itself. Recent theories of human causal inference seem to support the Occams razor approach to model selection highlighted above, but how can we support complex learning assuming said theories are correct?
Background
Over the course of the last half-century, developmental psychologists and cognitive scientists have endeavored to model human causal inference quantitatively. Some models were purely based on relative frequencies of candidate events and causes (Jenkins & Ward, 1965), effectively measuring their covariation. Over time, researchers altered these equations to account for the strength of the causal connection as well (Cheng, 1997). However, both pure covariation and strength-based equations were based on explicit assumptions about the structure of the relationship between candidate causes and the observed effect. Recognizing this shortfall, Glymour (1998) re-framed Chengs (1997) strength-based theory of inference as a type of graphical analysis known as a Causal Bayes Net (CBN) with a special type of parameterization known as noisy-OR. Based on Pearls (1988) treatise on graphical causal analysis, the CBN theory of human causal inference predicts that, given the proper inputs, humans will make simple causal inferences in a normative manner. We can use the scenario in the introduction, above, as an example (Figure 1, below).
O 0
A 1
Figure 1. Causal Bayes Net of the causal effect of oxygen (O) and fertilizer (F) on Algae growth (A) Observing Figure 1, above, we note that a directed edge (arrow) goes from O to A and from F to A, indicating that both O and F are independent causes of A. In addition, we note that each causal relationship has an associated weight () that indicates the strength of the proposed connection, based on observation of covariation. In the scenario above, Hannah screened off O as a potential cause, because the strength of the relationship between F and A was stronger. This highlights two points: 1) humans may infer simple causal relationships in a manner consistent with CBNs, and 2) this inferential process may prohibit humans from identifying complex causal relationships (Grotzer & Tutwiler, in preparation). Focusing on the first point, researchers over the last decade have used CBN-based theories to model and predict human causal inference across different tasks and ages (e.gSchulz & Gopnick, 2004; Griffiths &Tenenbaum, 2005). These findings all center on simple causal relationships, however. Researchers in the field of machine learning, for which CBN methods were originally developed (Pearl, 1988, 2000), have shown that, as systems become more complex, the computations for CBNs become intractable (Bishop, 2006). So, given that the core cognitive mechanism used to infer causal connections may force people to form overly simplistic
ISLS
543
ICLS2012
Volume 2: Posters
causal models, and that the mechanism cant be scaled up directly to infer more complex models, which are more likely to occur in ecosystems (Grotzer, 2002), how can we better support student learning therein?
Information Theory
One way to help support student understanding of complex systems, assuming a CBN paradigm, is through the application of information theory. The degree of surprise (i.e. information) in learning some piece of data x is given as Equation 1 in Table 1, below. The amount of information you have to transmit in order for x to update beliefs (i.e. entropy) is given as Equation 2 in Table 1, below. In general, entropy increases as distributions become more broad and uninformative. In other words, if students are exposed to data that they already expect (or that they dont know not to expect), then it takes much more data to override a prior belief. Finally, we can also calculate the relative entropy Equation 3, Table 1 (below), or the average addition information required to transmit a value (x) if we assume a distribution, q(x), which is not exactly the same as the distribution actually generating the data, p(x). In other words, if the student has the wrong model in mind, the amount of extra information, on average, that they have to gather before they correctly discern the value of x, which can then be used to update their prior belief, is very high. In essence, the amount of information needed becomes much greater as p(x) and q(x) diverge. Equation 1 h(x) = -ln*p(x) Equation 2 Equation 3
Table 1. Equations for 1) Information, 2) Entropy, and 3) Mutual Entropy
Impact on the Design of a Virtual Ecosystem

Assuming that students will make causal inferences in ways consistent with CBN theories, we can revise the design of the EcoMUVE, an ecosystems-based Multi-User Virtual Environment. The theoretical model above predicts that students will benefit most from data that is novel, but within their perceptual limits. As such, we can first present example causal relationships that are easily discernable as a training data set. Then, as students learn to discern relevant information, we will add layers of complexity to the system, longitudinally increasing the entropy, and observe when students begin to have trouble inferring causal connections. We also hypothesize that supporting students in model building/model selection will make learning within the EcoMUVE more efficient, due to the decrease in relative entropy. To that end, we will include more guided reflection on possible underlying models of interaction within the system at the end of each day. Future research can then directly compare groups with and without modeling support.
Significance
Students have trouble treating ecosystems as complex systems (Grotzer & Basca, 2003; Hmelo-Silver, Pfeffer, Malhotra, 2003). Based on a synthesis of modern psychological theories of human causal inference and machine learning theories on information detection and processing, the theoretically motivated design strategies outlined here indicate a step toward addressing that trend through the strengths of MUVEs (Dede, 2009). In addition, future avenues of research are opened by application of these theories.
References
Bishiop, C.M., (2006). Pattern Recognition and Machine Learning. New York, NY: Springer Science+Business Media, LLC. Cheng, P.W. (1997). From covariation to causation: A causal power theory. Psychological Review, 104, 367-405. Dede, C. (2009). Immersive interfaces for engagement and learning. Science, 323(5910), 66-69. Glymour, C. (1998). Learning Causes: Psychological Explanations of Causal Explanation. Minds and Machines, 8(1998), 39-60. Griffiths, T.L. &Tenenbaum, J.B. (2005). Structure and Strength in Causal Induction. Cognitive Psychology, 51, 334-384. Grotzer, T.A. (2002). Causal patterns in ecosystems: Lessons to infuse into ecosystems units. Cambridge, MA: President and Fellows of Harvard College. Grotzer, T.A., & Basca, B.B. (2003) Helping students to grasp the underlying causal structures when learning about ecosystems: How does it impact understanding? Journal of Biological Education, 38,(1)16-29. Grotzer, T.A., & Tutwiler, M.S. (in preparation) Causal Bayes Nets: A bridge too far? Hmelo-Silver, C.E., Pfeffer, M.G., & Malhotra, B.A. (2003, April). Fish swim and rocks sit: Understanding structures, behaviors, and functions in a complex system. Paper presented at the American Educational Research Association Annual Meeting, Chicago, IL. Pearl, J. (1988). Probabilistic Reasoning in Intelligent Systems. San Francisco, CA: Morgan Kaufman Publishers, INC.
ISLS
544
ICLS2012
Volume 2: Posters
From tacit knowing to explicit explanation: Mining student designs for evidence of systems thinking
Melissa Sommerfeld Gresalfi, Leon Gordon Indiana University, 1900 E. 10th St., Eigenmann 543, Bloomington, IN 47406 mgresalf@indiana.edu, leogordo@indiana.edu Sinem Siyahhan, Arizona State University, PO Box 873701, Tempe, AZ 85287 sinem.siyahhan@asu.edu
Introduction and Framing

It has become increasingly clear that students' experiences in schools do not match the kinds of experiences they are likely to have once they have completed school. The push to support "21st century" skills stems from this mismatch, and which skills need supporting is the topic of significant conversation. In this paper, we focus on one particular aspect of 21st century skills, which involves understanding the world not as a simple set of causeand-effect experiences, but rather as a set of complex systems. Systems thinking generally refers to a way of understanding the world as a set of systems that are made up of many elements, each of which have distinct behaviors, which change and interact, giving rise to emergent behavior. The advantages to understanding the world as a set of systems are many, but a chief advantage is that systems thinking allows students to understand and interpret the world across content areas (Goldstone & Wilensky, 2008). Unfortunately, supporting students to develop systems thinking has proven to be a significant challenge. First, systems thinking ideas are difficult (Hmelo-Silver & Pfeffer, 2004) and is also often counter-intuitive (Wilensky & Resnick, 1999). Systems thinking requires students to look for myriad contributions to system behaviors as opposed to simple cause and effect. Indeed, a key concept of systems thinking involves understanding that a small change can lead to a significant outcomean idea that flies in the face of many core assumptions we have about the world. Linda Booth-Sweeny (2001) points out that most of our experiences in the world, particularly as children, are explained in terms of causality, and we have very few opportunities to experience the world as a set of systems. Despite these challenges, the advantages of supporting students to understand systems are clear. The question is how to best go about doing it. The purpose of this poster is to examine what students can learn about systems when they engage tasks that require systems thinking to support their successful completion. In contrast to most existing approaches to teaching students about systems, which involves teaching students explicitly about systems, this poster considers data from a project that sought to support students' nascent understandings of systems thinking by engaging them in experiences that would support their tacit understanding of principles of systems thinking. Specifically, we present a case-study analysis of two students engaged in a video game design unit using a platform called Gamestar Mechanic. Our goal is not to demonstrate the efficacy of the unit for supporting all students to engage in systems thinking. Instead, this study is exploratory in that it interrogates what kinds of thinking about systems might be possible in the context of designing video games.
Methods
This paper focuses on two intentionally selected cases that present a contrast between ways that students demonstrated their understanding of systems. The cases are drawn from a project that examined how students engaged systems thinking concepts across different design-based modules. This study took place in the context of a two-week summer camp in a major city in the Midwestern United States. This camp was free to all students, but was voluntary, and thus students who participated in the camp had chosen to be there. The four cases from this study were selected based on initial observations of their engagement; because part of the data collection methodology involved interviewing students, cases were initially selected based on observations of who would be talkative in those interviews. From those ten cases, four were intentionally selected for further analysis based on indicators that they had some understanding of elements of systems thinking. Students were working on designs in the context of a module that focused on how to design videogames as an explicit focus; a secondary focus of the unit involved thinking of games as systems, and considering how to design well-functioning systems. Language of game design was overlapped with language of systems thinking. Students were thus introduced to the idea of systems thinking in the following terms: systems are made of up elements, which have particular behaviors, whose interactions shape and change the behavior of those elements and the resulting emergent dynamics of that system. Two methods of coding were leveraged for this analysis. First, videotapes and designed games were reviewed using an emergent coding approach. This initial coding phase was concerned with characterizing students thinking about their games, particularly with respect to the key ideas about systems that were covered in instruction. This process resulted in coding categories that documented: the number of components included in a design; the number of interactions among components included in the design, and evidence of intentionality
ISLS
545
ICLS2012
Volume 2: Posters
(with respect to the design of interconnections among components). We then examined the existing literature to consider what other indicators of systems thinking had been developed. We leveraged the coding scheme from Jacobsen (2001) that included eight categories, four of which were reflected in our data.
Results
The results of our coding is presented in two parts: what can be said about students tacit understanding based on the games they designed, and what can be said about their explicit understanding based on interviews conducted with the students about their games.
Tacit understanding
Students designed games reveal quite a bit about what they (at least tacitly) understand about systems, and looking comparatively across the two games helps to highlight particularly significant differences. In the case of Levon, his game included seven elements. The system interactions (i.e. the elements whose behaviors interact) are generally relatively simple; there are four interactions and they are relatively simple cause and effect. There is evidence of intentionality with respect to designing for interaction; for example, all prey generators are located in the corners, while all predator generators are included in the middle of the space. Based on this game, we could conclude that this student appears to have at least a tacit understanding of fundamental elements of systems thinking, although there isnt a sense of more sophisticated understandings about the ways that systems function in general. In contrast, Waylens game suggests an understanding of more systems thinking concepts. His game includes more elements (ten), although the number of elements alone is not indicative of conceptual understanding. In addition, Waylens game has almost twice the amount of system interconnection between elements, several of which represent fairly complex relationships (i.e.- shows levels of interconnection responsible for emergent outcomes for meeting the games goals (i.e.- prey & time limit), energy, health). Taken together, this suggests that Waylen has some kind of understanding that in a system, control is decentralized and based on system interactions. There is no simple way to win Waylens game; instead, myriad criteria need to be met in order for the game to be complete.
Explicit Understanding
In contrast to what students designed games suggest about what the students understand about systems, interviews with the students, which required that they explicitly discuss their reasoning, were quite different. While Waylen struggled to describe his thinking in the context of designing his game (discussing only the simplest of interconnections between elements, which did not reflect the complexity of the game), Levons interview revealed more understanding of systems than was suggested from his game. Specifically, in Levons interview he described how his design worked in ways that indicated that he understood that outcomes have multiple causes, rather than single, when he described how particular interactions among components would change behaviors in the game.
Conclusions
In this poster we examined what students can demonstrate they know when they engage tasks that require systems thinking to support their successful completion. We considered differences between what students can explicitly state about what they know about systems, as well as what they can demonstrate that they know tacitly through their designs. This is exploratory work that seeks to understand what students systems thinking might look like in this dramatically different context. However, even in this exploratory phase, two primary conclusions are clear. First, the activity of designing games can indeed support students to develop some sophisticated intuitions about systems, and second, that gaining insight into those intuitions, from the perspective of an analyst, is not a trivial task. Indeed, differences between what students can show that they know through their designs, versus what they can say that they know through interviews, suggest that assessment practices must be broad in order to capture a full picture of student thinking.
References
Booth Sweeny, L. (2001). When A Butterfly Sneezes. Waltham, MA: Pegasus Communications. Goldstone, R. L., & Wilensky, U. (2008). Promoting Transfer through Complex Systems Principles. Journal of the Learning Sciences, 17, 465-516. Hmelo-Silver, C. E., & Pfeffer, M. G. (2004). Comparing expert and novice understanding of a complex system from the perspective of structures, behaviors, and functions. Cognitive Science, 1, 127-138. Jacobsen, M. J. (2001). Problem solving, cognition and complex systems: Differences between experts and novices. Complexity, 6(3), 41-49. Wilensky, U. & Resnick, M. (1999). Thinking in levels: A dynamic systems perspective to making sense of the world. Journal of Science and Technology, 8(1), 3-19.
ISLS
546
ICLS2012
Volume 2: Posters
The Authority of Ideas: How Students Become Influential in Linguistically Heterogeneous Small Group Discussions
Jennifer M. Langer-Osuna, University of Miami, 5202 University Ave, Coral Gables, FL, USA, jlangerosuna@miami.edu Abstract: The objective of this poster is to explain how students in a linguistically heterogeneous classroom became influential during small group mathematical discussions. In particular, the study focuses on the role of the teacher on: (1) the perceived merit of students contributions; and students (2) position of intellectual authority; (3) access to the conversational floor; and (4) access to interactional space. This analysis extends our understanding of equitable classrooms to consider interactional dimensions of student influence. Research on collaborative group work has shown that when students are engaged in peer-to-peer discussions, the influence of students contributions often falls prey to issues of social domination (Anderson et al., 1997; Hogan, Nastasi, & Pressley, 2000). These student dynamics can be particularly acute in diverse classrooms that include students from marginalized communities, such as language learners (Moschkovich, 2010. This poster reports on a study that focused on the role of the teacher, who walks from group-to-group for short periods of interaction, in determining whose ideas become influential during collaborative group work. In particular it utilizes a promising framework to examine how students ideas are attended to, evaluated, and taken up by others during collaborative work and then focuses on the role of the teacher in how students positions of authority and mathematical ideas are negotiated during group work.
Theoretical and Methodological Approaches

In order to investigate the role of the teacher on how student ideas are attended to and taken up by others during group work, I draw on a framework I helped develop for modeling how student contributions to group discussions become more or less influential (Engle, Langer-Osuna, & McKinney de Royston, 2012). The framework is built on literatures about persuasion, argumentation, discourse, and classroom discussions to propose that each students level of influence in a discussion emerges out of the social negotiation of influence itself and the following four factors that interact with it: (1) the perceived merit of each students contributions; and each students (2) position of intellectual authority; (3) access to the conversational floor; and (4) access to interactional space. Data analyzed in this study was derived from a video record of one lesson in a fifth grade classroom in which a high proportion of English language learners (ELLs) are served. A linguistically heterogeneous student dyad, Ana and Jerome, was selected for analysis (one ELL and one English proficient). The teacher was participating in a professional development study focused on instructional practices that supported equitable, discussion-based mathematics classroom communities for linguistically heterogeneous classrooms. On the focal day, the teacher attempted to implement two new instructional practices: utilizing an open-ended mathematics problem and drawing on a greater variety of student answers as part of class-wide sense-making activity. Both practices were implemented with some difficulty on the focal day. In order to analyze student group interactions, the transcript of the video record of the focal lesson was uploaded to Atlas ti software, and words and actions relevant to each of the five types of components in the Influence Framework (which were operationalized in Engle, Langer-Osuna & McKinney de Royston, 2012; see Table 1) were coded: (a) signifying potential changes in Influence by agreement, uptake, and/or building on the ideas of a particular student; (b) evaluating the Perceived Merit of a conjecture; (c) the perceived Authority of the speaker; (d) management of the Conversational Floor, and (e) management of Interactional Space. The Hypertext function of the software was used to link components where a shift in one (e.g., uptake of a peers ideas toward the solution path) was clearly subsequent to a shift in another (e.g., a positively perceived evaluation of the ideas merit). These connected episodes were analyzed to explain key moments of student influence (Barron & Engle, 2007; Jordan & Henderson, 1995; Ochs, 1979). See Table 1 for definitions of each component. Table 1. Definitions for each component of the proposed framework, operationalized in Engle, Langer-Osuna & McKinney de Royston (2012) Framework component Influence Definition The degree to which the student is socially positioned by group members
ISLS
547
ICLS2012
Volume 2: Posters
as having swayed other students understanding or solution path Perceived Merit of a students The degree to which the students argument is socially positioned by group conjectures members as being of high quality, whether or not this corresponds with normative standards of quality Authority The degree to which the student is evaluated, acts, or is treated as a credible source of information by group members Access to the Conversational The degree to which the student can initiate turns when desired, complete Floor them without interruption, and control who else has access to the floor during group work Access to Interactional Space The degree to which the student is visually attended to and physically oriented to by group members when speaking or listening, and is able to affect the spatial access of others.
Findings: The Role of the Teacher in Unilaterally Affecting Student Influence

Our Influence Framework shows promise for investigating the forms of participation and the positionings of ELLs during small group work, and in particular highlights the role of the teacher in affecting unilaterally access to the interactional space and conversational floor, positions of authority, and the perceived merit of the Anas (ELL) and Jeromes contributions. Ultimately, Ana was positioned with authority and garnered considerable influence in directing the dyads solution path. What is particularly noteworthy about this case is that, as the poster will show in greater detail, Anas ideas werent necessarily normatively meritorious. Rather, the teacher, in an attempt to draw on a variety of approaches to the mathematical problem, inadvertently positioned problematic ideas as seemingly meritorious. The case poignantly reveals the power of the teacher in affecting whose ideas are attended to and how they are positioned by group members, even when the ideas are nonsensical. The poster will illustrate in greater detail the analysis of the dyads solution path, summarized here in the following analytical narrative: At the start of activity, Ana grabs scissors, glue, poster, and paper and begins to cut and glue mathematics word problem onto poster. Jerome waits. Teacher publicly positions Jerome as off-task 3 times, and needing to focus on Ana, positioning her as central to the activity. Ana takes up positioning and begins to issue directives to Jerome, increasing her authority and control of the floor. Ana gains teachers attention by successfully bidding for floor. Teacher positions Anas drawing as meritorious. Anas authority and influence increase while demoting Jeromes authority. Ana issues directives to Jerome to implement her ideas, which he does. Teacher publicly positions Ana as having good ideas twice, and directly afterwards Ana claims to have found the solution. Ana gains the floor and explains solution to teacher, who accepts answer. Ana revises solution and Jerome immediately takes up her revised solution. Ana gains the floor and explains solution to teacher, who asks Ana to present her solution to the class, increasing Anas authority.
References
Anderson, C., Holland, D. & Palincsar, A. (1997). Canonical and sociocultural approaches to research and reform in science education: The story of Juan and his group. The Elementary School Journal, 97(4), 359-383. Barron, B. & Engle, R. A. (2007). Analyzing data derived from video records. In S. J. Derry (Ed.), Guidelines for video research in education: Recommendations from an expert panel (pp. 24-43, 79-80). Chicago: Data Research Development Center. Available at: http://drdc.uchicago.edu/what/video-research.html Engle, R.A., Langer-Osuna, J.M. & McKinney de Royston, M. (2012). Explaining differential student influence in a classroom science discussion: the role of quality, authority, and access to the conversational floor. Under Review. Hogan, K. Nastasi, B. K. & Pressley, M. (2000). Discourse patterns and collaborative scientific reasoning in peer and teacher-guided discussions. Cognition and Instruction. (17)4, 379-432. Jordan, B. & Henderson, A. (1995). Interaction Analysis: Foundations and practice. The Journal of the Learning Sciences (4)1, 39-103. Moschkovitch, J. (2010). Language and mathematics education. Information age publishing. Ochs, E. (1979). Transcription as theory. In Developmental pragmatics (pp. 43-72). New York: Academic Press.
Acknowledgments
The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305A100862 to the University of Miami and by a University of Miami Provost Research Award Grant. The opinions expressed are those of the author and do not represent views of the Institute or the U.S. Department of Education.
ISLS
548
ICLS2012
Volume 2: Posters
The Power of Improvement Networks to Transform Educational Inquiry: A Preliminary Exploration

Jonathan Dolle, Carnegie Foundation for the Advancement of Teaching, dolle@carnegiefoundation.org Peter S. Wardrip, Jennifer L. Russell, University of Pittsburgh, psw9@pitt.edu, jrussel@pitt.edu Louis M. Gomez, UCLA, lgomez@ucla.edu Anthony S. Bryk, Carnegie Foundation for the Advancement of Teaching, bryk@carnegiefoundation.org Abstract: Educational Improvement Research is offered as an alternative to translational research and action research. Translational research often takes the worlds natural variation as noise to be controlled so that the efficacy of an intervention is accurately judged. Translational research, while rigorous, often looses contact with local contexts. Action research takes as its raison d'tre creating the conditions for and the assessment of effectiveness within local contexts, but often is criticized for failing to produce generalizable knowledge. We propose educational improvement research as a middle course. Undergirded by principles from quality improvement and networked improvement communities (NIC), educational improvement research aims to remain connected to settings of practice while producing reliable knowledge.
Introduction
The flow of educational knowledge between research and practice tends to move in one of two directions. In translational research, knowledge typically flows downstream, taking social science theories from fields like cognitive and behavioral science and applying them to the design of certain educational technologies, or interventions. This approach has had some notable successes. For example, writing interventions based on decades of social psychological research have been shown to have substantial effects on student achievement that persist for years (Yeager & Walton, 2011). A strength of this research is its explanatory power: when interventions spur progress on key problems, success can be explained by well-evidenced theories. At the same time, to work at scale, these interventions typically are propagated with a belief that a high degree of fidelity of implementation is required for success. Consequently, these interventions typically operate on small evidence-based kernels, fundamental units of behavioral influence that underlie effective prevention and treatment (Embry & Biglan, 2008, p. 75). In contrast, many forms of action research flow upstream. Rich investigation of one or more local practices or interventions are studied and analyzed to make sense of whats working and whats not. Action research focuses on specific local problems, exploring the understanding of different local actors within an organization or system. In education, it is most commonly associated with teachers working individually or in small groups on the felt needs in classrooms and schools. Action research has great value in that it organizes onthe-ground practitioners around problems of practice. Its methods are non-linear, characterized by cycles of planning, acting, observing, and reflecting on the changes in the social situations. Action research has many demonstrated successes addressing particular problems, but it tends to place much lower priority on understanding the causes, in terms of basic theory, that are at work in particular improvements. Consequently, successful action research projects tend to remain as locally-bound cases of innovation with little access to other contexts. Despite the strengths of each, neither approach is well suited to addressing complex problems educational practice at scale. These kinds of problems tend to be more causally diverse than those addressed by translational research, and they require coordinated action and organizational learning at greater scales than most action research. Of course, not all efforts to use educational inquiry to improve schooling fall in one of two camps. Many forms of program evaluation and design research (e.g. Phillips & Dolle, 2006) could be situated somewhere between these approaches. With few exceptions, these approaches have also fallen short of addressing many of the systemic problems of practice facing educational systems. To tackle this problem space, we introduce a collaborative learning structure known as a networked improvement community (NIC) (Engelbart, 2003). NICs combine the methods of quality improvement that have grown in popularity over the last five decades--but not significantly penetrated educational institutions--with a distributed network structure organized around a shared aim or purpose. The improvement orientation of NICs is not unlike the design-based implementation research recently described by Penuel et al (2011). Both focus on problems of practice, make use of iterative collaborative structures, develop useful theory through disciplined inquiry, and are concerned with building capacity for sustaining systemic change. What distinguishes NICs is its intentional formation organized to improve learning at scale.
ISLS 549
ICLS2012
Volume 2: Posters
Improvement Networks: A Third Way

Recently we have explored a new model of educational research and development that we call educational improvement research (Bryk & Gomez, 2008; Bryk, Gomez, & Grunow 2011). This model has two important dimensions that distinguish it from most educational inquiry. First, it builds on the practical intellectual heritage of Deming (2000), and others who pioneered principles of continuous quality improvement. This improvement science (Langley, Moen, Nolan, Nolan, Norman, & Provost 2009) seeks to bring the practice of disciplined inquiry to bear in the continuous improvement of products, services, or processes. As we are indebted to the current work in the Learning Sciences, we are also indebted to the efforts of Don Berwick and colleagues at the Institute for Healthcare Improvement (IHI) who pioneered a set of inquiry practices and conceptual frameworks that are now broadly applied to improving health services worldwide. Health services bears a family resemblance to education for many reasons not the least of which is that it is peopled by professionals who have a strong sense of individual agency when lives of users--students or patients--could be improved by relentless attention improvement. The second cornerstone consists of a model of social infrastructure for continuous improvement across organizations. Engelbart defines a NIC as a distinct organizational form that arranges human and technical resources so that the community is capable of getting better at getting better (Engelbart, 2003). The power of NICs can be understood by considering three domains of organizational activity. Alevel activities are the dayto-day work of carrying out the organizations primary business. In the case of schools this is the work of classroom teaching. Blevel activities are those designed to improve the ontheground work, typically taking place within a single organization. This work in school districts, for example, might include the professional development of instructors or the provision of data that improves school management. Clevel activity is much less common and typically involves crossorganizational co-ordination and learning aimed at getting better at getting better. Here institutions engage in concurrent development, working on problems and proposed solutions that have a strong family resemblance. Concurrent activity across organizations places relevant aspects of the context in focus and can help each local setting see its efforts from new vantage points. In principle this functions as an asset to local problemsolving and cross-organizational generalization. Engelbart (2003) observes that Clevel activity affords mechanisms for testing the validity of local knowledge, adjusting local understanding of the true nature of a problem, and advancing local support structures for improvement. NICs engaged in improvement research combine elements of both translational and action research in a way that facilitates cross-network learning and improvement of practice. Like translational research, NICs try to use and build upon existing theories in ways that improve practice. Like action research, NICs draw insights and innovations from practitioners experiences whose practice is transformed by network participation. The basic conjecture is that improvements at scale may best emerge from fleets of small, ongoing studies, coordinated across many contexts, rather than within a large field trial or through discrete local efforts carried out in isolated settings. This is a requirement for achieving effective and reliable outcomes at scale: diverse participants working in varied organizational contexts, but working within a common infrastructure that coordinates their inquiries.
References
Bryk, A.S., & Gomez, L.M. (2008). Ruminations on reinventing an R&D capacity for educational improvement. In F.M. Hess(Ed.), The future of educational entrepreneurship: Possibilities of school reform (pp. 181206). Cambridge: Harvard Education Press. Bryk, A.S., Gomez, L.M. & Grunow, A. (2011). Getting ideas in action: Building networked improvement communities in education. In M.T. Hallinan (Ed.), Frontiers in sociology of education (pp. 127-162). Dordrecht, Netherlands: Springer. Deming, W. E. (2000). Out of the crisis (1st MIT Press ed.). Cambridge, Mass.: MIT Press. Embry, D. D., & Biglan, A. (2008). Evidence-based kernels: fundamental units of behavioral influence. Clinical Child and Family Psychology Review, 11(1), 75-113. Engelbart, D. C. (2003, September). Improving our ability to improve: a call for investment in a new future. Paper presented at the IBM Co-Evolution Symposium. Penuel, W. R., Fishman, B. J., Cheng, B. H., & Sabelli, N. (2011). Organizing research and development at the intersection of learning, implementation, and design. Educational Researcher, 40(7), 331-337. Phillips, D. C., & Dolle, J. R. (2006). From Plato to Brown and beyond: theory, practice, and the promise of design experiments. In L. Verschaffel, F. Dochy, M. Boekarts & S. Vosniadou (Eds.), Instructional psychology: past, present, and future trends (1st ed., pp. 277-292). Amsterdam: Elsevier. Yeager, D. S., & Walton, G. M. (2011). Social-psychological interventions in education: they're not magic. Review of Educational Research, 81(2), 267-301.
ISLS
550
ICLS2012
Volume 2: Posters
Understanding Teachers Cultural Models about Technology

Jen Scott Curwood, The University of Sydney, Education Building A35, NSW Australia 2006 js.curwood@sydney.edu.au Abstract: This poster draws on educational psychology, cognitive anthropology, and sociolinguistics to build a theory of teacher learning. By using a microethnographic approach to discourse analysis, I show how secondary English teachers use language and contextualization cues within a learning community to reflect their cultural models, or everyday beliefs, about technology. The analysis suggests that the implementation of educational reforms, including those associated with technology, may be dependent upon teachers skills, values, and cultural models.
Introduction
Professional development is a cornerstone of educational reforms that seek to improve student achievement and engagement. Moreover, it can play a critical role in developing teachers instructional practices in the content areas, knowledge of standards-based assessment, and innovative use of new tools and strategies (Lawless & Pellegrino, 2007). Scholars have identified five core features of effective reform-oriented professional development, which includes a content area focus, opportunities for hands-on and active learning, coherence with previous professional experiences, collective participation with colleagues, and a considerable duration of contact hours (Desimone, 2009). However, due to issues with time and cost, professional development available to teachers remains inadequate. This is particularly evident in terms of technology integration. In this poster, I draw on educational psychology, cognitive anthropology, and sociolinguistic ethnography to build a theory of teacher learning. I then take an ethnographic approach to discourse analysis to show how teachers situated language within technology-focused professional development reflects their learning and sense making processes. This study was designed to address two gaps in the literature. First, it explores the role of situated language in constructing teachers cultural models, or everyday beliefs, about technology. Second, it examines micro-level interactions within a professional learning community to understand how teacher learning occurs in social and cultural contexts. The analysis suggests that the effectiveness of technology integration at a local level is closely tied to teachers cultural models. This analysis was guided by the following research questions: How does a discourse analysis of teachers situated language within a professional learning community reveal their cultural models? What are English teachers cultural models about technology?
Research Context
This ethnographic case study took place in during the 2009-2010 school year. I designed and facilitated professional learning communities at two high schools. (All names of cities, schools, and participants have been changed). In this paper, I focus on the learning community at Avon High School, which is located within close proximity to a large Midwestern city in the United States. Participants in this learning community included five English teachers. Over the year, they regularly shared their own lessons, brought in examples of student work, and engaged in critical dialogue about the role of technology in the English curriculum. They were veteran English teachers who had between 16 and 27 years of teaching experience.
Methods
I collected multiple forms of data, including: 1) initial survey of teachers knowledge of technology, content, and pedagogy; 2) audio recordings of two semi-structured interviews with each research participant; 3) video and audio recordings of ten learning community meetings; 4) field notes of my observations within the learning communities; 5) teachers mid-year written reflections; and 6) artifacts, including school district policies and teachers lesson plans. All told, this yielded 20 hours of videotaped discussions of the learning community as well as two hour-long interviews with each individual participant. In this analysis, I focus on an event, which Bloome, Carter, Christian, Otto and Shuart-Faris (2005) define as a bounded series of actions and reactions that people make in response to each other at the level of face-to-face interaction (p. 6). Within an event, utterances are acts that are part of a series of actions and reactions. In this analysis, I broke each utterance into a line, which generally consisted of one to two clauses. The focal event for this analysis involved a discussion on the use of social networks in the English curriculum. To answer my research questions, I examined teachers utterances within this singular event, which encompassed 61 lines and 11 turns at talk. I began with a transcript of the event and broke it down by speaker and then by utterance. I developed three layers of analysis, which build from the micro-level to the macro-level. This process of analysis will be more fully detailed in the poster, but the guiding questions were:
ISLS
551
ICLS2012
Volume 2: Posters
1. 2. 3.
Situated language and contextualization cues: How does the speaker use language and related contextualization cues within this utterance? Social interaction: How does this utterance contribute to social interaction? Cultural models: How does the speakers utterance reveal his or her cultural models of technology?
Findings
Quinn and Holland (1987) define cultural models as pre-supposed, taken-for-granted models of the world that are widely shared (although not necessarily to the exclusion of other, alternative models) by the members of a society and that play an enormous role in their understanding of that world and their behavior in it (p. 4). Within this four-minute event, teachers used language and contextualization cues in complex ways to advance the social interaction and express cultural models related to technology. By engaging in a three-layer approach to microethnographic discourse analysis, I was able to break the event into sections: 1) lines 1-13, which serve to initiate the event and introduce cultural models related to pedagogy and technology; 2) lines 14-50, which turn to cultural models on language and technology; and 3) lines 51-61, which conclude with cultural models related to learning and technology. Rebecca initiated the event when she turned our discussion from social networks to issues of teachers pedagogy and responsibility and stated: 1. what I'm saying is that I fe::el like as a teacher (.) 2. I need to teach kids how to use that (.) 3. so they won't be >shut out from access to the information< 4. >because they lack the skill< (2) In the poster, I will share all 61 lines and the related analysis. For this proposal, I will discuss just the first four lines, which show two competing cultural models of pedagogy. While Rebecca had no experience with social networks, she suggested in lines 1-2 that she had a responsibility as a teacher to instruct students on the function of such online spaces. In earlier interviews, Rebecca noted that many of her students were well versed in social networks, particularly with Facebook. However, Rebecca presented a cultural model of learning that was unilateral, from teachers to students. In effect, she didnt account for her students prior experience with social networks or consider how she could capitalize on this knowledge within her classroom. In Lines 3-4, Rebecca added that if students do not have the technical skills to access social networks, the consequence was that they will be denied access to information. In this respect, Rebecca held a cultural model that conceptualized knowledge as a commodity that could be given or withheld by an authority figure. In contrast, critical theorists propose a cultural model of knowledge as a social construction. Depending on which cultural model a teacher applies to their pedagogy, it will shape their instructional design and how they position teachers and students in the learning environment.
Implications
The National Education Technology Plan calls for episodic and ineffective professional development [to be] replaced by professional learning that is collaborative, coherent, and continuous (p. 40). Learning communities embody this kind of reform-oriented professional development (Curwood, 2011). By studying discourse within a professional learning community, I identified some of the ways that teachers used situated language and contextualization cues to contribute to the social interaction and express cultural models related to technology. In this four-minute event, a three-layer approach to microethnographic discourse analysis revealed some of our cultural models: 1) Knowledge is a commodity; 2) Knowledge is a social construction; 3) Asynchronous communication is an affordance of online spaces; 4) Synchronous communication is an affordance of physical spaces; 5) Teachers must take a critical approach to technology integration; and 6) Students future success depends on their access to and experience with technology. These cultural models directly inform teachers approach to technology integration, instructional design, and assessment within the English curriculum.
References
Bloome, D., Carter, S.P., Christian, B.M., Otto, S., & Shuart-Faris, N. (2005). Discourse analysis and the study of classroom language and literacy events. Mahwah, NJ: Lawrence Erlbaum. Curwood, J.S. (2011). Teachers as learners: What makes technology-focused professional development effective? English in Australia, 46 (3). Desimone, L.M. (2009). Improving impact studies of teachers professional development: Toward better conceptualizations and measures. Educational Researcher, 38 (3), 181-199. Lawless, K.A. & Pellegrino, J.W. (2007). Professional development in integrating technology into teaching and learning: Knowns, unknowns, and ways to pursue better questions and answers. Review of Educational Research, 77 (4), 575-614. Quinn, N. & Holland, D. (1987). Culture and cognition. In D. Holland and N. Quinn (Eds.) Cultural models in language and thought. Cambridge: Cambridge University Press.
ISLS 552
ICLS2012
Volume 2: Posters
Design-Based Implementation Research of Spreadable Educational Practices within the Participatory Learning and Assessment Network (PLAnet)
Rebecca C. Itow, Daniel T. Hickey, Learning Sciences Program, Indiana University rcitow@indiana.edu, dthickey@indiana.edu Abstract: This poster presents a secondary language arts module where open-source activities were (a) oriented to the theme of Romeo and Juliet, (b) aligned to a Common Core English standard, and (c) were organized around four principles of participatory assessment. The design of the module and the spread of the principles to the module and beyond show how ideas from the spread of media memes and new participatory approaches to assessment can extend design-based implementation research. Current efforts to catalogue and test 21st Century Skills are woefully misguided because they emphasize static decontexutalized knowledge and under-represent the importance of multi-modal writing (Hickey, Honeyford, Clinton, & McWilliams, 2010). The quickening evolution of digital knowledge networks means that we know very little about the actual contexts where our students will operate. But the most consequential contexts will surely be digital networks consisting of user-generated content that is persistent, searchable, and replicable (boyd, 2008), and will feature transactive interactions where media is customized for users personal preferences and shared control where content and expertise are co-created (Xenos & Foot, 2008). These networks will be characterized by participatory culture including low barriers to entry, support for creating and sharing, informal mentoring of newcomers, and a strong sense of social connection; as such they will be spaces where not every member must contribute, but all must believe they are free to contribute when ready and that what they contribute will be appropriately valued (Jenkins et al., 2006, p. 7). This reality is prompting growing interest in participatory education and curriculum for use in school and elsewhere. Unfortunately, test-based accountability and demands for achievement impact in strict experimental designs present significant obstacles for participatory learning in formal school contexts. Teachers need curricular resources (a) that support conventional literacies and academic knowledge teachers are accountable for, (b) which they can implement with reasonable levels of professional development and typical student access to networked computers, and (c) that are no more laborious for teachers than existing resources. This presentation features a case study of the development of just such a resource within a digital professional development network designed to support participatory learning while addressing accountability concerns.
Participatory Assessment and the Participatory Learning and Assessment Network

This module and the nascent PLAnet professional development network are organized around four core participatory assessment principles: (1) let contexts give meaning to concepts and skills: foster increasingly sophisticated discourse around valued concepts and skills by considering how they get their meaning from the contexts where they are used; (2) assess reflections rather than artifacts: rather than assessing student-created artifacts directly (i.e., with a rubric), assess student reflection on how creating the artifact gave meaning to valued concepts and skills; (3) downplay assessments: protect participation by using formal assessments to formatively assess curriculum rather than students; (4) Isolate tests: use tests only to evaluate the curriculumassessment ecosystem. These principles are rooted in the implications of situative theories of learning for assessment (e.g., Hickey & Anderson, 2007). This network was initiated in 2008 as collaboration between a University-based team of assessment specialists, Project New Media Literacies and a secondary English Language Arts (ELA) teacher. In 2009, the network expanded using Moodle to accommodate five new ELA teachers. In 2011, the network migrated to National Writing Projects Digital Is network to foster further growth.
The Consequences of Ignorance English Language Arts (ELA) Module

In Spring, 2011, the first author, with a ME.D from a progressive program, was implementing elements of participatory curriculum in her California ELA classroom. In advance of entering a doctoral program to join the aforementioned assessment research team headed by the second author, she elected to implement participatory assessment in her own classroom. She built a module using custom and open educational resources that were aligned to Common Core English standards (concerning character analysis and writing) and to a single classic text (Romeo and Juliet). The module included four activities: Discussion (in class and online) and role play, the mock trial, the digital poster, and the formal essay. For each activity in the module, the author created informal reflections during the activity, semi-formal reflections after the activity, and formal reflections on artifacts created in the activity. For the entire module she created a curriculum-oriented assessment to formatively assess
ISLS
553
ICLS2012
Volume 2: Posters
the module, and a standards-oriented test consisting of released items aligned to the targeted standards but independent of the curriculum to discreetly estimate impact on external achievement. The module was successful in that (a) it was manageable for the teacher and students, (b) the reflections revealed increasingly successful participation in increasingly formal discourse about the text and the practices, (c) the final formal essay showed enduring understanding of the concepts and skills in the standards, (d) students excelled on the formal assessment, and (e) test scores increased significantly.
Design-Based Implementation Research (DBIR)

This presentation shows how this effort embraced the four elements of what Penuel et al. (2011) labeled DesignBased Implementation Research (DBIR), and how participatory assessment can support DBIR. The first element of DBIR is that teams form around a focus on persistent problems of practice from multiple stakeholders perspectives. In our work, the team was formed around the persistent problem of the obstacles to participatory curriculum described above while the multiple perspectives were embodied in the three very different kinds of learning outcomes that participatory assessment aims to align and foster (shared participation, individual understanding, and aggregated achievement). The second element is to improve practice, teams commit to iterative, collaborative design. In close collaboration with the assessment team, the teacher made continual small refinements with the feedback she received from the informal reflections in each activity; additionally, she discussed the progression of the implementation with the researchers to make larger adjustments with the feedback from the more formal reflections as the module progressed. The third element of DBIR is teams develop theory related to both classroom learning and implementation through systematic inquiry. The design of the module was based on the participatory assessment framework and guiding principles, which evolved out of situative theories of cognition. The fourth element of DBIR is that it is concerned with developing capacity for sustaining change in systems. Because participatory assessment invites practitioners and researchers to work collaboratively to develop and iteratively refine modules with each implementation, the modules themselves stay relevant and useful for each classroom in which they are used. Teachers are encouraged to adapt, adjust, and remix these modules (and activities within the modules) as necessary to keep the content relevant for their curricular needs and student population.
Spreadable Educational Practices (vs. Disseminated Instructional Routines)

This presentation also illustrates the lessons that educational innovators and the PLAnet can learn from the reuse, remixing and adaptation of messages of viral memes by media scholars (e.g., Green and Jenkins, 2011). In contrast to centrally-tested evidence-based resources, participatory curricular resources need to spread in educational professional development networks the way that the familiar lolcats (cat pictures with humorous misspelled captions) spread in digital social networks. The module exemplifies what we call a Spreadable Educational Practice. While the Romeo and Juliet module was entirely different from other modules in the network, it was guided by the specific principles that emerged when the general participatory assessment principles were applied in other ELA contexts. Perhaps the best example of spread is the way that another teacher in the network in the Fall of 2011 successfully adapted the Romeo and Juliet activities to Of Mice and Men. This and other initial successes are now being used to gradually invite new teachers to join the network. Consistent with the intentions of the PLAnet and spread, the persistent digital traces of prior collaboration between the developing and implementation teachers are expected to be crucial resources, along with shared accounts, artifacts, and evidence from the specific implementations. boyd, d. (2008). Why teens MySpace: The role of digital publics in youth culture. In D. Buckingham (Ed.) (pp. 119-142) Youth, identity and digital media. Cambridge: MIT Press. Green, J., & Jenkins, H. (2011). Spreadable Media: How Audiences Create Value and Meaning in a Networked Economy. The Handbook of Media Audiences, 109127. Hickey, D. T., & Anderson, K. T. (2007). Situative approaches to student assessment: Contextualizing evidence to support practice. Yearbook of the National Society for the Study of Education: Evidence and Decision Making, 106, 264287. Hickey, D. T., Honeyford, M. A., Clinton, K. A., & McWilliams, J. (2010). Participatory assessment of 21st Century Proficiencies. In V. J. Schute & B. Becker (Eds.), Innovative assessment in the 21st century: Supporting educational needs (pp. 107-138). New York: Springer. Jenkins, H., Purushotma, R., Clinton, K., Weigel, M., & Robison, A. J. (2006). Confronting the challenges of participatory culture: Media education for the 21st century. Chicago: The MacArthur Foundation. Penuel, W. R., Fishman, B. J., Cheng, B. H., & Sabelli, N. (2011). Organizing research and development at the intersection of learning, implementation, and design. Educational Researcher, 40(7), 331337. Xenos, M., & Foot, K. (2008). Not your fathers internet: The generation gap in online politics. In L. Bennett (Ed.). Civic life online: Learning how digital media can engage youth (pp. 51-70). MIT Press.
References
ISLS
554
ICLS2012
Volume 2: Posters
Picture-Based Science Attitudes Assessment

Emily Johnson, Amy Bolling, Robb Lindgren, University of Central Florida, 3100 Technology Pkwy, Orlando, FL 32826-3281 Email: ekj@knights.ucf.edu, amy.bolling@gmail.com, robb.lindgren@ucf.edu Abstract: When assessing the effectiveness of science programs, it is important to accurately gauge subjects attitudes toward science. We designed an assessment that would elicit more authentic gut reactions to pictures of a variety of science and non-science objects and activities as opposed to text-based questions asking students to make detached reflections on their own attitudes toward science. This assessment was validated by administering it to middle school students along with a previously validated survey.
Attitudes Toward Science

When studying the effectiveness of science programs, it is important to accurately gauge students attitudes toward science. Current research in this field tends to rely primarily on assessment tools asking introspective questions with a Likert scale response. Some of these statements may be straightforward, Every citizen should understand science (Moore, 1997, p. 334), but most require the subject to reflect on her emotions, I would enjoy studying science (Moore, 1997, p. 335), and query an understanding of her own cognition, No matter how hard I try, I cannot understand science (adapted from Sandman, 1973 by Gogolin & Swartz, 1992, as cited in Rebello et al., 2011, p. 210). This introspective and metacognitive activity can be difficult even for adults and may be harder for children, typical targets for studies of science attitudes. While these commonly used, text-based tools are useful, we were interested in the potential advantages of using pictures to assess attitudes. In our contemporary culture, images are ubiquitous, but despite the pervasive nature of images, they are rarely used in research on attitudes towards science. Images have been used in research to assess attitudes toward commercials, movies, and other products (Thurstone, 1930; Soley, 2006). Market researchers have adapted picture-based assessments initially developed as psychological evaluations, aiming to study consumers non-conscious reactions. Psychiatrist-developed instruments such as the Thematic Apperception Test, or TAT, have been utilized by advertisers to assess the reactions of customers to commercials (Soley, 2006). With respect to attitudes in educational settings, pictures have been included in a limited capacity, predominantly in asking children to draw pictures representing their attitudes and to pictorially represent levels of a Likert Scale. Several tests have asked children to draw a person, themselves, or people with certain careers such as scientists (Goodenough, 1926; Oakland & Dowling, 1983; Chambers, 1983; Finson, 2002; Martin, 2004). These assessments give compelling evidence of the childrens attitudes toward the subjects of their drawings. Other assessments utilize pictures in lieu of words, aiming to discern the attitudes of younger children through illustrations of emotions (Kear, Coffman, McKenna, & Ambrosio, 2000).
Assessment Design
In the development of a picture-based assessment, we attempt to measure students attitudes toward science and related activities by asking them to rate their interest in pictures showing scientific activities and artifacts. By using images rather than statements, we are avoiding both the metacognitive task of responding to complex verbal questions and the reliance on the abstract notion of science which could be more negatively stereotyped than the specific scientific activities. For example, a middle-school-age student who claims that science is not interesting (or cool) may in fact be interested in a documentary about undersea exploration. Furthermore, at the middle school level, many students associate the word science with the course that they are being required to take at school, and often their opinions about a subject are more closely linked to the teachers personality and teaching style than the actual content of the course. In our picture-based assessment, students are presented with images of people doing science (see Figure 1) and objects related to science (see Figure 2) and asked to agree or disagree to one of two statements based on the type of picture: I would like to be doing what the people in the pictures are doing and I would like to find out more about the things in the pictures. This
Figure 1. Image of person doing science activities.
Figure 2. Image of science object.
ISLS
555
ICLS2012
Volume 2: Posters
makes the test very simple to answer, another aspect we hope will better gauge students true feelings toward science. Another obvious advantage pictures have over written or spoken language is the relative universality; by using a picture-based assessment, students with limited English speaking abilities will be assessed more completely and fairly using this methodology. We have been unable to identify any previous studies that have measured science attitudes in quite this way, but we anticipate that the indirect nature of posing the question of interest through an image of science in the world may lead to a more informative and honest response.

We administered the Picture-Based Science Attitudes Assessment and the modified Attitudes Toward Science Inventory (ATSI) (Rebello et al., 2011) to students at a rural middle school in central Florida using a withinsubjects design. During a non-academic enrichment period, students first took the picture-based assessment because of its more ambiguous nature, as we wished to avoid any science priming by the specific questions utilized by the ATSI. A total of 38 surveys were completed and used in this analysis. The students scores for all of the science-related pictures were averaged to produce a total attitude toward science score between 1 and 10, with a higher score representing a more pro-science attitude and a lower score representing a less enthusiastic attitude toward science. A similar average score was calculated from students responses to the 10question modified ATSI. The Pearson correlation of these two measures was .437 (p=.01). Based on the previously validated ATSI, we believe this correlation supports the validity of our picture-based assessment. The test was found to have high internal consistency with a Cronbachs alpha reliability coefficient of .826. While our findings are informative, we would also like to investigate the content validity of the picture-based assessment by comparing it with other measures, such as interviews or experiments that ask students to choose between science and non-science activities. This research could be extended in future studies of science attitudes by considering potential attitude differences between science in the real world and science in school.
References
Chambers, D. W. (1983). Stereotypic images of the scientist: The Draw-a-Scientist Test. Science Education, 67(2), 255-265. Finson, K. D. (2002). Drawing a scientist: What we do and do not know after fifty years of drawings. School Science and Mathematics, 102(7), 335-345. Gogolin, L., & Swartz, F. (1992). A quantitative and qualitative inquiry into the attitudes toward science of nonscience college students. Journal of Research in Science Teaching, 29(5), 487-504. doi: 10.1002/tea.3660290505 Goodenough, F. L. (1926). Measurement of intelligence by drawings. Yonkers: World Book Co. Kear, D. J., Coffman, G. A., McKenna, M. C., & Ambrosio, A. L. (2000). Measuring attitude toward writing: A new tool for teachers. The Reading Teacher, 54(1), 10-23. Martin, C. D. (2004). Draw a computer scientist. ACM SIGCSE Bulletin, 36(4), 11-12. Moore, R. W., & Foy, R. L. H. (1997). The scientific attitude inventory: A revision (SAI II). Journal of Research in Science Teaching, 34(4), 327-336. Oakland, T., & Dowling, L. (1983). The "Draw-A-Person Test": Validity properties for nonbiased assessment. Learning Disability Quarterly, 6(4), 526-534. Pell, T., & Jarvis, T. (2001). Developing attitude to science scales for use with children of ages from five to eleven years. International Journal of Science Education, 23(8), 847-862. Rebello, C. M., Witzig, S. B., Siegel, M. A., & Freyermuth, S. K. (2011). Attitude research in science education: Classic and contemporary measurements I. M. K. M. S. Saleh (Ed.) Assessment practices for understanding science-related attitudes (pp. 199-218). Sandman, R. S. (1973). The development, validation, and application of a multidimensional mathematics attitude instrument. Unpublished doctoral dissertation, University of Minnesota. Soley, L. (2006). Measuring responses to commercials: A projective-elicitation approach. Journal of Current Issues & Research in Advertising. Thurstone, L. L. (1930). A scale for measuring attitude toward the movies. The Journal of Educational Research, 22(2), 89-94. Weinburgh, M. E. & Steele, D. (2000). The modified attitudes toward science inventory: Developing an instrument to be used with fifth grade urban students. Journal of Women and Minorities in Science and Engineering 6(1), 87-94.
Acknowledgments
This material is based upon work supported by the National Science Foundation under grant DRL-1114621. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the granting agencies.
ISLS
556
ICLS2012
Volume 2: Posters
Connecting Visitors to Exhibits through Design: Exploring United States census data with CoCensus
Jessica A. Roberts, Leilah Lyons, Joshua Radinsky, Francesco Cafaro, University of Illinois at Chicago Email: jrober31@uic.edu, llyons@uic.edu, joshuar@uic.edu, fcafar2@uic.edu Abstract: This study presents preliminary results from a design-based research program examining ways to engage museum visitors in free-choice inquiry with geo-referenced census data, focusing on affording connections between big narratives about ancestry and migration, and little narratives about neighborhoods and personal identity. Preliminary findings have revealed affordances and tradeoffs in key design decisions that can be incorporated into ongoing iterations of the display.
Introduction
Designers of museum exhibits can benefit from attending explicitly to the ways visitors might connect their own life narratives (little narratives) to particular larger, social or big narratives, through the activity that occurs in the exhibit space (Rowe, Wertsch, & Kosyaeva, 2002). We are engaged in iterative design and study of a digital exhibit for a small history museum in Chicago, which plots tract-level census data onto interactive geographic information system (GIS) maps. Maps, especially small-scale local maps, afford connections to personal narratives locating where we live, where we were this morning, places we recognize. The maps design plays a large role in determining the extent to which these connections are afforded. In this paper we describe some of the design decisions we have made and how they relate to the conversations visitors have had around the display in initial testing. For each decision we discuss approaches and tradeoffs considered and how they can be further iterated to support visitors connections between big and little narratives.
Prior Work
Some work has been done to investigate users interpretations and learning around complex data maps, finding that design considerations for GIS maps in educational environments affect users interpretations (McCabe, 2009). We build off the assumption that museums are potentially ideal public spaces where personal, private, or autobiographical narratives come into contact with larger-scale, collective, or national narratives in mutually animating ways (Rowe et al., 2002). Our project explored embodied interaction as a means to join these narratives (Cafaro et al., 2011), but this work focuses on visualization design.
Methods
The first round of testing took place in situ at the Hull House Museum in the spring of 2011. Ten interviews of groups ranging from 1-3 visitors were conducted. Participants were asked for the one or two ancestry categories that best represented them, and were shown two different digital maps (described below) of that ancestry data plotted on Cook County. Visitors were asked open-ended questions about what they saw in the maps and what, if anything, they found surprising or confusing. They were also asked which map they preferred. Interviews were transcribed and open coding was conducted to characterize the nature of the discussions. The first round of coding revealed phenomena of interest, and a second round of coding was conducted with a small subset of codes in order to target issues for design of ongoing iterations. Questions of interest include: How do people interpret the display and different variations of it? How does the display mediate the noticing and exploration of patterns, the connection between big and little narratives, the construction of new narratives and/or activation of existing narratives about themselves, a place, and others?
Design Consideration - Ancestry Categories

A review of GIS and design literature (e.g. Krygier & Wood, 2011) led us to use graduated symbols to represent quantities (census tract populations) and hue to represent categories (ancestries). To constrain the number of categories to a scope reasonably distinguishable on the display, we joined ancestries of smaller populations (fewer than 45,000, or roughly 1% of the 2009 population) according to the United Nations Statistical Divisions regional categories. Some visitors struggled to make sense of the categories. For example, Ashley (all names are pseudonyms) initially struggled between choosing Western and Northern European. She stated she was Irish and German, adding, But, uh, Im more than just that, so Those are just smaller portions of my ancestry. This which box to check confusion led to frustration about how or whether Ashley was represented. Another visitor, Elliot, stated that his fathers family was from Poland and was Jewish. Very knowledgeable about Chicago demographics, while observing the Polish data he commented, the Catholic Poles distribution
ISLS
557
ICLS2012
Volume 2: Posters
was very much different from the Jewish Polish, a distinction not representable with the ancestry data set. Cultural distinctions problemetize the representation, potentially invoking additional little narratives that allow visitors to question the big narrative. The next design needs to balance the need to make the categories more intuitive against the desire to leverage the inherent contradictions of historical categories to prompt valuable conversations about identity.
Design Consideration - Anchors for Place Identification

Another design concern was the extent to assist visitors with place identification. The displays tested here included bodies of water and city and township boundary lines in the county but no other landmarks for reference. Viewing ancestry data on this map, visitors were able to identify parts of the region, including the downtown area and the suburbs, and some specific places, like Chinatown. Including the transit lines on the second map, gave participants additional orientation assistance. Interestingly, the addition of transit lines increased another phenomenon visitors generating hypotheses. In one case, Paige stated, Well you can see on this where people, where people moved to more, like along the lines of transportation and (inaudible) ... where all the jobs are and everything. In contrast to the first map, now she moved beyond general characterizations of the data to connect to the lived experiences behind the data, forming theories about settlement. Future iterations, therefore, will explore how different reference points afford orientation and hypothesis generation, balanced against spuriously obscuring the thematic data.
Design Consideration - Data Scaling

Extreme variations in size and population of census tracts posed complicated questions regarding data symbolization. We tested two options: one with seven class breaks, one with 32 equal interval classes (see Figure 1).
`
Figure 1. Natural breaks map (left) and equal interval map (right) displaying Polish population. Participants responses revealed inherent tradeoffs. Some preferred the natural breaks map, saying it was more appealing to the eye (Paula) and easier to identify patterns, but some felt it obscured the geography and other data sets. Elliot succinctly expressed his concern: You know the (natural breaks map) kind of smacks you in the face, but Im not sure whether smacking you in the face is something thats misleading or not. Preference for the equal interval map seemed rooted in its preservation of the geographic features, but the small range in bubble size also led to complaints that data patterns were hard to detect. The comments suggest that a hybrid design might be called for using natural breaks or perhaps logarithmic scaling to separate classes but with smaller, less overlapping bubbles.
References
Cafaro, F., Lyons, L., Radinsky, J., and Roberts, J. (2010). RFID localization for tangible and embodied multiuser interaction with museum exhibits. In Proc. of Ubicomp '10. ACM, New York, NY, 397-398. Krygier, J., & Wood, D. (2011). Making maps: A visual guide to map design for GIS (2nd ed. ed.). New York: Guilford Press. McCabe, C. A. (2009). Effects of data complexity and map abstraction on the perception of patterns in infectious disease animations. Unpublished MS, The Pennsylvania State University. Rowe, S. M., Wertsch, J. V., & Kosyaeva, T. Y. (2002). Linking little narratives to big ones: Narrative and public memory in history museums. Culture & Psychology, 8(1), 96.
Acknowledgements
This project is funded by NEH HD 51357 and the UIC IPCE Civic Engagement Research Fund Award.
ISLS
558
ICLS2012
Volume 2: Posters
Methods of analysis for identifying patterns of problem solving processes in a computer-supported collaborative environment
Shannon Kennedy-Clark, Australian Catholic University, Sydney, Australia, Email: shannon.kennedyclark@acu.edu.au Kate Thompson, Centre for Computer Supported Learning and Cognition, University of Sydney, Sydney, Australia, Email:kate.thompson@sydney.edu.au Abstract: Recently, there have been calls to undertake further analysis of learner interactions in collaborative computer supported environments that move beyond code and count. We present three different means of analyzing and interpreting conversational data recorded from learners sharing a computer to solve a virtual inquiry. We propose that through the use of group function analysis, discourse analysis and an analysis of periodicity we can gain an understanding of how learners interact on both macro and micro levels.
Introduction
Reporting on multiple measures of process is becoming more common (see for example (Evans, Feenstra, Ryon, & McNeill, 2011; Weinberger & Fischer, 2006)). An advantage of the use of multiple measures is the ability to report on interaction effects (Weinberger & Fischer, 2006) and relationships between data at different levels, particularly individual contributions to group processes (Ding, 2009). We describe our experiences capturing and analyzing student learning patterns as we reinterpret the data through the administration of different coding systems and visual representations. To understand the successes and failures of learning in a virtual world, learning processes need to be understood in addition to relationships to learning outcomes. We recorded the in world actions of four postgraduate and eight undergraduate education students in their interactions with a narrative-based virtual world, Virtual Singapura. The results are not presented in full; we demonstrate the results of the different treatments through the analysis of excerpts from two dyads.
A Group Functions Perspective

One question to be answered by this study was to ascertain if there were patterns in the collaborative problem solving that may indicate a successful problem solving strategy. The rationale for this is that if successful strategies can be identified then scaffolding can be incorporated into the learning activities this may be in the form of teacher instruction, prompts, discussion and work sheets, for example. A modified version of the DFCS from Poole & Holmes (1995) was adopted for the analysis of the collaborative problem solving (Figure 1).
(a) Dyad 1 (unsuccessful)
(b) Dyad 6 (successful) Figure 1. Decision-making process models
The opportunity to relate the synchronous collaboration data to the information about the interaction with the virtual world provided insights into the design and scaffolding required in this inquiry-learning task. However, coordinated analysis of data was time-consuming, and would benefit from some element of automation. Moreover, the use of a code and count approach has been deemed to lack the necessary rigor to provide a detailed understanding of how a group functions (Kapur, 2011).
A Systemic Functional Linguistics Perspective
ISLS
559
ICLS2012
Volume 2: Posters
We hypothesized that we could obtain a better understanding of the different ways in which dyads react to the degree of structure in a virtual world if a systemic functional linguistics approach to the analysis was applied. The data were coded using Collaborative Process Analysis Coding Scheme (CPACS) (Kennedy-Clark & Thompson, submitted). We identified patterns in each of the multiple code categories: action, content, attitude, tense, modality and pronouns. Within the content category, patterns ranged from simple navigation (Dyad 1) to a more cyclic process: plan, navigate, collect data (Dyad 6).
Visualizations of Periodicity
Periodicity is temporally constrained and shows the development or non development of waves of conversation across a temporal axis. After coding the data using CPACS, we were able to present a visual pattern of the waves of conversation. Figure 3 demonstrate the periodicity of the content code in Dyads 1 and 6.
Topic
Planning
Navigation
(a) Dyad 1
(b) Dyad 6 Figure 2. Periodicity of content code
Conclusions
The results suggest that helping students to develop routines for problem solving would be more effective that not developing routines. What is missing in this discussion is any mention of the temporal nature of the data, and the ways in which the groups processes change over time, or with regards to the context of their learning. Visualisations were found to be useful in identifying areas of overlap and the multiple reasons for a groups success.
References
Ding, N. (2009). Visualizing the sequential process of knowledge elaboration in computer-supported collaborative problem solving. Computers & Education, 52, 509-519. Evans, M. A., Feenstra, E., Ryon, E., & McNeill, D. (2011). A multimodal approach to coding discourse: Collaboration, distributed cognition, and geometric reasoning. International Journal of ComputerSupported Collaborative Learning, 6(2), 253-278. Kapur, M. (2011). Temporality matters: Advancing a method for analyzing problem-solving processes in a computer-supported collaborative environment. International Journal of Computer Supported Collaborative Learning, 6(1), 39-56. Kennedy-Clark, S., & Thompson, K. (submitted). Between the lines: The use of discourse analysis in a virtual inquiry to inform learning design. the International Journal of Virtual and Personal Learning Environments. Poole, M. S., & Holmes, M. E. (1995). Decision Development in Computer/Assisted Group Decision Making. Human Communication Research, 22(1), 90-127. Weinberger, A., & Fischer, F. (2006). A framework to analyze argumentative knowledge construction in computer-supported collaborative learning.
Acknowledgments
The Centre for Computer Supported Cognition and Learning at the University of Sydney would like to acknowledge Singapore Learning Sciences Laboratory (National Institute of Education) and faculty in Computer Engineering and in Art, Design, and Media at Nanyang Technological University for their ongoing support in the use of Virtual Singapura.
ISLS
560
ICLS2012
Volume 2: Posters
An Eye for Detail: Techniques for Using Eye Tracker Data to Explore Learning in Computer-Mediated Environments
Marcelo Worsley, Paulo Blikstein Transformative Learning Technologies Lab (TLTL), Stanford University, 520 Galvez Mall, CA, USA [mworsley, paulob] @stanford.edu Abstract: This paper describes two methods for analyzing student gaze in computer-mediated learning applications. More specifically, we demonstrate how to use eye-tracking data from an agent based modeling study to identify meaningful patterns in student learning processes. We do this by using techniques from network analysis and natural language processing which allow us to identify statistically significant differences among our two conditions. Finally, we conclude by motivating a larger study that will further utilize these techniques.
Introduction
Computer-mediated learning applications have become increasingly pervasive in todays technology-driven society. From intelligent tutoring systems, to educational games, to agent-based modeling environments, students are presented with a number of opportunities for computer-mediated learning. However, there remains a number of ways in which analyzing student learning in these environments can be quite challenging. Accordingly, many adopters of computer assisted learning resort to creating scripted environments that limit exploration. They also take advantage of techniques such as think-aloud and emote-aloud, to identify student learning processes. However, there are shortcomings in these approaches (Cooke and Cuddihy 2005). And while keyboard and mouse logging can capture student actions within the system, we wish to understand the complexities of student learning processes, especially at a micro-genetic scale. Accordingly, we will present two techniques that we recently developed to analyze data from a small scale study (n=4) of students interacting with an agent-based modeling environment.
Experimental Design
Four undergraduate engineering students were asked to participate in a thirty minute Netlogo study in which each explored three models related to the Ideal Gas Law. The four students were split into two conditions. The two conditions were identical except for the mode used to instruct the students about the Ideal Gas Law. The flow of the conditions was as follows: 1) Prior to interacting with the first model, students are informed that they will be asked to study each model for 3 to 5 minutes, after which they will be asked to described what is happening in the model (Mayer, 2009 in Jukka, 2010); 2) student interact with Netlogos Connected Chemistry Temperature and Pressure Model and provide a description of the model; 3) students read either an agent based modeling explanation of the gas laws, or a textbook based description of the gas law; 4) students interact with the Gas in a Box model and provide a description of what is happening in the model; 5) interact with the Adiabatic Piston model and provide a description of what is happening in the model; 6) Students watch the video of their eye tracking data and provide meta-cognition of their process (De Koning et al., 2010, Jarodzka et al., 2010 in Jukka, 2010). Finally, the study captured gaze, audio, video and student Netlogo logging history, but we will focus our attention on the gaze related data for this paper.
In order to analyze this data we used two techniques that build on previous eye-tracking data analysis techniques (Jukka, 2010, Conati &Merten, 2007, Slykhuis, Wiebe, Annetta, 2005, van Gog & Scheiter, 2010. For the first technique we wanted to investigate the relationship between student actions and the gaze events that take place between those actions. In order to accomplish this, we elected to segment the stream of gaze points using mouse clicks. This is to say that all of the gaze points that happen between two successive mouse clicks were grouped together. This mode of segmentation lets us consider how the various gazes that the student makes after clicking may be influential in determining their next behavior. For a given segment, we look at the total number of gaze points generated, and the number of unique gaze points that are generated. This distinction can theoretically be used to examine a couple of different ideas. First, a student that is focusing on a small number of elements may have a strong intuition about what will happen and need not look at more than a select number of elements (Canham and Hegarty, 2010, Jarodzka, 2010 in Jukka, 2010). A second explanation is that a student with a large number of unique gazes is able to process a larger cascade of causation and consider how changing a single element impacts several of the model's variables. To further hone in on the types of cognitive connections that may be at work as the students interact with the models, we borrow techniques from network analysis. Doing so allows us to examine the centrality of any given gaze point, and its relationship to other gaze points. To do this, we create an edge between all
ISLS 561
ICLS2012
Volume 2: Posters
temporally adjacent gaze points, and then use this network representation to look at a number of salient aspects in network analysis research: average degree, average shortest path and degree distribution. We conclude by encoding for patterns of behavior in the way that the students interact with the visual elements. More specifically, we look at the direction of each gaze; this is to say, whether the subject is moving their eyes from left to right, up to down, and any combination of these (left and up, right and up, etc.) Pairs of gaze points that are identical, i.e. those from a fixation point, are omitted for this portion of the analysis, since we are primarily interested in the direction that people were moving their eyes, when they moved them. Furthermore, we construct pairs and trios of directional movements, to see what types of patterns of movements people typically employed when interacting with these models. An example of a trio would be to look left, then right, and then left; or to look up, then down, then up these are examples of what we will refer to as A-B-A patterns.
Results
The results from this analysis technique are multifaceted, and there remain a number of dimensions to explore further. We will begin by describing a quantitative result that we observed. While our goal in this initial study was to develop a qualitative understanding of the differences in how people visually interact with Netlogo models, we did observe a statistically significant result between our two conditions. The participants that read the agent-based modeling description of the gas laws had statistically significantly larger numbers of unique gaze points compared to those that read the textbook description. This occurred despite the two groups not having a statistically different number of unique gazes in the initial exploration (that which occurred just prior to reading the different descriptions of the Ideal Gas Law). Another compelling observation arose from our analysis of the users' viewing patterns. Not surprisingly, most of the users primary gaze pattern was to look at one item, look at another, and then look back at the first item again (the A-B-A pattern). Students that read the agent-based modeling description were more likely to follow the A-B-A gaze pattern than students that read the textbook description. However, further analysis is needed to understand whether this indicated a more in-depth study of the model. In terms of the network analysis techniques that we used, our preliminary results indicate that there are only a few sets of gaze points that have degree higher than 2. This means that most gaze points are only observed once. However, when using gaze point compression, the average degree ranges from 2.8 to 5.2 edges, depending on the model, and the parameters for compression. Further work will need to be completed to see if there is pedagogically meaningful data encoded in network characteristics.

In this paper we have presented two approaches for looking at gaze data, and have generally argued that this form of data can be useful in gaining a better understanding of the learning processes that students engage when interacting with computer-based learning applications. These approaches borrowed proven techniques from network analysis and natural language processing. Beyond this, we have presented initial findings that compare how students' interactions with agent based models are dependent on the type of instruction that they received. That said we are cognizant that there may be many elements at play that require further exploration and more rigorous controls. Furthermore, we recognize the need to dive deeper into some additional features of the data. Despite the areas for improvement, adopting this form of analysis will become increasingly relevant, and necessary, as we move towards more technologically-enhanced learning environments as it affords accurate identification of student learning on a micro-genetic level.
References
Barrios, G., Gtl, V., Preis, A., Andrews, K., Pivec, M., Mdritscher, F., & Trummer, C. (2004). AdeLE: A Framework for Adaptive E-Learning through Eye Tracking, Proceedings of IKNOW 2004, Graz, Austria, 2004, pp.609-616. Conati, C. and Merten,C. (2007) Eye-tracking for user modeling in exploratory learning environments: An empirical evaluation, Knowledge-Based Systems, Volume 20, Issue 6, August 2007, Pages 557-574 Cooke, L.; Cuddihy, E., "Using eye tracking to address limitations in think-aloud protocol," Professional Communication Conference, 2005. IPCC 2005. Proceedings. International , vol., no., pp. 653- 658, 1013 July 2005 Jukka H., (2010)The use of eye movements in the study of multimedia learning, Learning and Instruction, Volume 20, Issue 2, April 2010, Pages 172-176 Slykhuis, D., Wiebe, E., Annetta, L. (2005) Eye-Tracking Students' Attention to PowerPoint Photographs in a Science Education Setting. Journal of Science Education and Technology. December 2005, 509-520 van Gog, T. and Scheiter, K. (2010) Eye tracking as a tool to study and enhance multimedia learning, Learning and Instruction, Volume 20, Issue 2, April 2010, Pages 95-99
ISLS
562
ICLS2012
Volume 2: Posters
Finding the Common Thread: Learners Intuitive Knowledge of General Patterns that Apply Across Domains
Hillary L. Swanson, University of California, Berkeley, hswanson@berkeley.edu Abstract: This study explores learners intuitive knowledge of the general patterns of change that apply across multiple domains. Examples of such patterns are oscillation, equilibration, and threshold. Intuitions of these patterns can be leveraged in the design of constructivist science instruction. Clinical interviews elicit pattern intuitions by showing participants highly schematic simulations designed to illustrate each pattern. Participants are then prompted to generate examples of phenomena that follow each pattern. Preliminary analysis suggests that students have rich intuitive knowledge regarding general patterns that transcend various domains.
Introduction
In this study I explore learners intuitive knowledge of the patterns of change that apply across multiple domains. This work is situated within a larger program that aims to design equitable science instruction by building normative understanding of dynamical systems topics on the foundation of students intuitive knowledge of patterns. Following a Knowledge in Pieces (KiP) perspective, I view intuitive knowledge as a rich and productive resource for learning that all students bring to the classroom, regardless of prior educational opportunity (diSessa, 1993; Smith, diSessa, & Roschelle, 1993). Intuitive knowledge of patterns, in particular, is powerful for learning because: 1. Patterns can be analyzed in a diversity of engaging contexts belonging to both psychosocial and physical domains. 2. Patterns provide an intuitive framework on which to ground the construction of normative understanding of important topics of both classical and cutting-edge science. 3. Pattern-finding is an inherently scientific activity. By looking for the patterns that transcend seemingly unrelated phenomena, students engage in an important practice of professional science.
I explored the intuitive pattern knowledge of three 9th grade students. The students were of ethnic minority and from low SES families, and had been designated by their schools academic support staff as at-risk of failing or dropping out of high school. Participants were intentionally selected from this population in order to gather evidence in support of the claim that all students, regardless of prior educational opportunity, come to school with a knowledge base richly populated with intuitions that are productive for learning. Prior work with high SES participants suggested that students can meaningfully engage in this subject matter and I anticipated finding similar engagement with students of low SES. Each student participated in a clinical interview that lasted about thirty minutes. During the interview participants were shown highly schematic simulations designed to illustrate each of four patterns relevant to the study of physical systems. These patterns are threshold, hysteresis, critical damping, and linear control. After viewing each simulation, participants were asked to describe the behavior of the elements in the simulation and then generate examples of physical or psychosocial phenomena that followed the same pattern, or respond to interviewer generated examples. For all examples, students were asked to explain mappings between examples and simulations.
Preliminary Analysis 1: Students have Rich Intuitive Knowledge Regarding Patterns

The participants generated a number of examples of both physical and psychosocial phenomena for each pattern simulation (see Figure 1, below). In general, I found that these students engaged well in the tasks and connected phenomena to simulations with relative ease. Broadly speaking, the work of these students seemed to be quite comparable with that of higher SES, academically oriented students.
Figure 1: Total number of student-generated examples of physical and psychosocial phenomena that follow the pattern of each simulation.
ISLS
563
ICLS2012
Volume 2: Posters
Preliminary Analysis 2: Intuitive Knowledge of Patterns Transcend Domains

Analysis of a small portion of an interview with one student, referred to as Anita, is reported below as an illustrative case where the same pattern was recognized in phenomena of both physical and psychosocial domains. Anita was shown a simulation designed to illustrate threshold. I schematize threshold in relation to a transition between two modes of action. Actions of the initial mode are characterized as approaching a transition. Final mode actions are the result of the transition. The point at which the transition occurs is the threshold. A series of three screenshots is provided in Figure 2 (below) to give the reader a sense of the elements and sequence of events in the threshold simulation. The simulation begins with two elements, a red circle and blue rectangle, at rest (Figure 2, left box). The user moves the blue rectangle horizontally to the left with the mouse until the rectangle reaches the circle and pushes it to the left (Figure 2, middle box). The circle reaches its threshold and reacts by flying leftward away from the rectangle (Figure 2, right box). In this simulation, the initial mode of action is illustrated by the rectangle pushing the circle. The final mode of action is illustrated by the circle flying away from the rectangle. The threshold is a location on the left side of the screen at which the circle transitions from being passively pushed along by the rectangle to actively flying away from it.
Figure 2: A series of screenshots show three moments in time to illustrate the sequence of events in the threshold simulation. Although Anita did not use the word threshold, she recognized examples of phenomena from both physical and psychosocial domains that followed the pattern illustrated by the simulation. She indicated a sense of threshold in the physical phenomenon (suggested by the interviewer) of pushing an object up a hill until it crosses the threshold of the peak and as a result, rolls down the hill on its own. As well, she indicated a sense of threshold in the psychosocial phenomenon of breaking a habit of skipping class (an example she generated on her own). Like if someone is like/ just they could/ say skipping class or something like that/ and you do it like constantly and then you try to break the habit and then after a while you just like/ you've had enough and you just go to class Her language indicates an intuitive sense of the threshold pattern. In her example, the initial mode of action is illustrated by skipping class constantly. The final mode of action is going to class. The threshold is the point at which youve had enough and a transition is made from the first mode of action to the second - from skipping class to attending it.
Future Work
Preliminary analysis suggests that students have rich intuitive knowledge concerning patterns that cut across physical and psychosocial domains. In order to more thoroughly characterize the nature of students intuitive knowledge regarding patterns and the phenomena they underlie, analysis will be conducted of the mappings made by participants between examples of physical and psychosocial phenomena and the simulation. Additional interviews will be conducted with students of non-dominant ethnic groups and low SES and contrasted with data collected from interviews with students of dominant ethnic groups and higher SES to provide statistical evidence in support of claims regarding the universality of these productive intuitive resources.
References
diSessa, A. A. (1993). Toward an epistemology of physics. Cognition and Instruction, 10(2-3), 105-225. Smith, J. P., diSessa, A. A., & Roschelle, J. (1993). Misconceptions reconceived: A Constructivist analysis of knowledge in transition. Journal of the Learning Sciences, 3(2), 115-163.
ISLS
564
ICLS2012
Volume 2: Posters
A case study of P2PU: New models for open and peer-focused learning
Monica Resendes, Stian Hklev OISE/University of Toronto, 252 Bloor Street W., Toronto, ON, CANADA E-mail: stian.haklev@utoronto.ca, monica.resendes@utoronto.ca Abstract: Supporting life-long learning is gaining in importance around the world. Collaborative peer-led open courses can potentially play an important role in this field, and Peer 2 Peer University, offers a unique platform for design-research aiming to improve students abilities to engage in collaborative deep inquiry-based learning. This paper introduces a course that ran on P2PU, discusses the unique affordances of open courses, and how these can be preserved while promoting students integrative work on ideas.
Introduction
The advent of the knowledge society has resulted in the creation of educational programs dedicated to promoting so-called 21st century competencies in traditional classrooms (e.g., Partnership for 21st Century Skills, the 21st Century Learning Initiative). As society shifts from an information to a knowledge-based economy, demand for increased access to meaningful learning opportunities for people at all ages, and in all life-situations, has grown dramatically (Hargreaves, 2003). This rise in demand for non-traditional forms of education is a global phenomenon--the Obama administration recognized the hundred thousands of workers who need retraining due to the shifting nature of the US economy, through the creation of a $2 billion fund for the creation of Open Educational Resources (US Department of Labor, 2011) and China is creating a range of programs to reach their goals of 350 million people involved in life-long learning by the year 2020 (Gu, 2010). Open courses that use Open Educational Resources and take advantage of the affordances of new network technologies to enable people to learn together in a peer-to-peer fashion have been developing for several years. Technological development and innovation in the area of Open Educational Resources has proceeded at a very quick pace, so that careful and systematic research about the specific challenges, opportunities and problems that open courses face is needed, particularly with respect to how these new educational platforms can support student achievement and help them develop competencies integral to success in the 21st century.
The research site, Peer 2 Peer University

The Peer 2 Peer University (P2PU) provides an interesting platform for research projects designed to tease out the new challenges and opportunities inherent in open courses. P2PU is a grassroots open education project that embodies a model for lifelong learning based on the principles of openness, community, and peer learning. It has developed rapidly in the three years since its inception, both technologically and in terms of its user-base. However, fundamental questions about how the P2PU model supports learning and challenges conventional educational structures have yet to be systematically explored, and research into how Open Educational Resources can support 21st century competencies like collaboration and knowledge creation are in their infancy. For example, although research dedicated to exploring automated learning analytics and forms of assessment in open courses is growing (see for example Liddo, et al., 2011), innovative methods for tracing and assessing the growth and spread of ideas in these environments, as exhibited in the discourse students generated online, remains scarce. The research reported in this paper seeks to contribute to this objective by exploring the unique affordances of open courses in facilitating and supporting the sharing of ideas and deep discussions in a distributed, networked educational environment. More specifically, this study focuses on a case analysis of an experimental P2PU course that ran in the spring of June 2011 entitled Introduction to the field of ComputerSupported Collaborative Learning (CSCL). The main objectives of this study are as follows: i.) to determine the extent to which ideas and concepts spread across the various platforms that were available to course participants, including P2P forums, twitter, blogs, and etherpad (a collaborative writing platform), ii.) to test designs for tracing, aggregating and visualizing the key ideas that students are engaging in their dialogue.
Method
The research reported in this paper is, at this stage, exploratory in nature. It represents the first steps in a larger program of research that adopts a design-based approach (Brown 1992) dedicated to improving the P2PU learning model. For this particular case study, we will adopt a grounded theory (Glaser & Strauss, 1967) approach in order to identify and categorize types of ideas, related to key concepts that emerged through shared discourse. The goal will be not only to investigate how the sharing of key ideas and concepts between the
ISLS
565
ICLS2012
Volume 2: Posters
individual students and across distributed platforms could be encouraged, but also to engage in the iterative design of innovative tools to help give feedback to students about the nature and content of their discourse.
Course Design
Students were offered the opportunity to apply to become core members, who committed themselves to participate actively, and were the only ones with write-access to the P2PU platform, or to follow the course. For followers receive automatic updates, but no approval is necessary, and no social commitment is made. We were interested in whether ideas and contributions from the course members would be taken up and amplified through the online networks of the followers hoping to hear from some of the followers (who resemble lurkers, as mentioned in for example Meiszner, 2010).
Participants and Dataset

Participants for this study include 55 participants, include 8 students, 2 two course facilitators, and 45 followers. The dataset for this study is comprised of all output generated from participants during the eightweek course. Data includes student written contributions in various forms, including blog entries, tweeter feeds, all contributions to approximately six discussions as archived on Etherpad documents, postings on the P2PU discussion forums, and group emails. All students also wrote a public reflection on their participation during the final course, and a survey was conducted among the followers.
Results and Future Work

Our preliminary findings showed very little engagement by the formal followers, although one person reported having followed the entire course closely, and going so far as to change his MA thesis topic based on new ideas gained through the passive participation in the course. However, we also found numerous examples of course amplification through participants existing social networks, extending the reach of the course to individuals that would not have been reached otherwise, and bringing new perspectives into the discussions. Preliminary analysis of the discussions also shows strong student engagement and a high level of engagement with the weekly topics, but a lack of integration of knowledge across topics and time. Students also reported that they found that the distribution of the discourse among multiple open platforms, such as the builtin discussion forum, the students individual blogs, and the weekly chats contributed to this difficulty. Given the strong advantages of using open platforms, and the ability of students individual blogs to tap into students existing social networks, this calls for research on how to enable knowledge integration across a number of distributed Web 2.0 platforms. Next steps will include a second iteration of the Introduction to Computer-Supported Collaborative Learning course on P2PU, incorporating a custom-built RSS feed reader, capable of aggregating content across both blogs (uniquely including blog comments), discussion forums and text chats. This will be combined with an interface allowing students to quickly visualize the discourse around different keywords, regardless of where the contributions were made. This research will contribute to broader discussions dedicated to developing emerging innovations for open learning by exploring the extent to which this course was successful at supporting the spread of ideas and networked interaction in open, distributed learning environments.
References
Brown, A.L. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. The Journal of the Learning Sciences, 2(2), 141-178. Corneli, J. & Danoff, C.J. (2011) Paragogy: Synergizing individual and organizational learning. In: The Open Knowledge Conference (OKCon), Berlin, Germany. De Liddo, A.; Buckingham Shum, S.; Quinto, I.; Bachler, M. and Cannavacciuolo, L. (2011). Discourse-centric learning analytics. In: LAK 2011: 1st International Conference on Learning Analytics & Knowledge, 27 Feb - 01 Mar 2011, Banff, Alberta . Downes, S. (2010). New Technology Supporting Informal Learning. Journal of Emerging Technologies in Web Intelligence, 2(1). doi:10.4304/jetwi.2.1.27-33 Hargreaves, A. (2003). Teaching in the knowledge society: Education in the age of insecurity. Maidenhead: Philadelphia Open University Press. Knowles, M. S. (1980) The modern practice of adult education: From pedagogy to andragogy. Chicago: Follett. Meiszner, A. (2010). The emergence of free / open courses lessons from the Open Source movement. PhD thesis at Institute of Education, the Open University. Siemens, G. (2004). elearnspace. Connectivism: A Learning Theory for the Digital Age. Retrieved February 22, 2009, from http://www.elearnspace.org/Articles/connectivism.htm. Stahl, G. (2006). Group cognition: Computer support for building collaborative knowledge. Cambridge, MA: MIT Press.
ISLS
566
ICLS2012
Volume 2: Posters
UltraLite Collaboration: A Low-Cost Toolkit to Promote Collaborative Learning in the Classroom

Claire Rosenbaum, Paulo Blikstein, TLT Lab, Stanford University, CA, USA Email: rosenbaum.claire@gmail.com, paulob@stanford.edu Patti Schank, Ken Rafanan, Jeremy Roschelle, SRI International, CA, USA Email: patricia.schank@sri.com, kenneth.rafanan@sri.com, jeremy.roschelle@sri.com Abstract: The UltraLite Collaboration toolkit is a low-cost, collaborative electronic quiz book that encourages students to work together and share their reasoning when answering teacherauthored questions. Group dialog occurs in teams of three students within a school classroom. Students do not receive feedback about their individual answers, but rather about the groups level of agreement, and thus must discuss their answers. Previous work suggests significant growth in content-related dialogue using similar, but much costlier tools.
Introduction
Primary school students typically work either individually or contribute to class-wide sharing of answers (Hiebert & Grouws, 2007). Students seldom have the opportunity to explain their thought processes or listen to peer reasoning (Stigler & Hiebert, 1997). More extroverted or content knowledgeable learners often dominate unscaffoleded group discussions. UltraLite Collaboration Toolkits aim is to facilitate small group discussions where students share ideas and receive content feedback, structuring dialogue such that all students participate.
Foundational Work
This work is grounded in both learning sciences theory and the contextual classroom user testing of TechPALS handheld collaborative toolkits (Roschelle et al., 2009). The UltraLite Collaboration toolkit leverages Resnicks (1998) Digital Manipulatives concept by equipping students with physical, computationallyaugmented answer tokens that elicit feedback from the game board, and ultimately, from their peers. The nature of group interaction facilitated by UltraLite incorporates Adams & Hamms (1996) findings that consensus building discussion is only achievable in small student groups, not in large classrooms. As peers build small group connections and view each other as knowledge sources rather than competitors, students construct a climate more likely to emphasize understanding over memorization (Zurita & Nussbaum, 2004). This project drew upon previous TechPALS prototypes, which were handheld Computer Supported Collaborative Learning (CSCL) devices with a custom-designed Eduinnova platform (Roschelle et al., 2009). These platforms underwent two cycles of research, in which TechPALS was introduced into three fourth-grade California classrooms. Positive results were shown compared to the control group, with the treatment group testing significantly higher on post-test assessments (Roschelle et al., 2010). Additionally, the TechPALS toolkit group exhibited greater student collaboration along several metrics. However, major implications from these studies suggested that the overall cost of ownership should be significantly lower, the academic content needed to be more customizable for each classrooms needs, technical issues needed to be streamlined, and initial training time for students and teachers needed reduction (Roschelle et al., 2010).
Current Design
The UltraLite Collaboration prototype addresses design insights revealed by earlier work while also facilitating the same goal of collaborative classroom learning. The overall content creation and interaction platform in the UltraLite prototype differs from previous work. First, the teacher uses open source custom software to author quiz books (Figure 1, a) with tailored academic content for the needs of the classroom. Groups of three students receive an UltraLite game board (Figure 1, b) and each student gets an answer totem (Figure 1, c). Then, as a group, the students check out one of the several teacher-authored paper quiz books. After the students toggle the booklet ID (corresponding to a particular answer key) into the game board (Figure 1, e), they progress through the books content. When the group approaches a new question, each student individually submits their answer with their personal answer tag (Figure 1, c). After all students have submitted answers, the game board gives feedback in the form of colored lights (Figure 1, d). In the current interaction mode, the group as a whole receives feedback if all the submitted answers are correct (green light) or if at least one student (not identified) submitted a different answer (red light). Specific correct or incorrect answers are purposely left unidentified by the toolkit, thus encouraging group discussion before answer resubmission. This feature was seen in previous work of TechPALS and contributed to greater levels of student collaboration through higher frequencies of giving an answer, giving an explanation, reading a question aloud, making a collaborative move, directing a peer, disagreeing with another student, and asking an on-topic academic question. (Roschelle et al., 2009).
ISLS
567
ICLS2012
Volume 2: Posters
Figure 1. The UltraLite Collaboration Prototype. The teacher-authored booklet (a), game board (b), answer pieces (c), answer LEDs (d), book ID selector (e), and page selector (f). The cardboard design was used for low-cost, but many other materials are usable. The physical UltraLite Collaboration toolkit is comprised of three main components. Students use the game board (Figure 1, b) and tangible answer tags (Figure 1, c), while the teacher leverages the content generating software. The game board contains all of the programmed computational components of UltraLite Collaboration: a low-cost microcontroller and circuitry, three colored LEDs to signal feedback, a book ID selector, a page number selector, and three docking spots for students to submit their answers with an answer tag. These answer tags contain embedded resistors of three different values that correspond to one of three answer choices. For a student to submit an answer, he or she lines up the respected answer portion of his or her answer tag to the dock of the game board. This completes a simple resistor circuit that the game board logic circuit interprets as a particular submitted answer choice. Each kit runs on a 9V battery with a bill of materials around $5 $7 (USD). One of the flexibilities of the UltraLite Collaboration Toolkit is that teachers may author their own content. An open-source formatting platform enables teachers to print their own booklets of questions and answers for student interaction scaffolding. In addition to using the software to create unique quiz books, teachers may also use a set of pre-existing booklets as well. The software collates the questions in several patterns, each with a color-coded book ID code. This technique assures students do not receive the questions in the same order during every interaction, and minimizes the risk of students simply memorizing a string of answers as a workaround.
Conclusion and Further Work

The design of UltraLite Collaboration was guided by the need for ultra-low cost kits that are not only deployable in classrooms, but could also be used in museums, on field trips, and other informal settings hence the choice for a simple but functional feature set and elementary electronics. This design eliminates the complexities of deploying laptops and tablets into classrooms while offering a technological platform that could enhance teaching and collaboration is real settings, in a scalable way. The vision is that the combination of teacher authorability, low cost, ease of deployment, and visible gains in collaborative work will lower the threshold for adoption among the formal and informal educational communities.
References
Adams, D., & Hamm, M. (1996). Cooperative learning: Critical thinking and collaboration across the curriculum. Springfield, IL: Charles C. Thomas Ltd. Hiebert, J., & Grouws, D. A. (2007). The effects of classroom mathematics teaching on students learning. In F. K. Lester, Jr., (Ed.), Second handbook of research on mathematics teaching and learning (pp. 371404). Charlotte, NC: Information Age Publishing. Resnick, M., Martin, F., Robert, B., Borovoy, R., Colella, V., Kramer, K., & Silverman, B. (1998). Digital manipulatives: New toys to think with. CHI'98, ACM Press, 281-287. Roschelle, J., Rafanan, K., Bhanot, R., Estrella, G., Penuel, B., Nussbaum, M., & Claro, S. (2009). Scaffolding group explanation and feedback with handheld technology: Impact on students' mathematics learning. Association for the Educational Communications and Technology, (58), 399-419. Roschelle, J., Rafanan, K., Estrella, G., Nussbaum, M., & Claro, S. (2010). From handheld collaborative tool to effective classroom module: Embedding CSCL in a broader design framework. Computers & Education, (55), 1018-1026. Stigler, J. W., & Hiebert, J. (1997). Understanding and improving classroom mathematics instruction: An overview of the TIMSS video study. Phi Delta Kappan, 78(1), 1421. Zurita, G., & Nussbaum, M. A. (2004). Constructivist mobile learning environment supported by a wireless handheld network. Journal of Computer Assisted Learning, 20, 235-243.
ISLS
568
ICLS2012
Volume 2: Posters
Learning to Graph: A Comparison Study of Using Probe or Draw Tools in a Web-Based Learning Environment
Libby F. Gerard, Amber Zertuche, & Marcia C. Linn, UC Berkeley, 4407 Tolman Hall, Berkeley, CA 94720 Email: libby.gerard@gmail.com, anzertuche@gmail.com, mclinn@berkeley.edu Abstract: Research suggests that motion probes can significantly enhance student understanding of position/time graphs. New, online drawing tools offer opportunities to augment or even replace motion probesespecially in classrooms that allow little space for 35 students to move. We compare how 8th grade students (N=315) learned to construct, interpret and critique graphs using motion probes or drawing tools. Post-unit individual and studentpair outcome measures indicate probes and drawing activities both significantly improve student science learning. Embedded assessments illustrate unique affordances of the precision allowed with drawing tools and the immediate feedback provided by probes in online units.
Rationale
Graphing is a critical concept in the development of mathematical and scientific thinking (Leinhardt, Zaslavsky & Stein, 1990), and a key practice used in personal and policy decisions during life (National Research Council, 2010). Linking graphs to other symbolic systems in order to interpret and communicate evidence is challenging for students to learn. For instance, those who can solve graphing problems in mathematics are often unable to access this knowledge in science. Students often interpret graphs as pictures (Leinhardt et al, 1990). Previous research suggests advantages of using motion probes to support student learning about graphing (e.g. Zucker, Tinker, Staudt, Mansfield, Metcalf, 2008; Sokoloff, Laws & Thornton, 2007). New drawing tools can augment or even replace motion probes for schools with large class sizes. Prior work illustrates the potential of drawing tools to help students make connections between multiple representations of scientific concepts (Zhang & Linn, 2011). We examine the relative benefits of probes and drawing tools to support student learning about graphing in the Web-based Inquiry Science Environment (WISE, http://wise.berkeley.edu, Linn & Eylon, 2011).
Methodology
Curriculum, Study Design, Participants, Data Sources, and Scoring
This study compares an activity using motion probes to a similar activity using drawing tools embedded in a WISE inquiry unit called Graphing Stories. The unit and assessments were designed to support knowledge integration learning processes (Linn & Eylon, 2011). The probes connect seamlessly to the WISE environment, allowing students to collect, interpret, and utilize the probe data within WISE Students took approximately five 50-minute class periods to complete Graphing Stories, spending approximately two of the class periods on the probe/draw activities (see Table 1). The probe/draw activities guided students to interpret, construct and critique graphs of increasingly complex verbal descriptions of motion. Two 8th grade teachers and their 315 students participated in this study in an urban-fringe, moderately diverse (30% non-white), affluent (6% reduced lunch), California school district. Students completed 2 embedded assessment items in pairs immediately after the probe/draw activities and 2 pre/post assessments individually before and after the unit. We scored these assessments using 5-point knowledge integration rubrics to capture students increasingly integrated understanding of direction, start and end points, distances and rate in graph construction and interpretation (Liu, Lee, Hofstetter, & Linn, 2008). In addition, we utilized WISE log data, classroom video, field notes, and end-of-unit teacher interviews to generate case studies of students in each condition. The WISE log data documented each version of student drawings, critique, and explanations in the probe/draw activities. The video captured eight student pairs, four using probes and four using draw tools
Results
Students in both the probe and draw groups made significant pre/post gains in their understanding of position-time graphs (p<.001). There were no significant differences between the draw and probe groups on either the pretest or the posttest. Students in the draw group constructed more precise graphs and verbal interpretations on the embedded assessments than students in the probe condition (7.46 versus 6.5 out of 10 total points). These students graphs and verbal descriptions were more likely to accurately represent direction, start and end points, distances, and rate. Students in the probe condition alternatively tended to construct graphs and interpretations that represented the correct direction but included only roughly similar start and end points, distances, and rates. This suggests
ISLS
569
ICLS2012
Volume 2: Posters
that the draw tools encouraged students to attend to the point-by-point details of graphs (local interpretation) while the probe condition encouraged students to attend to the graphs general shape or intervals of increase or decrease (global interpretation) (Leinhardt et al, 1990). Students in the draw condition generated more non-normative ideas consistent with pictorial interpretations of graphs than students in the probe condition (McDermott et al, 1987). A substantial minority (28%) of student responses in the draw condition interpreted the graphs as a picture, describing the peaks as going up and down a hill or as velocity graphs, going fast then slow. The probe condition on the other hand elicited significantly fewer such non-normative ideas (11% of student responses). This finding suggests that the probes more effectively helped students to distinguish graphs as representations of motion rather than as literal pictures, relative to the drawing tools. Our video observations show that the immediate feedback from the probe enabled students to see that their pictorial interpretations did not result in their expected graph support this conclusion. The field notes and video suggest greater engagement by students using probes. Students who used the probes negotiated with their partner about how far to walk, how long, and how fast, while, students using the draw tools engaged in minimal if any peer discussion, choosing instead simply to alternate their attention between the verbal description of motion and graph construction. Both teachers reported that probes were more engaging to students, and according to one teacher gave the students and teachers, the feeling of doing real science. Students in the probe group struggled at times to focus on the relationship between their movement and the graphical representation, attending instead to data collection logistics and interference from others. Table 1. Student Generated Walking Instructions and Corresponding Graphs Student-Generated Instructions for Graph C* Corresponding Graph Generated Using Draw Tools at 0.5 meters, walk forward briskly for 2 seconds and you will reach 2 meters. then for 3 seconds slowly walk forward for .25 meters. Now turn around and walk back at a normal pace for another 5 seconds and 2 meters. now just stand still for 2 seconds. Student-Generated Instructions for Graph C* Start half a meter away from the motion sensor, walk 1.5 meter quickly in 2 seconds. Than slowly walk .25 of a meter in 3 seconds. Then walk 2 meters back towards the starting point in 5 seconds. Then stand still for 2 seconds. Corresponding Graph Generated Using Probe Tools
Discussion
Although drawing provides a viable alternative to using motion probes to learn about position/time graphs there are tradeoffs. Combining these approaches could strengthen outcomes from units on motion.The opportunity,to do real science remains a unique advantage of probes. Future research will explore combining these activities.
References
Leinhardt, G. Zaslavsky, O. & Stein, M-K. (1990). Functions, Graphs, and Graphing: Tasks, Learning and Teaching. Review of Educational Research, 60(1), 1-64 Linn, M.C. & Eylon, B-S. (2011). Science learning and instruction: Taking advantage of technology to promote knowledge integration. New York: Routledge. Liu, O., Lee, H., Hofstetter, C., & Linn, M. (2008). Assessing knowledge integration in science: Construct, measures, and evidence. Educational Assessment, 13(1), 33-55. Sokoloff, D., Laws, P. & Thornton, R. (2007). Real Time Physics: Active Learning Labs Transforming the Introductory Laboratory, European. Journal of Physics, 28, S83-S94. Zhang, Z., & Linn, M. C. (2011). Can generating representations enhance learning with dynamic visualizations? Journal of Research in Science Teaching, 48(10), 1177-1198 Zucker, A.A., Tinker, R., Staudt, C., Mansfield, A., & Metcalf, S. (2008). Learning science in grades 3-8 using probeware and computers: Findings from the TEEMSS II Project. Journal of Science Education and Technology, (17), 42-48
Acknowledgments
We acknowledge the support of National Science Foundation Grants DRL-0918743, DRL-1119670, and DRL0822388 in conducting this research. Views are those of the authors and not necessarily the views of the NSF.
ISLS
570
ICLS2012
Volume 2: Posters
Bifocal Biology: Combining Physical and Virtual Labs to Support Inquiry in Biological Systems
Tamar Fuhrmann, Daniel Greene, Shima Salehi, Paulo Blikstein Transformative Learning Technologies Lab (TLTL), Stanford University, 520 Galvez Mall, CERAS 232, CA, USA. tamarrf@gmail.com, dkgreene@stanford.edu, shimasalehi87@gmail.com, paulob@stanford.edu Abstract: In this paper we describe a pilot study of an approach to STEM inquiry learning called Bifocal Modeling (Blikstein, 2010) with a group of high school students studying bacterial growth. Students grew real bacteria, and then collaboratively designed a conceptual agent-based model of bacteria to mimic the observed growth. Observations and student notes suggest that the activity helped students demonstrate their knowledge of bacterial growth by formalizing it from a list of unorganized facts into an accurate pseudo-computational model. In completing their task, they also critically reflected on the assumptions built into the modeling activity itself, and demonstrated familiarity with some core principles of complex systems.
Introduction
As computational tools continue to improve, virtual models and simulations of scientific phenomena have become increasingly popular options for science learning. Alongside this trend, a new opportunity has emerged: combining and comparing virtual and physical laboratories. Research suggests that a combination of virtual models and hands-on lab activities is more effective than either alone (Liu, 2006) and that sequencing a virtual model after a physical one is more effective than the reverse (Gire et al. 2010; Smith et al. 2010). However, the literature has focused entirely on pre-designed physical and virtual models, overlooking the fact that creating and critically evaluating models is an important part of scientific practice and a valued learning outcome. (Levy & Wilensky, 2008; Blikstein 2010). Similarly, the literature has also under-explored the potential to support students comparisons between the physical and virtual models (Blikstein & Wilensky, 2007, Blikstein, 2010). Smith and collaborators (2010) noted that particular scaffolds in virtual and physical models could foster discussion about the models and their content. In this paper, we present a pilot study to address both of these concerns. Using a type of scientific inquiry activity called Bifocal Modeling, high school students designed virtual and physical models of bacterial growth in order to develop both content knowledge and critical model-evaluation skills. This study is part of a larger project that investigates how students learn by examining the differences between ideal and real systems, normally by means of building and connecting virtual computer models and physical, sensor-enabled systems.
Methods
The authors conducted a pilot study in a laboratory setting with four high school students, all female and ranging from 9th to 11th grade. Two had previously learned some general information about bacteria in class, but knew nothing about the growth patterns of bacteria. The workshop lasted for a total of about five hours, split across three afternoon sessions. In the first session, students were introduced to the goal of understanding what makes bacteria grow. They designed an experiment to compare bacteria growth by collecting samples from different parts of their environment and preparing them in a Petri dish. They also watched a time-lapse video of bacterial growth. In the second session, the students were grouped into two pairs to do web research on the bacterial growth curve and write down the variables that they thought would be relevant. For the third and final session, the authors conducted a variation of paper modeling in which students collectively designed and ran an agentbased model of bacterial growth on a whiteboard. This required articulating the variables in the model and agent-level rules. The students would run the model by enacting its rules on the board to progress the model by a single time step at a time, and then stop to add or change rules and variables. The authors scaffolded the modeling session with minimal questions. After the whiteboard activity, students interacted with pre-made computer models of bacterial growth and were led in guided inquiry about the limitations of the models.
Data and Discussion

Students were given two open-ended questionnaires about bacteria and the growth curve - once before and once after the entire session. They were also videotaped during all activities, their computer usage was documented with the screen-capture software, the researchers took field notes, and all of their notes and sketches in all three sessions were preserved. Based on these data, we observed four main themes:
Iteratively improving the virtual model to resolve mismatch
ISLS
571
ICLS2012
Volume 2: Posters
The students used a strategy of iterative debugging by running their whiteboard model, comparing the results to the behavior of the physical bacteria, and resolving the perceived differences between the two by adding rules and variables to the virtual model. They repeated this process a total of four times in the 1.5 hours of the third session, developing an increasingly accurate model in the process. For example, a student observed at one point after running the virtual model that their bacterial growth curve was unlike the real one, which only began to increase after a short delay to acclimate. They used this to come up with the following rule: If a bacterium is in the first generation, it has to wait two time steps before reproducing. Upon running the model again, the students could see from the resulting curve that they had successfully created the lag phase of real bacterial growth.
Converging on appropriate variables

During their internet search in the second session, the students wrote down a large number of variables that are irrelevant to bacterial growth. However, during the whiteboard virtual modeling session, they consciously rejected most of the variables without prompting, and only included those that were necessary to define the shape of the growth curve. This suggests that the modeling activity helped them to clarify their understanding of the process of bacterial growth by understanding the relative importance of different variables.
Critically evaluating the assumptions of models

The students in the workshop spontaneously reflected on the underlying assumptions of their models. For example, time in the whiteboard model (and in computer models) is represented as a series of discrete steps called ticks. One student realized that they had no agreed conversion between ticks and real time. She asked, Do bacteria get food and moisture each minute? how can we translate the tick into real time? Though they did not entirely resolve their questions about representing time and space in their model, the students were asking the right questions; that is, they were asking questions about the assumptions that models make about the world, which are at the heart of scientific critical thinking and modeling (Blikstein & Wilensky, 2007).
Translating between micro and macro perspectives

A final theme that emerged was the continual switching of perspectives, from the rules for an individual bacterium to the emergent behavior of its entire colony. With no prior academic knowledge of agent-based modeling or complex systems, the students in this study described and manipulated a complex system at two levels. While Bifocal Modeling is not inherently bound to a complex-systems framework, the process of modeling a phenomenon and trying to match it to a reference pattern may be an effective way to engage with complex systems.
Conclusions and next steps

Based on our observations, the opportunity to build and compare virtual and real models helped students improve and refine their theoretical models. In the process, we claim that they demonstrated content knowledge about bacterial growth and critical meta-modeling skills. Future work will keep developing Bifocal Modeling as a platform for real-time linking of physical and virtual models, and for collaborative programming with computational media.
References
Blikstein, P., & Wilensky, U. (2007). Bifocal modeling: a framework for combining computer modeling, robotics and real-world sensing. Paper presented at the Annual meeting of the American Research Education Association, Chicago, IL. Blikstein, P. (2010). Connecting the science classroom and tangible interfaces: the bifocal modeling framework. Proceedings of the 9th International Conference of the Learning Sciences, Chicago, IL. Gire, E., Carmichael, A., Chini, J. J., Rouinfar, A., Rebello, S., Smith, G., & Puntambekar, S. (2010). The Effects of Physical and Virtual Manipulatives on Students Conceptual Learning about Pulleys. Proceedings of the 9th International Conference of the Learning Sciences, Chicago, IL, pp. 937-943. Levy, S. T., & Wilensky, U. (2008). Inventing a mid-level to make ends meet: Reasoning through the levels of complexity. Cognition and Instruction, 26(1), 1-47. Liu, X. (2006). Effects of combined hands-on laboratory and computer modeling on student learning of gas laws: A quasi-experimental study. Journal of Science Education and Technology, vol. 15, pp. 89-100. Smith, G., Gnesdilow, D., & Puntambekar, S. (2010). Examining the Combination of Physical and Virtual Experiments in an Inquiry Science Classroom. Proceedings of the Computer Based Learning Science Conference.
ISLS
572
ICLS2012
Volume 2: Posters
Design-Based Research in Practice: A technology-based classroom experiment that explores how students use interactive virtual representations to order groups of fractions
Maria Mendiburo, Gautam Biswas, and Ted Hasselbring, Vanderbilt University, Nashville, TN. USA. Email: {maria.mendiburo, gautam.biswas, t.hasselbring} @vanderbilt.edu Abstract: This paper reports the results of a mixed-methods classroom experiment that examines how students use interactive virtual representations to order groups of fractions. The results of the experiment suggest that students can learn to construct virtual representations of fractions with minimal instruction, but their ability to construct virtual representations of fractions is not always aligned with their ability to correctly order groups of fractions represented symbolically.
Introduction
In June of 2011, we began the second year of a design-based research project called Helping At-Risk Students Learn Fractions (HALF). The long-term goal of the project is to design a fractions intervention curriculum that adapts assessments to students level of understanding, and then uses interactive virtual representations within a proven classroom instructional model to help students improve their conceptual understanding of fractions. We conducted this experiment to understand if students were able to use the HALF computer system to construct correct virtual representations of fractions and if constructing virtual representations of fractions improved students ability to order groups of fractions represented symbolically. The experiment also investigated the correct and incorrect strategies students use to order fractions.
Background
USA students lack of understanding of fractions concepts is well documented (e.g. National Mathematics Advisory Panel, 2008). To address this problem, the Rational Number Project (RNP) team conducted a large number of investigations over a period of two decades and then used the results of those investigations to design an experimental, manipulative-based fractions curriculum. The curriculum placed particular emphasis on the use of multiple physical models and translations within and between pictorial, verbal, real world, and symbolic representations of fractions. An evaluation of the curriculum with over 1600 fourth and fifth graders showed that students who used RNP materials significantly outperformed students in a control condition on four of six subscales: concepts, order, transfer, and estimation (Cramer, Post, & delMas, 2002). Despite the positive impact of manipulative-based instruction, teachers often struggle to implement manipulatives in classrooms due to a variety of practical and pedagogical difficulties. For example, manipulative pieces often become lost or broken, and in a typical classroom of twenty or more students, teachers cannot monitor the activity of every student while they are using the manipulatives. As a result, teachers may not be able to provide students with appropriate goals for using the manipulatives or provide formative feedback. However, many of these practical and pedagogical difficulties may be eliminated if teachers use virtual rather than physical manipulatives (Clements & McMillan, 1996). We compared virtual and physical fractions manipulatives in previous research, and our results showed that virtual manipulatives are as effective as physical manipulatives but are significantly more time-efficient (Mendiburo, 2010). Therefore, we chose to include interactive virtual representations rather than physical manipulatives in the HALF fractions intervention.
Methodology
We conducted the two-day experiment reported in this paper in three classrooms of mixed-ability middle school students located in Middle Tennessee. The students were introduced to the interactive virtual representations and then asked to complete three sets of practice problems that required students to order groups of four fractions represented symbolically. The computer system required students to construct virtual representations prior to answering each of the first five problems. During the second set of five problems, students were allowed to choose whether or not they wanted to construct virtual representations prior to answering each problem. The system required students to solve the last five problems without constructing virtual representations. A researcher collected field notes while the students completed all three sets of practice problems, and the computer system collected log data. We used the data from the computer log files to select ten students with different performance characteristics to interview individually. During the interviews, we asked the students questions intended to help us understand their perceptions of the functionality of the interactive virtual representations and their usefulness as a tool for solving fractions problems. We also asked the students to talk aloud as they solved practice problems with and without the interactive virtual representations.
ISLS
573
ICLS2012
Volume 2: Posters
Results
The results of the experiment suggest that students can learn to build virtual representations of fractions with minimal instruction, but their ability to build virtual representations of fractions is not always aligned with their ability to correctly order groups of fractions represented symbolically. For all of the students in this experiment, the percentage of correct virtual representations when the students were required to construct virtual representations and when the students had the option to construct virtual representations was 83.4% and 87.0% respectively. In contrast, the percentage of correct responses to the practice problems under these two conditions was 56.6% and 61.3% respectively. This suggests that some students who quickly learned to construct virtual representations could not make accurate connections between the representations and the correct answers to the practice problems. During the individual interviews, students stated that they felt the tool we designed was easy to use to create virtual representations, and they had few suggestions for ways to improve the tool. However, the results of the two problems we asked each student to solve by talking aloud during the interview align with the assessment results; they show that students who construct accurate virtual representations of fractions vary substantially in their ability to correctly order fractions represented symbolically. Analysis of students solutions revealed multiple correct and incorrect strategies for comparing fractions, and since the questions were multistep problems that required comparing four fractions rather than just two fractions, the response to a single question given by a single student often included multiple strategy codes. The most common correct strategy code was the reference point strategy, in which students relate the size of two fractions to a third number (Behr et. al, 1984 & Cramer et. al, 2002). We found the students who used the reference point strategy typically used one-half as the reference point. The students teacher told our research team prior to the start of the intervention that she heavily emphasized understanding the relative size of one-half in comparison to other fractions in her most recent teaching about fractions, which probably explains the predominant use of the reference point strategy in the students responses. However, two of the three students who answered the questions incorrectly also used the reference point strategy, but they applied the strategy incorrectly. These students answers showed evidence of the whole number dominance strategy a very common incorrect strategy in which students compare fractions by making separate comparisons of the numerators and the denominators using the ordering of whole numbers (Behr, et. al, 1984).
Discussion
In future versions of the system, we plan to include instruction in the system that emphasizes the inverse relationship between the numerator and denominator of a fraction and shows that the whole number dominance strategies is an incorrect method for reasoning about fractions. We plan to scaffold this instruction for students of different ability-levels, and we plan to differentiate instruction based on teacher or computer-based assessment of the particular correct and incorrect strategies used by a student when making comparisons between fractions. The computer will collect data about students actions with the manipulatives during each phase of instruction, and it will add and remove scaffolds to the virtual representations using the formative data it collects. With these improvements, we expect the system to have a significant positive impact on students conceptual understanding of fractions.
Acknowledgements
The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305A100110 to Vanderbilt University. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.
References
Behr, M., Wachsmuth, I., Post T., & Lesh R. (1984). Order and equivalence of rational numbers: A clinical teaching experiment. Journal for Research in Mathematics Education, 15(5), 323-341. Clements, D., & McMillan, S. (1996). Rethinking 'concrete' manipulatives. Teaching Children Mathematics, 2(5), 270-279. Cramer, K., Post, T., & delMas, R. (2002). Initial fraction learning by fourth- and fifth-grade students: A comparison of the effects of using commercial curricula with the effects of using the Rational Number Project curriculum. Journal for Research in Mathematics Education, 33(2), 11-144. Retrieved from http://www.jstor.org.proxy.library.vanderbilt.edu/ Mendiburo, M. (2010) Virtual manipulatives and physical manipulatives: Technologys impact on fraction learning. (Unpublished doctoral dissertation). Peabody College of Vanderbilt University, Nashville. National Mathematics Advisory Panel. (2008). Foundations for success: The final report of the National Mathematics Advisory Panel. Retrieved from www2.ed.gov/about/bdscomm/list/mathpanel/report/final-report.pdf
ISLS 574
ICLS2012
Volume 2: Posters
Learning by Collaborative Design and Evaluation

Shannon Kennedy-Clark1, shannon.kennedy-clark@acu.edu.au; Vilma Galstaun2, vilma.galstaun@sydney.edu.au; Martin Parisio2, martin.parisio@sydney.edu.au; Kate Anderson2, k.anderson@sydney.edu.au 1. Learning and Teaching Centre, Australian Catholic University, Sydney, Australia, 2. Faculty of Education and Social Work, University of Sydney, Sydney, Australia, Abstract: This paper presents the preliminary findings of a learn-technology-by-design conceptual framework in an ICT in education unit of study. One-hundred and twenty preservice teachers participated in the study. Data sources included annotated video feedback, reflective journals and pre- and post-tests. Initial findings demonstrate that the design process deepened the pre-service teachers understandings of the affordances of Internet technology and that pre-service teachers had increased their technological, pedagogical and content knowledge on completion of the course.
Introduction
This paper presents the preliminary findings of a learn-technology-by-design study aimed at developing preservice teachers technological, pedagogical and content knowledge (TPACK). The limited and inadequate amount of training that pre-service teachers often receive before entering a classroom means that, in many cases, they feel that they do not have the technical support, the skills, and a pedagogical rationale for implementing information and communication technologies (ICT) in the classroom (Hu, Wong, Fyfe, & Chan, 2010; Lee, 1997; Mishra & Koehler, 2006). Concomitantly, pre-service teachers have been shown to lack the confidence to use ICTs in the classroom (Angeli, 2004; Lee, 1997). This paper discusses how technology-mediated peer review helped pre-service teachers improve their understanding of the affordances of the Internet for student learning and the issues involved in designing web-based activities that promote learning.
Research Design
We used a collaborative approach in both the design of the tasks and the critical feedback through a peer review process. Mishra and Koehler (2006) advocate an approach of learn-technology-by-design for preparing for technology integration. Design-based learning activities are usually carried out in groups and pre-service teachers develop deeper understanding through the experiences of both dialogue and reflection in action. Rather than providing a generic skills course, tutorial sessions were designed so they adapted to the learning needs of the pre-service teachers. This allowed the focus of the lesson to be on the use of innovative tools to model effective ICT integration in teaching. Learn-technology-by-design tasks are accomplished in the environments where pre-service teachers feel comfortable working collaboratively and are encouraged to use ICT tools to build a learning environment where pre-service teachers could benefit from peer interactions and working collaboratively. There have been numerous studies on cooperative and collaborative learning which have focused on benefits, such as motivation, social cohesion, and higher levels of learning (Johnson & Johnson, 2002; Slavin, 1996). We adopted an ADDIE (Analysis, Design, Development, Implementation, and Evaluation) (Dick, Carey, & Carey, 2001) model for the learn-technology-by-design task to guide the design process.
Data Collection and Analysis

This study involved two different cohorts of 4th year pre-service teachers in Bachelor of Education degrees: 14 pre-service teachers in the primary education program and 106 pre-service teachers in the secondary combined degrees program. Pre-service teachers attended their respective courses for 2 hours a week over a 10 week period during Semester 2, 2011. The task required pre-service teachers to develop a teaching and learning web resource. The design task was carried out via two separate activities that were completed in pairs (Figure 1). In order to accomplish the design task, pre-service teachers would need first to become familiar with the school curriculum and their subject area. They identified a specific topic or concept that they believed to be difficult for teachers to teach and for students to learn. The first activity required pre-service teachers to develop a website using the Google Sites tool. The pre-service teachers then evaluated two of their peers websites. The pre-service teachers were provided with a scaffold to guide meaningful and effective feedback to their peers about their web resource. The evaluations were recorded using Camtasia screen capture software, which recorded the audio, webcam and onscreen movements. The recordings were uploaded to a video annotation platform (EVA). There were four stages of the data analysis. Firstly, pre-and post-test TPACK surveys were administered and were measured for differences. Secondly, pre-service teachers reflective journals were analyzed using a constant comparative method. Thirdly, the video recordings were transcribed and then coded to
ISLS 575
ICLS2012
Volume 2: Posters
determine patterns and themes. Finally, on the basis of the preceding steps, the participants ICT literacy was examined and a narrative of the main results was constructed. Collaborative Learn-technology-by-design
1. Storyboard/planning 2. Website development 3. Peer video feedback 4. Annotation of peer feedback 5. Revision of website 6. Submission of website
Individual ongoing reflection and evaluation of process Individual reflection and evaluation of collaborative design process Figure 1. Workflow of collaborative and individual components of the assessment.
Preliminary Findings and Conclusions

The courses described in this paper tried to prepare pre-service teachers to be fluent in both pedagogy and basic design theory so that they may use newly acquired ICT skills to produce educationally sound web-based learning resources (Hu, et al., 2010). This study extends earlier research by Hu, et. al. (2010), who found that the pre-service teachers became too close to their products to detect the inadequacies of their design. As shown in this study, the feedback obtained from peer assessment helped pre-service teachers modify their designs. The process worked as a source of support, ideas and encouragement crucial for the accomplishment of the set task. Such an experience should encourage pre-service teachers to value peer review for both pre-service education and future professional growth. Pre-service teachers displayed engagement in the activities of recording evaluations and watching peer videos for the peer assessment task. The capturing of peer assessment in the form of video and audio created a novelty aspect within the task that was reflected in the pre-service teachers engagement in the task. Preservice teachers were able to access the video evaluations off-campus and this garnered a positive response. The way ICT was integrated in the teaching and learning processes of the course meant we were able to demonstrate its effectiveness. Moreover, pre-service teachers were able to experience the tools and pedagogical strategies firsthand, which is supported by Merrills (2002, p. 43) assertion that learning is best promoted when new knowledge is demonstrated to the learner. The ADDIE model (Dick, et al., 2001)used in the project task to guide pre-service teachers through the development of their web-based resource strongly focuses on evaluation. This aspect of the process was significant in that pre-service teachers learned about evaluating their designed artifacts [resources] and the design process itself. Above all, we have been able to evaluate the integration of ICT approach adopted in the design of the course through the administration of pre-and post-tests adapted from Schmidt et al.s (2009) instrument. Early analysis has shown significant increases in pre-service teachers TPACK. This research is noteworthy on a practical level because it demonstrates successful student engagement with ICTs to support and enhance learning.
References
Angeli, C. (2004). The effects of case-based learning on early childhood pre-service teachers' beliefs about the pedagogical uses of ICT. Journal of Educational Media, 29(2), 139 -151. Dick, W., Carey, L., & Carey, J. O. (Eds.). (2001). The systematic design of instruction. (5th ed.). New York: Addison-Wesley. Hu, C., Wong, W. Y., Fyfe, V., & Chan, H. (2010). Formative Evaluation via Technology-Mediated Peer Assessment. Paper presented at the World Conference on Educational Multimedia, Hypermedia and Telecommunications 2010 Toronto, Canada. Johnson, D. W., & Johnson, R. T. (2002). Learning Together and Alone: Overview and Meta-analysis. Asia Pacific Journal of Education, 22(1), 95 - 105. Lee, D. (1997). Factors influencing the success of computer skills learning among in-service teachers. British Journal of Educational Technology, 28(2), 139 -141. Merrill, M. (2002). First principles of instruction. Educational Technology Research and Development, 50(3), 43-59. Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A framwork for integrating technology in teacher knowledge. Teachers College Record, 108(6), 1017-1054. Schmidt, D., Baran, E., Thompson, A. D., Mishra, P., Koehler, M. J., & Shin, T. S. (2009). Technological Pedagogical Content Knowledge (TPACK): The Development and Validation of an Assessment Instrument for Preservice Teachers. Journal of Research on Technology in Education, 42(2), 123-149. Slavin, R. E. (1996). Research on Cooperative Learning and Achievement: What We Know, What We Need to Know. Contemporary Educational Psychology, 21(1), 43-69.
ISLS
576
ICLS2012
Volume 2: Workshop Abstracts
Bifocal Modeling: Combining Virtual and Physical Experiments in Real Time Using Low-cost Sensors and Open-source Computer Modeling
Paulo Blikstein, Tamar Fuhrmann, Daniel Greene, Shima Salehi, Claire Rosenbaum, Marcelo Worsley Transformative Learning Technologies Lab (TLTL), Stanford University, 520 Galvez Mall, CERAS 232, CA, USA [paulob, dkgreene, mworsley] @stanford.edu, tamarrf@gmail.com, shimasalehi87@gmail.com, rosenbaum.claire@gmail.com The past two decades have witnessed a true scientific revolution: Science is becoming computational. A recent report from the National Science Foundation on Simulation-Based Engineering Science states that computing is an indispensable tool for resolving a multitude of scientific and technological problems facing our country (Oden, et al., 2006). Computer simulation is not just a verification or visualization tool anymore it can generate first level hypotheses and give scientists new lenses into natural phenomena. It greatly improves not our sight (as does the microscope), but rather our insight into mechanisms within complex systems. (Vicsek, 2002). As computational tools continue to improve, virtual models and simulations of scientific phenomena have become increasingly popular options for inquiry science. Whereas the practice of using virtual models and sensors in realtime might already is familiar in scientific circles, exploring how to bring this powerful technique to classrooms is a relatively new, but promising, endeavor. Bringing the use of simulation and virtual models from the research laboratory to the classroom has been a popular practice amongst education designers. Virtual models are used in schools, to display data, to simulate complex phenomena, and to enable student experimentation in domains that might be costly, impractical, or dangerous (Jaakkola & Nurmi, 2008, Finkelstein et al., 2005, Klahr, Triona & Williams, 2007, Zacharia 2008a,b, Resnick & Wilensky, 1998). The potential of a combination of virtual and physical models for science learning has been documented for a wide range of ages and domains (Liu et al. 2006, Jaakkola & Nurmi, 2008, Zacharia, 2010,). For example, Liu et al. (2006) showed that the combined computer modeling and hands-on laboratories were more effective than either alone in promoting students conceptual understanding of the relationship between temperature and pressure in gases. Jaakkola et al. (2008) showed that the simulationlaboratory combination environment led to learning gains compared to either simulation or laboratory activities alone in electricity. Zacharia (2010) examined undergraduate students conceptual understanding of heat and temperature, and his results indicated that when students experimented with the combination of physical manipulative and virtual manipulative, they showed a greater degree of conceptual understanding than those that used physical manipulative alone. Recent studies also suggest that sequencing physical manipulation activities before virtual activities is superior to the opposite order, e.g., Gire et al. (2010) showed that students who used real pulleys before the simulation achieved higher scores on questions having to do with force and mechanical advantage. However, traditional virtual and physical experiments do not communicate with each other; most studies present the two activities as separate entities that simply follow each other. Virtual models and simulations are typically unaffected by real-time physical input, and they are separated in time and space from typical lab activities. Bifocal Modeling is a platform and pedagogical approach created to address that gap (Blikstein & Wilensky 2005, 2006a, 2007; Blikstein, 2010). Bifocal Modeling (BM) that provides students with the opportunity to build and refine their own physical and computer models alongside real-world sensors data. When building a bifocal model, students have three main tasks. First they construct a physical model to explore a scientific phenomenon, using electronic sensors and robotic materials to build their own automated data-gathering science lab. Second, they design their virtual computer model of the same phenomenon using computer modeling software. Finally, there are two options for the real time connection. Either students (a) compare both the physical model and the virtual system to validate and debug their initial model and hypotheses, or (b) they use the physical data to drive the computational experiment. Using special hardware interfaces or camera-based sensing, students can see side by side the results of both models, easily contrasting them. We have developed such models in physics (pendulum, heat transfer, Newtonian motion, fluid dynamics, electricity), chemistry (gas laws, diffusion, chemical reactions), and biology (bacteria growth, population growth). A core component of the BM framework is that students engage in not only building physical and virtual models, but also connecting them and investigating their commonalities and differences. BM framework (Blikstein & Wilensky, 2006b, c) merges three tools that have been successfully (but separately) used in STEM education, both in formal and informal settings: computer modeling, probeware (electronic sensors), and robotics. One crucial aspect of BM is the development of low-cost, easily deployable sensing technologies. We are developing a range of open-source hardware and software to make the deployment of such activities feasible in public schools.
ISLS
577
ICLS2012
Research on BM has shown that the real-time connection between real and physical models is beneficial in a number of ways, for example, offering multiple entry points (i.e., model first or hands-on experiments first), scaffolding complex reflection and argumentation about scientific models, and allowing students to develop crucial STEM skills such as meta-modeling, identifying and quantifying relevant parameters and variables, understanding error, precision, and time/scale discrepancies between real and ideal systems.
References
Blikstein, P., & Wilensky, U. (2005). NetLogoLab curriculum. Evanston, IL: Center for Connected Learning and Computer Based Modeling, Northwestern University. Retrieved from http://ccl.northwestern.edu/curriculum/NetLogoLab Blikstein, P., & Wilensky, U. (2006). A Technological Platform for Trans-Media Scientific Exploration. Paper presented at the Interaction Design for Children Conference 2006, Tampere, Finland. Blikstein, P., & Wilensky, U. (2006). 'Hybrid modeling': Advanced scientific investigation linking computer models and real-world sensing. Paper presented at the Seventh International Conference of the Learning Sciences, Bloomington, USA. Blikstein, P., & Wilensky, U. (2006). The Missing Link: A Case Study of Sensing-and-Modeling Toolkits for Constructionist Scientific Investigation. Paper presented at the 6th IEEE International Conference on Advanced Learning Technologies (ICALT 2006), Kerkrade, The Netherlands. Blikstein, P., & Wilensky, U. (2007). Bifocal modeling: a framework for combining computer modeling, robotics and real-world sensing. Paper presented at the Annual meeting of the American Research Education Association, Chicago, IL. Blikstein, P. (2010). Connecting the science classroom and tangible interfaces: the bifocal modeling framework. Paper presented at the Proceedings of the 9th International Conference of the Learning Sciences Volume 2, Chicago, Illinois. Finkelstein, N. D., Adams, W. K., Keller, C. J., Kohl, P. B., Perkins, K. K., Podolefsky, N. S. Reid, S. and LeMaster, R. (2005). When learning about the real world is better done virtually: A study of substituting computer simulations for laboratory equipment, Physical Review Special Topics - Physics Education Research, vol. 1, pp. 010103-010110. Gire, E., Carmichael, A., Chini, J., Rebello, S., & Puntambekar, S. (2010). The Effects of Physical and Virtual Manipulatives on Students Conceptual Learning About Pulleys. Proceedings of the 9th International Conference of the Learning Sciences. Jaakkola, T., & Nurmi, S. (2008). Fostering elementary school students' understanding of simple electricity by combining simulation and laboratory activities. Journal of Computer Assisted Learning, 24(4), 271283. Klahr, D., Triona, L. M., & Williams, C. (2007). Hands on what? The relative effectiveness of physical versus virtual materials in an engineering design project by middle school children. Journal of Research in Science Teaching, 44(1), 183-203. Liu, X. (2006) Effects of combined hands-on laboratory and computer modeling on student learning of gas laws: A quasi-experimental study. Journal of Science Education and Technology, vol. 15, pp. 89-100. Oden, J. T., T. Belytschko, et al. (2006). Simulation-Based Engineering Science: Revolutionizing Engineering Science Through Simulation. N. S. Foundation. Papert, S. (1980). Mindstorms : children, computers, and powerful ideas. New York, Basic Books. Resnick, M., & Wilensky, U. (1998). Diving into Complexity: Developing Probabilistic Decentralized Thinking Through Role-Playing Activities. Journal of the Learning Sciences, 7(2), 153-171. Schweikardy, E., and Gross, M., D.(2006). roBlocks: a robotic construction kit for mathematics and science education" ACM New York, NY, USA . Resnick, M., Berg, R., and Eisenberg, M. (2000). Beyond Black Boxes: Bringing Transparency and Aesthetics Back to Scientific Investigation Journal of Learning sciences, Volume 9, Issue 1. Vicsek, T. (2002). The bigger picture. Nature 418: 131. Zacharia, Z. C., Olympiou, G. and Papaevripidou, M. (2008).Effects of experimenting with physical and virtual manipulatives on students' conceptual understanding in heat and temperature. Journal of Research in Science Teaching, vol. 45, pp. 1021-1035. Zacharia, Z. C. & Constantinou, C. P. (2008). Comparing the influence of physical and virtual manipulatives in the context of the Physics by Inquiry curriculum: The case of undergraduate students conceptual understanding of heat and temperature. American Journal of Physics, 76(4&5), 425-430. Zacharia, Z. C. and Olympiou, G. "Physical versus virtual manipulative experimentation in physics learning," Learning and Instruction, 2010.
ISLS
578
ICLS2012
Classroom orchestration: Moving beyond current understanding of the field

Yannis Dimitriadis, University of Valladolid, Valladolid, Spain, yannis@tel.uva.es Pierre Dillenbourg, EPFL, Lausanne, Switzerland, pierre.dillenbourg@epfl.ch Miguel Nussbaum, Pontificia Universidad Catlica de Chile, Chile, miguel.nussbaum@gmail.com Chee-Kit Looi, Nanyang Technological University, Singapore, cheekit.looi@nie.edu.sg Jeremy Roschelle, Stanford Research Institute International, USA, jeremy.roschelle@sri.com Abstract: This paper describes the main features of a pre-conference workshop on Classroom orchestration, which aims to consolidate current understanding of the field. This workshop aims to deepen the communitys shared knowledge on the issues related to productive coordination of supportive interventions, across multiple learning activities through multiple tools occurring at multiple social levels. An analysis through case studies of classroom orchestration may provide a revised map of the field and provide cues for support to teachers and other significant stakeholders.
General description
This half-day workshop is related to the ICLS 2012 theme of The Future of Learning since the sustainability of innovative teaching and learning practices highly depends on effective scenarios in the complex classroom ecosystem (Luckin, 2008), integrating multiple activities and tools at multiple social planes and contexts. Such multiplicity demands a holistic perspective implementing a flexible orchestration layer, either through technological or conceptual tools. The Assessment & Teaching of 21st Century Skills project (http://atc21s.org) has among others the aims of developing in students communication and collaboration, critical thinking and problem solving, creativity and innovation, flexibility and adaptability, initiative and self direction, productivity and accountability, and leadership and responsibility. The question that arises is not only how do we change the curriculum and classical assessments to reach these goals but also how do we organize the classroom, the available resources and the teachers role to achieve this transformation (Dimitriadis, 2011). Thus we need to know how all these processes are scaffolded within a classroom and how does the teacher organizes her time and the available resources to cover curricular needs, maintain the classroom rhythm and create collaboration spaces, while permitting the students to learn, think and reflect at their own pace. Orchestration has emerged recently as a driving concept and metaphor, which reflects some of the aforementioned challenges for the future of learning, especially the one supported by technology or focusing in collaborative activities (Dillenbourg, Jrvel, & Fischer, 2009). The complexity of the related issues has motivated a vivid debate in the community that took form as: a. b. c. d. A report (Dillenbourg, 2011) of the major European on orchestration, which is considered as one out of three great challenges by the Stellar European Network of Excellence (NoE) on Technology Enhanced Learning The workshop (Nussbaum, Dillenbourg, Fischer, Looi, & Roschelle, 2011) on How to integrate CSCL in classroom: Orchestration organized with extraordinary success at the CSCL 2011 conference in Hong Kong A collective paper, currently under review at the journal of Computers and Education by a significant number of senior researchers including a collection and synthesis of answers to a position paper by P. Dillenbourg on Design for Orchestration A high number of papers in the related literature that employ and explore the concept of orchestration, that were partially analyzed in a recent synthesis (Prieto, Holenko Dlab, Gutirrez, Abdulwahed, & Balid, 2011)
Such a debate has shown some convergence on the characteristics and graphical representations of integrated pedagogical scenarios, as well as the need to empower teachers in the real-time management of such scenarios, so that they can flexibly enact designs and technologies (either ICT-based or not) in a minimalist way, taking into account the practicalities and time-space constraints of classrooms. On the other hand, several issues are still subject of discussion, as e.g. the power and limitations of the underlying metaphor, the design of adequate orchestration and orchestrable technologies, the right balance between planning and enactment, external and internal scripts, or the ways to advance towards a better usability at the classroom level. This workshop aims to build on the considerable recent advances on classroom orchestration in the community of learning sciences, and especially in the field of Technology Enhanced Learning. Thus, it is
ISLS
579
ICLS2012
expected to deepen the discussion and analysis of the divergent and convergent aspects, expecting to shed light on the concept, issues and eventual approaches towards effective integration of pedagogical scenarios in current and future classroom ecosystems. On the other hand, it aims to provide a common ground with other related fields, as e.g. the field dealing with Learning Design and Pedagogical Patterns that explores principles, methods and tools for sound pedagogical design, or the field of Human Computer Interaction that aims to provide modest technologies for data mining, visualization and regulation actions during a flexible enactment of learning designs. The workshop is based on significant scenarios submitted by the participants and analyzed by senior researchers from the viewpoint of orchestration, aiming to provide a revised landscape of orchestration and the associated challenges and issues. It is expected that the workshop will produce: a. b. c. d. Critical analysis and collective synthesis regarding case studies described in the accepted papers from the orchestration point of view Production of a new concept map related to the main issues on orchestration Generation of a synthesis paper based on the workshop proceedings after the workshop Eventual joint publication of the papers presented in the workshop and an overview of the current trends on orchestration in an international journal or book, to be defined
References
Dillenbourg, P. (2011). Trends in orchestration. Second research and technology scouting report. Stellar European Network of Excellence (NoE) on Technology Enhanced Learning deliverable D1.5. Published on-line on August 18, 2011 and retrieved on March 16, 2012 from http://www.stellarnet.eu/kmi/deliverables/20110818_stellar___d1.5___trends-in-orchestration.pdf. Dillenbourg, P., Jarvela, S. & Fischer, F (2009). The evolution of research on computer-supported collaborative learning: from design to orchestration. In Balacheff, S. Ludvigsen, T. de Jong, A. Lazonder & S. Barnes (Eds.) Technology enhanced learning: Principles and Products. (pp. 3-20). Berlin:Springer. Dimitriadis, Y. (2011). Supporting teachers in orchestrating CSCL classroom. In A. Jimoyannis (Ed.) Research on e-learning and ICT in education. (pp. 71-82), Berlin:Springer. Luckin, R. (2008). The learner centric ecology of resources: A framework for using technology to scaffold learning. Computers & Education, 50(2), 449-462. Nussbaum, M., Dillenbourg, P., Fischer, F., Looi C-K., & Roschelle J. (2011). How to integrate CSCL in classroom life: orchestration. In Proceedings of the 2011 CSCL Conference vol. 3, 1199, Hong-Kong, China. Prieto, L.P., Holenko Dlab, M., Gutirrez, I., Abdulwahed, M. & Balid, W. (2011). Orchestrating technology enhanced learning: a literature review and a conceptual framework, Int. J. Technology Enhanced Learning, 3(6), 583598.
ISLS
580
ICLS2012
Re-thinking Roles in Future Learning: Teachers as Curriculum Media Designers and Students as Teacher Collaborators
Eric Hamilton, Pepperdine University (eric.hamilton@pepperdine.edu) Gina Chaves, Los Angeles Charter Alliance (gchaves@laalliance.org) Wendy Chaves, Los Angeles Charter Alliance (wchaves@laalliance.org) Israel Ramirez, Green Dot Schools (ram71_98@yahoo.com)
Overview
This workshop seeks an exchange of approaches to formal learning environments that rely on different roles, agency, participation and design. Under what conditions or approaches can teachers and learners undertake different identities and practices en route to productive and personalized learning communities? The organizing team will share research that explores intergenerational collaboration around teaching and learning, especially by which students collaborate with teachers in creating digital media that can scaffold peer learning. The primary but not exclusive area of research is mathematics. The work that will seed this discussion, shared by both teachers and researchers, focuses on means by which creativity around cognition and content (how can digital media be imaginatively used to enhance content learning) can result in routinely immersive or engrossing flow experiences in learning, visualizing, or otherwise representing scientific or mathematical ideas or behavior. The intended audience includes those exploring issues related to immersion, flow, creativity, and intergenerational collaboration. The session will seek vigorous theoretical and empirical contributions from attendees. Over two years of observation as teachers functioning in more creative roles suggest deeper immersion in content, more reflective activity in unpacking student misconceptions, and greater coherence, imagination and subtlety in presentation. Similarly powerful dynamics have been observed as students collaborate with teachers in digital media making. As students collaborate with teachers in content-based media making, interesting intergenerational dynamics consistently emerge. Students see themselves differently--as capable of handling teaching responsibilities, for example. This is a significant identity shift that stimulates and requires new patterns of self-explanation, digital representation of mathematical ideas, and formulation of multiple viewpoints of mathematical structure. This identify shift also stimulates and requires different forms of collaborative receptiveness, in the form of giving and receiving trusted critiques to and from teachers, investment in the success of peers in learning, and new motivational dynamics. In these roles, students report seeing subject matter nuances differently, as subtleties not only to understand but to clarify to others through digital expression. They see the teacher differently. Student-tutors, like their teachers, formulate unanticipated and inventive moves.
Additional Pathways
The use of design as a forum for shifting roles and identities for teachers and students in learning environments is only one avenue of an array of possibilities for future learning environments. The workshop seeks contributions from those whose research similarly implies conceptual and sophisticated shifts in how students and teachers interact with each other in the formation of high performance and personalized learning communities.
Acknowledgments
The authors wish to thank both the US Department of Educations Institute for Education Sciences and the US National Science Foundation for support of the research that is the basis for this workshop. We additionally wish to acknowledge the support of Techsmith, Inc., for contributions of software licenses to support this research.
ISLS
581
ICLS2012
Teachers as Designers of Technology-Enhanced Learning Materials

Yael Kali, University of Haifa, Mt. Carmel, 31905, Haifa, Israel. Email: yael.kali@edtech.haifa.ac.il Susan McKenney, Open University of the Netherlands & Twente University, PO Box 2960, 6401DL Heerlen, the Netherlands. Email: susan.mckenney@ou.nl Abstract: Research on factors affecting curriculum implementation has pointed to the importance of involving teachers, to varying degrees, in shaping the learning scenarios in their own classrooms. While the benefits of Teachers as Designers (TaD) are acknowledged in literature, far less is known about ways of shaping that involvement to yield those benefits. Research is needed to understand how teachers learn through design, how such activities may be supported, and how teacher involvement in design partnerships with researchers impacts the quality of the artefacts created, their implementation, and ultimately, student learning. This workshop speaks to that need by bringing together researchers and practitioners interested in further exploring various TaD aspects.
Scientific foundations
Teacher participation in design can take the form of near-to-practice involvement in the form of critical reflection on and redesign of ones own practice (Raval, McKenney, & Pieters, 2010); evidence-based customization (Gerard, Spitulnik,& Linn, 2010); or teacher design teams collaborating within an educational affiliation (Voogt, Almekinders, Van den Akker, & Moonen, 2005). Yet teacher participation may also be realized as creating exemplary materials to be used in the classrooms of others (McKenney, 2005) sometimes designed in multi-professional expert teams (Kali, Markauskaite, Goodyear, Ward, 2011). The Teachers as Designers (TaD) line of research is recently gaining increased interest as free online tools that enable simple authoring (e.g. Google Apps) are becoming widespread, and new authoring environments and pedagogical design guidelines for technology-enhanced learning are provided by the Learning Design community (The Learning Design Grid, 2012) Teacher involvement in educational design stands to yield multiple benefits. First, teachers find reflection on and re-design of their own practice insightful (Davis & Varma, 2008). Consistent with the notions of constructionism (Harel & Papert, 1991) and learning by design (Kolodner et al., 2003), this process, if appropriately sculpted, can contribute to teacher professional development (George & Lubben, 2002; Kali, & Ronen-Fuhrmann, 2011; Mishra & Koehler, 2005). Second, teacher involvement in curriculum design can positively impact the quality of implementation infusing a healthy reminder of practical realities into the design team ambitions (Knings, Brand-Gruwel, & van Merrinboer, 2007), and/or increasing ownership and commitment for implementation (Carl, 2009). Third, high-quality teacher involvement in curriculum and instructional design or customization can yield improvement of student learning (Corcoran & Siladner, 2009; Gerard, Spitulnik,& Linn, 2010). While the benefits of teacher involvement in educational design are acknowledged in literature, far less is known about ways of shaping that involvement to yield those benefits. Research is needed to understand how teachers learn through design; how teacher design activities may be supported; and how teacher involvement in design impacts the quality of the artifacts created, their implementation, and ultimately, student learning. Existing conceptual foundations for such work are urgently in need of bolstering, and will definitely play a critical role in the future of learning. This workshop speaks to that need by bringing together researchers and practitioners interested in further exploring various TaD aspects. Participants will share existing TaD research and practice, discuss areas needing additional research, and actively engage in synthesis activities. With the ultimate aim of improving the quality and relevance of research related to this theme, this workshop will help generate a conceptual foundation for understanding the notion of TaD, to be shared among a broader community, possibly via a special issue of a scientific journal.
ISLS
582
ICLS2012
References
Carl, A. E. (2009). Teacher empowerment through curriculum development: theory into practice: Juta & Company. Corcoran, T., & Siladner, M. (2009). Instruction in high schools: The evidence and the challenge. The Future of Children,19(1), 157-183. Davis, E. A., & Varma, K. (2008). Supporting teachers in productive adaptation. In Y. Kali, M. C. Linn, M. Koppal & J. E. Roseman (Eds.), Designing Coherent Science Education: Implications for curriculum, instruction, and policy (pp.94-122). N.Y.: Teachers College Press. George, J. M., & Lubben, F. (2002). Facilitating teachers' professional growth through their involvement in creating context-based materials in science. International journal of educational development, 22(6), 659-672. Gerard, L. F., Spitulnik,M., & Linn, M. C.,(2010). Teacher use of evidence to customize inquiry science instruction. Journal of Research in Science Teaching, 47(9), 1037-1063. Harel, I. E., & Papert, S. E. (1991). Constructionism: Ablex Publishing. Kali, Y., & Ronen-Fuhrmann, T., (2011). Teaching to design educational technologies. The International Journal of Learning Technology (IJLT). 6(1),4-23. Kali, Y., Markauskaite, L., Goodyear, P., & Ward, M-H. (2011). Bridging multiple expertise in collaborative design for technology-enhanced learning. Proceedings of the Computer Supported Collaborative Learning (CSCL) conference (pp. 831-835). ISLS. Konings, K. D., Brand-Gruwel, S., & van Merrienboer, J. J. G. (2007). Teachers' perspectives on innovations: Implications for educational design. Teaching and Teacher Education, 23(6), 985-997. Kolodner, J. L., Crismond, D., Fasse, B., Gray. J., Holbrook,J., & Puntembakar,S. (2003). Putting a StudentCentered Learning by Design Curriculum into Practice: Lessons learned. Journal of the Learning Sciences, 12, 495 - 548. Mishra, P., & Koehler, M. J. (2005). What happens when teachers design educational technology? The development of Technological Pedagogical Content Knowledge. Journal of Educational Computing Research, 32(2), 131-152. McKenney, S. (2005). Technology for Curriculum and Teacher Development: Software to Help Educators Learn while Designing Teacher Guides. Journal of Research on Technology in Education, 28(2), 167190. Raval,H., McKenney, S., & Pieters, J. (2010). A conceptual model for supporting para-teacher learning in an Indian NGO. Studies in Continuing Education, 32(3),217-234. The Learning Design Grid (2012). Retrieved from http://www.ld-grid.org/ Voogt, J., Almekinders, M.,van den Akker, J., & Moonen, B. (2005). A blended in-service arrangement for classroom technology integration: impacts on teachers and students. Computers in Human Behavior, 21(3), 523-539.
ISLS
583
ICLS2012
Inducing Learners Creativeness and Encouraging Learning across Disciplines

Moseli Mafa, Lesotho College of Education: moseli3712@g mail.co m Palesa Khotso, Lesotho College of Education, khotsopalesa@yahoo.com Mamotena Jankie, Lesotho College of Education, mamjankie@yahoo.co.uk Mako mosela Qhobela, National Un iversity of Lesotho, m.qhobela@nul.co.ls Abstract: This workshop is motivated by emerging global challenges such as poverty, climate changes, pandemics and diseases epidemics. Therefore the need to collaborate across faculties to creatively respond to the challenges. This workshop seeks to expose some innovative teaching/ learning practices and methods that can enhance learners creativity and/or learning across disciplines. Presentations are made, followed by discussions and constructions of model(s), aimed at informing educators and researchers on creativeness and/or transdisciplinary knowing.
Workshop Description
This workshop is triggered by the contemporary challenges such as poverty, pandemics, economic decline and climate change which seem to force us to think that in order to overcome such challenges we need more creativeness and collaborated efforts from various disciplines. Reports on ICSL and CSCL indicate that other scholars have already made attempts to sensitize people on the importance of collaborated efforts, as a current strategy for responding to prevailing and future challenges. However, there is need to focus more on creativeness that could be achieved when various faculties collaborate and compliment each other (Sternberg and William, 2008). The workshop runs over three sessions. In the first session organizers and instructors provide the purpose and theoretical framework. In the next session, participants break into groups based on similar thematic aspects; to present, discuss, comment and construct draft learning/teaching model(s) fro m their deliberations. The third session converges all participants, to discuss and make reconstructions for further refining and consolidation of the model(s) so as to improve learners creativity and/or elevate learning across disciplines. Future activ ities are also recommended. The activities stated above are aimed at answering the questions: Why is there a need to visit teaching and learning strategies? What research and ways of learning, are suitable for adequately responding to emerging global challenges? How can we transform learn ing to become more adaptive, relevant and responsive? Why are collaborated efforts towards global challenges necessary? Why is there a need to merge theories in relation to analysis of a situation? What modes of learn ing can induce learners creat iveness and/or learning across disciplines? How do different disciplines (across faculties) co mplement each other? What impact is made by learners creat iveness in their learning? How can we better assist learners to think mo re critically and act with greater degree of creativeness? What are the challenges facing research on creativity? How can (trans-disciplinary) research on learners creativeness be better carried-out and encouraged?
The workshop constructs on creativity, multi-disciplinarity and trans-disciplinarity as part of transformative learning in order to equip a future people in their response to emerg ing global challenges (Songca, 2006). After the workshop, the model(s) will be piloted at some higher learning institutions to find if indeed they can render people more creative, adaptive and responsive to emerging changes and their dire consequences to life on Earth (OSullivan, 1999). Further research on the model(s) shall be carried-out, to investigate ways of imp roving them; to be more appropriate, practicable and effective.
ISLS
584
ICLS2012
References
OSu llivan, E.(1999). Transformative Learning: Educational Vision for the 21st Century. Toronto: University of Toronto Press. Songca, R. (2006). Transdisciplinarity: The dawn of emerging approach to acquiring Knowledge. International Journal of African Renaissance Studies,1 (2) 221-232. Sternberg, J. &Williams, N. (2008). Teaching for Creativ ity: Two Dozen Tips, Centre for Development and Learning [available at www.cdl.org/resource-library/article on 23rd November, 2011].
ISLS
585
ICLS2012
Developing a Competitive Educational Research Proposal for NSFs Division of Research on Learning
Sandra Toro Martell, Janet L. Kolodner, National Science Foundation, 4201 Wilson Boulevard, Arlington, VA, 22230, smartell@nsf.gov, jkolodne@nsf.gov Abstract: This half-day workshop on the morning of Tuesday, July 3rd, will help researchers interested in submitting proposals to the NSF Division of Research on Learning (DRL) by building capacity among learning sciences researchers to write competitive and high quality proposals that are likely to be successful. The workshop is comprised of multiple short sessions and provides multiple opportunities for discussion and questions, including ample time for interaction with NSF program officers. The content of the workshops includes: 1) the contexts of STEM educational research in DRL; 2) characteristics of and significant changes in DRLs major programs, including Research on Education and Learning or REAL (formerly REESE), Advancing Informal Science Learning or AISL (formerly ISE); and Faculty Early Career Development (CAREER); 3) NSFs proposal review process and merit review criteria; 4) characteristics of competitive proposals; and 5) common weaknesses in poorly-rated proposals. Novice and experienced proposers are equally welcome. All workshop participants will better understand: DRLs major research programs and priorities; the NSF review process; Program Officers experience of what makes a good proposal; Program Officers observations of common weaknesses in proposals; and the mechanics of proposal preparation. Researchers must apply for participation by submitting a document that includes: a letter of interest, a description of prior proposal-writing experience, a short summary of workshop expectations (maximum of 250 words); a research idea for discussion during small-group breakout sessions; and a brief CV (2 pages).
The primary goal of the workshop is to provide an overview of the proposal-writing process and NSFs review criteria for researchers potentially submitting proposals to DRL. Novice and experienced proposers are equally welcome. While the basic structure of NSF and DRL is unchanged this year, DRLs programs are revised, as are the priorities and emphases of the Administration. There has also been some change in the interaction between the NSFs Directorate for Education and Human Resources (EHR) and the U.S. Department of Education.
Rationale
The primary rationale is to build capacity among learning sciences researchers to write competitive and high quality proposals that are likely to be successful. In this regard, it is important to understand the larger context of NSF goals for scientific research. The NSF context includes: emphasis on potentially transformative research; NSF culture of evolving programs to meet dynamic opportunities; evaluation and accountability expectations; research and development to promote educational reform; and knowledge and evidence base for change.
Goals
Workshop participants will better understand: DRLs major research programs and priorities; the NSF review process; Program Officers experience of what makes a good proposal; Program Officers observations of common weaknesses in proposals; and the mechanics of proposal preparation. Presentations will set the context for the priorities of NSF, EHR, and DRL. There will be a brief discussion of NSFs current structure and organization and funding levels. The bulk of the time will be spent on the NSF proposal review process, what makes a proposal competitive, and how DRL thinks about rigor in education research. As described in the syllabus, participants will have ample interaction with Program Officers. This will be accomplished two ways. First, participants will critique and discuss prearranged scenarios which demonstrate proposal strengths or weaknesses highlighted in the presentation. Second, participants will have opportunities to describe and get feedback on their own research ideas.
ISLS
586
ICLS2012
Syllabus
The workshop is comprised of multiple short sessions to vary the format and provide opportunities for interaction. This format was piloted for a mini-course at the 2011 AERA Annual Meeting, with positive response from course participants. Session 1 [45 minutes] begins with an introduction to DRLs funding programs that are relevant to learning sciences researchers: Research and Evaluation in Education, Science and Engineering (REESE), Informal Science Education (ISE); and Faculty Early Career Development (CAREER; and NSF-wide initiative). It will describe the programs objectives, topical strands, types of awards, and provide examples of projects supported by each program. The session will then continue with an overview of the NSF review timeline and merit criteria intellectual merit and broader impact and DRL-specific expectations and procedures. The purpose of this session is for participants to understand the programs foci and the review process. In Session 2 [1.5 hours], the workshop will split into smaller groups, each led by a DRL PO. During this time, the participants will first read a scenario a short passage designed to stimulate discussion about strengths and weaknesses of a proposal. While reading each scenario, participants will be encouraged to note their thoughts about the passage, and may also be asked to respond to a prompt to help seed the discussion. Multiple scenarios will be addressed during this session. The scenarios are likely to include: 1) brief passages that address important proposal elements such as national significance, role of STEM content, connections to theory-building or testing, and rigor and evidence in study design; and 2) brief summaries of potential research projects or study ideas, which POS occasionally receive from potential applicants. The scenarios provided will be prepared by Pos as exemplars. It is expected that these will not be drawn from real proposals but if any passages from real proposals are used, these will first be approved by the respective PIs and cleared through NSFs Office of the General Counsel. After a break [15minutes], the small groups will rejoin for Session 3 [45 minutes]. Participants will shift from discussion of pre-arranged scenarios to discussion of their own questions or individual proposal ideas. This format allows participants to bring up questions or points of discussion about the programs or about the review process in a smaller and more congenial environment. To support open dialogue, participants are invited to share a brief summary of a potential research topic or study. As mentioned above, Pos commonly receive brief summaries of this type from researchers and are asked to provide feedback on both the selection of the appropriate program and on the potential strengths and weaknesses of the idea. After having read and discussed the scenarios, participants may be more comfortable with their small group to mention their research ideas. Preregistered participants will receive an email prior to ICLS that will remind them to prepare this summary in advance. All participants will also have several minutes at the beginning of this session to write or update the summary. Participants can opt to share their summary with the group directly or to have the DRL PO share with the group to retain anonymity. Session 4 [45 minutes] will explore the qualities of successful (and unsuccessful) proposals. This presentation will include discussion of common strengths of highly-rated proposals and weaknesses of poorly-rated proposals. The Program Officers will use the strengths and weaknesses that participants identified in the small-group discussion as the entr into the discussion of general strengths and weaknesses. The session will then address e proposal-writing tips that reflect POs experience reading and reviewing proposals. It will also include clear examples that will relate directly to the two NSF merit review criteria and to the DRL programs. This session is expected to be an open dialogue with ample opportunity for participants questions.
ISLS
587
ICLS2012
Digital Ecosystems for Collaborative Learning: Embedding Personal and Collaborative Devices to Support Classrooms of the Future
Roberto Martinez1, James Slotta3, Pierre Dillenbourg4, Andrew Clayphan 1, Mike Tissenbaum3, Beat Schwendimann2 and Chirstopher Ackad1 1 School of Information Technologies, 2Faculty of Education and Social Work University of Sydney, Sydney, NSW 2006, Australia 3 Ontario Institute for Studies in Education, University of Toronto, Ontario, Canada 4 CRAFT, Ecole Polytechnique Fdrale de Lausanne (EPFL) roberto@it.usyd.edu.au, jslotta@oise.utoronto.ca, pierre.dillenbourg@epfl.ch, andrew.clayphan@sydney.edu.au, mike.tissenbaum@utoronto.ca, beat.schwendimann@sydney.edu.au, christopher.ackad@sydney.edu.au Abstract: Multi-touch tables, interactive whiteboards, motion sensitive interfaces, physical and tangible computers, all present enticing new functional affordances for learning. However, this constitutes a problem space for design, rather than any specific solution. What forms of learning can now be supported by a multi-user touch screen? How can such learning be incorporated into K-12, university, or informal learning designs? As new commercial offerings become available, it is timely for learning scientists and educators to explore how to best make use of these tools at the classroom. This workshop will offer a venue to discuss how to develop novel technology into supportive tools and intelligent mediators between peers activity to build the classroom of the future. We will discuss the role of technology and its limitations along with the roles of teachers and students. This workshop gives participants the opportunity to experience such a classroom, share their work, discuss practical challenges and define an agenda for future work.
Motivation and Themes

The expanded abilities to connect students with peers, support collaborative work, and aggregate user contributions holds great promise for the design of materials, assessments, activities, and interactions. Yet, designing curriculum that deeply integrates such media is challenging, particularly if done comprehensively so that the entire course is organised according to the social and collaborative products or exchanges that typify collaborative environments. Face-to-face collaboration offers unique advantages compared with other ways of learning by affording continuous communication, dynamic transition between individual and group work, and productivity in completing tasks (Olson, Teasley, Covi, & Olson, 2002). However, groups of students often do not just spontaneously collaborate. Students require teacher support and attention at different times to make collaboration effective. One of the main challenges for teachers in managing collaborative learning at traditional classrooms is that teachers attend specific groups at a time and often only see the final product (Race, 2001). There are a number of emerging and well grounded technologies that are promising to improve face-toface collaborative learning, and there is growing interest on how they can enhance classroom activities. These include: multi-display groupware; interactive surfaces, such as multi-touch tabletops, electronic whiteboards, tablets, smart phones; motion sensitive interfaces; and physical and tangible computing. They all provide novel interaction spaces that can enrich collaborative work by providing access to digital content and also by capturing the learners digital footprints. Such spaces can offer teachers and researchers new opportunities to orchestrate the class (Dillenbourg et al., 2011), offer an ecological learning environment (Moher, Gnioli, Jaeger, Wiley, & Lopez Silva, 2011) and inspect the collaborative process through real-time tools that help improve teachers awareness in the classroom (Martinez, Yacef, Kay, & Schwendimann, 2012). The nature of learning and instruction may be transformed by such new technologies, but we must consider how to re-organise our instruction around social, cooperative principles, including user-contributed content, shared experiences and emergent learning goals. It is not straightforward to design instruction that integrates students use of shared displays or technologically augmented learning environments. As a field, we require exemplars of coherent designs in order to inform our theoretical models of learning in such a community-oriented approach. These new forms of learning and instruction provide unique
Website: https://sites.google.com/site/iclsdecl/
ISLS
588
ICLS2012
opportunities to connect individual students, small groups, and the entire class in complex and increasingly interconnected configurations that could not be achieved without the technological scaffolds that support them.
Purpose and Contribution

Interactive surface devices (for example, multi-touch tabletops, interactive whiteboards, tablets and personal devices) promise to improve collaborative learning. There is growing interest on how to enhance classroom activities through interactive surface devices. Tabletops and whiteboards have been habitually used as spaces, where people interact face-to-face to discuss ideas and build artefacts together. Tablets and other mobile devices create personal and private spaces that can connect people anywhere and at anytime. The promise of new technology is that they offer a tangible shared space that can enhance collaborative work by providing access to rich digital content or carrying this content in the pocket or the backpack. The purpose of this workshop is to explore how new tangible, ubiquitous and collaborative learning technologies can transform the traditional classroom, allowing for new forms of, collaborative learning, and inquiry-oriented instruction.. This workshop aims to engage the research community in discussions about the best practices for adaptation of advanced technologies to educational contexts, and integrating technology within a pedagogical or theoretical perspective. In this workshop, participants will collaboratively develop an understanding of the current state of interactive surfaces, mobile and ecological technology integration: first, through the demonstration of several successful implementations in authentic classroom settings; and second, through a panel discussion with experts in the field sharing their personal experiences in enacting these complex pedagogical and technical designs; and finally, through hands-on interactions with collaborative and distributed surface technologies, turning the workshop itself into an advanced technology augmented classroom. The goal of this workshop is to address several key issues within this emerging field including: What forms of collaborative inquiry can be empowered by collaborative and distributed surface technologies? What theoretical models and frameworks form the basis of learning from these technologies? What kinds of learning outcomes/measures can be obtained? What technology innovations are exciting? What are recent innovations in classroom technologies? What problems or obstacles might be present? What are the defining issues for this area of research? How can research collaborations make a difference? What is the role of open source and online communities in these classrooms?
Conclusion
The workshop will focus on the discussion of two main questions: 1) What are the Challenges for collaboration in real classrooms that technology has the potential to address?; and 2) What are some possible solutions and the agenda for the next steps in research to bring these to practice? This workshop recognises that the use of technology in the classroom needs to respond to the needs of educators and learners, and must be guided by a strong pedagogical/collaborative perspective about learning. The only way to design and implement truly enhanced collaborative learning for classrooms of the future is through the integration of the experience of researchers, educators, designers, ethnographers and computer scientists who explore, build, or work with educational technology that provides support to students and teachers in the classroom (Slotta, 2010). Our multidisciplinary vision is that Educators will leverage these tools to effectively deliver next-generation learning experiences.
References
Dillenbourg, P., Zufferey, G., Alavi, H., Jermann, P., Do-Lenh, S., & Bonnard, Q. (2011). Classroom orchestration: The third circle of usability. Paper presented at the International Conference on Computer Supported Collaborative Learning 2011. Martinez, R., Yacef, K., Kay, J., & Schwendimann, B. (2012). An interactive teachers dashboard for monitoring multiple groups in a multi-tabletop learning environment. Paper presented at the Intelligent Tutoring Systems. Moher, T., Gnioli, A., Jaeger, A., Wiley, J., & Lopez Silva, B. (2011). Embodied Learning for Embedded Spaces. Paper presented at the Computer-Supported Collaborative Learning. Olson, J. S., Teasley, S., Covi, L., & Olson, G. (2002). The (currently) unique advantages of collocated work. Distributed work, 113-135. Race, P. (2001). A briefing on self, peer & group assessment: York (United Kingdom) : Learning and Teaching Support Network. Slotta, J. D. (2010). Evolving the classrooms of the future: The interplay of pedagogy, technology and community. In K. Mkitalo-Siegl, F. Kaplan, J. Zottmann & F. Fischer(Eds.). Classroom of the Future. Orchestrating collaborative spaces. (215-242).
ISLS
589
ICLS2012
Tightening research-practice connections: Applying insights and strategies during design charrettes
Susan McKenney, Open University of the Netherlands & Twente University, PO Box 2960, 6401DL Heerlen, the Netherlands, susan.mckenney@ou.nl Kimberley Gomez, University of California, Los Angeles. Box 951521, Los Angeles, CA 90095-1521. kimgomez@ucla.edu Brian Reiser, Northwestern University, Room 339, 2120 Campus Drive, Evanston, IL 60208-0001 Abstract: Design charrettes feature hands-on activities for capturing, analyzing and developing the knowledge, values, and vision of its participants. In this workshop, using a design charrette approach, participants will (a) consider how their research informs formal and informal practice, (b) learn about a variety of outlets for bringing research to practice audiences, and (c) consider who might benefit from learning about the research. Participants will discuss different modes of research-practice interaction, and their implications for the production and use of new knowledge. Individuals will analyze their current approaches to knowledge dissemination for use and participants will share existing strategies to stimulate fruitful and mutually informing research-practice connections. Participants will designs their own research-practice connections, both through individual projects and through the ISLS community.
Background
Educational research has long been criticized for its weak link with practice. Explicit attempts to close the research-practice gap have been underway for over four decades. These efforts have included design team approaches and researcher-teacher models aimed at making practice the object of investigation. Shrader, Williams, Whitcomb, Finn and Gomez (1998), for example, described a research for practice approach, in the Learning Sciences, that involved working collaboratively with teachers to design, enact, and refine science materials. Lampert (1992) and Ball (Lampert & Loewenberg-Ball, 1998) taught in classrooms and made their practice an object of study. Yet, while a robust, growing body of knowledge now exists to describe how policymakers and educators access, value and use research, much of the work that would be useful happens in silos, is known to a few, and is rarely leveraged by policy makers, school administrators and teachers to improve educational practice. We have learned much about what aspects of evidence-based practice and research utilization in other fields can be applied to education, yet, how to share current knowledge, generate and share new knowledge, and walk the knowledge-sharing communicative path between research and practice remains a significant challenge in the Learning Sciences. Internationally, enormous efforts have been launched to improve the practical relevance and actual use of research knowledge, especially in the fields of education and health care. However, both the scholarly insights and effective practices have yet to become widely spread. In addition, even though researchers are becoming increasingly required to disseminate research findings among practitioners, few graduate programs devote serious attention to preparing researchers for the task and many researchers find it daunting. The proposed workshop addresses this problem by (a) sharing insights and practices from existing projects that stimulate fruitful research-practice connections during knowledge production and/or use; and (b) facilitating the design of strategies through which ISLS can tighten research-practice connections, both individually and as a community.
Theoretical underpinnings
Informed by the work of Rogers (1969), and review of over 2600 research studies, Havelock (1971) published a landmark report on the dissemination and use of scientific outputs. Havelock identified seven general factors that could account for how scientific outputs are taken up and used: linkage, structure, openness, capacity, reward, proximity and synergy. He identified several modes in which those factors can be seen: social interaction; research, development and diffusion (RDD); and problem solving. More recently, attention has also been given not only to the use of scientific knowledge for educational practice (e.g. Hargreaves, 1999; Levin, 2004), but also to how it is produced (Vanderlinde & van Braak, 2010). Specifically, there is growing attention for how researchers and practitioners can collaboratively bear the responsibility for both producing and using relevant knowledge in education. Burkhardt and Schoenfeld (2003) identify seven models to describe the relationship between research and practice, five of which feature strong divisions of labor, relate more to evidence-based practice and align well with the RDD model described by Havelock (the reading model; the summary model; the professional
ISLS
590
ICLS2012
development model; the policy model; the long route) and two of which show more characteristics of Havelocks problem solving model (design experiments; and the engineering model). De Vries and Pieters (2007) add an eighth model which shares elements of Havelocks social interaction model and highlights equal collaboration: knowledge communities. Each of these models denotes different assumptions and expectations regarding the roles of practitioners and researchers in the generation and application of theoretical understanding. Taken together, three broad types of research-practice interactions during knowledge production can be distinguished today: RDD, design research and communities.
Workshop structure
To meet these aims, the workshop is divided into two main stages. The first stage is intended to sensitize and inform participants by sharing insights and strategies from existing projects that stimulate fruitful researchpractice connections during knowledge production and/or use. Strategies for sharing (emerging) insights will be discussed in terms of: content (focus), form (products and activities), medium (face-to-face, online, etc.) and time (sustained, bursts, frequency, etc.). During the second stage of the workshop, participants will be encouraged to consider, and design, specific strategies for tightening research-practice connections that can be put into action in the short to medium term. Participants may choose to develop action plans related to individual projects, or to a broader (sub-) set of ISLS work. Participants will be involved in design charrettes that will undertake the tasks of: establishing a focus, considering time, determining the form and choosing the media to enhance research-practice interactions in a project that is currently underway or recently completed.
References
Burkhardt, H., & Schoenfeld, A. (2003). Improving educational research: Toward a more useful, more influential, and better-funded enterprise. Educational Researcher, 32(9) 3-14. De Vries, B., & Pieters, JM (2007). The meaning of conferences, knowledge communities and knowledge conscious schools in a crackled education. Educational Studies, 84, 233-240. Hargreaves, D. (1999). The Knowledge-creating school. Journal of Education Studies, 47(2) . Lampert, M. (1992). On teaching , in R. Glaser (Ed.), Advances in instructional psychology, 4. Erlbaum, Hillsdale, NJ. Lampert, M., & Loewenberg-Ball, D. (1998). Teaching, multimedia and mathematics: investigations of real practice. New York: Teachers' College Press. Levin, B. (2004). Making research matter more. Educational Policy Analysis Archives. 12(56). Shrader, G., Williams, K., Lachance-Whitcomb, J., Finn, L.E., & Gomez, L.M. (1999). Work in the work circle: Collaborative design to improve teaching practice. Paper presented at Spencer Conference on Collaborative Research for Practice, New Orleans, LA. Vanderlinde, R., & van Braak, J. (2010). The gap between educational research and practice: views of teachers, school leaders, intermediaries and researchers. British Educational Research Journal, 36(2), 299-316. .
ISLS
591
ICLS2012
Analyzing collaborative learning at multiple levels

Gerry Stahl, Drexel University, USA, Gerry@GerryStahl.net Keith Sawyer, Washington University in St. Louis, USA, ksawyer@wustl.edu Heisawn Jeong, Hallym University, South Korea, heisawnj@gmail.com Dan Suthers, University of Hawaii, USA, suthers@hawaii.edu Abstract: The workshop aims to understand how learning that takes place at individual, small group and collective (community or networked) levels might connect to each other. The workshop will consist of two data sessions, during which a data presenter will describe the background of the data and multiple analysts will present results of preliminary analyses. Workshop participants will then collaboratively explore data excerpts and discuss how different levels of analyses and associated theories are connected. This workshop seeks to understand how learning takes place in the interplay between individual, small group and collective (institutions, community, or networked) levels of activity in both offline and online settings. So far, most of the research reported at ICLS and CSCL has addressed learning mainly at one level, be it at the individual, small-group, or community level. To date, there has been relatively little focused research on the connections among these levels (e.g., data and analysis on how individual, small-group and community-level processes or phenomena influence, constrain or mediate each other). Given the possibility that a lot more learning in the future will occur collaboratively, often in networked communities, this workshop is appropriate for the theme of ICLS 2012. The workshop will explore connections across levels based on the analysis of actual data. The workshop will consist of two data sessions, during which a data presenter will describe the background of the data and multiple analysts will present preliminary analyses pertaining to different levels. Workshop participants will then collaboratively explore and discuss how different levels might relate to each other in small group discussions, which will then be shared. Through these activities, the workshop aims to address questions of how learning at different levels might be connected to each other. More specifically: How do theories at different levels of analysis or levels of learning relate to each other and how cansuch theories explain phenomena across levels? Do the different levels of analysis require different theories? Are there theoretical perspectives that themselves bridge the levels of analysis/levels of learning? How do lower-level phenomena (individual and small-group activity) lead to emergent phenomena? How might these emergent phenomena then act as resources, contexts, or values that in turn are available for future individual and small-group learning? How do activities at one level (e.g., interaction patterns or group structure) influence other levels? How does learning take place through the interplay between individual and collective agency? How are advances in community knowledge (knowledge building) driven by small-group activity? These issues cannot be separated from the methodological issues such as: What constitutes data at each level? How can methods for small-group analysis (e.g., interaction analysis) be coordinated with methods for community or societal analysis (e.g., structural social-network analysis)? How can (for example) sequential analysis of interaction, content analysis and social-network analysis be coordinated to address these issues? How can we use multiple levels of analysis to figure out where to "dive in" for maximum explanatory power, for example to make sense of results at the community or societal level, or to find small-group sources of innovation? What practical techniques (such as different forms of triangulation or visualization techniques) might help connect different levels of analysis? Is there a corresponding need for different analytic approaches and associated tools? This workshop builds on the Productive Multivocality workshop series successfully conducted at CSCL 2007, ICLS 2008, CSCL 2009, ICLS 2010, Alpine Rendezvous 2010, and Alpine Rendezvous 2011. This workshop is the second with the theme of Connecting Levels, with the first such workshop held at CSCL 2011 in Hong Kong. It is different from the earlier series in that it addresses how learning at different levels can be coordinated toward a more comprehensive theory of learning and cognition. We will continue our reflections on the interplay between levels in Sydney at ICLS 2012.
ISLS
592
ICLS2012
Volume 2: Author Index
GVOLUME 2: AUTHOR INDEX
A Abrahamson, Dor, 99 Ackad, Chirstopher, 588 Adams, Karlyn, 489 Aditomo, Anindito, 416, 441 Ainley, Mary D., 25 Aleven, Vincent, 134 Allaire, Stephane, 455 Anderson, Kate, 575 Anmarkrud, istein, 479 B Bachrach, Jessica, 49 Bagley, Elizabeth, 256 Baker, Ryan, 134 Barron, Brigid J., 25 Barth-Cohen, Lauren, 291, 306 Bass, Michelle, 217 Beauvineau, Yves, 505 Becker, Bernd, 515 Bell, Philip, 142 Berland, Matthew, 134 Bertram, Johanna, 157 Bielaczyc, Katerine, 409 Bientzle, Martina, 162 Bijker, Monique, 197
ISLS
593
ICLS2012
Birchfield, David, 99 Biswas, Gautam, 251, 573 Black, John B., 99 Blikstein, Paulo, 7, 134, 561, 567, 571, 577 Bokhorst, Franziska, 162 Bolling, Amy, 555 Boshuizen, Henny P. A., 197 Brand-Gruwel, Saskia, 197 Bransford, John, 41 Brten, Ivar, 479 Bricker, Leah, 142 Bryk, Anthony, 549 Buehrer, Manuel, 515 Bull, Susan, 33 Burd, Elizabeth, 187 C Cafaro, Francesco, 557 Calvo, Rafael, 416 Cecilie F. Jahreie, 9 Cerniak, Gabriele, 33 Chai, Ching Sing, 72 Chan, Carol K. K., 351, 361, 465 Chan, Yuan Y., 361 Chang, Chih-Hsuan, 497 Chang, Hsin-Yi, 461, 521 Chang, Mido, 475 Charles, Elizabeth, 207
ISLS
594
ICLS2012
Chaves, Gina, 581 Chaves, Wendy, 581 Chen, Jennifer King, 469 Cheng, Britte, 49 Chiu, Chieh-Hsin, 495 Chiu, Jennifer, 17 Chiu, Ming Ming, 356 Choi, Sung-Youn, 316 Cisterna, Dante, 527 Clark, Dav, 481 Clark, Doug, 86 Clarke-Midura, Jody, 110 Clayphan, Andrew, 588 Clegg, Tamara, 126 Cober, Rebecca, 64 Cohen, Sarah, 481 Constantinou, Constantinos, 507 Cosejo, David D., 86 Cruse, Julie, 99 Curwood, Jen Scott, 551
D Dalgarno Barney, 227 Davenport, Jodi, 110, 381 Davis, Pryce, 91, 477 de Jong, Ton, 17
ISLS
595
ICLS2012
De Wever, Bram, 118 Deater-Deckard, Kirby, 475 DeBarger, Angela Haydel, 505 Dede, Chris, 110 DeLiema, David, 99, 311 DeVane, Ben, 126 Dierking, Lynn D., 326 Dillenbourg, Pierre, 4, 579, 588 Dimitriadis, Yannis 579 Dolle, Jonathan, 549 Dookie, Lesley, 491, 286 Drake, Joel, 321 E Eberle, Julia, 386 Eisenberg, Michael, 99 Ely, Robert B. W., 25 Esmonde, Indigo, 286, 491 Evans, Michael A., 78, 391, 475 F Fadjo, Cameron, 99 Falk, John H., 326 Feiten, Linus, 515 Ferguson, Leila, 479 Fielding-Wells, Jill, 149 Fischer, Frank, 118, 386, 446 Fong, Cresencia, 64
ISLS
596
ICLS2012
Friedrich, Luisa Amelie, 237 Fueyo, Vivian, 192 Fuhrmann, Tamar, 571, 577 G Gegenfurtner, Andreas, 459, 517 Georgiou, Yiannis, 533 Gerard, Libby, 569 Gillespie, Amy, 525 Gnoli, Alessandro, 9, 64 Gobert, Janice, 110 Goldman, Susan R., 5, 126 Gomez, Kimberley, 590 Gomez, Louis M., 5, 549 Gonzlez, Carlos, 471 Gordon, Leon, 545 Greene, Daniel, 571, 577 Gresalfi, Melissa, 49, 545 Grotzer, Tina, 543 H Hakkarainen, Kai, 346 Haklev, Stian, 565 Halatchliyski, Iassen, 162 Halb, Wolfgang, 33 Halverson, Erica Rosenfeld, 217, 276 Hamilton, Eric, 581 Hand, Brian, 154, 503 Hasselbring, Ted, 573
ISLS
597
ICLS2012
Haun-Frank, Julie, 326 Heddy, Benjamin, 501 Herman, Phillip, 246 Hershkovitz, Arnon, 134 Hershkowitz, Rina, 376 Hesse, Friedrich, 33 Hewitt, Jim, 485 Hickey, Daniel, 541, 553 Higgins, Steve, 78, 187 Hmelo-Silver, Cindy, 126, 489, 537 Hnizda, Michaela, 391 Hodge, Kumi, 451 Hong, Huang Yao, 72, 495, 497 Hoppe, H. Ulrich, 523 Horn, Michael, 78 Horstman, Theresa, 142 Hsiao, Ya Ping (Amy), 157 Hsiao, Ying-Ting, 431 Huber, Stefan, 157 Hung David, 509 Hussain, Abid, 341 I Ioannou, Andri I., 533 Itow, Rebecca, 541, 553 J Jacobson, Michael, 7, 86 Jamalian, Azadeh, 281
ISLS
598
ICLS2012
Jamaludin, Azilawati, 509 Jankie, Mamotena, 584 Jeong, Heisawn, 592 Jochen, Rick, 126 Johnson, Emily, 555 Johnson, Matthew, 33 JohnsonGlenberg, Mina C., 99 Jordan, Michelle, 424 Jornet, Alfredo, 9 Joyce-Gibbons, Andrew, 187 Jump, Vonda, 493 K Kaendler, Celia, 515 Kali, Yael, 582 Kalyuga, Slava, 457 Kapon, Shulamit, 511 Kapur, Manu, 2 Karabinos, Michael, 381 Kaufmann, Iris, 473 Kawasaki, Jarod N., 311 Kay, Judy, 3, 241 Kearney, Matthew, 453 Keifert, Danielle, 91 Kempler- Rogat, Toni, 489 Kennedy-Clark, Shannon, 559, 575 Khotso, Palesa, 584 Kim, Beaumie, 401
ISLS
599
ICLS2012
Kim, Mi Song, 401 Kinnebrew, John, 251 Kirshner, Ben, 326 Knuth, Randy, 41 Kobiela, Marta, 202 Koedinger, Kenneth, 110,134 Koh, Hwee Ling, 72 Kolin, Konstantin ., 7 Kollar, Ingo, 118, 446 Kolodner, Janet L., 1, 586 Kossowski, Michal, 33 Koziupa, Tatyana, 99 Krajcik, Joseph, 316 Krakowski, Moshe, 261 Krange, Ingeborg, 9 Krauskopf, Karsten, 157 Kyza, Eleni A., 531, 533 L Laine, Erkka, 517 Lam, Irene Chung-man, 519 Langer-Osuna, Jennifer, 547 Lasry, Nathaniel, 207 Law, Nancy, 7, 301 Lazonder, Ard, 17 Lee, Chwee Beng, 72 Lee, Mark J. W., 227 Lee, Shuh Shing, 467
ISLS
600
ICLS2012
Lee, Silvia Wen-Yu, 521 Lee, Tiffany, 41 Lee, Victor, 172, 321 Lehrer, Richard, 91, 202, 296 Lei, Chunlin, 351 Leutner, Detlev, 237 Lim-Breitbart, Jonathan, 469 Lindgren, Robb, 99, 555 Linn, Marcia, 86, 469, 569 Lira, Matthew, 406 Liu, Shiyu, 483 Looi, Chee-kit, 212, 579 Louca, Loucas, 331, 507, 513 Lu, Jingyan, 301 Luckin, Rose, 33 Lui, Michelle, 9, 64, 78 Lund, Kristine, 232, 271 Lundh, Patrik, 49 Lyons, Leilah, 557 M Madeira, Cheryl, 64 Mafa, Moseli, 584 Magnifico, Alecia Marie, 276, 411 Maher, Carolyn, 537 Makar, Katie, 149 Maldonado, Roberto Martinez, 241 Malzahn, Nils, 523
ISLS
601
ICLS2012
Marbouti, Farshid, 431 Martell, Sandra Toro, 586 Martin, Taylor, 134 Martinez, Roberto, 588 Matsuzawa, Yoshiaki, 465 Matuk, Camillia, 469 Mazur-Palandre, Audrey, 271 McCall, Marty, 110 McCann, Colin, 64 McElhaney, Kevin, 469 McKenney, Susan, 582, 590 McNaughton, James, 78 McNeill, David, 391 Mehus, Siri, 91 Mendiburo, Maria, 573 Mercier, Emma, 78, 187 Merritt, Joi, 527 Miller, Andrea, 451 Miller, David, 469 Miyake, Naomi, 5, 57, 72, 86, 182 Moher, Tom, 9, 64 Molinari, Gaelle, 232 Moorthy, Savitha, 505 Mulder, Yvonne, 17 Munn, Maureen, 41 N Nicolaidou, Iolie, 531
ISLS
602
ICLS2012
Norton, Anderson, 475 Novellis, Francesco, 9 Nckles, Matthias, 473 Nussbaum, Miguel, 7, 579 O Oeberst, Aileen, 162 Ohlsson, Stellan, 86 Okita, Sandra, 57, 281 Olson, Izabel Duarte, 529 Opfermann, Maria, 237 Oshima, Jun, 182, 465 Oshima, Ritsuko, 182, 465 Oura, Hiroki, 41 Oviatt, Sharon, 451 Ow, John, 409 Oztok, Murat, 485 P Pachman, Mariya, 457 Pagliaro, Claudia, 493 Paik, Sunhee, 409 Palius, Marjory, 537 Panciera, Katherine, 157 Papademetri-Kachrimani, Chrystalla, 513 Parisio, Martin, 575 Parlin, Mary Ann, 493 Penney, Lauren, 91 Penuel, William, 326, 505
ISLS
603
ICLS2012
Perry, Rachel, 453 Peters, Vanessa L., 118,126 Petrick, Carmen, 99 Pinkard, Nichole D., 5, 142 Pressick-Kilborn, Kimberley, 49 Prusak, Naomi, 376 Q Qhobela, Makomosela, 584 Quek, Choon Lang, 72 Quellmalz, Edys, 110 R Rachel, Alexander, 17 Radinsky, Joshua, 557 Raes, Annelies, 118 Rafanan, Ken, 567 Rafferty, Anna, 381 Ralambondrainy, Henri, 463 Ramirez, Israel, 581 Ranney, Michael, 481 Rebello, Carina M., 366 Rebello, N. Sanjay, 366 Recker, Mimi, 172 Reimann, Peter, 17, 33, 72, 341, 441 Reinholz, Daniel, 481 Reiser, Brian, 590 Renninger, K. Ann, 25, 49 Resendes, Monica, 565
ISLS
604
ICLS2012
Reynolds, Rebecca, 356 Rhodes, Suzanne, 487 Richard, Halverson, 487 Rick, Jochen, 78 Roberts, Jessica, 557 Roll, Ido, 134 Roschelle, Jeremy, 567, 579 Rose, Carolyn, 57 Rosenbaum, Claire, 567, 577 Rouse, Rob, 296, 535 Rowe, Deborah, 525 Roy, George, 192 Rummel, Nikol, 5, 266 Russ, Rosemary S., 477 Russell, Jennifer, 549 Russell, Kamala, 391 S Salehi, Shima, 571, 577 Sandoval, William A., 311 Sato, Elissa, 177 Savio-Ramos, Caroline, 99 Sawyer, Keith, 592 Sayre, Eleanor, 366 Scalone, Giovanna, 41 Schank, Patti, 567 Schellens, Tammy, 118 Schmeck, Annett, 237
ISLS
605
ICLS2012
Schoerning, Emily, 154 Schwarz, Baruch B., 371, 376 Schwendimann, Beat, 241, 588 Scopelitis, Stephanie, 406, 539 Scoresby, Jon, 493 Segedy, James, 251 Seitamaa-Hakkarainen, Pirita, 346 Seppnen, Marko, 459 Sester, Sebastian, 515 Shaffer, David Williamson, 256 Shah, Niral, 222 Shahar, Nitza, 371 Shechtman, Nicole, 49 Shelton, Brett, 493 Shen, Chia, 78 Sherin, Bruce, 134 Shin, Namsoo, 316, 396 Shinohara, Mayumi, 535 Shouse, Andrew, 41 Shutt, Kari, 41 Sigley, Robert, 537 Silva, Brenda Lpez, 9, 64 Simon, Jean, 463 Sinatra, Gale, 501 Sinha, Suparna, 489 Siyahhan, Sinem, 545 Skoulia, Thea, 331
ISLS
606
ICLS2012
Slotta, James D., 78, 588 Slotta, Jim, 9, 17, 64, 118, 346 Smrdal, Ole, 9 Songer, Nancy, 118, 126 Southavilay, Vilaythong, 416 Stahl, Gerry, 592 Stegmann, Karsten, 386 Stevens, Reed, 91 Stevens, Shawn Y., 396 Stieff, Mike, 406, 539 Strackeljahn, Andrea, 541 Strfling, Nicole, 157 Strms, Helge I., 479 Sun, Daner, 212 Sutherland, LeeAnn M., 316 Suthers, Dan, 592 Svihla, Vanessa, 177 Swanson, Hillary, 563 Sweller, John, 457 T Takeuchi, Miwa, 491 Tan, Lynde, 401 Taylor, Jonte, 503 Tee, Meng Yew, 467 Teplovs, Christopher, 361 Therrien, William, 503 Thompson, Kate, 441, 559
ISLS
607
ICLS2012
Tian, Stella W., 361 Timms, Michael, 110, 381 Tissenbaum, Mike, 64, 78, 118, 436, 588 Trinidad, Gucci, 49 Tutwiler, M. Shane, 543 Tzialli, Dora, 331, 507 U Unal, Zafer, 192 Utz, Wilfrid, 33 V Vahey, Phil, 192 van Aalst, Jan, 157, 162, 361, 465 Van Horne, Katie, 41, 142 van Strien, Johan, 197 Vanover, Charles, 192 Varma Keisha, 483 Vatrapu, Ravi, 33, 341 Viilo, Marjut, 346 Vilma, Galstaun, 575 Vogel, Freydis, 446 Vosniadou, Stella, 86 Vye, Nancy, 41 W Wagh, Aditi, 426 Walker, Richard, 49 Wan, Wing-San, 361 Wan, Zhihong, 336
ISLS
608
ICLS2012
Wardrip, Peter, 549, 246 Weber, Katrin, 515 Wecker, Christof, 17, 118 Wegner, Elisabeth, 473 Westermann, Katharina, 266 Whittaker, Chris, 207 Wichmann, Astrid, 157 Wiedmann, Michael, 515 Wiesner, Hartmut, 17 Wilensky, Uri, 426 Williams, Michelle, 527 Wing-mui So, Winnie, 336 Wise, Alyssa Friend, 431 Worsley, Marcelo, 134, 561, 577 Wu, Hsin-Kai, 521 Y Yacef, Kalina, 241, 416 Yaron, David, 381 Ye, Lei, 172 Yeung,,Yau-yuen, 336, 519 Yoon, Susan, 167 York, Adam J., 326 Z Zertuche, Amber, 569 Zhang, Baohui, 7, 212 Zhao, Yuting, 431 Ziebarth, Sabrina, 523
ISLS
609
ICLS2012
Zingaro, Daniel, 485 Zywica Jolene, 246
ISLS
610

ICLS2012 Proceedings Vol 2 2012

Uploaded by

Copyright:

Available Formats

ICLS2012 Proceedings Vol 2 2012

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

ICLS2012 Proceedings Vol 2 2012

Uploaded by

Copyright:

Available Formats

10th International Conference of the Learning Sciences

The Future of Learning

Centre for Research on Computer Supported Learning & Cognition (CoCo)

THE FUTURE OF LEARNING

ICLS2012 would like to thank our sponsors:

Volume 2: Short Papers, Symposia, and Abstracts

ORGANISERS AND COMMITTEES CONFERENCE CHAIRS

Twente, the Netherlands

Sydney, Australia EARLY CAREER WORKSHOP

University, Taiwan PRE-CONFERENCE WORKSHOPS AND SPECIAL SESSIONS

Germany PAPER REVIEW COORDINATION

Kong, Hong Kong

Kong, Hong Kong DOCTORAL CONSORTIUM

Volume 2: Short Papers, Symposia, and Abstracts

Volume 2: Short Papers, Symposia, and Abstracts

Volume 2: Short Papers, Symposia, and Abstracts

Volume 2: Short Papers, Symposia, and Abstracts

Volume 2: Short Papers, Symposia, and Abstracts

Volume 2: Short Papers, Symposia, and Abstracts

ABOUT "COCO" AT THE UNIVERSITY OF SYDNEY

Volume 2: Short Papers, Symposia, and Abstracts

Volume 2: Short Papers, Symposia, and Abstracts

Volume 2: Short Papers, Symposia, and Abstracts

Lifelong Learning as a Driver for Designing Pervasive Technologies

Classroom Orchestration: Interweaving Digital and Physical Workflows

INVITED SPECIAL SESSIONS

Gathering Evidence of Scientific Argumentation Practices: From Pre-Kindergarten to High 41 School

Robot Facilitation as Dynamic Support for Collaborative Learning

Volume 2: Short Papers, Symposia, and Abstracts

Madeira, Colin McCann, Tom Moher, Jim Slotta, Mike Tissenbaum

Critical Aspects in Learning with Technologies

Interactive Surfaces and Spaces: A Learning Sciences Agenda

Instructional Approaches to Promote Conceptual Change

You're It! Body, Action, and Object in STEM Learning

Scripting Science Inquiry Learning in CSCL Classrooms

Learning as Identity Formation: Implications for Design, Research, and Practice

Language Formality, Learning Environments, and Student Achievment

Unraveling Idea Development in Discourse Trajectories

Investigating Emergent Dynamics to Understand Interactions in Small Professional Development Groups

Volume 2: Short Papers, Symposia, and Abstracts

Victor Lee, Lei Ye, Mimi Recker

Scaffolding Collaborative Sensemaking during Critique of Explanations in TechnologyEnhanced Science Curriculum

Collaborative Reading Comprehension with Communication Robots as Learning Partner

Multi-Touch Technology to Support Multiple Levels of Collaborative Learning in the Classroom

Measuring Sophistication of Epistemic Beliefs Using Rasch Analysis

Establishing a Mathematical Practice in a Middle School Classroom

Identity and Digital Media Production in the College Classroom

Dimensions of Scaffolding in Technology-Mediated Discovery Learning Environments

Motivation in the Learning Sciences: Connecting to Practice by Design

Supporting Student Learning using Conversational Agents in a Teachable Agent Environment

Epistemic Mentoring in Virtual and Face-to-Face Environments

Volume 2: Short Papers, Symposia, and Abstracts

New Evidence on Productive Failure - Building on Students' Prior Knowledge is Key!

Bidirectional artifact analysis: A method for analyzing creative processes

Understanding Influence in Collaborative Group Work: The Importance of Artifacts

Engaging in Design: Reflections of Young Paper Engineers

296 301 306

Using Critical Reading Tools to Facilitate the Learning of Argumentation Skills

Changing Explanations about Sand Dune Movement

Towards Teaching Analytics: Repertory Grids for Formative Assessment