Evaluation Methodology

This study examined business students' perceptions of four objectives (i.e., Enjoyment, Learning, Motivation, and Career Application) across five teaching technologies (i.e., Projector, PowerPoint, Video, the Internet, and Lecture), business professors' effective application of technologies, and students' academic performance. We collected data from 215 students at a regional state university in the USA. We developed Students' Perceptions of Technology Scale, SPOTS, specifically for the present study, used the most rigorous criteria, and investigated reliability, convergent and discriminant validity, exploratory and confirmatory factor analysis, and measurement invariance of this scale. Mean scores revealed that Video conveyed the highest amount of Enjoyment. PowerPoint provided the highest amount of Learning and Motivation. The Internet provided the highest Career Application for future jobs. Younger students preferred Video, whereas older students favored Lecture. Regression results showed that the use of Video for Learning, Projector and Lecture for Enjoyment, PowerPoint for career and Motivation, and the Internet for Learning contributed to professors' teaching effectiveness. Students' high ratings for professors' effective use of the Lecture method and low expectation for the use of a Projector predicted their self-reported GPA (academic performance). Professors may use a different mix of technologies in the classroom and use them creatively in order to promote the most Learning for students and satisfy students' Learning needs and objectives.

In this paper, several issues regarding the epistemology of technological pedagogical content knowledge (TPCK) are first raised for the purpose of clarifying the construct. Specifically, the transformative and integrative views are juxtaposed for exploring the epistemology of TPCK, and, at the end, the transformative view is adopted concluding that TPCK is a unique body of knowledge that is constructed from the interaction of its individual contributing knowledge bases. Then, ICT-TPCK is introduced as a strand of TPCK, and is described as the ways knowledge about tools and their affordances, pedagogy, content, learners, and context are synthesized into an understanding of how particular topics that are difficult to be understood by learners or difficult to be represented by teachers can be transformed and taught more effectively with technology in ways that signify its added value. One model for the development and another for the assessment of ICT-TPCK are then discussed. Technology Mapping is proposed as a situative methodology for the development of ICT-TPCK, and three forms of assessment, namely, expert assessment, peer assessment, and self-assessment are proposed for assessing teachers' competencies to teach with technology. The paper also reports on the empirical findings of a study that was undertaken to investigate the impact of the proposed models on student learning within the context of two design tasks in a pre-service primary teacher education course. Repeated measures within-subject effects were tested and the results indicated that ICT-TPCK competency significantly improved over the course of a semester. The results of this study clearly show that the theoretical models proposed herein can positively impact the development of ICT-TPCK. Lastly, these results can be used as baseline data in future studies that may be conducted to further validate or improve the proposed models in different contexts.

Although many researchers have studied different factors which affect E-Learning outcomes, there is little research on assessment of the intervening role of readiness factors in E-Learning outcomes. This study proposes a conceptual model to determine the role of readiness factors in the relationship between E-Learning factors and E-Learning outcomes. Readiness factors are divided into three main groups including: technical, organizational and social. A questionnaire was completed by 96 respondents. This sample consists of teachers at Tehran high schools who are utilizing a technology-based educating. Hierarchical regression analysis is done and its results strongly support the appropriateness of the proposed model and prove that readiness factors variable plays a moderating role in the relationship between E-Learning factors and outcomes. Also latent moderated structuring (LMS) technique and MPLUS3 software are used to determine each variable’s ranking. Results show that organizational readiness factors have the most important effect on E-Learning outcomes. Also teachers’ motivation and training is the most important factor in E-Learning. Findings of this research will be helpful for both academics and practitioners of E-Learning systems.►A framework of the effects of readiness factors on E-Learning outcomes is proposed. ►Readiness factors include three categories: technical, organizational and social. ►A questionnaire was completed by 96 respondents. ►Organizational readiness factors have the most important effect on E-Learning. ►Also teachers’ motivation and training is the most important factor in E-Learning.

In this paper, several issues regarding the epistemology of technological pedagogical content knowledge (TPCK) are first raised for the purpose of clarifying the construct. Specifically, the transformative and integrative views are juxtaposed for exploring the epistemology of TPCK, and, at the end, the transformative view is adopted concluding that TPCK is a unique body of knowledge that is constructed from the interaction of its individual contributing knowledge bases. Then, ICT-TPCK is introduced as a strand of TPCK, and is described as the ways knowledge about tools and their affordances, pedagogy, content, learners, and context are synthesized into an understanding of how particular topics that are difficult to be understood by learners or difficult to be represented by teachers can be transformed and taught more effectively with technology in ways that signify its added value. One model for the development and another for the assessment of ICT-TPCK are then discussed. Technology Mapping is proposed as a situative methodology for the development of ICT-TPCK, and three forms of assessment, namely, expert assessment, peer assessment, and self-assessment are proposed for assessing teachers' competencies to teach with technology. The paper also reports on the empirical findings of a study that was undertaken to investigate the impact of the proposed models on student learning within the context of two design tasks in a pre-service primary teacher education course. Repeated measures within-subject effects were tested and the results indicated that ICT-TPCK competency significantly improved over the course of a semester. The results of this study clearly show that the theoretical models proposed herein can positively impact the development of ICT-TPCK. Lastly, these results can be used as baseline data in future studies that may be conducted to further validate or improve the proposed models in different contexts.

Earlier studies have suggested that higher education institutions could harness the predictive power of Learning Management System (LMS) data to develop reporting tools that identify at-risk students and allow for more timely pedagogical interventions. This paper confirms and extends this proposition by providing data from an international research project investigating which student online activities accurately predict academic achievement. Analysis of LMS tracking data from a Blackboard Vista-supported course identified 15 variables demonstrating a significant simple correlation with student final grade. Regression modelling generated a best-fit predictive model for this course which incorporates key variables such as total number of discussion messages posted, total number of mail messages sent, and total number of assessments completed and which explains more than 30% of the variation in student final grade. Logistic modelling demonstrated the predictive power of this model, which correctly identified 81% of students who achieved a failing grade. Moreover, network analysis of course discussion forums afforded insight into the development of the student learning community by identifying disconnected students, patterns of student-to-student communication, and instructor positioning within the network. This study affirms that pedagogically meaningful information can be extracted from LMS-generated student tracking data, and discusses how these findings are informing the development of a customizable dashboardlike reporting tool for educators that will extract and visualize real-time data on student engagement and likelihood of success.

Despite recent advances of electronic technologies in e-learning, a consolidated evaluation methodology for e-learning applications is not available. The evaluation of educational software must consider its usability and more in general its accessibility, as well as its didactic effectiveness. This work is a first step towards the definition of a methodology for evaluating e-learning applications. Specific usability attributes capturing the peculiar features of these applications are identified. A preliminary user study involving a group of e-students, observed during their interaction with an e-learning application in a real situation, is reported. Then, the proposal is put forward to adapt to the e-learning domain a methodology for systematic usability evaluation, called SUE. Specifically, evaluation patterns are proposed that are able to drive the evaluators in the analysis of an e-learning application.

This paper compares and contrasts two projects in order to better understand the complex issues surrounding the use of technology to support parental involvement with schools and their children's learning. The Becta-funded ICT Test Bed evaluation (2002)(2003)(2004)(2005)(2006) had the intention of saturating schools (in three areas of social deprivation) with a range of technologies, including 23 elementary schools. The ESRC/EPSRC/DTI-funded Homework project (2003)(2004)(2005)(2006) used participatory design methods to develop and evaluate technology to link home and school in a elementary school in the South East. Both projects shared a common theoretical foundation, that of socio-cultural theory. The theory influenced the evaluation methodology employed in both projects and in the Homework project it additionally influenced the design of the technology intervention. Findings suggest that technologies with readily accessible and interactive resources that are flexible can help develop parental engagement. However, simpler and less resource hungry solutions such as the use of websites and email can provide opportunities for quick wins. In relation to transporting technology between home and school, there are issues for both staff and parents. Without purposeful use, these challenges act as a barrier once they outweigh the novelty effect. We conclude that parental needs are complex and that engagement needs to be sensitively scaffolded rather than focussing on the technology. Participatory design offers an effective means of addressing this and should be the starting point. The technology should facilitate independence and mediate access to a shared space for collaborative activity. The content and guidance needs to be purposeful and relevant, offering a means to integrate learning across the learner's broader context, including school and home seamlessly.

For a particularly long time, automatic diagnosis of diabetic retinopathy from digital fundus images has been an active research topic in the medical image processing community. The research interest is justified by the excessive potential for new products in the medical industry and possible reductions in healthcare costs. However, the maturity of state-of-the-art algorithms cannot be judged due to the lack of commonly accepted and representative image database with a verified ground truth and strict evaluation protocol. In this study, problems and issues related to the database are discussed from medical, image processing, and security perspectives. Based on the discussion, an evaluation methodology is proposed and a prototype image database with the ground truth is described. The database is made publicly available for benchmarking diagnosis algorithms. By utilizing the proposed database, it is now possible compare different algorithms, and correspondingly, analyze their maturity for technology transfer from the research laboratories to the medical practice.

This paper develops personalized models for different university degrees to obtain the risk of each student abandoning his degree and analyzes the profile for undergraduates that abandon the degree. In this study three faculties located in Granada, South of Spain, were involved. In Software Engineering three university degrees with 10,844 students, in humanities nineteen university degrees with 39,241 students and in Economic Sciences five university degrees with 25,745 students were considered. Data, corresponding to the period 1992 onwards, are used to obtain a model of logistic regression for each faculty which represents them satisfactorily. These models and the framework data show that certain variables appear repeatedly in the explanation of the drop out in all of the faculties. These variables are, among others, start age, the father's and mother's studies, academic performance, success, average mark in the degree and the access form and in some cases also, the number of rounds needed to pass. Students with weak educational strategies and without persistence to achieve their aims in life have low academic performance and low success rates and this implies a high risk of abandoning the degree. The results suggest that each university centre could consider similar models to elaborate a particular action plan to help lower the drop out rate reducing costs and efforts. As concluded in this paper, the profile of the students who tend to abandon their studies is dependent on the subject studied. For this reason, a general methodology based on a Data Warehouse architecture is proposed. This architecture does most of the work automatically and is general enough to be used at any university centre because it only takes into account the usual data the students provide when registered in a course and their grades throughout the years.

This study aimed to compare the efficacy of collaborative learning in face-to-face and online groups. Fifty psychology majors learnt the same professional skill (a community evaluation methodology) in two seminars taught over a two month period by the same teacher online and face-to-face. Participants in both seminars achieved similar growth in level of professional competence, academic self-efficacy, social self-efficacy and self efficacy for problem solving among members. Post-course evaluation of collaborative experience showed no significant differences between online and face-to-face seminar participants in perceived social presence, cooperation and satisfaction with the learning experience. Our results support the claim of advocates of third generation distance education methodologies that computers can be an effective enabler, not only of independent learning, but also of collaborative learning. Furthermore, computer-supported collaborative learning environments are as efficient as collaborative learning in face-to-face seminars in developing social presence and increasing professional competencies and self-efficacy.

During the school year 2000-2001, the Ministry of Education and Culture of Cyprus, the Department of Computer Science of the University of Cyprus and the bank of Cyprus implemented the first pilot program on distance learning in elementary schools of Cyprus, called ''ODYSSEUS''. The objective of this paper is to present the evaluation methodology and discuss the most important results with regard to the learning effectiveness and social implications that were realised in the students of two elementary schools that participated in the program. The combination of the new educational technologies with the traditional learning pedagogical model, the aid of learning actions that cause interaction between the students into the virtual classroom and the appointment of school as the most important point of social integration, constitute the more important conclusions of this research. # (P.S. Anastasiades). the world is to help their citizens come up to the needs of the new digital era (2001/C 292/02). As part of this endeavour, the role of our educational system is of utmost importance. The preparation of tomorrow's citizens is incumbent on the elementary schools, which ought to include the new technological applications in the educational process. The integration of new technologies in the sensitive area of a school classroom is not an easy task . The use of computers and educational multimedia, the establishment of network infrastructures and the provision of videoconference systems ensure the technological aspect of this endeavour . The technological tools and applications is the first step in the transition from the current, conventional educational system, to the new model of the virtual classroom and the hybrid school environment . At the same time, it is necessary to develop a pedagogic model, which will constitute the necessary theoretical basis and will mark out the framework where the new educational technologies will be contained.

Despite the advances of the electronic technologies in e-learning, a consolidated evaluation methodology for e-learning applications does not yet exist. The goal of e-learning is to offer the users the possibility to become skillful and acquire knowledge on a new domain. The evaluation of educational software must consider its pedagogic effectiveness as well as its usability. The design of its interface should take into account the way students learn and also provide good usability so that student's interactions with the software are as natural and intuitive as possible. In this paper, we present the results obtained from a first phase of observation and analysis of the interactions of people with e-learning applications. The aim is to provide a methodology for evaluating such applications.

This study aimed to compare the efficacy of collaborative learning in face-to-face and online groups. Fifty psychology majors learnt the same professional skill (a community evaluation methodology) in two seminars taught over a two month period by the same teacher online and face-to-face. Participants in both seminars achieved similar growth in level of professional competence, academic self-efficacy, social self-efficacy and self efficacy for problem solving among members. Post-course evaluation of collaborative experience showed no significant differences between online and face-to-face seminar participants in perceived social presence, cooperation and satisfaction with the learning experience. Our results support the claim of advocates of third generation distance education methodologies that computers can be an effective enabler, not only of independent learning, but also of collaborative learning. Furthermore, computer-supported collaborative learning environments are as efficient as collaborative learning in face-to-face seminars in developing social presence and increasing professional competencies and self-efficacy.

In order to meet the requirements of 4G mobile networks targeted by the cellular layer of IMT-Advanced, Next Generation Mobile WiMAX devices based on IEEE 802.16m will incorporate sophisticated signal processing, seamless handover functionalities between heterogeneous technologies and advanced mobility mechanisms. This survey provides a description of key projected features of the physical (PHY) and medium access control (MAC) layers of 802.16m, as a major candidate for providing aggregate rates at the range of Gbps to high-speed mobile users. Moreover, a new unified method for simulation modeling, namely the Evaluation Methodology (EVM), introduced in 802.16m, is also presented.

This paper presents a quantitative approach to multimodal discourse analysis for analyzing online collaborative learning. The coding framework draws together the fields of systemic functional linguistics and Activity Theory to analyze interactions between collaborative-, content- and technology-related discourse. The approach is used to examine how the task subject matter, the activity design, and the choice of interface affected interaction and collaboration for a computing course conducted in a web-conferencing environment. The analysis revealed the critical impact of activity design on the amount and type of discourse that transpired. Student-centred designs resulted in over six times more student discourse as compared to teacher-centred designs and created a learning environment where students took greater ownership over the tasks and contributed more to the content-based discussion. The paper also incorporates a rationale for the approach to coding and a reflection on its efficacy for discourse analysis in technology-based learning environments.

In the present study, we reject the view that computer use can be studied as an isolated variable in a learning environment. Our main objective is to develop an instrumental tool to measure different types of educational computer use in the classroom. This builds on a comprehensive review of the literature about computer use in education. This review helped to construct a questionnaire to identify a typology of computer use in primary education. In addition, the questionnaire was enriched by input of experts in this field. The questionnaire was presented to a sample of 352 primary school teachers. The input from a first subsample was used to carry out an exploratory factor analysis; the second subsample was used to verify the identified factor structure via confirmatory factor analysis. A three-factor structure of computer use in primary education was identified: 'the use of computers as an information tool', 'the use of computers as a learning tool,' and 'learning basic computer skills'. The three-factor structure was confirmed in the confirmatory factor analysis. The results underpin a number of meaningful differences in the current practice of computer use in primary education.

Adaptive Hypermedia Systems (AHS) can be developed to accommodate a variety of individual differences, including learning style and cognitive style. The current research is an attempt to examine some of the critical variables, which may be important in the design of an Adaptive Educational System (AES) based on student's cognitive style. Moreover, this paper describes the design issues that were considered for the development of the system that are reported in the relevant literature Throughout the development of the system, formative evaluation was an integral part of the design methodology. The results of the formative evaluation were used to improve our system in order to make the instruction more effective and efficient. Furthermore, the recommendations resulted from the formative evaluation could be seen as some points worth considering from designers of AHS. #

A challenge that faces the footwear market is the improvement of casual shoe comfort, being shock absorption one of the most significant properties for comfort. However, the methodologies available for the quantification of the shock absorption properties in materials for footwear applications are limited. With this work, an evaluation methodology of the shock absorption properties of rubber materials is proposed and the relationship between different test methods is established. Eight different commercial compact heeled rubber materials at six different thicknesses were prepared and characterized according to their physical and impact absorption properties. From the experimental values, the following correlations were defined: energy compression/hardness and energy compression/maximum deceleration. Shock absorption prediction curves were also determined from the experimental maximum deceleration values. Results present an excellent tool for industrial applications since they allow correlation of shock absorption properties and heel thickness, and also to predict the contribution of rubber material composition to the final product shock absorption properties.

Design has been recognized for a long time both as art and as science. In the sixties of the previous century design-oriented research began to draw the attention of scientific researchers and methodologists, not only in technical engineering but also in the social sciences. However, a rather limited methodology for design-oriented research has been developed, especially as to the social sciences. In this article we introduce evaluation methodology and research methodology as a systematic input in the process of designing. A designing cycle is formulated with six stages, and for each of these stages operations, guidelines and criteria for evaluation are defined. All this may be used for a considerable improvement of the process and product of designing.

The ARPA MT Evaluation methodology effort is intended to provide a basis for measuring and thereby facilitating the progress of MT systems of the ARPA- sponsored research program. The evaluation methodologies have the further goal of being useful for identifying the context of that progress among developed, production MT systems in use today. Since 1991, the evaluations have evolved as

An evaluation methodology based on fuzzy computing with words aimed at measuring the information quality of Web sites containing documents is presented. This methodology is qualitative and user oriented because it generates linguistic recommendations on the information quality of the content-based Web sites based on users' perceptions. It is composed of two main components, an evaluation scheme to analyze the information quality of Web sites and a measurement method to generate the linguistic recommendations. The evaluation scheme is based on both technical criteria related to the Web site structure and criteria related to the content of information on the Web sites. It is user driven because the chosen criteria are easily understandable by the users, in such a way that Web visitors can assess them by means of linguistic evaluation judgments. The measurement method is user centered because it generates linguistic recommendations of the Web sites based on the visitors' linguistic evaluation judgments. To combine the linguistic evaluation judgments we introduce two new majority guided linguistic aggregation operators, the Majority guided Linguistic Induced Ordered Weighted Averaging (MLIOWA) and weighted MLIOWA operators, which generate the linguistic recommendations according to the majority of the evaluation judgments provided by different visitors. The use of this methodology could improve tasks such as information filtering and evaluation on the World Wide Web.

This paper describes the development and experimental evaluation of an e-laboratory platform in the field of robotics. The system in its current configuration is designed to enable distance training of students in real scenarios of robot manipulator programming. From a technological perspective, the research work presented in this paper is directed towards the adaptation of concepts and techniques developed in the field of telerobotics and virtual reality, and their integration in such e-laboratory settings. This paper focuses particularly on the educational impact of such systems. The goal is to assess the performance of e-laboratory scenarios in terms of the efficacy of training provided to students. The results of a pilot experimental study are presented, providing a comparative evaluation for three training modalities: real, remote, and virtual training on robot manipulator programming. The experiments were conducted according to an evaluation protocol specially designed for the considered target training task, using scoring charts to obtain quantitative performance measures and assess the performance of the student groups participating in the course. Training, as a dynamic process, is approached according to a classical three dimensional model, and performance scores are accordingly assessed in these dimensions (namely: low-level versus mid and high-level skills and understanding). The obtained results reveal certain differences between the three groups, particularly as related to the low-level skill training score, giving some insight about the training 'dimensions' that are expected to be mostly affected by the absence of physical (or realistic virtual) presence in a real hands-on experimentation. Statistical analysis indicates, however, that, despite these apparent differences, such e-laboratory modules can be integrated quite effectively in practical scenarios, creating virtual training environments that can provide adequate learning elements, as related particularly to mid and high-level skill acquisition. Further work and large-scale studies are still needed, though, in order to explore the extent to which such a general conclusion is valid in different training settings, and to form the basis of a more theoretical evaluation for a comprehensive understanding of the pedagogical differences between real, virtual, and remote learning/training methodologies and experiences.

The concepts of well-being and quality of life concern evaluative judgements. There is insufficient understanding in current literature that these judgements are made variously due to the use of not only differing values and differing research instruments but also differing standpoints, differing purposes, and differing theoretical views and ontological presuppositions. The paper elucidates these sources of differences and how they underlie the wide diversity of current conceptions.

This article reports on an evaluation methodology development study for K-12 school and university partnerships. The method is based on . Learning by expanding: An activity-theoretical approach to developmental research. Helsinki: Orienta-Konsultit Oy activity systems analysis that allows researchers to examine qualitative datasets of complex human interactions. This study was designed for participants to evaluate partnership relations and activities. We investigated how the use of activity systems analysis in K-12 school and university partnership evaluation meetings affected participant communication processes. In this study, during a 1-day retreat K-12 school staff and university staff used a modified activity systems model to identify persisting institutional tensions in their program that often trigger miscommunications and strain their relations. During the discussion sessions, study participants collaboratively examined their partnership relations and identified strategies for overcoming difficulties. Additionally, in subsequent monthly partnership meetings during the school year, participants examined findings from the evaluation to design and implement improvement strategies. The results from this methodology development and implementation study provided researchers and partnership participants a new means to (a) evaluate partnership activities, (b) identify institutional barriers, (c) plan future activities, and (d) listen to and incorporate the ideas of less vocal staff members into the planning of future activities. r

Internal self-diagnosis and external evaluation of the obtained results are essential for any automatic system. In the long run these factors are of major importance for the relevance of the system for practical applications. Obviously, this statement is also true for image analysis in photogrammetry and remote sensing. However, so far only relatively little work has been carried out in this area. This paper deals with the external evaluation of automatic road extraction algorithms by means of comparison to manually plotted linear road axis used as reference data. The comparison is performed in two steps: (1) Matching of the extracted primitives to the reference network; (2) Calculation of quality measures. Each part depends on the other: the less tolerant is matching, the less exhaustive the extraction is considered to be, but the more accurate it looks. Therefore, the matching process is an important part of the evaluation process. The quality measures proposed for the automatically extracted road data comprise completeness, correctness, quality, redundancy, planimetric RMS differences and gap statistics. They aim at evaluating exhaustivity as well as assessing geometrical accuracy. The evaluation methodology is presented and discussed in detail. Results of a comparative evaluation of three different automatic road extraction approaches are presented. They show the overall status of the road extractors, as well as the individual strengths and weaknesses of each individual approach. Thus, the applicability of the evaluation method is proven.

Authorship attribution supported by statistical or computational methods has a long history starting from 19th century and marked by the seminal study of on the authorship of the disputed Federalist Papers. During the last decade, this scientific field has been developed substantially taking advantage of research advances in areas such as machine learning, information retrieval, and natural language processing. The plethora of available electronic texts (e.g., e-mail messages, online forum messages, blogs, source code, etc.) indicates a wide variety of applications of this technology provided it is able to handle short and noisy text from multiple candidate authors. In this paper, a survey of recent advances of the automated approaches to attributing authorship is presented examining their characteristics for both text representation and text classification. The focus of this survey is on computational requirements and settings rather than linguistic or literary issues. We also discuss evaluation methodologies and criteria for authorship attribution studies and list open questions that will attract future work in this area.

Eye-movement measures were found to be highly sensitive to the demands of visual and auditory in-vehicle tasks as well as driving task demands. Two newer measures, Percent road centre and Standard deviation of gaze, were found to be more sensitive, more robust, more reliable, and easier to calculate than established glance-based measures. The eye-movement measures were collected by two partners within the EU project HASTE to determine their sensitivity to increasingly demanding in-vehicle tasks by means of artificial, or surrogate, In-vehicle Information Systems (S-IVIS). Data from 119 subjects were collected from four routes: a motorway in real traffic with an instrumented vehicle, a motorway in a fixed base simulator, and from rural roads in two different fixed base simulators. As the visual task became more difficult, drivers looked less at the road centre area ahead, and looked at the display more often, for longer periods, and for more varied durations. The auditory task led to an increasing gaze concentration to road centre. Gaze concentration to the road centre area was also found as driving task complexity increased, as shown in 1369-8478/$ -see front matter Ó (J.L. Harbluk), johan.a.engstrom@volvo.com (J.A. Engströ m). 1 Tel.: +1 613 998 1971; fax: +1 613 990 2913. 2 Tel.: +46 31 322 27 80. www.elsevier.com/locate/trf Transportation Research Part F 8 (2005) 167-190

This paper describes an empirical study of fun, usability, and learning in educational software. Twenty Wve children aged 7 and 8 from an English primary school participated. The study involved three software products that were designed to prepare children for government initiated science tests. Pre and post tests were used to measure the learning eVect, and observations and survey methods were used to assess usability and fun. The Wndings from the study demonstrate that in this instance learning was not correlated with fun or usability, that observed fun and observed usability were correlated, and that children of this age appeared to be able to diVerentiate between the constructs used to describe software quality. The Fun Sorter appears to be an eVective tool for evaluating products with children. The authors discuss the implications of the results, oVer some thoughts on designing experiments with children, and propose some ideas for future work. 

The fast evolution of digital video has brought many new multimedia applications and, as a consequence, has increased the amount of research into new technologies that aim at improving the effectiveness and efficiency of video acquisition, archiving, cataloging and indexing, as well as increasing the usability of stored videos. Among possible research areas, video summarization is an important topic that potentially enables faster browsing of large video collections and also more efficient content indexing and access. Essentially, this research area consists of automatically generating a short summary of a video, which can either be a static summary or a dynamic summary. In this paper, we present VSUMM, a methodology for the production of static video summaries. The method is based on color feature extraction from video frames and k-means clustering algorithm. As an additional contribution, we also develop a novel approach for the evaluation of video static summaries. In this evaluation methodology, video summaries are manually created by users. Then, several user-created summaries are compared both to our approach and also to a number of different techniques in the literature. Experimental results show -with a confidence level of 98% -that the proposed solution provided static video summaries with superior quality relative to the approaches to which it was compared.