Evaluation Methodology

The paper presents an empirical study with a digital educational game (DEG) called 80Days that aims at teaching geographical content. The goal of the study is twofold: (i) investigating the potential of the eye-tracking approach for evaluating DEG; (ii) studying the issue of vicarious learning in the context of DEG. Twenty-four university students were asked to view the videos of playing two micro-missions of 80Days, which varied with regard to the position of the non-player character (NPC) window (i.e. lower right vs. upper left) and the delivery of cognitive hints (i.e. with vs. without) in this text window. Eye movements of the participants were recorded with an eye-tracker. Learning effect and user experience were measured by questionnaires and interviews. Significant differences between the pre-and post-learning assessment tests suggest that observers can benefit from passive viewing of the recorded gameplay. However, the hypotheses that the game versions with cognitive hints and with the NPC window on the upper left corner can induce stronger visual attention and thus better learning effect are refuted.

Literature Based Discovery (LBD) is a process that searches for hidden and important connections among information embedded in published literature. Employing techniques from Information Retrieval and Natural Language Processing, LBD has potential for widespread application yet is currently implemented primarily in the medical domain. This article examines several published LBD systems, comparing their descriptions of domain and input data, techniques

Information visualizations have been shown useful in numerous laboratory studies, but their adoption and use in real-life tasks are curiously under-researched. We present a field study of ten programmers who work with an editor extended with a fisheye view of source code. The study triangulates multiple methods (experience sampling, logging, thinking aloud, and interviews) to describe how the visualization is adopted and used. At the concrete level, our results suggest that the visualization was used as frequently as other tools in the programming environment. We also propose extensions to the interface and discuss features that were not used in practice. At the methodological level, the study identifies contributions distinct to individual methods and to their combination, and discusses the relative benefits of laboratory studies and field studies for the evaluation of information visualizations. This paper aims to combine and make contributions within two themes: the methodology of information visualization evaluation and fisheye views for supporting programmers. Next we summarize the relevant literature for each theme.

One of the major challenges in camera document analysis is to deal with the page curl and perspective distortions. In spite of the prevalence of dewarping techniques, no standard for their performance evaluation method exists with most of the evaluation done to concentrate in visual pleasing impressions. This paper presents an objective evaluation methodology for document image dewarping techniques. First,

This paper presents an objective evaluation methodology for document image binarization techniques that aims to reduce the human involvement in the ground truth construction and consecutive testing. A skeletonized ground truth image is produced by the user following a semi-automatic procedure. The estimated ground truth image can aid in evaluating the binarization result in terms of recall and precision as well as to further analyze the result by calculating broken and missing text, deformations and false alarms. A detailed description of the methodology along with a benchmarking of the six (6) most promising state-of-the-art binarization algorithms based on the proposed methodology is presented.

This article describes a approach to quantify the quality of thresholding-based segmentation on Brazilian bank checks. The checks are previously acquired by scanner and soon after, their colored patterns are decomposed according to the HSI model. Several thresholding algorithms are then applied and an automatic evaluation of the results is carried out. The evaluation methodology is based on "performance assessment" of the character recognition rate from an OCR.

Geographically-grounded situational awareness (SA) is critical to crisis management and is essential in many other decision making domains that range from infectious disease monitoring, through regional planning, to political campaigning. Social media are becoming an important information input to support situational assessment (to produce awareness) in all domains. Here, we present a geovisual analytics approach to supporting SA for crisis events using one source of social media, Twitter. Specifically, we focus on leveraging explicit and implicit geographic information for tweets, on developing place-time-theme indexing schemes that support overview+detail methods and that scale analytical capabilities to relatively large tweet volumes, and on providing visual interface methods to enable understanding of place, time, and theme components of evolving situations. Our approach is user-centered, using scenario-based design methods that include formal scenarios to guide design and valid...

The emergence of public housing estate renewal programs in Australia in the last decade has been one of the most prominent developments in social housing policy. These programs have undertaken a broad mix of renewal activity, ranging from outright physical redevelopment and stock replacement for sale, to community development type initiatives to improve social and employment outcomes for residents. However, while a number of evaluations of these programs have been undertaken, the development of evaluation methodology has lagged behind that of other countries. This in part is due to the lack of federal government interest or involvement in these programs which are essentially state specific. The article reviews the evaluations that have been undertaken in the last 10 years in Australia and assesses the relative importance of qualitative methodologies in these evaluations. Despite a strong focus among policy makers on value-for-money aspects of renewal, the authors show that qualitative methods have been commonplace, if limited in range, and argue this is a result of both the difficulty of obtaining comparative quantitative information especially when comparing dissimilar programs between states. In this context, qualitative methods are more easily managed by researchers and offer more insightful assessments than quantitatively based approaches. The article concludes by arguing for a national evaluation methodology to assist in more rigorous evaluations and the extension of qualitative evaluation methods.

Design has been recognized for a long time both as art and as science. In the sixties of the previous century design-oriented research began to draw the attention of scientific researchers and methodologists, not only in technical engineering but also in the social sciences. However, a rather limited methodology for design-oriented research has been developed, especially as to the social sciences. In this article we introduce evaluation methodology and research methodology as a systematic input in the process of designing. A designing cycle is formulated with six stages, and for each of these stages operations, guidelines and criteria for evaluation are defined. All this may be used for a considerable improvement of the process and product of designing.

Context Interest 1 in the teaching of communication skills in medical schools has increased since the early seventies but, despite this growing interest, relatively limited curricular time is spent on the teaching of communication skills. The limited attention to the teaching of these skills applies even more to the physicians' clinical years, when attention becomes highly focused on biomedical and technical competence. Continuing training after medical school is necessary to refresh knowledge and skills, to prohibit decline of performance and to establish further improvements. Objective This review provides an overview of evaluation studies of communication skills training programmes for clinically experienced physicians who have ®nished their undergraduate medical education. The review focuses on the training objectives, the applied educational methods, the evaluation methodology and instruments, and training results. Methods CD-ROM searches were performed on Med-Line and Psychlit, with a focus on effect-studies dating from 1985. Results Fifteen papers on 14 evaluation studies were located. There appears to be some consistency in the aims and methods of the training programmes. Course effect measurements include physician self-ratings, independent behavioural observations and patient outcomes. Most of the studies used inadequate research designs. Overall, positive training effects on the physicians' communication behaviour are found on half or less of the observed behaviours. Studies with the most adequate 2 designs report the fewest positive training effects. Conclusion Several reasons are discussed to explain the limited ®ndings. Future research may bene®t from research methods which focus on factors that inhibit and facilitate the physicians' implementation of skills into actual behaviours in daily practice.

The information society has developed with differing levels of success in Western Europe and the United States, as well as in Eastern Europe. E-government – a significant element of the information society – has an important and leading role in some countries, while elsewhere it lags well behind the development level seen in the private sector, non-governmental organisations, and even among citizens. Different strategies and action plans have been adopted in many countries, with different results being achieved against a backdrop of optimistic forecasts and hopes. Though Slovenia has become a new member of the European Union, Slovenian public administration still retains many characteristics of the old political system and is attempting to successfully apply egovernment in practice, both in terms of back-office and front-office operations. Its development has been underway for just a few years, and includes some areas that match the national strategy and plans, and some areas that d...

Human-robot interaction (HRI) for mobile robots is still in its infancy. Most user interactions with robots have been limited to tele-operation capabilities where the most common interface provided to the user has been the video feed from the robotic platform ...

Two of the most critical requirements in support of producing reliable face-recognition systems are a large database of facial images and a testing procedure to evaluate systems. The Face Recognition Technology (FERET) program has addressed both issues through the FERET database of facial images and the establishment of the FERET tests. To date, 14,126 images from 1199 individuals are included in the FERET database, which is divided into development and sequestered portions. In September 1996, the FERET program administered the third in a series of FERET face-recognition tests. The primary objectives of the third test were to (1) assess the state of the art, (2) identify future areas of research, and (3) measure algorithm performance on large databases

Sustainable environmental management is contingent on having an effective environmental planning system. A new methodology for designing and evaluating environmental planning systems is described and applied to a case study evaluation of the Canadian environmental planning process. The methodology is based on eight international best practice principles for environmental planning and 45 indicators. The research illustrates the benefits of the evaluation methodology in identifying how to improve environmental planning systems to achieve desired results. The methodology is applicable to a wide variety of jurisdictions.

The use of qualified panels applying specific evaluation methods for sensory quality control increases the reliability of the results, even more if the method is accredited by official accreditation bodies.Based on the work developed with young red wine, this paper describes all the steps carried out after having defined the method: assessor selection, basic and specific training, qualification of assessors

Despite recent advances of electronic technologies in e-learning, a consolidated evaluation methodology for e-learning applications is not available. The evaluation of educational software must consider its usability and more in general its accessibility, as well as its didactic effectiveness. This work is a first step towards the definition of a methodology for evaluating e-learning applications. Specific usability attributes capturing the peculiar features of these applications are identified. A preliminary user study involving a group of e-students, observed during their interaction with an e-learning application in a real situation, is reported. Then, the proposal is put forward to adapt to the e-learning domain a methodology for systematic usability evaluation, called SUE. Specifically, evaluation patterns are proposed that are able to drive the evaluators in the analysis of an e-learning application.

In this paper, several issues regarding the epistemology of technological pedagogical content knowledge (TPCK) are first raised for the purpose of clarifying the construct. Specifically, the transformative and integrative views are juxtaposed for exploring the epistemology of TPCK, and, at the end, the transformative view is adopted concluding that TPCK is a unique body of knowledge that is constructed from the interaction of its individual contributing knowledge bases. Then, ICT-TPCK is introduced as a strand of TPCK, and is described as the ways knowledge about tools and their affordances, pedagogy, content, learners, and context are synthesized into an understanding of how particular topics that are difficult to be understood by learners or difficult to be represented by teachers can be transformed and taught more effectively with technology in ways that signify its added value. One model for the development and another for the assessment of ICT-TPCK are then discussed. Technology Mapping is proposed as a situative methodology for the development of ICT-TPCK, and three forms of assessment, namely, expert assessment, peer assessment, and self-assessment are proposed for assessing teachers' competencies to teach with technology. The paper also reports on the empirical findings of a study that was undertaken to investigate the impact of the proposed models on student learning within the context of two design tasks in a pre-service primary teacher education course. Repeated measures within-subject effects were tested and the results indicated that ICT-TPCK competency significantly improved over the course of a semester. The results of this study clearly show that the theoretical models proposed herein can positively impact the development of ICT-TPCK. Lastly, these results can be used as baseline data in future studies that may be conducted to further validate or improve the proposed models in different contexts.

Earlier studies have suggested that higher education institutions could harness the predictive power of Learning Management System (LMS) data to develop reporting tools that identify at-risk students and allow for more timely pedagogical interventions. This paper confirms and extends this proposition by providing data from an international research project investigating which student online activities accurately predict academic achievement. Analysis of LMS tracking data from a Blackboard Vista-supported course identified 15 variables demonstrating a significant simple correlation with student final grade. Regression modelling generated a best-fit predictive model for this course which incorporates key variables such as total number of discussion messages posted, total number of mail messages sent, and total number of assessments completed and which explains more than 30% of the variation in student final grade. Logistic modelling demonstrated the predictive power of this model, which correctly identified 81% of students who achieved a failing grade. Moreover, network analysis of course discussion forums afforded insight into the development of the student learning community by identifying disconnected students, patterns of student-to-student communication, and instructor positioning within the network. This study affirms that pedagogically meaningful information can be extracted from LMS-generated student tracking data, and discusses how these findings are informing the development of a customizable dashboardlike reporting tool for educators that will extract and visualize real-time data on student engagement and likelihood of success.

The fast evolution of digital video has brought many new multimedia applications and, as a consequence, has increased the amount of research into new technologies that aim at improving the effectiveness and efficiency of video acquisition, archiving, cataloging and indexing, as well as increasing the usability of stored videos. Among possible research areas, video summarization is an important topic that potentially enables faster browsing of large video collections and also more efficient content indexing and access. Essentially, this research area consists of automatically generating a short summary of a video, which can either be a static summary or a dynamic summary. In this paper, we present VSUMM, a methodology for the production of static video summaries. The method is based on color feature extraction from video frames and k-means clustering algorithm. As an additional contribution, we also develop a novel approach for the evaluation of video static summaries. In this evaluation methodology, video summaries are manually created by users. Then, several user-created summaries are compared both to our approach and also to a number of different techniques in the literature. Experimental results show -with a confidence level of 98% -that the proposed solution provided static video summaries with superior quality relative to the approaches to which it was compared.

For a particularly long time, automatic diagnosis of diabetic retinopathy from digital fundus images has been an active research topic in the medical image processing community. The research interest is justified by the excessive potential for new products in the medical industry and possible reductions in healthcare costs. However, the maturity of state-of-the-art algorithms cannot be judged due to the lack of commonly accepted and representative image database with a verified ground truth and strict evaluation protocol. In this study, problems and issues related to the database are discussed from medical, image processing, and security perspectives. Based on the discussion, an evaluation methodology is proposed and a prototype image database with the ground truth is described. The database is made publicly available for benchmarking diagnosis algorithms. By utilizing the proposed database, it is now possible compare different algorithms, and correspondingly, analyze their maturity for technology transfer from the research laboratories to the medical practice.

A challenge that faces the footwear market is the improvement of casual shoe comfort, being shock absorption one of the most significant properties for comfort. However, the methodologies available for the quantification of the shock absorption properties in materials for footwear applications are limited. With this work, an evaluation methodology of the shock absorption properties of rubber materials is proposed and the relationship between different test methods is established. Eight different commercial compact heeled rubber materials at six different thicknesses were prepared and characterized according to their physical and impact absorption properties. From the experimental values, the following correlations were defined: energy compression/hardness and energy compression/maximum deceleration. Shock absorption prediction curves were also determined from the experimental maximum deceleration values. Results present an excellent tool for industrial applications since they allow correlation of shock absorption properties and heel thickness, and also to predict the contribution of rubber material composition to the final product shock absorption properties.

Despite the advances of the electronic technologies in e-learning, a consolidated evaluation methodology for e-learning applications does not yet exist. The goal of e-learning is to offer the users the possibility to become skillful and acquire knowledge on a new domain. The evaluation of educational software must consider its pedagogic effectiveness as well as its usability. The design of its interface should take into account the way students learn and also provide good usability so that student's interactions with the software are as natural and intuitive as possible. In this paper, we present the results obtained from a first phase of observation and analysis of the interactions of people with e-learning applications. The aim is to provide a methodology for evaluating such applications.

This paper proposes a structured methodology for the evaluation of alternative Electronic Commerce technologies to be introduced into SMEs' (Small to Medium Enterprises) every-day business practices. The main tools used by the methodology are Discrete Event Simulation and Activity Based Costing/ Management (ABC/ABM). The enterprise strategy definition is supported by the GRAI methodology, whereas the business process modeling is carried out with the use of IDEF-0 and IDEF-3 activity modelling tools. The application of the methodology and the results produced are demonstrated for a Greek medium-sized enterprise operating in the Clothing Industry. Conclusions and further research efforts are presented in the end of the paper.

This paper develops personalized models for different university degrees to obtain the risk of each student abandoning his degree and analyzes the profile for undergraduates that abandon the degree. In this study three faculties located in Granada, South of Spain, were involved. In Software Engineering three university degrees with 10,844 students, in humanities nineteen university degrees with 39,241 students and in Economic Sciences five university degrees with 25,745 students were considered. Data, corresponding to the period 1992 onwards, are used to obtain a model of logistic regression for each faculty which represents them satisfactorily. These models and the framework data show that certain variables appear repeatedly in the explanation of the drop out in all of the faculties. These variables are, among others, start age, the father's and mother's studies, academic performance, success, average mark in the degree and the access form and in some cases also, the number of rounds needed to pass. Students with weak educational strategies and without persistence to achieve their aims in life have low academic performance and low success rates and this implies a high risk of abandoning the degree. The results suggest that each university centre could consider similar models to elaborate a particular action plan to help lower the drop out rate reducing costs and efforts. As concluded in this paper, the profile of the students who tend to abandon their studies is dependent on the subject studied. For this reason, a general methodology based on a Data Warehouse architecture is proposed. This architecture does most of the work automatically and is general enough to be used at any university centre because it only takes into account the usual data the students provide when registered in a course and their grades throughout the years.

In line with recent research, the question this paper raises is whether or not gender differences also exist in e-learning. This study is based on a sample of 1,185 students who are doing on-line courses at the Universidad de Granada in Spain. The main conclusion is that female students are more satisfied than male students with the elearning subjects that make up the sample. Furthermore, we find that female students assign more importance to the planning of learning, as well as to being able to contact the teacher in various ways.

Art museums and galleries must increasingly demonstrate their social relevance and provide evidence of their educational value. A large number of audiences who repeatedly visit galleries are from public and private schools, including university and adult education groups, who demand that museums maintain their relevance within the education curriculum. This paper examines the critical role of visitor studies to reveal how people are learning from their museum experiences. It argues for evaluation to become an integral and ongoing part of museum education programs by demonstrating the increasingly self-critical and self-enquiring role that museum education staff must adopt to better meet visitor needs within the context of increasing professionalism, accountability, customer service and educational reform. The paper is a summative evaluation case study that employs both qualitative and quantitative methods to evaluate the effectiveness of the National Gallery of Victoria’s literacy program, ‘Art and Language.’ The study outlines the methodology related to the design and implementation of an evaluation strategy that can be adopted for future evaluation processes. Based upon evaluation findings, the study lists suggestions for improving the program and reinforces the usefulness of evaluation in proving the importance of literacy programs in museum education. The paper concludes by providing recommendations to increase program evaluation activities across all museum education programs.

Background: The Course Experience Questionnaire (CEQ) provides general feedback from graduates on teaching quality in degree programs, but is not targeted to medical courses. This study aimed to develop and pilot graduate survey questions that complement the existing CEQ questions, and that reflect issues relevant to medical courses.

Although many researchers have studied different factors which affect E-Learning outcomes, there is little research on assessment of the intervening role of readiness factors in E-Learning outcomes. This study proposes a conceptual model to determine the role of readiness factors in the relationship between E-Learning factors and E-Learning outcomes. Readiness factors are divided into three main groups including: technical, organizational and social. A questionnaire was completed by 96 respondents. This sample consists of teachers at Tehran high schools who are utilizing a technology-based educating. Hierarchical regression analysis is done and its results strongly support the appropriateness of the proposed model and prove that readiness factors variable plays a moderating role in the relationship between E-Learning factors and outcomes. Also latent moderated structuring (LMS) technique and MPLUS3 software are used to determine each variable’s ranking. Results show that organizational readiness factors have the most important effect on E-Learning outcomes. Also teachers’ motivation and training is the most important factor in E-Learning. Findings of this research will be helpful for both academics and practitioners of E-Learning systems.►A framework of the effects of readiness factors on E-Learning outcomes is proposed. ►Readiness factors include three categories: technical, organizational and social. ►A questionnaire was completed by 96 respondents. ►Organizational readiness factors have the most important effect on E-Learning. ►Also teachers’ motivation and training is the most important factor in E-Learning.

Authorship attribution supported by statistical or computational methods has a long history starting from 19th century and marked by the seminal study of on the authorship of the disputed Federalist Papers. During the last decade, this scientific field has been developed substantially taking advantage of research advances in areas such as machine learning, information retrieval, and natural language processing. The plethora of available electronic texts (e.g., e-mail messages, online forum messages, blogs, source code, etc.) indicates a wide variety of applications of this technology provided it is able to handle short and noisy text from multiple candidate authors. In this paper, a survey of recent advances of the automated approaches to attributing authorship is presented examining their characteristics for both text representation and text classification. The focus of this survey is on computational requirements and settings rather than linguistic or literary issues. We also discuss evaluation methodologies and criteria for authorship attribution studies and list open questions that will attract future work in this area.

This study examined business students' perceptions of four objectives (i.e., Enjoyment, Learning, Motivation, and Career Application) across five teaching technologies (i.e., Projector, PowerPoint, Video, the Internet, and Lecture), business professors' effective application of technologies, and students' academic performance. We collected data from 215 students at a regional state university in the USA. We developed Students' Perceptions of Technology Scale, SPOTS, specifically for the present study, used the most rigorous criteria, and investigated reliability, convergent and discriminant validity, exploratory and confirmatory factor analysis, and measurement invariance of this scale. Mean scores revealed that Video conveyed the highest amount of Enjoyment. PowerPoint provided the highest amount of Learning and Motivation. The Internet provided the highest Career Application for future jobs. Younger students preferred Video, whereas older students favored Lecture. Regression results showed that the use of Video for Learning, Projector and Lecture for Enjoyment, PowerPoint for career and Motivation, and the Internet for Learning contributed to professors' teaching effectiveness. Students' high ratings for professors' effective use of the Lecture method and low expectation for the use of a Projector predicted their self-reported GPA (academic performance). Professors may use a different mix of technologies in the classroom and use them creatively in order to promote the most Learning for students and satisfy students' Learning needs and objectives.

In order to meet the requirements of 4G mobile networks targeted by the cellular layer of IMT-Advanced, Next Generation Mobile WiMAX devices based on IEEE 802.16m will incorporate sophisticated signal processing, seamless handover functionalities between heterogeneous technologies and advanced mobility mechanisms. This survey provides a description of key projected features of the physical (PHY) and medium access control (MAC) layers of 802.16m, as a major candidate for providing aggregate rates at the range of Gbps to high-speed mobile users. Moreover, a new unified method for simulation modeling, namely the Evaluation Methodology (EVM), introduced in 802.16m, is also presented.

In this paper, several issues regarding the epistemology of technological pedagogical content knowledge (TPCK) are first raised for the purpose of clarifying the construct. Specifically, the transformative and integrative views are juxtaposed for exploring the epistemology of TPCK, and, at the end, the transformative view is adopted concluding that TPCK is a unique body of knowledge that is constructed from the interaction of its individual contributing knowledge bases. Then, ICT-TPCK is introduced as a strand of TPCK, and is described as the ways knowledge about tools and their affordances, pedagogy, content, learners, and context are synthesized into an understanding of how particular topics that are difficult to be understood by learners or difficult to be represented by teachers can be transformed and taught more effectively with technology in ways that signify its added value. One model for the development and another for the assessment of ICT-TPCK are then discussed. Technology Mapping is proposed as a situative methodology for the development of ICT-TPCK, and three forms of assessment, namely, expert assessment, peer assessment, and self-assessment are proposed for assessing teachers' competencies to teach with technology. The paper also reports on the empirical findings of a study that was undertaken to investigate the impact of the proposed models on student learning within the context of two design tasks in a pre-service primary teacher education course. Repeated measures within-subject effects were tested and the results indicated that ICT-TPCK competency significantly improved over the course of a semester. The results of this study clearly show that the theoretical models proposed herein can positively impact the development of ICT-TPCK. Lastly, these results can be used as baseline data in future studies that may be conducted to further validate or improve the proposed models in different contexts.

The purpose of this article is twofold. First, evaluation scoring systems for triple bottom line (TBL) reports to date are examined and potential methodological weaknesses and problems are highlighted. In this context, a new assessment methodology is presented based explicitly on the most widely acknowledged standard on non-financial reporting worldwide, the Global Reporting Initiative (GRI) guidelines. The set of GRI topics and performance indicators was converted into scoring criteria while the generic scoring devise was set from 0 to 4 points. Secondly, the proposed benchmark tool was applied to the TBL reports published by Greek companies. Results reveal major gaps in reporting practices, stressing the need for the further development of internal systems and processes in order to collect essential non-financial performance data. A critical overview of the structure and rationale of the evaluation tool in conjunction with the Greek case study is discussed while recommendations for future research on the field of this relatively new form of reporting are suggested.

Performance evaluation of visual tracking algorithms is a complex task requiring consideration of the robustness, accuracy and failure modes of any proposed method. Both artificial and real data sets are typically employed, with quantitative, distancebased measures of tracking accuracy supported by qualitative, manual analysis of robustness and failure modes. Failure is usually taken to mean dissociation of the tracker with its target, and is identified by eye. Although distance measures are valid given artificial data, manually generated ground truth is not sufficient to allow them to be used reliably on real data sets. This paper presents an alternative evaluation methodology in which quantitative measures, in the form of statistical tests, are applied to the evaluation of both accuracy and robustness. These tests then form the basis of a tool that automatically identifies tracking failures, focusing attention on a well-defined set of events.