Preliminary Assessment Results Of The Nc Lsamp Project

2006-1101: PRELIMINARY ASSESSMENT RESULTS OF THE NC-LSAMP PROJECT Marcia Williams, North Carolina A&T State University Marcia Williams is the Coordinator of Sponsored Programs for the College of Engineering at North Carolina A&T State University and Statewide Coordinator for the NC Louis Stokes Alliance for Minority Participation (NC-LSAMP). She received a B.S. in Industrial Technology (Manufacturing) from North Carolina A&T State University, and a MBA from Wake Forest University. She has sixteen years of experience in sponsored program administration. Gerald Watson, North Carolina A&T State University Jerry Watson is a Ph.D. candidate in Industrial Engineering at North Carolina A&T State University. Xiaochun Jiang, North Carolina A&T State University Xiaochun Jiang is an Assistant Professor in the Department of Industrial and Systems Engineering at North Carolina A&T State University. He received his Ph.D. in Industrial Engineering with a minor in experimental statistics from Clemson University in 2001. He is a member of ASEE, IIE, SME, and HFES. Sanjiv Sarin, North Carolina A&T State University Sanjiv Sarin is the Associate Dean of the College of Engineering at North Carolina A&T State University. He received a B.Tech. in Chemical Engineering from the Indian Institute of Technology, New Delhi and a Ph.D. in Industrial Engineering from the State University of New York at Buffalo. He is a member of ASEE and IIE, and is a registered Professional Engineer in North Carolina. Page 11.1010.1 © American Society for Engineering Education, 2006 The North Carolina Louis Stokes Alliance for Minority Participation (NC LSAMP) project is an ongoing project aimed to substantially increase the number of underrepresented minorities who will contribute significantly in science, technology, engineering and mathematics, engineering, and technology (STEM) areas, especially in graduate degree programs. To assess the effectiveness of the project, student information from the lead institution, NC A&T State University has been collected for analysis. To assess the effectiveness of the project, two control groups were carefully chosen for years 2003 2004 and 2004 2005. Statistical analysis was performed to ensure the compatibility between the control group and the experimental group. Student cumulative GPAs were recorded for each group over the past two years. Preliminary results from the statistical analysis indicated that the NC LSAMP project has had a very significant positive impact on student performance. Participants of the NC LSAMP project performed significantly better than their counterparts in the control group for both years. Although this project is ongoing, it has the potential to significantly impact the retention and graduation rates of underrepresented STEM students. There has been significant shortage in minority students majoring in science and engineering 1. The shortage of minority doctorate students is even more serious 2. The National Science Foundation's Louis Stokes Alliances for Minority Participation (LSAMP) program is a comprehensive, multidisciplinary, undergraduate program designed to increase substantially the quantity and quality of students, especially African American, Hispanic, and Native American students, who successfully complete science, technology, engineering, and mathematics (STEM) baccalaureate degree programs, and increasing the number of students interested in, and academically qualified for and matriculating into programs of graduate study3. The North Carolina Louis Stokes Alliance for Minority Participation (NC LSAMP), funded by the National Science Foundation, formed a partnership between eight institutions (four minority schools and four majority schools) within the University of North Carolina system. Partner institutions include North Carolina A&T State University as lead campus, Fayetteville State University, North Carolina Central University, North Carolina State University, University of North Carolina at Chapel Hill, University of North Carolina at Charlotte, University of North Carolina at Pembroke and Winston Salem State University. Page 11.1010.2 The primary goal of the NC LSAMP project is to substantially increase the number of minority students earning B.S. degrees, and subsequently pursuing M.S. and Ph.D. degrees in science, technology, engineering, and mathematics (STEM) disciplines. The Alliance is achieving this goal by systematically enhancing recruitment, retention, access, and opportunities to education, internships, and research in these fields. So far, the combined efforts of partner institutions have resulted in a variety of programs and activities geared towards enhancing overall student success. Major initiatives include supplemental instruction, bridge programs, undergraduate research, and internships. In addition, an annual research conference sponsored by the Alliance showcases faculty mentored research projects completed by students. The lead institute, North Carolina Agricultural and Technical State University (NCA&T), was established in 1891 and is a public, comprehensive, land grant university committed to fulfilling its fundamental purposes through exemplary undergraduate and graduate instruction, scholarly and creative research, and effective public service. NCA&T is a Historically Black College and University (HBCU), located in Greensboro, North Carolina and enrolls over 10,000 students with about 89% of them being African Americans. It is the leading producer of African American engineers in the nation. The university offers programs at the baccalaureate, masters and doctoral levels with emphasis on engineering, science, technology, business, education, agriculture, and other academic areas. Basic and applied research is conducted by faculty in university centers of excellence, in inter institutional relationships, and through significant involvement with several public and private agencies. The university also conducts major research through engineering, transportation, and its extension programs in agriculture. In order to assess the impact of this program, an experimental study needs to be conducted. ! As part of the assessment effort, a longitudinal study is being and will continue to be conducted at NCA&T4. In this study, five separate cohort groups from NC A&T will be tracked over a five year period – Freshmen, Sophomore, Junior, Senior undergraduate students and first year graduate students. Demographic information such as age, gender, and race, and academic information such as SAT score, High school GPA as well as GPA for each semester of all students participating in the project at NC A&T for both year 2003 2004 and 2004 2005 has been entered to a database developed to help the assessment4. Furthermore, for each year of student participation, details of NC LSAMP activities involved have also been recorded. In order to assess the effectiveness of the project, control groups consisting of students that have no involvement in the project need to be established to compare with the experimental group. Two control groups were carefully chosen for years 2003 2004 and 2004 2005. Statistical analysis was performed to ensure the compatibility between the control groups and the experimental groups. Specifically, demographic data similarities between the two groups were established using the Fisher’s Exact Test while academic credentials similarities were compared using the Wilcoxon rank sum test. For both years, the statistical analysis revealed that the control group and the experimental group were not significantly different in terms of both demographic background and academic credentials. Therefore, the two groups are compatible. In order to evaluate the effectiveness of the project, students cumulative GPAs were recorded by semester for each group over the past two years. Preliminary results from the statistical analysis indicated that the NC LSAMP project has had a very significant positive impact on student performance. Participants of the NC LSAMP project performed significantly better than their counterparts in the control group for each semester of both years of the study. Page 11.1010.3 The following sections will detail the construction of control groups and the preliminary results from the statistical analysis. ! " Demographic information as gender, and race, as well as academic information such as SAT score and High school GPA was used to construct control groups for year 2003 2004 and year 2004 2005, respectively. The selection process employed to match students from the experimental group with students from the control group is described below: 1. The experimental group was divided into five cohorts: Freshmen, Sophomores, Juniors, Seniors, and First Year Graduate students. 2. Data for each student in the experimental group was retrieved based on selection criteria: gender, ethnicity, high school GPA, and SAT scores. Gender—classify them into male and female. Ethnicity –classify them into the following categories: African American, Native American High school GPA SAT 3. For each student in the experimental group, the selection criteria identified in step 2 was used to randomly select compatible students from the STEM student pool. First, five cohorts from the STEM students were established and gender was used to categorize students into two groups for each cohort. Next, Ethnicity was used to narrow the sample for each cohort. Since the population of Native Americans enrolled at school is extremely small, a randomly selected African American student was used when no Native American students available. Finally, SAT score and high school GPA were used for each cohort group. The same number of students was randomly chosen to match the experimental group. Furthermore, once a student was selected from the control group, the record of that student was marked to prevent that student from being selected again in the future. The following table summarizes the basic statistics for both control and experimental groups. Year 2003 2004 #$ % & # ( ( * % ) % & # 4 0 4 0 N/A N/A 3.548 970 & 0 0 0 0 N/A % ' 12 9 15 0 31 23 57 3 % 8 5 13 0 3.496 1056 3.371 1027 4 0 4 0 3.492 1031 % ' 12 9 15 0 31 23 58 2 % 8 5 13 0 3.478 3.218 3.415 3.130 ( ) 92 Mean 3.482 1043 ( ) 92 3.334 Page 11.1010.4 * % ( ( & 0 0 0 0 N/A ) 990 998 1050 955 10 2 12 0 % 0 0 0 0 3.383 1067 0.000 0.000 10 2 12 2 % 0 0 0 0 1022 Year 2004 2005 #$ % & # ( ( * % ) % & # ( ( & 1 0 1 0 10 3 13 0 3.900 1210 3.339 1051 & 1 0 1 0 10 3 13 0 % ' 7 2 9 0 3.078 1017 % ' 7 2 9 0 ( ) 35 Mean 3.396 1132 ( ) 35 3.830 3.118 3.410 3.349 0.000 3.331 * % 1210 1028 1001 1059 0000 1079 ) ( Graduate—first year graduate students AA – African American NA—Native American Table 2.1 Statistics for control and experimental groups for year 2003 2004 and 2004 2005 One of the key issues to ensure the validity of the experiment is to make sure the two groups are compatible. Statistical analyses have been performed based on the screening criteria. Specifically, Fishers Exact test, a statistical test used to determine if there are nonrandom associations between two categorical variables, was performed on ethnicity and gender and Wilcoxon Rank Sum test, a test used in nonparametric statistics to compare the locations of two populations to determine if one population is shifted with respect to another, was performed on the other variables: SAT and High School GPA. The following subsections provide the detailed analyses. !! " $ " Ethnicity and gender are discrete variables. To evaluate the compatibility of control group and experimental group for their ethnicity background and gender, Fisher’s Exact Test was used. Unlike ethnicity and gender, academic performance such as high school GPA, and SAT score are continuous variables. Hence, Wilcoxon Rank Sum Test was used to assess the academic compatibility of the two groups. !! + !,,- !,,. / Page 11.1010.5 First, the Fisher’s Exact Test was performed on entire groups and results indicated there is no statistical difference between the control group and the experimental group ( ). A histogram is provided in Figure 2.2.1.1. Fisher’s Exact Test then was performed for each of the five cohorts and no statistical difference was found for the cohorts in their ethnicity background: freshman (no members in this cohort), sophomore ( ), junior ( ), senior ( ), and first year graduate student ( ). Figure 2.2.1.2 provides ethnic information for senior students. " "# Figure 2.2.1.1 Students by Ethnicity ! " Figure 2.2.1.2 Senior Students by Ethnicity Next, Fisher’s Exact Test was performed on data based on gender and results can be seen from Figures 2.2.1.3 and 2.2.1.4. No statistical difference was found for all students ( ), or each of the five cohorts: freshman (no members in this cohort), sophomore ( ), junior ( ), senior ( ), and first year graduate student ( ). ! # $ % % & " "# & ! " Figure 2.2.1.3 Students by Gender Figure 2.2.1.4 Senior Students by Gender Page 11.1010.6 Wilcoxon Rank Sum Test on entire groups was performed on high school GPA (HSGPA) and results indicated there is no statistical difference between the control group and the experimental group ( ). A histogram is provided in Figure 2.2.1.5. Wilcoxon Rank Sum Test then was performed for each of the five cohorts and no statistical difference (all p values are greater than significance level α which was chosen to be 0.05) was found at the 0.05 significance level for the cohorts in their high school GPA: freshman (no members in this cohort), sophomore ( ), junior ( ), senior ( ), and first year graduate student ( ). Figure 2.2.1.6 provides high school GPA information for senior students. '$ '$ '$ '$ '$ '$ '$ '$ Figure 2.2.1.5 High School GPA Figure 2.2.1.6 Senior Students HSGPA Wilcoxon Rank Sum Test on entire groups was performed on SAT scores and results indicated there is no statistical difference between the control group and the experimental group ( ). A histogram is provided in Figure 2.2.1.7. Wilcoxon Rank Sum Test then was performed for each of the five cohorts and no statistical difference was found for the cohorts in their high school GPA: freshman (no members in this cohort), sophomore ( ), junior ( ), senior ( ), and first year graduate student ( ). Figure 2.2.1.8 provides SAT information for senior students. ! ! ! ! ) ) ( ( Figure 2.2.1.7 Participants SAT !!! Figure 2.2.1.8 Senior Participants SAT !,,. !,,0 / First, the Fisher’s Exact Test was performed on ethnicity for entire groups and results indicated there is no statistical difference between the control group and the experimental group ( ). A histogram is provided in Figure 2.2.2.1. Fisher’s Exact Test then was performed for each of the five cohorts and no statistical difference was found for the cohorts in their ethnicity background: freshman ( ), sophomore ( ), junior ( ), and senior ( ). Figure 2.2.2.2 provides ethnic information for senior students. $ "# "# $ $ " $ $ ! " Page 11.1010.7 Figure 2.2.2.1 Students by Ethnicity $ " "# $ $ ! " Figure 2.2.2.2 Senior Students by Ethnicity Next, Fisher’s Exact Test was performed on data based on gender and results can be seen from Figure 2.2.2.3. No statistical difference was found for all students ( ), or each of the five cohorts: freshman ( ), sophomore ( ), junior ( ), and senior ( ). $* ! # " "# % & ! " Figure 2.2.2.3 Students by Gender Wilcoxon Rank Sum Test on high school GPA for entire groups was performed and results indicated there is no statistical difference between the control group and the experimental group ( ). A histogram is provided in Figure 2.2.2.5. Wilcoxon Rank Sum Test then was performed for each of the five cohorts and no statistical difference was found for the cohorts in their high school GPA: freshman ( ), sophomore ( ), junior ( ), and senior ( ). Figure 2.2.2.6 provides high school GPA information for senior students. " Figure 2.2.2.5 High School GPA " Figure 2.2.2.6 Senior Student HSGPA Page 11.1010.8 Wilcoxon Rank Sum Test on SAT score for entire groups was performed and results indicated there is no statistical difference between the control group and the experimental group ( ). A histogram is provided in Figure 2.2.2.7. Wilcoxon Rank Sum Test then was performed for each of the five cohorts and no statistical difference was found for the cohorts in their high school GPA: freshman ( ), sophomore ( ), junior ( ), and senior ( ). Figure 2.2.2.8 provides SAT information for senior students. " ! " ) ) ( ( Figure 2.2.2.7 SAT ! Figure 2.2.2.8 Senior Student SAT Simple statistics such as central tendency measures (e.g., mean, median), and variability measures (e.g., variance, standard deviation) have been computed for the data collected for both academic years. Statistical analyses using Wicoxon Rank Sum Test were performed. The following are the results from the analyses. - + !,,- !,,.1 Table 3.1.1 2003 2004 Students Cumulative GPA Semester " " !* Fall 2003 Spring 2004 Fall 2004 Spring 2005 2003 2004 Students Cumulative GPA Experimental Control Wilcoxon Rank Sum Test P value - -2- ,02 , ,,,! - -34 - 04 , ,,30 - -20 - ,.. , ,!,5 - ..0 - ,. , ,,0! Significant? Y Y Y Y ' ' ' ' & + & + $ Figure 3.1.1 2003 2004 Students GPA by Semester Page 11.1010.9 Clearly, by looking at the chart above, we can find that students in the experimental group— those who were supported by NC LSAMP project performed better than those in the control group. Results from Wicoxon Rank Sum test also supported this conclusion. Statistically, students performed better than those in the control group for all semesters as seen in table 3.1.1. -!+ !,,. !,,0 Table 3.2.1 2004 2005 Students Cumulative GPA 2004 2005 Students Cumulative GPA Semester Experimental Control Wilcoxon Rank Sum Test P value Fall 2004 -- . - ,-, ,0!4 Spring 2005 - ,22 ! 4,5 , , 55 Significant? N Y !* ' ' " " " ' ' ' & + $ Figure 3.2.1 2004 2005 Students GPA by Semester Looking at the chart above, it is obvious that students in the experimental group—those who were supported by NC LSAMP project performed better than those in the control group. Results from Wilcoxon Rank Sum test also supported this conclusion. Although statistically, students performed the same as those in the control group for Fall 2004, they did perform better than those in the control group for Spring 2005. Therefore, it can be concluded that by far, the NC LSAMP project is working very well. However, in order to fully assess the effectiveness of the program, we need to continue to follow student performance and collect longitudinal data for analysis upon completion of the project. Nevertheless, it is safe to say this project is very promising. . Although the NC LSAMP project is still ongoing, it has the potential to significantly impact the retention and graduation rates of underrepresented STEM students. Furthermore, this project will have positive impact on the number of underrepresented STEM students who enroll in the graduate programs. Page 11.1010.10 0 6 7 " We would like to acknowledge the National Science Foundation for supporting the NC LSAMP project. 2 1. Oguntimein, G. B., Leigh Mack, P., Davy, B., and Wheatland, J.(2004), “Research experience program for undergraduates in a Historically Black College and University.” Proceedings of the 2004 American Society for engineering Education Annual Conference & Exposition. 2. Mervis, J. (1998), “Wanted: A better Way to boost number of minority Ph.D.s”, Science 281, 1268 1270. 3. NSF LSAMP Program: http://www.ehr.nsf.gov/hrd/amp.asp 4. Jiang, X., Sarin, S., and Williams, M. (2005), “Assessment of NC LSAMP project: A longitudinal Study”, Proceedings of the 2005 American Society for engineering Education Annual Conference & Exposition, Portland, OR. 8 " 6 9 Marcia Williams is the Coordinator of Sponsored Programs for the College of Engineering at North Carolina A&T State University and Statewide Coordinator for the NC Louis Stokes Alliance for Minority Participation (NC LSAMP). She received a B.S. in Industrial Technology (Manufacturing) from North Carolina A&T State University, and a MBA from Wake Forest University. She has sixteen years of experience in sponsored program administration. She is a member of ASEE. ' 9 Jerry Watson is a Ph.D. candidate in Industrial Engineering at North Carolina A&T State University. : ' " Xiaochun Jiang is an Assistant Professor in the Department of Industrial and Systems Engineering at North Carolina A&T State University. He received his Ph.D. in Industrial Engineering with a minor in experimental statistics from Clemson University in 2001. He is a member of ASEE, IIE, SME, and HFES. ; Sanjiv Sarin is the Associate Dean of the College of Engineering at North Carolina A&T State University. He received a B.Tech. in Chemical Engineering from the Indian Institute of Technology, New Delhi and a Ph.D. in Industrial Engineering from the State University of New York at Buffalo. He is a member of ASEE and IIE, and is a registered Professional Engineer in North Carolina. Page 11.1010.11