ALL-MS-Assessment-Report 2019-2020

Assessment Report, 2019-2020
Masters Degree Program

Adult Learning and Leadership
MS ADULT LEARNING AND LEADERSHIP ASSESSMENT 1
Assessment Report, 2019-2020 ............................................................................................................................. 5

Introduction. .......................................................................................................................................................................................5
Student Learning Outcomes ...............................................................................................................................................................5
Evaluation Approach. ....................................................................................................................................................... 6

Direct Measures. ................................................................................................................................................................................6
Indirect Measures. ..............................................................................................................................................................................7
Internal and external validity threats. .................................................................................................................................................7
Direct Measure Findings ....................................................................................................................................................................7
SLOs summaries and results. ......................................................................................................................................................10
Student self-assessments. ............................................................................................................................................................15
Indirect Measure Findings ...............................................................................................................................................................15
Student narrative self-assessment of SLOs .................................................................................................................................15
End of program reports ...............................................................................................................................................................23
Inter-Reliablity Statistical Tests. ......................................................................................................................................................30
Cohen’s Kappa. ...........................................................................................................................................................................30
Spearman’s Rho statistical test for inter-reliability. ....................................................................................................................31
Assessing Evaluations and Gender. ............................................................................................................................................33
Assessment Report Review and Recommendation. ..................................................................................................... 36

List of Tables
Table 1. Masters of Adult Learning and Leadership SLOs. ................................................................................ 6
Table 2. Direct Measures....................................................................................................................................... 7
Table 3. Items Key to Enrollment. ...................................................................................................................... 24
Table 4. Students Assessment of Program Quality ............................................................................................ 24
Table 5. Kappa Assessment for Mutual Agreement. .......................................................................................... 31
Table 6. Rater Inter-Reliability, Spearman Rho. ............................................................................................... 32
Table 7. Review and Recommendations. ............................................................................................................ 36
List of Figures
Figure 1. Student Populations/Ratings, AYs 2013/2014 thru 2019/2020..................................... 8

Figure 2. Yearly Average SLO Rating. ......................................................................................... 9
Figure 3. Disbursement of Populations Across Delivery Methods. ............................................. 9
Figure 4. Average SLO Ratings Across Sites and Online Platform. ......................................... 10
Figure 5. SLO 1 Literature Integration. ..................................................................................... 11
Figure 6. SLO 2. Research Process. .......................................................................................... 12
Figure 7. SLO 3. Social Justice Issues. ..................................................................................... 12
Figure 8. SLO 4. Technological Impacts................................................................................... 13
Figure 9. SLO 5. Written Communication Skills. ..................................................................... 13
Figure 10. SLO 6. Synthesize Information. ............................................................................... 14
Figure 11. SLO 7. Moral and Ethical Responsibilities. ............................................................ 14
Figure 12. Students Self-Assessments vs Raters' Assessment, Consolidated. ........................... 15
Figure 13. Student Self-Assessment Narratives, SLO 1, Literature Integration. ..................... 16
Figure 14. Student Self-Assessment Narratives, SLO 2, Research Process.............................. 17
Figure 15. Student Self-Assessment Narratives, SLO 3, Social Issues. .................................... 18
Figure 16. Student Self-Awareness Narrative, SLO 4, Technological Impacts........................ 19
Figure 17. Student Self-Awareness, SLO-5, Written Communications. ................................... 20
Figure 18. Student Self-Awareness, SLO 6, Synthesize Information. ...................................... 21
Figure 19. Student Self-Awareness, SLO 7, Moral and Ethical Responsibilities. .................... 22
Figure 20. Percentage of Students Responding by Semester..................................................... 23
Figure 21. Student Access of Portfolio Site. ............................................................................... 25
Figure 22. Main Reasons to Enroll in Program. ....................................................................... 26
Figure 23. Current Operational Themes. .................................................................................... 27
Figure 24. End of Program Survey, Sustain in Program. ......................................................... 28
Figure 25. End of Program Survey, Recommended Improvements. ......................................... 29
Figure 26. How to Market Program. .......................................................................................... 30
Figure 27. Cisgender Comparison, AY 2018-2019. ................................................................... 33
Figure 28. Cisgender Comparison, AY 2019-2020. .................................................................... 34

Figure 29. Gender Comparison Within Population- Female, AYs 2017-2020. ........................ 34
Figure 30. Gender Comparison Within Population Male, AYs 2017-2020. ............................. 35
Assessment Report, 2019-2020
Introduction. The purpose of the 2019-2020 MS Adult Learning and Leadership Assessment
Report was to conduct an annual formative assessment to review the focus of the program, to
collect, analyze, and summarize data, and to formulate decisions regarding program
improvements in curricula, facilitations, and evaluation of artifacts. Additionally, through the
use of students’ self-assessments and reflection papers, the assessment sought to “give voice” to
the adult learners participating in the program. The assessment report served as an informative
platform to stimulate dialogue amongst stakeholders, to add to knowledge of adult learning
theories, and to improve the program (Fitzpatrick, Sanders, & Worthern, 2011).
The formative assessment applied a practical-participative evaluation (P-PE) approach

where the assessment served as a collaborative effort between administrators and primary
stakeholders consisting of faculty members, department heads, and senior administrators.
During the course of the academic year, stakeholders initiated three major changes and
underwent a significant event that impacted the assessment process. First, stakeholders deleted
the requirement for students to upload their Entry Reflection Essay within the Portfolio Canvas
site. Second, stakeholders revised the wording for Student Learning Outcome (SLO) 1,
Literature Integration, to better reflect students’ breadth and depth of literature in adult education
field. Third, administrators blinded all portfolio artifacts and reflection essays prior to releasing
the files for evaluation. Finally, the context of the assessment was through the lens of the onset
and continuance of the COVID 19 pandemic in early spring and summer, 2020. COVID 19
mitigation plans resulted in major changes in the schedule, delivery of course content, learning
approaches and modalities, and the facilitation of the program.
Student Learning Outcomes. The Masters of Science in Adult Learning and Leadership
program encompassed seven student learning outcomes. Table 1 below listed the SLO number,
title, and learning outcomes students must master to achieve educational outcomes.
Table 1. Masters of Adult Learning and Leadership SLOs.

SLO # Title Learning Outcomes
Knowledge
1 Literature Articulate an understanding of the breadth and depth of the
Integration literature in the field of adult education.
2 Research Demonstrate understanding of the research process.
Process
3 Social Issues Demonstrate an understanding of social issues affecting adult
education.
4 Technological Demonstrate knowledge of the impact of technology on adult
Impacts education and adult learning.
5 Written Demonstrate effective written communication skills.
Communication
Skills
6 Synthesize Demonstrate the ability to synthesize complex information.
Information
Attitudes and Professional Conduct
7 Moral and Recognize moral and ethical responsibilities within the adult
Ethical education profession and practice professional ethics.
Responsibilities
Evaluation Approach. The assessment used a mixed method, component typology

encompassing both direct and indirect measures (Rallis & Rossman, 2003). The direct measure
consisted of a quantitative research methodology and statistical tools using evaluators’ ratings of
students’ artifacts and reflective essays from SLOs 1 though 7 and the final student essay. The
indirect measures consisted of qualitative research methodologies using narrative themes from
students’ self-assessments and end of program survey instruments.
Direct Measures. Administrators assessed learning outcomes through two components

of the MS portfolio. The first component consisted of artifacts to demonstrate satisfaction of
SLOs from completed courses (assignments within the courses). The second component was a
narrative reflection essay summarizing students’ evidence of knowledge, skills, and attitudes
related to fields of study. Faculty members assessed blind portfolios using a Likert rating scale
and an evaluation rubric. Two faculty members evaluated each portfolio submission. During the
course of the 2019-2020 academic year, raters reviewed 35 portfolios. Ratings took place at the
completion of fall, spring, and summer semesters. Table 2, Direct Measures, contains specific
details regarding the content of the portfolio product.
Table 2. Direct Measures.

Item Title Content
Performance based Artifacts § Products (i.e. any paper, presentation, video,
assessment podcast) composed during designated coursework.
§ Requires two artifacts for each SLO.
Essay SLO § Two-page paper that addresses the topic of the
reflection SLO.
essays § Reflects upon the knowledge, understanding, and
synthesis the student achieved during the adult
education courses (the program).
Essay Final § Completed as the final requirement after all essays
Reflection and products are submitted for assessment.
Essay. § Three to five-page essay where student reflects
upon growth and change as he/she progresses
through the program.
Indirect Measures. All students completed two components: 1) a self-assessment of their

progress in the acquisition of knowledge, skills, and attitudes and 2) an end-of-program survey
containing summated Likert scale statements and open-ended questions.
Internal and external validity threats. Several methods were used to address internal
and external validity threats. First, to avoid instrument decay associated with scoring fatigue,
administrators randomly divided students’ portfolios amongst 14 pairs of faculty members.
Second, the assignment of portfolios was purposive in order to support statistical testing of
representative populations. Third, administrators sought to alleviate the potential for raters’
biases using three methods: blinding portfolios, reviewing rubric standards with students and
evaluators, and analyzing historical patterns of analyses from previous assessments.
Direct Measure Findings. The direct measure findings consisted of various quantitative
comparisons between goals and actuals, sites, and individual SLOs. The following tables,
figures, and narratives provided brief descriptions of programmatic execution. The findings
reflected the data in the context of a declining student population over a 7-year period. As
depicted in Figure 1, the population ranged from a high of 88 students in AY 2014-2015 to a low
of 38 students in AY 2019-2020.
The major reason for the decline in student population was a change in students’
demographics at the Fort Leavenworth Center. The average student population at this Center
between academic years 2013 thru 2018 was 52 students or approximately 65% of the total
student population. Due to policy changes and the option for a Government fully funded
graduate program at the Command and General Staff College, the Fort Leavenworth Center
population decreased to 36 students (62%) in AY 2018-2019 and to 22 students (58%) in AY
2019-2020. The decrease of students in this demographic led to the findings being slightly less
reflective of the Fort Leavenworth Center. For the first time since the inception of the program
assessment period, the number of female students being assessed (N=20) was greater than males
being assessed (N=18).
Figure 1. Student Populations/Ratings, AYs 2013/2014 thru 2019/2020.
Student Populations/Ratings, AYs 2013/2014 thru 2019/2020

208
210
180
180
158
150 148
150
119
120
88
90 83 79
76 73 76
58
60
38
30
0
2013-2014 2014-2015 2015-2016 2016-2017 2017-2018 2018-2019 2019-2020
# of Ratings # of Students
Figure 2 displayed the yearly SLO average for the past 7 academic years. The green line
with data points indicated the average ratings for AY2019-2020 where each student received 2
evaluations for a total of 76 ratings. Staff and faculty established a program objective of
achieving a proficient (3.0) or higher, 75% percentage level rating for each SLO. With the
exception of SLOs 1 and 2, AY2019-2020 met the objective of a 75% or greater proficiency for
5 of the 7 SLOs. That said, SLO 2 recorded the highest average rating (2.9) in the program’s
history. Ratings for SLO 1, SLO 2, SLO 3, SLO 4, and SLO 7 set new program goals.
Figure 2. Yearly Average SLO Rating.
Yearly (N=7) Average SLO Rating

3.3 3.22
3.2 3.14 3.14 3.13
3.1 3.06
3.0 2.95
2.90
2.9
2.8
2.7
2.6
2.5
2.4
SLO1 SLO2 SLO3 SLO4 SLO5 SLO6 SLO7
2014-2015 2015-2016 2016-2017
2017-2018 2018-2019 2019-2020
Figure 3 displayed the disbursement of students across the two learning centers and
online delivery. All of the centers and online encountered decreases in student populations.
Figure 3. Disbursement of Populations Across Delivery Methods.
Delivery Method, AY 2018-2019 Delivery Method, AY 2019-2020

40 36 40
35 35
30 30
25 25 22
20 16 20
15 15 12
10 7 10
5 4
5
0 0
Olathe Fort Leavenworth Online Olathe Fort Leavenworth Online
Figure 4 depicted the average SLO ratings per learning center and online. In comparison
to AY 2018-2019 ratings, the 2019-2020 ratings demonstrated greater variability between SLO
point spreads. A point spread was defined as the difference between the high SLO average and
the low SLO average for the two learning centers and online learning. The comparison of point
spreads provided indicators of the quality of delivery at each site.
AY 2019-2020-point spreads ranged from a low of .23 (SLO 6) to a high of .63 (SLO 2).
In contrast, AY 2018-2019-point spreads ranged from a low of .09 (SLO 3) to a high of .39 (SLO
6). Figure 4 depicted an example of large point spread (.63) for SLO 2 where students at Olathe
received much lower ratings then students at Fort Leavenworth and Online. In most cases,
students choosing online delivery demonstrated higher ratings then students attending face-to-
face or remote learning centers. That said, due to COVID 19 mitigation planning, 100% of the
student populations transitioned to online learning in the spring and summer terms.
Figure 4. Average SLO Ratings Across Sites and Online Platform.
Average SLO Ratings Acros Methods of Delivery

3.40
3.33
3.26 3.27
3.23
3.21
3.20
3.13 3.13 3.13
3.00 2.98
3.00 2.94
2.92
2.88 2.88
2.80 2.75
2.73 2.75
2.60
2.38
2.40
2.20
SLO1 SL02 SL03 SL04 SL05 SL06 SL07
Olathe Fort Leavenworth Online
SLOs summaries and results. Figures 5 through 12 provided overviews of the direct
measures’ data and longitudinal results for each SLO. As shown in the horizontal axis, students
received a rating of unsatisfactory (uncommon), basic, proficient, or distinguish. The line charts
depicted the slopes of cumulative SLOs’ ratings over AYs 2013-2014 through 2019-2020. Note
that as the student population decreased over the academic years, the height of the slopes also
decreased. The green comment box denoted whether the program achieved the faculty goal of
75% of students achieve a rating of proficient or distinguished. The percentage figure reflects
the number of students achieving a proficient rating plus the number of students achieving a
distinguished rating divided by the number of portfolio ratings (N=76). The yellow text line
within the green box described the percentage of change in evaluators’ ratings in comparison to
the previous academic year. In most cases (86%), students met the goal of achieving 75% or
higher ratings as proficient or distinguished. Excluding SLO 2 and SLO 7, the average range in
SLO ratings ranged from a -3% decrease to a 1.5% increase.
Figure 5. SLO 1 Literature Integration.

SLO1. Literature Integration
2013-2014 2014-2015 2015-2016 2016-2017 2017-2018 2018-2019 2019-2020
100 Met Goal: 75% Achieved

Overall rating: 76%
Increase of 1%
80
60
39
40
18 19
20
0
0
Unsatisfactory Basic Proficient Distinguished
Most notably for SLO 2, while students did not achieve the goal of 75%, the overall
rating of 70% was significant. The average ratings for AYs 2013-2019 was 46%. The higher
ratings appeared to be attributed to certain student populations. Students attending the fall term
averaged 2.55. Students attending the spring and summer terms recorded average scores of 2.98
and 3.17. The location of delivery also impacted ratings where average ratings for Fort
Leavenworth (2.92) and online (3.0) were offset by Olathe (2.38).
Figure 6. SLO 2. Research Process.
SLO 2. Research Process

2013-2014 2014-2015 2015-2016 2016-2017 2017-2018 2018-2019 2019-2020
100
Not Met Goal: 75%

Overall rating: 70%
Increase of 23%
80
60
40 38
23
20
15
0
0
Figure 7. SLO 3. Social Justice Issues.
SLO 3. Social Justice Issues

2013-2014 2014-2015 2015-2016 2016-2017 2017-2018 2018-2019 2019-2020
100
Met Goal: 75%

90
Overall rating: 84%
80 Increase of 2%
70
60
50
40 36
30 28
20
12
10
0
0
Figure 8. SLO 4. Technological Impacts.

SLO 4. Technological Impacts
2013-2014 2014-2015 2015-2016 2016-2017 2017-2018 2018-2019 2019-2020
100
Met Goal: 75%
90
Overall rating: 89%
Decrease of 1%
80
70
60
50
42
40
30 26
20
8
10
0
0
Figure 9. SLO 5. Written Communication Skills.

SLO 5. Written Communications
2013-2014 2014-2015 2015-2016 2016-2017 2017-2018 2018-2019 2019-2020
100
Met Goal: 75%
90 Overall rating: 88%
Decrease of 1%
80
70
60
48
50
40
30
19
20
9
10
0
0
Figure 10. SLO 6. Synthesize Information.

SLO 6. Synthesize Information
2013-2014 2014-2015 2015-2016 2016-2017 2017-2018 2018-2019 2019-2020
100
Met Goal: 75%

90 Overall rating: 79%
Decrease of 2%
80
70
60
50
39
40
30
21
20 16
10
0
0
Figure 11. SLO 7. Moral and Ethical Responsibilities.
SLO 7. Moral and Ethical Responsibilities

2013-2014 2014-2015 2015-2016 2016-2017 2017-2018 2018-2019 2019-2020
100
90
Met Goal: 75%
Overall rating: 79%
80 Decrease of 8%
70
60
50
40
31
29
30
20 16
10
0
0
Student self-assessments. At the conclusion of their program, students (N=35)

completed a self-assessment of their understanding and knowledge of the learning outcomes. As
Figure 12 showed, similar to previous academic years, in most cases, except SLO 5, students
self-assessed ratings were higher than actual ratings.
Figure 12. Students Self-Assessments vs Raters' Assessment, Consolidated.
Student-Self vs Rater Assessment, Consolidated
Student Self-Assessment Avg (N=35) Rater Average
3.60 3.58
3.54
3.49
3.40
3.27
3.25
3.20
3.22
3.14 3.09
3.14 3.13
3.03
3.06
3.00
2.95
2.90
2.80
SLO1 SL02 SL03 SL04 SL05 SL06 SL07
Indirect Measure Findings. All students completed two components: 1) a self-assessment of

their progress in the acquisition of knowledge, skills, and attitudes and 2) an end-of-program
survey containing summated Likert scale statements and open-ended questions.
Student narrative self-assessment of SLOs. Figures 13 through 19 contain sunburst

charts incorporating students’ narratives. The charts reflected common themes associated with
students’ comments regarding their self-assessments. While in all cases students provided
numerical assessments, a lower number of students provided comments regarding their
experiences.
Figure 13. Student Self-Assessment Narratives, SLO 1, Literature Integration.
SLO 1, Literature Integration
Thoroug
h knowle
2
dge,
lit Ti
er e c
at y
lea ur onc dm
rn e , ep e
in ad auth ts t en 11
g/ u o th s,
lea lt ors e ng litie
an r i
de
rsh d St ab
ip
,7
Co
nf
id
en
t
Trouble knowing what

information to look
for, 2 Improved
nt
, wi
ll e me
sure , 3 ov
o
exp r time pr
ited Im
ve d
Lim rove o e AP
Ne
imp A ski
lls
r e fi
ne
d,
5
& a eor g
ts, l th pin
uth ies,
,5
Inte educ
ors
cep t al kee
ract
Focus on implications
con aigh e in
&
ion tors , 3
str uggl
vs writing, 2
sw
a
Str
ith
pee
rs
Literature Integration, N= 40
Slightly over a half of the students (52.5%) indicated they saw improvements in their
integration of literature within both their reflection papers and artifacts. Students who
indicated the need for additional improvement cited difficulties in remembering or
understanding all the concepts and applying the new knowledge to their materials. A few
students indicated their skills sets would improve with more exposure to literature.
Figure 14. Student Self-Assessment Narratives, SLO 2, Research Process.
SLO 2, Research Process
Unmotivated,
Ne pra
ds ea tteerr
ed cti
ho es a
, 9 rch
fo ce,
et r re
rf 6
m on a g
ur
p d
th
as re
gr q u i
er
Ac
Ne
ed
Im
pr
ov
em
en
t
Thoroughly
understood pr
ocess,
3
Improv
ed
nt
onfide
C
I ca
nn
con ow a
ten naly
t, z
in 6 e
nt
if de es, 6
n i
Co bilit
a
language/resou
Provided me th
to understa
tand
nd,, 55
rces
e
Research Process, N= 36
A large proportion of students, 81%, felt they saw improvements in their use of research
methods or felt confident in their research abilities. These students felt they understood the
processes and the program increased their abilities to select and utilize research methods.
Students who indicated the need for additional improvement (17%) felt they needed
additional practice.
Figure 15. Student Self-Assessment Narratives, SLO 3, Social Issues.
SLO 3, Social Issues
re
Ca nd t
sp
ni o
o
s,
de iss
m
nt ue
ste
ify s,
10 f sy
& 8
o
us
cio
ns
Co
Co
nf
ide
nt
Need to
develop
stance ,
2
Improved
,3
eberg t
ip of the ic en
T em
rov Exh
mp au
refl sting s
e dI ecti
Ne e
on, lf-
5
,6
ssu to
Ar
es
an re
ea ost
rst mo
di
Id ,3
Most compellin
m
de d
ev
un Nee
elo
of program, 3
pe
d
g part
Social Issues, N= 40
Seventy-two percent of students felt they gained the knowledge to apply content in their
personal or professional lives. Slightly greater than half of the students (52%) felt the topic
was both enlightening and exhausting as they underwent self-reflection and self-awareness of
their biases. Several students stressed the need for further self-improvement as they felt
overwhelmed by the complexity of social issues in their environment or conflicted by their
stances.
Figure 16. Student Self-Awareness Narrative, SLO 4, Technological Impacts.
SLO 4, Technological Impacts
Not profici
opp xplor
Lim ities
ort
e
ent,
un , 2
itd
1
e
to
ht
Nee
hlig g
ig nin
o h lear , 8
d
Lik t
Imp
e le d es
in more Ab e an ach
int
r
egr prac s o
ove
u pp r
ati t a
ng ice
m
,3
ent
Confident
ove d
Impr
G
r on to reat
tte ng y ols ar
Be andi log 7 in sen
st no rs, to
der tech rne ol al of
un ow s lea kit
,3
h ect
aff
Signif d my
increa s, 3
abilitie
icantly
se
Technological Impacts, N= 27
Fifty-two percent of students felt confident in their technological abilities as they practice
such techniques in their professional careers and the COVID 19 environment. Students
demonstrated the same confidence in Figure 12, where students’ self-assessed average rating
was 3.49. Several students commented on how the content increased their knowledge in the
adult learning information technology techniques. A small number (11%) of students felt
the need for greater amounts of practical exercises to explore the various learning techniques.
Figure 17. Student Self-Awareness, SLO-5, Written Communications.
SLO 5, Written Communication Skills
importance of
Understood flo
Exc mmu
research, 1
co
ell nic
w&
ed ati
in
wr n, 4
itte
o
n d 1
ce s, 1
Con
han tion
En nica
fide
mu
nt
c om
Occa
mec sional f
hani l
cs co ow and
ncer
ns, 4
Improved
ent
em
rov
mp
Ne ed I
8
s,
r es
og
pr
All tten
in
wr
ow co 7
k
or
i
ed mm
W
me un
to icat
ref ion
ine s,
my
Written Communications, N= 35
The largest percentage (51.4%) of students saw improvements in their communication skills.
Most students felt their skills were enhanced and the program led to a refinement in their
writing techniques. However, a marked percentage (34.2%) of students felt the need for
improvement. Most students indicated issues with the flow and mechanics of their writing
while others cited difficulties with transitioning from business conversation type writing to
scholastic writing. For this SLO, students’ reflection of their writing competencies were
closely aligned with evaluators’ assessment ratings where Figure 12 depicted only a 1.6%
difference in assessment ratings.
Figure 18. Student Self-Awareness, SLO 6, Synthesize Information.
SLO 6, Synthesize Information
I st nthes
sy
rug
gle s, 5
dw
i
/
e
siz
the , 9
n
sy fo
Ne
to x in
ed
y e
ilit pl
imp Ab com
rov
em
ent
Grea
tly im
prov
ed, 3
ed
Improv Conf
iden
t
of
pa ble g my
Ca rmin 5
o is,
nsf s
tra naly
a
Ca to atel
n b get y e
ad
rin he xpl
eq
gs r an ain
u
co d , 7
ive, 3 my
nc
ep
pers ; Express
ts
pect
ating
Liber
Synthesize Information, N= 32
Similar to SLO 4, a high percentage (59%) of students felt they were confident in their
abilities to synthesize information. Figure 12 demonstrated students’ confidence with the
average student self-assessment rating set as 3.58, the highest of all SLOs. In contrast, the
average evaluator rating for SLO 6 was 3.06. Several found it “liberating” to be able to
assemble well-reasoned thoughts and present arguments which justified their perspectives.
Figure 19. Student Self-Awareness, SLO 7, Moral and Ethical Responsibilities.
SLO 7, Moral and Ethical Responsibilities
differ in open m
main
respo settings a , 3
Reco ities in
ta
ent
nsibil
gnize
ind
nd
St
r
kn ong nd d
ow b sta an
led ase der ure
ge of un at 11
,4 to of n els,
ame xity t lev
Co C le en
mp er
co diff
nfi
de
nt
Have room to grow, 4
Improved
ent
vem
Im pro
e d
Ne
ea to
ar e
,6
is or
th m
in uch
ar m
le ed
pe
Exp ectiv
Ne
rsp
an es, 9
de
d
Moral and Ethical Responsibilities, N= 37

Over 54% of students felt improvement in their moral and ethical responsibilities
competencies. Students commented on the program leading them to extend their perspectives
and to opening their mind to the complexity of issues at different levels of engagement with
adult learners. Still, 27% of the students felt they needed to improve in this skill set. For those
students expressing confidence in their abilities, most referred to their professional occupation
as being the cornerstone for their high level of moral and ethical competence.
End of program reports. The following exhibit reflected the students’ responses for a
total of 35 responses out of a possible 36 students or a 96.5% response rate. Figure 20 provided
the sample population as defined by each semester. The end of program report consisted of 7
close-ended questions.
Figure 20. Percentage of Students Responding by Semester.
Responses by Semester
25
21
20
15
10 8
6
5
0
Fall, 2019 Spring, 2020 Summer, 2020
Table 3 displayed the cumulative findings of students’ responses to the Question 4 (Q4),
How important were each of these items in your decision to enroll in Adult Learning and
Leadership program at K-State? Students responded to the questions via the use of a 5-point
Likert Scale rating where the responses ranged from Not important (NI) to Very Important. (VI).
We rank ordered the students’ responses. The item receiving the highest percentage of students’
combined responses “More Important and Very Important” was listed first with the remaining
items in decreasing percentage order. Notably, the item rated #1, Being able to study adult
learning and education, rated the highest amongst students demonstrating an increase in
percentages by 31% over the previous year. Program fitting into schedule dropped to third place
decreasing in percentage by 12%. While the name of the degree program remained as the lowest
consideration, the ability to study leadership increased by 11%. Thus, the combination of
studying adult learning and leadership presented the greatest percentage increase (42%) of
interest in students enrolling into the degree program.
Table 3. Items Key to Enrollment.
Importance of Topic Rated by Percentage. Highest % of Increase/Decrease

Positive Responses AY 2018-2019
1. Being able to study adult learning and 86% +31%
education.
2. Convenience of the course. 86% -5%
3. Program fitting into schedule. 83% -12%
3. Academic reputation of university. 83% +4
5. Being able to study leadership. 66% +11%
6. Academic reputation of degree. 60% -13%
7. Name of the degree program. 57% -2%
Table 4 displayed the cumulative findings of students’ responses to the second question,
On a scale of poor to excellent, how would you rate the quality of these items during your
program? Students responded to the questions via the use of a 5-point Likert Scale rating where
the responses ranged from Poor to Excellent. The items were ranked by students’ (N=35)
responses using the rating of excellent as the pacing item. and Excellent”. With the exception of
two items demonstrating minor increases, most of the percentage of excellent ratings decreased.
The decreases may be attributed to rapid program adjustments due to COVID 19 mitigation plans
or students encountering difficulties adjusting to virtual delivery.
Table 4. Students Assessment of Program Quality.
Ranking of Topics as Rated Excellent by Highest % of Increase/Decrease

Percentage. Excellent AY 2018-2019
Responses
1. Helpfulness of faculty. 60% N/A
2. Quality of instruction. 57% +2%
3. Communications & responses to questions. 51% -12%
4. Receipt of notifications from department. 49% -10%
4. Fairness of grading. 49% -15%
4. Quality of overall course content. 49% +4%
5. Instructors’ accessibility. 43% -23%
5. Clarity of degree requirements. 43% -16%
Ranking of Topics as Rated Excellent by Highest % of Increase/Decrease

Percentage. Excellent AY 2018-2019
Responses
5. Quality of academic advising. 43% -11%
6. Canvas site which provides information. 40% -24%
6. Program length. 40% -5%
7. Accessibility of academic advising. 34% -23%
7. Portfolio canvas course. 34% -24%
7. Interaction opportunities with faculty. 34% -18%
8. Courses schedule. 29% -21%
Figure 21 displayed students’ responses to “When did you first access the portfolio site in
canvas?”. Most students (44.4%) accessed the portfolio site at the start of their program of
study. This was an increase of 11% over the previous year.
Figure 21. Student Access of Portfolio Site.
Q12. First Access to Portfolio Site in Canvas (N=27)

14
12
12
10
8 7
6
6
4
2
2
0
First course of First semester of Midpoint of End of program
starting the starting program program
program
The end of program survey included a new question by querying students with “What
were the three main reasons you enrolled in the MS in Adult Learning and Leadership?”. Figure
22 provided the cumulative total of the students’ responses (N=103). A major impetus for
students to enroll focused on increasing students’ knowledge, interests, and continuance of
learning with the 3 reasons combined accounting for 60% of the responses. Thirty-three percent
of students indicated their top 3 choices dealt with increased opportunities for advancement or
job change.
Figure 22. Main Reasons to Enroll in Program.
Q9. Three Main Reasons to Enroll in Program (N=103).

35
31
30
25
22 22
20
15 13
10 8
5 2 2 2
1
0
r
e
n
er
t
e
e
er
ed
he
en
io
tim
ng
dg
oy
oy
st
at
Ot
em
ha
le
pl
re
pl
uc
e
ow
rc
th
m
em
nc
te
ed
te
ee
kn
in
va
at
e
e
n
ar
d
tiv
n
d
or
re
a
tio
Ia
an
/c
m
ec
or
ob
cu
op
ng
lls
or
f
os
aj
es
a
i
hi
ef
st
sk
of
pr
ti
et
Be
te
on
ni
y
ts
om
fa
a
m
tu
st
i
en
cil
e
so
or
ts
g
ov
fa
m
in
pp
ou
re
en
pr
pp
To
eo
ab
ui
im
te
eq
s
re
as
To
ea
or
tr
ui
cr
as
m
eq
ee
in
n
tr
e
m
ar
us
To
ee
To
le
To
m
To
To
Question 10 posed an open-ended question “What is your current occupation?”. The data
contained in Figure 23 reflected assorted occupational titles which were then relabeled as
common themes. As indicated, the military accounted for half of the responses (N=16) with the
remaining population in single digits.
Figure 23. Current Operational Themes.
Q 10. Current Occupation (N=32)
3% 3% 3%
3%
3%
19%
50%
3%
3%
7%
3%
Therapeutic Recreation Military Extension Agent

HR Specialist Financial Hr Manager
Educator/Trainer Clinical Researcher Law Enforcement
Program Manager Job Hunting
Figures 24, 25, and 26 displayed students’ responses to series of open-ended questions. The
questions focused on what areas in the program should be sustained, suggestions for program
improvements, and suggestions for marketing the Adult Learning and Leadership program.
Administrators reviewed students’ comments, identified themes, and bundled responses into a
donut graph with the inner loop containing primary themes and the outer loop providing
additional fidelity of comments. Similar to previous years, comments regarding sustainment of
program elements (81%) were much higher than recommended improvements (19%).
Figure 24. End of Program Survey, Sustain in Program.

Q6. Sustain in Program (N=39)
With
facilit
ators
,3
0
,1
ulty
c
fa
W le
ith e ab
pe dg
e rs, le
5 ow
En
Kn
ga
ge
Le t/In
me
ve te
n
l o ra
f cti
on
s
Books
inexpe
nsive b
good, ut still
1
Delivery of
Instruction
tion &
l reflec
Critica ness , 4
aware
ce
an
lev
Re
Fl
sc exib
he il
du ity
m le & in c
et
ho de our
ds liv se
, 8 ery
n, my
sio o
es y t
5
of lit
pr cabi
Diversity in cou
ship, 1
pli
w/leader lt
and assignments
u
Ap
ion of ad
Integrat
rse material
learning
,2
Figured 24 addressed the question, “Think about your entire experience in the master’s degree
program in adult learning and leadership. What is the best aspect of the K-State adult learning
and leadership program? The inner loop of the sunburst chart contained three major themes:
level of engagement, delivery of instruction, and relevance. Similar to earlier data (Table 3)
students valued the diversity of material, facilitators’ expertise, and the relevance of material to
both their professional career and their personal self-awareness. Students also prized the level
of engagement and interactions with their peers, more so than interactions with facilitators.
Figure 25. End of Program Survey, Recommended Improvements.

Q7. Improve Program (N=9)
Disc grams,
pro
oun
t on
n
ri
, 2 lie
line
1
m ear
og rse
pr cou
ra
ch
ar
se
Re
On
lin
ec
no atalo
ta
ccu g an
rat d lin
Ot
e,
1 ks
he
r
Course
conten Remain
ess in t
timelin flexible
Lack of g rad es, 1 schedu
to CGSC
of le, 1
posting
y
ult
Fac
s
1 date
Co pa
ur ct l
im
se en
at ent
s w gt
cc nm
e,
hic hen
ina ssig
ur
h s ed
a
oc , 1
se
ial
Readings too hard to
ur
ly
retain content, 1
Co
Figured 25 addressed the question, “Do you have any concerns about this graduate program
that you would like to share with the department?”. The inner loop of the sunburst chart
contained three major themes: faculty, course content, and other. Most of the comments dealt
with delays in the posting of assignments and grades, changes in learning approaches, and the
need for flexibility in students’ schedule. Remarkably, only a small number of students
mentioned issues associated with the program’s adaptations to COVID 19 mitigation plans.
Figure 26. How to Market Program.
Q8. Suggestions for Marketing Program (N=19)

Target Audience Outreach Marketing Communications
Marketing Communications Target Audience
Improve presence on Focus on more leader

Facebook page development
Show correlation
between SLOs and SAMs graduates
future assignments discuss program
Highlight ability to take

online courses prior to Post on un-official CGSC
CGSC pages
Market program
for law
Reach out to personnel enforcement Market towards
accepted to CGSC earlier officers military spouses
Testimonial from online
Post flyers in housing areas programs Outreach
Use more social media Attend different professional meetings for Continue improving
platforms Peers completed program recruitment the website
Figured 26 contained the results for the question, “If you have a suggestion of how we could
reach people in your field or generally market the program, would you please share?”.
Students’ suggestions focused on 3 areas; target audience, marketing communications, and
outreach. Most of the comments pertained to the Fort Leavenworth Center. Due to COVID
mitigation plans, direct communications with incoming students were curtailed by the
Installation Education Office. This limited the ability to correspond with incoming students.
The face-to-face education fair was cancelled with a virtual fair conducted later after course
start. Complicating matters, access to the installation was also limited. The introduction of
the Government funded Army University program also impacted students’ decision making.
Student not affiliated with the Fort Leavenworth Center suggested greater
involvement in professional meetings and continuing to improve social media and website
communications.
Inter-Reliablity Statistical Tests.

.1912 indicated the raters were in none or slight agreement. Raters’ agreement equated to
49.6%. However, the percentage of agreement was higher then the previous academic year of
43.7%.
Table 5. Kappa Assessment for Mutual Agreement.

AY 2019-2020
Kappa:
Rater 2
Rater 1 Basic Proficient Distinguished
Basic 5 11 9 25
Proficient 3 31 24 58
Distinguished 3 11 24 38
11 53 57 121
Agreement 5 31 24 60
Total: Agree = 60 49.59%

Total evaluations: 121 Z=2.705672
Kappa 0.1912
None or Slight Agreement 95% confidence interval: .142925
Spearman’s Rho statistical test for inter-reliability. The statistical test was used to
measure inter-reliability correlations between pairs of raters. The analysis encompassed twelves
pairs of raters who reviewed students’ portfolios. At a minimum, the matched pairings had to
contain 14 or greater pairs of SLO ratings. Due to the lack of sufficient data for comparative
analyses, two raters’ pairings were not included in the analysis. We compared raters’ evaluations
of students’ SLOs and the final essay. Of the 12 pairs, 3 pairs demonstrated a statistically
significant correlation (p value < .05) between 3 pairings; pairings # 1, #3, and #9. All 3 pairings
demonstrated a positive correlation. In comparison to the previous academic year where only 1
raters’ pairing demonstrated a strong correlation, the additional pairings demonstrated
improvement in raters’ inter-reliability. Table 6 reflected the Spearman Rho statistical test
results.
Table 6. Rater Inter-Reliability, Spearman Rho.
Rater Inter-Reliability using Spearman Rho (2-Tailed Test)

Pairing #1 (P1) Pairing #2 (P2) Pairing #3 (P3) Pairing #4 (P4)
Evaluator 11 Evaluator 12 Evaluator 13 Evaluator 14 Evaluator 13 Evaluator 15 Evaluator 16 Evaluator 15
Sample Size: 28 Sample Size: 35 Sample Size: 14 Sample Size: 14
R-Value: .00416 R-Value: .09251 R-Value: .65514 R-Value: .09245
p-value: .00416** p-value: 0.53173 p-value: .00588** p-value: .73346
R-Value: .22311 R-Value: .13814 R-Value: .31285 R-Value: -.44544
p-value: .40621 p-value: .60991 p-value: .23808 p-value: .08379
R-Value: .40458 R-Value: .09251 R-Value: .3721 R-Value: .09245
p-value: .04988* p-value: .53173 p-value: .15583 p-value: .73346
Legend: p value < .05* p value <.01**

Assessing Evaluations and Gender. To further studies on portfolio assessments, we

posed the research question, “Given the blinding of portfolios, were there significant differences
in the ratings of cisgender male and cisgender female students’ in comparison to previous
academic years? In support of the research, administrators removed students’ identities from the
portfolio submissions. However, in a few cases, administrators were unable to remove
identifying markers. This was normally due to artifacts where students presented videos or
conducted taped interviews. We compared the evaluation ratings for “blinded” portfolios with
portfolios from previous academic years to compare and contrast patterns of evaluations.
Figures 27 displayed findings. For ease in comparisons, all figures used the same vertical and
horizontal axis scales.
When comparing males’ and females’ portfolios pre- and post- blind process, we noted
the parallel, horizontal lines depicting scores demonstrated less variability. AY 2019-2020
demonstrated smaller parallel gaps between male and female students’ ratings (.1 or less) in 5 of
the 8 ratings. Figures 27 and 28 depicted a comparison of the average SLO ratings for student
populations AY 2018-2019 and AY 2019-2020 (Blind). Of the 8 data points, male students’
scores increased in 5 SLOs with the largest increase of .15. Female students’ ratings increased in
6 SLOs with the greatest increase of .33. While females’ ratings in SLO2 increased by a
percentage in AY 2019-2020, 5.4%, they continue to lag behind their male counterparts who
demonstrated an increase of 9.5% during the same period.
Figure 27. Cisgender Comparison, AY 2018-2019.
Gender Comparison, AY 2018-2019
3.45 3.37
3.33
3.24
3.25 3.18
3.14 3.15 3.16 3.15
3.06
3.05 3.00 3.00 2.99
2.87 2.87
2.84
2.85
2.65
2.50
2.45
SLO1 SL02 SL03 SL04 SL05 SL06 SL07 FE
Male Female
Figure 28. Cisgender Comparison, AY 2019-2020.
Gender Comparison, AY 2019-2020 (Blind)

3.52
3.45
3.29
3.24
3.25 3.20 3.21
3.17 3.17 3.18
3.12 3.11
3.10
3.05 3.01
2.93
2.88
2.87
2.85
2.64
2.65
2.45
Male Female
Based on the initial findings, we conducted further analysis to compare previous

academic years focusing on gender comparisons within same gender populations. The data
indicated female students’ ratings were greater than previous 2 academic years in 4 of the 8 SLO
ratings. The largest increase in ratings occurred with SLO4 where females’ average rating
increased by 8%. Figure 29 depicted the comparison of academic years.
Figure 29. Gender Comparison Within Population- Female, AYs 2017-2020.
Gender Comparison Within Population- Female , AYs 2017-2020

3.45
3.26
3.24
3.25 3.20 3.21
3.18 3.17
3.15 3.15
3.13
3.06
3.05 3.00 3.01 3.01
2.99
2.93 2.93
2.9
2.88
2.87
2.84
2.85
2.79
2.64
2.65
2.50
2.45
2019-2020 2018-2019 2017-2018

We also compared the male students’ ratings for the last 3 academic years. Figure 30
showed male students’ ratings were equal to or greater than the previous 2 academic years in 3 of
the 8 SLO ratings.
Figure 30. Gender Comparison Within Population Male, AYs 2017-2020.
Gender Comparison Within Population- Male , AYs 2017-2020

3.52
3.45
3.37
3.33
3.29
3.24
3.25
3.17 3.18 3.18
3.16
3.14
3.10
3.1 3.12 3.12 3.11
3.05 3.06
3.05 3.02
3.00
2.88
2.87 2.87
2.85
2.74
2.65
2.45
2019-2020 2018-2019 2017-2018
Given the multiple changes in the program and the COVID 19 event, it is not clear if the
blinding of the portfolios alone led to significant differences in ratings when compared to
previous years. However, the data indicates the variability between male and female students’
ratings decreased. Further research is required to address the research question.
Assessment Report Review and Recommendation.

Table 7. Review and Recommendations.
Action Item Findings and Recommendations

75% • AY2019-2020 met the objective of a 75% or greater proficiency for 5 of the 7 SLOs.
Proficiency • Ratings for SLO 1, SLO 2, SLO 3, SLO 4, and SLO 7 set new program goals.
Level
Recommendation: Continue the proficiency level of 75% as a metric.
Research • SLO 2 recorded the highest average rating (2.9) in the program’s history.
Process, SLO • Last year’s assessment recommended “Faculty discuss the merits of creating an online research methods course or
2 hybrid research methods course.”
• While COVID 19 mitigation plan served as the impetus for the transition of research methods to an online delivery, the
impact of transition led to the highest average ratings in the program’s history with students attending the spring and
summer terms recording average scores of 2.98 and 3.17. Students’ average ratings in the fall term were a 2.55.
Recommendation: The merits of creating an online research methods course or hybrid research methods course should
continue to be explored as a program option.
End of • Students number one reason to enroll rose sharply (31%) from the previous year, Being able to study adult learning and
Program education, resonated with students.
Reports • The ability to study leadership increased by 11%. The combination of studying adult learning and leadership presented
the greatest percentage increase (42%) of interest in students enrolling into the degree program.
• The Convenience of the Course (-5%), Program Fitting into Schedule (-12%), and Academic Reputation (-13%) and
Name of the Degree (-2%) declined.
• With the exception of Helpfulness of Faculty (N/A), Quality of Instruction (+2%) and Quality of Overall Course
Content (+4%), all other Program Quality topics declined.
• Students rarely cited the implementation of COVID 19 mitigation measures in narratives. Thus, the direct and indirect
impact of measures on quality and students’ perceptions of their experience was unknown.
Recommendation: Faculty discuss the merits of findings and the implications for AY 20-21 and AY 21-22.
Inter- • The statistical tests indicated inter-reliability remains a concern with mutual agreement being none or slight.
Reliablity of • The inter-reliability between rater pairings improved from 1 pairing to 3 pairings.
Faculty • Correlations between blinding of portfolios and adjustments in rating patterns demonstrated decrease variability and
Ratings increased in ratings for certain SLOs and student populations.
Recommendation: Further discussions on the topic given the data in the report and the role of inter-reliability in the
performance assessment

ALL-MS-Assessment-Report 2019-2020

Uploaded by

Copyright:

Available Formats

ALL-MS-Assessment-Report 2019-2020

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

ALL-MS-Assessment-Report 2019-2020

Uploaded by

Copyright:

Available Formats

Assessment Report, 2019-2020

Masters Degree Program

Assessment Report, 2019-2020 ............................................................................................................................. 5

Evaluation Approach. ....................................................................................................................................................... 6

Assessment Report Review and Recommendation. ..................................................................................................... 36

Figure 1. Student Populations/Ratings, AYs 2013/2014 thru 2019/2020..................................... 8

Figure 28. Cisgender Comparison, AY 2019-2020. .................................................................... 34

Assessment Report, 2019-2020

The formative assessment applied a practical-participative evaluation (P-PE) approach

Table 1. Masters of Adult Learning and Leadership SLOs.

Evaluation Approach. The assessment used a mixed method, component typology

Direct Measures. Administrators assessed learning outcomes through two components

Table 2. Direct Measures.

Indirect Measures. All students completed two components: 1) a self-assessment of their

Student Populations/Ratings, AYs 2013/2014 thru 2019/2020

Figure 2. Yearly Average SLO Rating.

Yearly (N=7) Average SLO Rating

Figure 3. Disbursement of Populations Across Delivery Methods.

Delivery Method, AY 2018-2019 Delivery Method, AY 2019-2020

Figure 4. Average SLO Ratings Across Sites and Online Platform.

Average SLO Ratings Acros Methods of Delivery

Olathe Fort Leavenworth Online

Figure 5. SLO 1 Literature Integration.

100 Met Goal: 75% Achieved

SLO 2. Research Process

Not Met Goal: 75%

Figure 7. SLO 3. Social Justice Issues.

SLO 3. Social Justice Issues

Met Goal: 75%

Figure 8. SLO 4. Technological Impacts.

Figure 9. SLO 5. Written Communication Skills.

Figure 10. SLO 6. Synthesize Information.

Met Goal: 75%

Figure 11. SLO 7. Moral and Ethical Responsibilities.

SLO 7. Moral and Ethical Responsibilities

Student self-assessments. At the conclusion of their program, students (N=35)

Indirect Measure Findings. All students completed two components: 1) a self-assessment of

Student narrative self-assessment of SLOs. Figures 13 through 19 contain sunburst

Figure 13. Student Self-Assessment Narratives, SLO 1, Literature Integration.

SLO 1, Literature Integration

Trouble knowing what

Figure 14. Student Self-Assessment Narratives, SLO 2, Research Process.

SLO 2, Research Process

Figure 15. Student Self-Assessment Narratives, SLO 3, Social Issues.

SLO 3, Social Issues

Figure 16. Student Self-Awareness Narrative, SLO 4, Technological Impacts.

SLO 4, Technological Impacts

Figure 17. Student Self-Awareness, SLO-5, Written Communications.

SLO 5, Written Communication Skills

Figure 18. Student Self-Awareness, SLO 6, Synthesize Information.

SLO 6, Synthesize Information

Figure 19. Student Self-Awareness, SLO 7, Moral and Ethical Responsibilities.

SLO 7, Moral and Ethical Responsibilities

Have room to grow, 4

Moral and Ethical Responsibilities, N= 37

Figure 20. Percentage of Students Responding by Semester.

Table 3. Items Key to Enrollment.

Importance of Topic Rated by Percentage. Highest % of Increase/Decrease

Ranking of Topics as Rated Excellent by Highest % of Increase/Decrease

Ranking of Topics as Rated Excellent by Highest % of Increase/Decrease