Six-Step Assessment Process:

Student Learning Outcomes &

Performance Outcomes

Assessment: Basics

09.20.17
 Get ready for WSCUC

2012 2015 2016

Reaccredited Interim Mid-Cycle
(7 yrs) Report Report

Areas of Concern:
- Integrated strategic plan
- Assessment
- Student success
- Funding
Fall 2018 Spring 2019 Fall 2019
Self-Study Off-Site On-Site
Due Review Visit
Where do we carry out assessment

University Learning Goals

University Strategic Plan Goals

Alignment
Division Division POs/SLOs

Unit Unit POs/SLOs

 Six-step assessment process*

What do we want our students to learn and/or our units to accomplish?

How are we documenting the

How are we doing?
assessment AND improvement
How do we know?
activities/results?

What evidence do we
What changes are we
need to know to
making?
determine whether we
Are the changes working?
are successful?

How do we use data to confirm/improve our practices?

*AEEC Spring 2014

 Step 1: Develop SLOs/POs
• A statement

SLO PO
• Significant & essential learning • Measurable end results or
that students achieve at the consequences of activities,
end of a program services, or program

• Knowledge; Skill; Attitude • Variety of results

• Focus on student learning • Focus on operational

effectiveness

SLO or PO?
Depends on the nature of the outcome, not the function of the unit
 Where do outcomes come from

• “Top-down” vs. “Bottom-up”

Alignment
University
• Adapt from existing “best
practices”
Division • Engage faculty/staff

• Involve important but often

Unit forgotten stakeholders (students,
alumni, employers, etc.)
Mission…Goals…Outcomes…Objectives…

Holistic vision of the values and philosophy of

Mission an institution/division/department/unit

Broad, general statements about general aims,

Goals purpose or expectations
Unit/Program-centered

Clear, specific “operational definitions” of goals

Outcomes Customer/Learner-centered
 What are good outcomes
• Customer/Learner-centered, not unit/program-centered
• Aligned with the mission and goals of university, division, college, etc.
• Focus on “high-priority learning”
• Real (not aspirational)
• Simple language
• Specific, clear and concise
• Demonstrable and measurable
• Discrete (no “double-barrel” statements)
• Manageable (more is not better)

*Adapted from Mary Allen workshop (2006) & ALA (2016)

 Sound SLOs are Active

LEVELS of SLOs BLOOM’S TAXONOMY EXAMPLE ACTION VERBS

(Bloom et al., 1956)

Evaluation Assess, Conclude, Criticize, Justify,Value

Increasing level of complexity

Synthesis Assemble, Create, Design, Produce, Reconstruct

Analysis Analyze, Compare, Differentiate, Experiment, Solve

Application Apply, Demonstrate, Modify, Practice, Use

Comprehension Convert, Explain, Interpret, Paraphrase, Report

Knowledge Define, Describe, List, Name, Outline

 Outcome examples*

Unit PO SLO

Students receive adequate support Students demonstrate

Career Center and feedback on their job seeking professionalism in the job seeking
effort. process.

McNair Scholars are provided with McNair Scholars can identify

Center for
step-by-step guidance to transition necessary steps for a successful
Scholars
into graduate school. transition into graduate school.

Student Assessment Scholars will

Student Assessment Scholars can
Office of receive high quality training to
apply basic research skills to carry
Assessment carry out learning assessment
out learning assessment activities.
activities.

*Suggested, not actual, outcomes

Case Study: Step 1
Step 2: Identify methods and measures learning

• We are already and always assessing how we are doing and/or

how our students are learning

• The evidence/measures already in place are NOT always the

best place to start
• Do the measures address the outcomes?
 A bit of vocabulary

Direct vs. Indirect

Value-added vs. Absolute

“Add-on”/
Embedded vs.
External
 Choosing the right measure

• Valid: Are you measuring the outcome?

• Reliable: Are the results consistent?
• Actionable: Do the results clearly tell you what students
can or cannot do?
• Triangulation: Are there multiple lines of evidence for
the same SLO?
• Meaningful and engaging: Are faculty engaged? Do
students care?
• Sustainable: Can the process be managed effectively
within the program context?
 Direct measure examples

SLO PO

• Exam/Quiz • Quantity & quality of service

• Paper/Presentation • Completion/Usage/Error rate
• Project/Portfolio • Analysis of processing time
• Recital/Exhibition • Needs analysis/Gap analysis
• Peer evaluation • Customer/Supervisor evaluation
 Indirect measure examples
Us
ea
ss
up
SLO PO ple
me
nt
al
evi
den
• Self-reflection essay • Customer survey** ce

• Self-report survey • Interview

• Interview • Focus group
• Focus group • Comparison to best
• Report by alumni, practices in the profession
employer, etc.
**Surveys are not always indirect
assessment

Direct evidence helps tell us “what”, and indirect evidence helps tell us “why”.
 Triangulating direct and indirect measures

Career Center:
SLO: Students demonstrate professionalism in the job seeking process

DIRECT INDIRECT

- Scenario-based exam questions

- Self-assessment survey
- Rubric scoring of job seeking
- Graduate survey
materials
- Alumni interview
- Interviewer/employer
evaluation
 Triangulating direct and indirect measures
Center for Scholars:
PO: McNair Scholars are provided with step-by-step guidance to transition into
graduate school.

DIRECT INDIRECT

- # of guidance or advising - Survey of graduate school

sessions provided per student admission officers
- % of students successfully - Comparison of successful
transitioned into graduate transition rate with peer
school institutions
- Student survey on quality of
service provided
 Triangulating direct and indirect measures
Office of Assessment:
SLO: Student Assessment Scholars can apply basic research skills to carry out learning
assessment activities.

DIRECT INDIRECT

- Exam questions on relevant

research skills - End-of-program survey on
- Final project (paper/presentation) self-perceived skills
- Department (i.e client) evaluation - Self reflection essay
- Peer evaluation - Focus group
Case Study: Step 2
 What are rubrics
• Scoring guides that explicitly classify learning products/behaviors
into categories that vary along a continuum.
• No one format - Flexible!

Performance Levels
Criteria

Basic Detailed
elements: performance
description
 Step 3: Determine criteria for success

• A performance standard:

• What level of performance is good enough?

• Pre-determined!

• Supported by historical data, reasonable expectations,

theoretical frameworks...
 Criteria for success examples*

Program
PO Measures Criteria of success
/Unit

- 75% of students received a score of

- Rubric scoring of job seeking
Students demonstrate “adequate” or higher
Career materials
professionalism in the job - 90% of students self-report as
Center - Student self-assessment
seeking process “confident” in demonstrating
survey
professionalism

- % of students successfully
McNair Scholars are - 95% of students successfully
transition into graduate school
Center for provided with step-by-step transition to graduate school
- Comparison of successful
Scholars guidance to transition into - Success rate higher than the average
transition rate with peer
graduate school. of fellow CSUs
institutions

- Student scholars on average receive

Student Assessment 3.5 or higher (out of 5) evaluation
Scholars can apply basic - Department (i.e client) score from the departments
Office of
research skills to carry out evaluation - Focus group participants express
Assessment
learning assessment - Focus group consensus that the Student
activities. Assessment Scholars have the
necessary research skills

*Suggested, not actual, outcomes

Case Study: Step 3
 Step 4: Collect and analyze data

• Sampling!
• Relevant, Representative, and Reasonably sized
• Determined by the outcome and unit context
• Moderate sample size is sufficient (e.g. “50-80” rule; 20-30%).
• Very large sample size is rarely needed.
• If homogenous population, small samples are sufficient.
Case Study: Step 4
Step 5: Plan and execute improvement actions

• Review the assessment findings

Weigh the pig
Small changes
• Determine plan for change (if needed)
matter

Possible Changes for SLO Possible Changes for PO

• Curriculum • Infrastructure
• Pedagogy • Program design Feed the pig
• Faculty support • Service delivery
• Student support • Tools used

• Resources
• Assessment plan
• More data collection?
Weigh the pig
• Don’t forget to re-assess the improvement actions! NILOA (2014)
 Improvement actions example 1

• Associated Students, Inc.:

• Student employees reported significant growth in oral communication skills, but not
in written communication. Supervisor evaluation of student skills do not support
students’ positive self-report.

• Unit will 1) develop plans to investigate possible areas of improvement for student oral
communication development; 2) review the nature of student employee positions; 3) adjust
the outcome to truly reflect the skills applicable to student employee positions.
 Improvement actions example 2

• Disability Support Services:

• In the new student orientation survey, over 80% of students expressed satisfaction
of the overall quality of the orientation; over 80% of students indicated that they
knew how to access DSS services.
• Data exceeded the criteria of success - outcome met

• Unit will 1) review open-ended questions on the survey to identify specific ways to further
improve the orientation program; 2) develop a direct assessment measure; 3) attend a
disability related assessment conference to improve assessment plan.
 Improvement actions example 3

• Business Communication

• Student writings of a case analysis were graded using the CLASS rubric, and found
that students had the greatest deficiencies in “Strategy”.

• Program 1) collected additional demographic data to narrow down weakness population;

2) offered faculty development workshop on case analysis; 3) emphasized use of topic
sentences and supporting evidence; 4) provided sample professional documents for use in
classroom and homework exercises.

• Writing communication scores improved 17% between 2009 and 2012

Step 6: Document assessment activities

Tell a
coherent
story

Weigh the pig

AGAIN
Case Study: Step 5 & 6
 A multi-year assessment plan

• What to plan for:

• Timeline

• Process

• Participants

• Steps to turn assessment results into improvement actions

• Self-evaluation/Reflection of the assessment process

 A multi-year assessment plan (cont.)

• Guidelines: Outcome is not

for only 1 year
• Start with a small number of outcomes

• Determine a realistic assessment plan cycle, i.e. how long (e.g. 7 years) to
complete meaningful assessment of all outcomes

• Create a multi-year assessment plan that assesses 1-2 outcomes a year

• Consider overlapping assessment (of new outcomes) and improvement

(of assessed outcomes) activities

• Make sure assessment involves the entire unit

 assessment@fullerton.edu

www.fullerton.edu/assessment