Slide Presentation Handout For SCE ASC 2014

11/3/2014
Accreditation Council for Graduate Medical Education
Next Accreditation System, SelfStudy Visits and Clinical

Learning Environment Review
American Society of Cytopathology
November 14, 2014
Laura Edgar, EdD, CAE
Executive Director, Milestone Development and
RC for Medical Genetics, Pathology and Radiation
Oncology
Presentation Topics
Review of Next Accreditation System
Self-Study Visits
Clinical Learning Environment Review (CLER)
Next Accreditation System
2012 Accreditation Council for Graduate

Medical Education (ACGME)
11/3/2014
ACGME Mission
We improve health care by assessing and
advancing the quality of resident physicians'
education through accreditation.
ACGME Vision
We imagine a world characterized by:
a structured approach to evaluating the competency of all
residents and fellows;
motivated physician role models leading all GME programs;
high-quality, supervised, humanistic, clinical educational
experience, with customized formative feedback;
residents and fellows achieving specialty-specific
proficiency prior to graduation; and
residents and fellows prepared to become Virtuous
Physicians who place the needs and well-being of patients
first.
ACGME Stakeholders
Stakeholders of the ACGME's accreditation
process are:
Residency programs and their sponsoring institutions

Residents
Medical students
Specialty boards of the American Board of Medical
Specialties (ABMS)
Patients
Payers
Government
General public

11/3/2014
Goals of
The Next Accreditation System
To begin the realization of the promise of
Outcomes
To reduce the burden of accreditation
To provide accountability for outcomes (in
tandem with ABMS) to the Public
ACGME 2012-2013
9,265 Residency and Fellowship Programs
133 Specialty and Subspecialty areas of medicine
117,717 Residents and Fellows
Where did we come from?

2002 Six Core competencies in PR
Work started in 2011
Core and Detailed Process
Outcome in Requirements
New policies and procedures
ADS rebuilt to prepare for NAS
Annual update: free text replaced by data
Scholarly activity replaced CVs
Milestones 1.0 developed

11/3/2014
Where are we?

2013
Phase 1 Programs begin using NAS and report
milestones
2014
Phase 2 Programs begin using NAS
Phase 1 and surgical Fellowships report milestones
2015
Phase 2 Fellowships report milestones
Continuous Accreditation Model

Review programs every 10 years with self-study
Leave Good Programs alone
Good Programs can innovate detailed standards
Identify weak programs earlier
Site visit or progress report from weak programs
Weak programs held to detailed standards

RRC review programs based on RRC set
performance indicators and thresholds
High performing programs moved to consent
agenda
Programs with potential problems require
more information with a request for
additional information or site visit
Every program will receive an accreditation
decision letter every year

11/3/2014

Annual Program, Faculty, & Resident Update
5 year first-time Board pass rate
Case Logs
Resident Survey
Faculty Survey
Scholarly Activity of Core Faculty
Scholarly Activity of Residents
Milestones
Program Review in the NAS
ACCURACY AND
COMPLETENESS COUNT
Annual Data Collection

Every program submits data every year
Every program is reviewed every year
Site visit only if RC asks for it after review of
program

11/3/2014
Annual Data Collection Common Errors

Outdated or missing information: certification
dates, updates to resident list, updates to faculty list
Low Board Pass Rates
Incomplete Case Log Data
Insufficient Faculty participation on Faculty Survey
Scholarly Activity for Residents and Faculty not
entered
ACCURACY AND
COMPLETENESS
COUNT
Program Reviews
Applications
for New
Programs
Continued
Accreditation
Accreditation
with Warning
Initial
Accreditation
Close look
6%
Structure
Core Processes
Detailed Processes
No outcomes yet
STANDARDS
Structure
Core Processes
Detail Processes
Outcomes
Closer look
1%
Data review
Structure
Core Processes
Detailed Processes
Outcomes
Structure
Core Processes
Outcomes
Adverse Actions
Less than 1%
2014 Accreditation Council for Graduate Medical Education

11/3/2014
Self-Study Visit
20
Ten Year Self-Study Visit

Annual Program Evaluation (PR-V.C.)
Resident performance
Faculty development
Graduate performance
Program quality
Documented improvement plan
SelfStudy
PROCESS
SelfStudy
VISIT
Ongoing Improvement
Beyond Minimal Standards
Yr 0
Yr 1
Yr 2
Yr 3
Yr 4
Yr 5
Yr 6
Yr 7
Yr 8
Yr 9
Yr 10
APE
APE
APE
APE
APE
APE
APE
APE
APE
APE
The 10-year Self-Study: Scope

Assesses current performance and ongoing
improvement effort
Covers the period between Self-Study Visits
Initially: the period since last accreditation
review
Eventually, a 10-year interval
Reviews program improvement activities,
successes achieved, and areas still in need of
improvement
Uses data from successive Annual Program
Evaluations, ACGME data, other relevant
information
Conducted 1 year before the Self-Study Visit date

11/3/2014
The 10-year Self-Study: Timeline

Time prior to Self
Study Visit
11-12 months
ACGME Actions
1. Sends summary of
actions/follow-up
from Annual Data
Review
6-11 months
4 months
Program Actions
1. Aggregates data from
Annual Program
Evaluations
1. Conducts Self Study
1. Sets FINAL SelfStudy Visit Date and

informs program
10 days
1. Completes ADS data

update
2. Uploads Self-Study
summary to ADS
Self-Study Process
Self-Study: Participants
Program Leadership
Faculty
Residents/Fellows
Coordinators
Potentially
Institutional Representatives
Others

11/3/2014
Self-Study: Key Processes

Data Gathering
Annual Program Evaluations, ACGME Annual
Data
Focus on data gathering as a learning exercise
Evaluate strengths and areas for improvement
Explore opportunities and threats
Reflect stakeholder (residents, faculty, and
relevant others) participation, input and
perspective
Offer evidence to support conclusions
Self-Study: Key Processes

Interviews
Verify and validate data
Identify areas that have been resolved and
areas and priorities for improvement
Identify program strengths
Review and revise program aims
Assess and validate strengths, weaknesses,
opportunities and threats
Ten Year Self-Study Visit

Not a traditional site visit
Assess broader unit of educational environment
Review core & subspecialty programs together
Will be implemented in 2016

11/3/2014
Self-Study Visit
Team of site visitors
Review the department self-study
Conduct a PIF-less site visit
Validate most recently submitted Annual Data
Potentially serve as a vehicle for:
Description of salutary practices
Accumulation of innovations in the field
Self-Study Visit
Review annual program evaluations (PR-V.C.)
Response to citations
Faculty development
Judge program success at CQI
Learn future goals of program
Verify compliance with Core and Outcome
requirements
Self-Study Visit
Webinar available for download
Coming: Article in Sept 2014 JGME on Self-Study
preparation for programs with early SSV dates

10
11/3/2014
Clinical Learning Environment

Review
32
Clinical Learning Environment Review

Designed to improve how clinical sites engage
resident and fellow physicians in learning to
provide safe, high quality patient care
CLER is intended to be a list of expectations, not
requirements
Each Sponsoring Institution will have a CLER
Visit every 18-24 months
Clinical Learning Environment Review

Provide teaching hospitals, medical centers,
health systems, and other clinical settings
affiliated with ACGME-accredited institutions
with periodic feedback that addresses:
Patient safety
Health care quality
Care transitions
Supervision
Duty hours, fatigue management and mitigation
Professionalism

11
11/3/2014
CLER Visit
Meetings scheduled include:
Senior Leadership (CEO and DIO must attend initial
and exit meetings)
DIO
Quality and Patient Safety Leadership
Group Meeting (30 attendees familiar with resident
experience)
Resident and Fellow Group
Faculty Member Group
Program Director Group
CLER Visit - Walking Rounds

Resident guides from different programs,
preferably senior residents from core
programs
Guides should be comfortable navigating
to all areas of the hospital or medical
center, and have general awareness of
ambulatory clinic locations and hours
CLER Focus Patient Safety

Reporting of adverse events, close calls (near
misses)
Education on patient safety
Culture of safety
Resident/fellow experience in patient safety
investigations and follow-up
Clinical site monitoring of resident/fellow
engagement in patient safety
Clinical site monitoring of faculty member
engagement in patient safety
Resident/fellow education and experience in
disclosure of events

12
11/3/2014
CLER Focus Health Care Quality

Education on quality improvement
Resident/fellow engagement in quality
improvement activities
Residents/fellows receive data on quality metrics
Resident/fellow engagement in planning for quality
improvement
Resident/fellow and faculty member education on
reducing health care disparities
Resident/fellow engagement in clinical site
initiatives to address health care disparities
CLER Focus Care Transitions

Education on care transitions
Resident/fellow engagement in change of duty
hand-offs
Resident/fellow and faculty member engagement
in patient transfers between services and locations
Faculty member engagement in assessing
resident-related patient transitions of care
Resident/fellow and faculty member engagement
in communication between primary and consulting
teams
Clinical site monitoring of care transitions
CLER Focus Supervision

Education on supervision
Resident/fellow perception of the adequacy of
supervision
Faculty member perception of the adequacy of
resident/fellow supervision
Roles of clinical staff members other than
physicians in resident/fellow supervision
Patients and families, and GME supervision
Clinical site monitoring of resident/fellow
supervision and workload

13
11/3/2014
CLER Focus Duty Hours, Fatigue

Management and Mitigation
Culture of honesty in reporting of duty hours
fatigue and burnout
Resident/fellow engagement in fatigue
management and mitigation
Faculty member engagement in fatigue
management and mitigation
Clinical site monitoring of fatigue and burnout
CLER Focus Professionalism

professionalism
Resident/fellow attitudes, beliefs, and skills related
to professionalism
Faculty engagement in training on professionalism
Clinical site monitoring of professionalism
CLER Pathways to Excellence

Available on the
ACGME webpage

14
11/3/2014
Resources
ACGME website has a variety of resources
including:
Specialty-specific webinars
General webinars on broad topics (e.g., NAS, CLER)
FAQs
JGME
PD Virtual Handbook
We are here to help

Executive Director: Laura Edgar, EdD, CAE
ledgar@acgme.org 312-755-5029
Accreditation Administrator: Erin Berryhill
eberryhill@acgme.org 312-755-5045
ADS Representative: Raquel Running
webads@acgme.org 312-755-7111
Cytopathology Fellowship
Training and the Next
Accreditation System
Milestones in Context
Steve Black-Schaffer, MD
Mass General Hospital

15
11/3/2014
Educational Objectives
Understand the background and purpose behind the
Cytopathology Milestones and the new accreditation
requirements for reporting of cytopathology fellowships
in 2015
Develop a plan to implement the Cytopathology Milestones
within cytopathology fellowship programs' evaluation
systems
Create a clinical competency committee that meets all
ACGME requirements
Lead a clinical competency committee in the review of
cytopathology fellows' performance for ACGME
accreditation and fellow assessment
Some basic considerations

1 Milestones are an accreditation requirement, so we need
to use them, but they potentially offer us practical benefit:
2 There are a multitude of good fellowship programs, but
these are quite variably, and often loosely, structured.
3 While flexibility is good, a tighter structure is not only
required henceforth, but can strengthen our training.
4 Why might this be? Lack of structure slows us down:
5 Excellent fellows may be delayed in advancing by the
absence of clearly-defined curricular expectations.
6 Poor fellowship performance is often not as promptly
documented or as usefully characterized as it could be.
Where does accreditation come from?
Dr. Wesley Y. Naritoku, Oct. 2011

16
11/3/2014
Where does "accreditation" come from?
Abraham Flexner
American Medical Association Council on
Medical Education (Carnegie Foundation)
Medical Education in the United States and
Canada: 1910
What did Flexner do?

He assessed and reported on all 155 medical
schools with regard to their:
Requirements for admission
Number and training of faculty
Endowment and tuition income
Availability and quality of teaching
laboratories for preclinical sciences
Availability and quality of teaching hospital
physicians and surgeons for clinical training
NOTE: these are process and structure
What impact did this have?

It standardized educational requirements
for medical school entry (at least two years
of collegiate biology, chemistry and physics).
It standardized a national curriculum of two
years of preclinical science (in anatomy,
physiology, bacteriology, pathology and
pharmacology) followed by two years of
supervised clinical practice in a teaching
hospital.
NOTE: this is time in training

17
11/3/2014
Medical education the short form:

Elementary School
Junior High School
High School
College
Medical School
Flexner Report
Medical Practice
The origins of accreditation (process + structure)
1847 American Medical Association.

1910 Carnegie Foundation published Flexner Report.
1914 American Medical Association published list of hospitals
with approved internships.
1920 American Medical Association Council on Medical
Education and Hospitals organized committees to
recommend preparation deemed essential to expertise in 11
clinical and 4 basic science specialties.
1928 American Medical Association Council on Medical
Education and Hospitals published "Essentials of Approved
Residencies and Fellowships" and list of approved
residencies and fellowships.
1972 Liaison Committee for Graduate Medical Education.
1981 Accreditation Council for Graduate Medical Education.
2001 Accreditation Council for Graduate Medical Education
Outcome Project
The ACGME Outcome Project

1st baby step away from
process and structure
Not a wholly successful enterprise
Not good enough to skip time in training
Not intuitive (Practice-Based Learning and
Improvement & Systems-Based Practice how
to understand/measure on teaching service?)
Raw competencies are really hard to digest!
(Hence the milestones.)

18
11/3/2014
ACGME's Six General Medical Competencies
Patient Care
Medical Knowledge
Practice-Based Learning & Improvement
Interpersonal & Communication Skills
Professionalism
Systems-Based Practice
FOR MORE INFO...
http://www.acgme.org/outcome/comp/compFull.asp
Objectives
&
Curriculum
Medical education the longer form:

Elementary School
Middle School
High School
College
Medical School
Medical Residency
Flexner Report
Outcome Project
Medical Fellowship
Medical Practice

19
11/3/2014
Flexner Report vs. Outcome

Project Key Differences:
Flexner Report defined the essential processes of
undergraduate medical education (UME)
Qualifications for admission to UME
Organization and curriculum of UME
Outcome Project defines the required outcome of

graduate medical education (GME)
General competencies for GME
An educational model for GME
Education is ultimately about trainees, so could

program and trainee assessments be integrated?
GME Program has
Accreditation
Structures and Processes
GME Trainee achieves

Competencies and Outcomes
Milestones
Outcomes
Challenge connect
Trainee Achievement (what is really wanted) to
Program Assessment (what is really done)
Accreditation and certification:

Prior to the NAS, two quite separate processes:
Specialty training programs were accredited by
their Review Committee of the ACGME
Specialist physicians were certified
by their Board of the ABMS
The common aim of both was to ensure that
graduate medical trainees were consistently
competent when they went into practice

20
11/3/2014
So the NAS represents

a confluence of accreditation and certification
concepts
Bringing elements of both together, with the
common aim of ensuring the competence of
medical practitioners
The history of certification (time in training +

boards)
1915 National Board of Medical Examiners.

1917 American Board of Ophthalmology.
1924 American Board of Otolaryngology.
1930 American Board of Obstetrics and Gynecology.
1932 American Board of Dermatology.
1932 National Board of Medical Examiners Committee on Specialists.
1933 American Board of Medical Specialties.
1935 American Board of Orthopaedic Surgery.
1935 American Board of Pediatrics.
1935 American Board of Psychiatry and Neurology.
1935 American Board of Radiology.
1935 American Board of Urology.
1936 American Board of Internal Medicine.
1936 American Board of Pathology.
Accreditation and
certification
So the ACGME has relied on
program process and structure
and the ABMS has relied on

time in training
(plus board examination).
We know about pathology boards;

what is time in training for pathology?

21
11/3/2014
Autopsy (Hospital) Weeks

20
2013 PRODS Survey on Curriculum

Naritoku, Powell, Black-Schaffer
15
10
5
0
0 3 6 9 12 15 18 21 24 27 30 33 36 39 42 45 48 51 54 57
Number of Weeks
Cytopathology Weeks
40
Surgical Pathology Weeks
10
8
30
6
20
4
10
2
0
0 3 6 9 12 15 18 21 24 27 30 33 36 39 42 45 48 51 54 57
Number of Weeks
0
0 6 12 18 24 30 36 42 Number
48 54 60
of Weeks
66 72 78 84 90 96 102108114
Anatomic Pathology Weeks

10
Clinical Pathology Weeks

20
15
6
10
4
5
Number of Weeks
0
8
15
23
30
38
45
53
60
68
75
83
90
98
105
113
120
128
135
143
0
0
8
15
23
30
38
45
53
60
68
75
83
90
98
105
113
120
128
135
143
Number of Weeks
So this is why the NAS was adopted
Programs are all different

And their outcomes are different
Programs are only evaluated q 5 years
Trainees are only uniformly evaluated at
the end of their time in training
Need more timely assessments
Need more specialty granularity
Milestones
Need some national uniformity
What drives the "need to do something now"?
Public perception of lack of patient safety:
Public Citizen
Institute of Medicine
Medicare Payment Advisory Commission
Dr. Wesley Y. Naritoku, Nov. 2013

22
11/3/2014
Sammy Almashat, M.D., M.P.H. and Sidney Wolfe, M.D.

(Public Citizen to Dr. David Michaels, Assistant Secretary of Labor for
Occupational Safety and Health, November 3, 2011)
By deferring to the ACGME, the Obama administration is

choosing to ignore the organizations track record, relying
instead on the ACGME response to the IOM report.
However, these new rules serve as yet more evidence of
the need for federal oversight.
Graduate Medical Education Reform Act of 2012 (S.3201) introduced
on 05/17/12 by Senators Jack Reed (D-R.I.) & Jon Kyl (R-Ariz.)
This drives "the need to do something now"!

http://www.citizen.org/hrg1981 accessed on May 21, 2012
Outcome Project vs. Milestones

Project Key Differences:
Outcome Project defined the required outcome
for a GME program as the achievement of the
general competencies by its trainees
General competencies for GME
An educational model for GME
Global
Periodic
Milestones Project defines the developmental

milestones of continuous progression toward
proficiency for the trainees in a GME program
Levels of performance to proficiency Individual
For each of the general competencies Continuous
The CEO's First Column The Next Step in

the Outcomes-Based Accreditation Project
Thomas J. Nasca, MD, MACP, Chief Executive Officer
[W]e must agree on the "Milestones" of Competency
development in each discipline.
At completion of training, Milestones are the level of
performance expected at entry into unsupervised practice in
each specialty,
[and are] the levels of clinical competence required to gain
eligibility for ABMS certification.
At earlier levels, they constitute "developmental" milestones
to offer assurance that residents and fellows attain
appropriate educational goals.

23
11/3/2014
Medical education even longer form:
Elementary School
Middle School
High School
Individual Accomplishment
College
Medical School
Matriculation
Early Training
Mid-Training
Late Training
Commencement
Medical Residency
Medical Fellowship
Programmatic Potential
Flexner Report
Milestone Project
in semiannual
Outcome
Project
increments by
competency
Medical Practice
Where Will the "Milestones" Take Us?

The Next Accreditation System.
We anticipate that this information will be available to
each RRC on a twice yearly basis.
In the theoretical example [following], you can see that
the program's performance is at or above expectation
in five of the six domains of clinical competency, but
falls more than two standard deviations from the mean
in Practice-Based Learning and Improvement.
So what is different under the NAS?

No longer one-time, high-stake testing at four or five years,
whether for accreditation or toward certification
Instead, close to continuous assessment of trainees, and
thereby of programs
No longer program requirements only for global adherence
to overall standards of training
Instead, specific national standards for training and for
competence at each level

24
11/3/2014
How does the NAS actually work?

Semiannual Milestones assessed by a
Clinical Competency Committee and
reported to the ACGME
Annual Trainee and Faculty Surveys by
the ACGME
Annual reporting of ABP Board results to
the ACGME
Reviewed annually by the ACGME RC for
Pathology
Milestones for Cytopathology

Who developed them?
Cytopathology Milestones Working Group
Stephen Black-Schaffer MD (chair), Diane Davey MD, Laura
Edgar EdD CAE, Wesley Naritoku MD PhD
Pathology Fellowship Milestones Advisory Group

Bruce Alexander MD (CP Fellowship chair), Julia Iezzoni MD
(ACGME Pathology RC chair), Rebecca Johnson MD (ABP
CEO), Wesley Naritoku MD PhD (AP Fellowship chair)
The Cytopathology Milestones

Cytopathology Milestones Work Group drafted:
18 milestone sets / developmental progressions
Divided among the Six General Competencies:
2 Patient Care milestone sets
2 Medical Knowledge milestone sets
2 Practice-Based Learning and Improvement milestone sets
3 Interpersonal and Communication Skills milestone sets
4 Professionalism milestone sets
5 Systems-Based Practice milestone sets

25
11/3/2014
Cytopathology Milestones
Five Levels for each milestone set / developmental progression
Level 1: The fellow demonstrates milestones expected of an
incoming fellow.
Level 2: The fellow is advancing and demonstrates additional
milestones, but is not yet performing at a mid-fellowship level.
Level 3: The fellow continues to advance and demonstrate
additional milestones, consistently including the majority of
milestones targeted for fellowship.
Level 4: The fellow has advanced so that he or she now substantially
demonstrates the milestones targeted for fellowship. This level is
designed as the graduation target.
Level 5: The fellow has advanced beyond performance targets set
for fellowship and is demonstrating aspirational goals which might
describe the performance of someone who has been in practice for
several years. It is expected that only a few exceptional fellows will
reach this level.
Cytopathology Milestones PC12

The Patient Care 1 milestone set => development in
awareness of patient safety.
Maintenance of specimen identity and integrity.
Analysis and troubleshooting specimen problems.
Patient safety module of MOC => Level 5 milestone.
The Patient Care 2 milestone set => perform FNAB,

recognize, treat / triage complications.
ACGME case log => part of developmental progression.
Level 3 => obtain adequate FNAB material.
Level 4 => know / perform US-guided FNAB / core biopsy.
Cytopathology Milestones MK12

The Medical Knowledge 1 milestone set => diagnosis.
Interpret / diagnose cytopathology specimens, order /
interpret diagnostic adjuncts (immunocytochemistry,
molecular testing, flow cytometry).
The Medical Knowledge 2 milestone set => screening.

Cervical cancer => advise on guidelines, follow-up, ancillary
studies.

26
11/3/2014
Cytopathology Milestones PBLI12

The Practice-Based Learning and Improvement 1
milestone set => scholarly activity.
Research, evidence-based presentations, preparation /
submission peer-reviewed articles => begin early =>
completion.
The Practice-Based Learning and Improvement 2

milestone set => understand / apply evidence-based
laboratory testing / result utilization.
Cytopathology Milestones ICS123

The Interpersonal and Communication Skills 1
milestone set => interactions with other healthcare
providers, patients, families.
Appropriate FNAB procedure manners, patient / family
communication skills.

milestone set => conflict resolution.
Laboratory management skills => conflict among
employees.

milestone set => provide clinical consultation,
prepare / perform multidisciplinary conferences.
Cytopathology Milestones PROF

The Professionalism 1 milestone set => constructively
receive feedback.
Fellows educating medical students / training residents =>
constructively provide feedback.
The Professionalism 2 milestone set => honesty /

integrity => identify limitations, accept responsibility.
The Professionalism 3 milestone set => cultural
competency.
Awareness of bias, ability to counteract its effects.
The Professionalism 4 milestone set => physical,

intellectual, emotional health.

27
11/3/2014
Cytopathology Milestones SBP123

The Systems-Based Practice 1 milestone set =>
laboratory regulation / compliance.
Level 1 => HIPAA.
Progression => state, federal and professional society
regulations => inter-laboratory comparison programs =>
corrective measures.
The Systems-Based Practice 2 milestone set => role

of cytopathologist within healthcare system.
resource utilization / personnel / finances.
Level 5 => manage personnel / develop laboratory budget.
Cytopathology Milestones SBP45

technical issues in the laboratory.
Knowledge => gyn / non-gyn specimen preparation.
Advise medical staff => specimen collection / preservation.

quality improvement, risk management, laboratory
safety .
Quality improvement & patient safety project participation
=> identify / assign early => completion => Level 4.
Four Alpha Test Sites 2012-2013

Cleveland Clinic Foundation (CCF)
Massachusetts General Hospital (MGH)
University of Southern California/Los Angeles
County+University of Southern California Medical Center
(USC)
University of Virginia (UVA)
Alpha site PD panel reported to fellowship directors in
November 2013 at ASC

28
11/3/2014
Clinical Competency Committee

CCC composed of board-certified cytopathology
faculty members, potentially also including other
physicians, as well as PhDs and cytotechnologists
Cover all divisions of the cytopathology laboratory
Meet at least semi-annually, to review all available
performance reports and evaluation tools, and assess
each fellow's milestone levels achieved
Serve in an advisory role to the program director, who
reports the fellows' milestone levels to ACGME online
Milestones Evaluation Process

Faculty and fellows acquaint themselves with the milestones
Fellows self-evaluate; the self-evaluations do not go to
Clinical Competency Committee and are not part of the
Competency Committee assessment process
Clinical Competency Committee assesses each fellow's level
of achievement on every milestone set, based on their own
experiences with the fellows and all other assessment tools
The program director compares the Clinical Competency
Committee and self-assessments and reviews them both
with each fellow; only the Competency Committee
assessment is reported to the ACGME
Redesigning Evaluations for

the New Accreditation System
Paul N. Staats, M.D.
Director, Cytopathology Fellowship
Chair, Pathology Residency Clinical Competency Committee
University of Maryland Medical Center

29
11/3/2014
Overview
A few general thoughts on evaluations
Tailoring monthly evaluations to the
milestones
Branching out: expanding the range of
evaluations
Why evaluate?

30
11/3/2014
Why evaluate?
Provide feedback to trainee
Assessment for learning
Provide feedback about trainee to program

Assessment of learning
Provide feedback about program to program

Provide feedback about program to ACGME
What are the qualities that make

for useful evaluations?
What are the qualities that make

for useful evaluations?
Timely
Specific
Constructive
Multiple evaluators

31
11/3/2014
Monthly Evaluations
Workhorse of most programs evaluation
system
Filled out by attendings
Cumulative experience of entire month
Questions sometimes vague or general
Subjective scales common
Probably dont ask the questions your CCC
needs to evaluate milestones
Redesigning Monthly Evals
Know thy Milestones!

Break down big Milestones into smaller pieces
Scale questions pop out
Decide where you want to evaluate each
question
Replace, dont just add
But keep what currently works

32
11/3/2014

Evaluation question:
Rate the fellows ability in ordering ancillary
testing (e.g. IHC, Flow Cytometry, Molecular
testing):
1. Understands concepts
2. Actively observes
3. Proposes appropriate testing at signout
4. Orders appropriate testing independently
5. Teaches others
Appropriate location: Attending monthly eval
Try to replace a similar question you already have
Branching Out:
Suggested Evaluation Methods
Direct Observation
360 Evaluations
Portfolio
Case Logs
Inservice Examination
Examination
Simulation/Role Play
Review of Report
Records
Narrative
Checklist
Quality Assessment
Retrospective peer
review
Team leader
performance evaluation
Direct Observation
Clinical Performance Ratings - Monthly,
rotation, semi-annual or annual ratings of
resident performance
Focused Observation and Evaluation Supervisor/attending observation of individual
resident-patient encounters, operations,
specimen preparation, etc., and concurrent
(same day) evaluation
PC1-2, MK1-2, SBP1-5, PBLI1-2, PROF1-4, ICS13

33
11/3/2014
360 Evaluations
360 Assessments - Evaluation by MDs
(supervisors, residents, medical students) and
non-MDs (nurses, technicians, social workers,
PAs) using the same or similar evaluation
forms
PC1-2, MK1-2, SBP1-5, PROF1-4, ICS1-3
Who should be included?
360 Evaluations
Who should be included?
Cytotechs
Prep techs
Other department staff
Residents
Clinicians/clinical staff
Portfolio
Resident Project Report (Portfolio) Evaluation of resident work products, such as
reports of research studies, practice
improvement, or systems-based improvement
Other Portfolio - Evaluation of resident
performance based on other
work/performance products not included
above, e.g., audiotapes, slide presentations
PC1-2, MK1-2, SBP1-5, PBLI1-2

34
11/3/2014
Case Logs
Review of Case or Procedure Log - Review of
number of cases or procedures performed and
comparison against minimum numbers
required
PC1-2
In Service Examination
In-training Exams - A multiple-choice exam
developed by an external vendor
MK1-2
Examination
In-house Written Exams - A multiple choice exam
developed by residency program faculty
Multimedia Exam - A computer based multiple
choice or branching question exam in which
authentic visual and auditory patient information
is presented as question information
Formal Oral Exam - "Mock" oral exam in which an
examiner asks residents questions about what to
do in a clinical scenario presented verbally or role
played by the examiner
SBP1-5, PBLI1-2

35
11/3/2014
Examination
Simulation/Role Play
Role-play or Simulations - Residents are
evaluated based on their performance on
assigned responsibilities in a staged replica of
a potentially real situation, e.g., mobilization
of medical team in a multi-victim accident,
confrontation of an "impaired" colleague,
negotiation with administration regarding
facilities and equipment upgrade
PC1-2, SBP1-5, PROF1-4
Review of Report Records

Review of Patient Chart/Record - Involves
abstraction of information from patient
records, such as tests ordered, and
comparison of findings against accepted
patient care standards
MK1-2

36
11/3/2014
Narrative
Resident Experience Narrative (Portfolio) Evaluation of performance based on residents'
narratives of critical incidences or other
experiences, usually accompanied by
reflection on the event, e.g., what happened,
why, what could have been done differently
MK1-2, SBP1-5, PROF1-4, ICS1-3
Checklist
No ACGME definition. Use your imagination!
Go wild!
One example might be an FNA performance
checklist see handout
PC1-2
Other suggested methods
Quality Assessment: MK1-2

Retrospective Peer Review: MK1-2
Team leader performance evaluation: SBP 1-5

37
11/3/2014
ASC Can Help

ASC Program Directors Website
http://www.cytopathology.org/cytopathologyfellowship-programs
PEC
Cyto eJournal
Cyto eConferences
PBL Program for Cytology Education
Case Studies
Sound Bites
Summary
Evaluations are primarily for learners and programs, not the
ACGME
Take advantage of the milestones to make your evaluations
better
Low-hanging fruit first:
Revised monthly evals
Other methods you already use
Branch out:
Multiple evaluation types
Objective outcome measures
Be creative!
Please share
Clinical Competency
Committees for
Cytopathology
Results of Summer 2014 Survey
Deborah Chute, MD
Cleveland Clinic

38
11/3/2014
Data Source
A 15 question survey was sent to
cytopathology fellowship programs in July and
August, 2014
Distributed via ASC Listserv and PRODS
Listserv
Designed to identify current status and needs
for cytopathology fellowship CCCs
71 responses collected
Basic Background Data

Number of fellows per program:
Average 2 (range 1-6)
Number of faculty per program:

Average 6 (range 1-17)
Fellowship Program
Director
Teaching Faculty
CCC Member
CCC Chair
Cytology supervisor (2)

Associate Fellowship PD
We dont have a fellowship
Other

39
11/3/2014
CCC Details
Number of CCC members planned:
Average: 4.8 (range 3-10)
How many cytopathology CCC members will

be on other CCCs?
Average: 2 (range 0-7)

40
11/3/2014
Will you have non-MDs on CCC?

Yes: 48%
Cytotechnologist
Prep Technologist
PhD Scientist
All CCC members
are MDs
Program coordinator (6)

Specialist in
Cytotechnology (1)
Other
Yes, we are mapping

our current evals to
the Milestones
Yes, we are creating
new evaluations
No, we have not
started yet

41
11/3/2014
Problems Mapping Milestones
Not applicable haven't started yet (9)

No problems (4)
Time consuming (3)
Milestones are more detailed than old
evaluations and difficult to correlate (3)
Milestones are lengthy (1)
Difficult to change evaluations (1)
Why Not Start Mapping Milestones
Not enough time since release (8)

Doing residency milestones first (3)
Still forming CCC (2)
Waiting for ASC guidelines (1)
Will use separate Milestone assessment (1)
Revolted by the prospect milestones dont
seem to measure attributes of successful
residents (1)
Summary
Most programs have started to create CCCs

Many will have non-MD members
Many faculty will be on multiple CCCs
Many programs have started mapping
milestones to evaluations but with difficulty
Time is the biggest barrier

42
11/3/2014
Specific ACGME requirements for

creating a CCC
Who can be a member or chair of
a CCC
Examples of individual fellowship
CCC policies
Suggestions on conducting CCC
meetings efficiently
What is the role of the program
director in the CCC?
How to map Milestones to
Assessments
Mock CCC, mapped

evaluations already made,
how to deal with problem
fellows
Other
How Are We Helping?

Specific ACGME requirements
for creating a CCC
Who can be a member or
chair of a CCC
Examples of individual
fellowship CCC policies
Suggestions on conducting
CCC meetings efficiently
What is the role of the
program director in the CCC?
Laura Edgars Presentation

Handout
Laura Edgars Presentation
How to map Milestones to

Assessments
Paul Staats Presentation
Other
Mock CCC today!
Summary
Most programs have started to create CCCs

Many will have non-MD members
Many faculty will be on multiple CCCs
Many programs have started mapping
milestones to evaluations but with difficulty
Time is the biggest barrier

43
11/3/2014
Milestone Project:
Mock CCC
American Society of Cytopathology
November 14, 2014
Laura Edgar, EdD, CAE
Executive Director, Milestone Development
and
RC for Medical Genetics, Pathology and
Radiation Oncology
Presentation Topics
Review of Milestone Template and Scoring
Clinical Competency Committees
Mock CCC Activity
137
Milestone Template
139

44
11/3/2014
Option to select
Not yet
achieved Level 1
Milestones are progressive

over time. There is no
prescribed speed at which
residents must complete a
milestone set.
Selecting a response box in

the middle of a level implies
that milestones in that level
and in lower levels have
been substantially
demonstrated.
Selecting a response box on the line in

between levels indicates that
milestones in lower levels have been
substantially demonstrated as well as
some milestones in the higher level(s).
What is the
Clinical
Competency
Committee?
141
Clinical Competency
Committee
Composed of a minimum of 3
faculty members
Non-physician members can be
appointed
Reviews all evaluations by all
evaluators semi-annually
Reviews residents against
milestones semi-annually
Make recommendations for
progress promotion, remediation
and dismissal

45
11/3/2014
How does the CCC

actually work?
143
Who should be on the

CCC?
Decision for PD
May be Chair
May be Member
Consider:
Representation from each
major site
Subspecialty
representation
Dedication to education
How do we DO the
evaluation?
There is no one-size-fits all approach
Organization and a trial run are key
try reviewing a recently graduated
resident
With so few fellows in the program,
best to have everyone review
information at one time

46
11/3/2014
How do we DO the
evaluation?
Understand the milestones & their
use
Leave personal bias at the door
Review all evaluations for each
fellow
Consider the source(s)
For each fellow, decide the milestone
narrative that best fits that resident
The Clinical Competency

Committee
Avoids common problematic
issues:
I dont like to give negative
evaluations
I spent little time working with this
resident
Herd mentality: positive or negative
Grade inflation
Vague statements:
I just didnt like this resident, but I
cant put my finger on it
Hearsay: Ive heard she is lazy
Clinical Competence
Committee
Operativ
e
Performa
nce
Rating
Nursing
Scales
and
Ancillary
Personne
l
Evaluatio
OS ns
CE
Peer
Evaluatio
ns
2012 Accreditation
Council for Graduate
Medical Education
(ACGME)
Moc
k
Oral
s
End of
Rotation
Evaluati
RI ons Si
SE
m
La
b
Self
Evaluati
ons
Clinical
Competenc
e
Committee
Assessment
of Milestones
Cas
e
Logs
Unsolici
ted
Comme
nts
Student
Evaluatio
ns
Clinic
Work
Place
Evaluati
ons
Patient /
Family
Evaluati
ons

47
11/3/2014
Implementation
First Milestone reporting for Pathology is
December 2014
First Milestone reporting for
Cytopathology (and other Pathology
Subspecialties) is December 2015
149

48

Slide Presentation Handout For SCE ASC 2014

Uploaded by

Copyright:

Available Formats

Slide Presentation Handout For SCE ASC 2014

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Slide Presentation Handout For SCE ASC 2014

Uploaded by

Copyright:

Available Formats

11/3/2014

Accreditation Council for Graduate Medical Education

Next Accreditation System, SelfStudy Visits and Clinical

Accreditation Council for Graduate Medical Education

Next Accreditation System

2012 Accreditation Council for Graduate

Residency programs and their sponsoring institutions

2012 Accreditation Council for Graduate

Where did we come from?

2012 Accreditation Council for Graduate

Where are we?

The Next Accreditation System

Continuous Accreditation Model

The Next Accreditation System

2012 Accreditation Council for Graduate

The Next Accreditation System

Program Review in the NAS

Annual Data Collection

2012 Accreditation Council for Graduate

Annual Data Collection Common Errors

Accreditation Council for Graduate Medical Education

2012 Accreditation Council for Graduate

Accreditation Council for Graduate Medical Education

Ten Year Self-Study Visit

The 10-year Self-Study: Scope

2012 Accreditation Council for Graduate

The 10-year Self-Study: Timeline

1. Sets FINAL SelfStudy Visit Date and

1. Completes ADS data

2012 Accreditation Council for Graduate

Self-Study: Key Processes

Self-Study: Key Processes

Ten Year Self-Study Visit

2012 Accreditation Council for Graduate

2012 Accreditation Council for Graduate

Accreditation Council for Graduate Medical Education

Clinical Learning Environment

Clinical Learning Environment Review

Clinical Learning Environment Review

2012 Accreditation Council for Graduate

CLER Visit - Walking Rounds

CLER Focus Patient Safety

2012 Accreditation Council for Graduate

CLER Focus Health Care Quality

CLER Focus Care Transitions

CLER Focus Supervision

2012 Accreditation Council for Graduate

CLER Focus Duty Hours, Fatigue

CLER Focus Professionalism

CLER Pathways to Excellence

2012 Accreditation Council for Graduate

We are here to help

2012 Accreditation Council for Graduate

Some basic considerations

Where does accreditation come from?

Dr. Wesley Y. Naritoku, Oct. 2011

2012 Accreditation Council for Graduate

Where does "accreditation" come from?

What did Flexner do?