Patricia Surdyk, PhD - Association for Hospital Medical Education

The ACGME Outcome Project:
How Far Have We Come
with the Competencies
Patricia M. Surdyk, PhD
Senior Project Manager
Susan R. Swing, PhD
Director of Research
Association for Hospital Medical Education
May 13, 2005

The ACGME Outcome Project:
Do You Know Where You’re Going
with the Competencies

Objectives
Identify expectations of the ACGME regarding the progress
residency programs are making integrating the Competencies
Explain how the ACGME is measuring progress made by
residency programs during Residency Review Committee site
visits
Discuss retrospective data for programs that have been site
visited to date
Reflect on insights gained through experience

Whatever we measure,
we tend to improve.

“The Role of Assessment in a Learning
Culture”
“Assessment in the service of
learning…”
Lorrie A. Shepard
http://www.aera.net/pubs/er/arts/29-07/shep01.htm

How should we change what we do so that
faculty and residents look to assessment as a
source of insight and help instead of an
occasion for meting out rewards and
punishment
Paraphrased from Lorrie Shepard, AERA
Presidential Address, April 2001

Common Program Requirements
Common Program Requirements
V.A. Program Design
1. Format
2. Goals and Objectives
V.B. Specialty Curriculum
V.D. ACGME Competencies
VII.A. Resident
1. Formative Evaluation
2. Final Evaluation
VII.C. Program

Expectations for Assessment
Accurately assesses general competencies
Provides for regular and timely feedback
Supports performance improvement
Verifies that residents can practice competently
and independently
Provides useful data for assessing program
effectiveness

…it seems important to start with the forthright
acknowledgement that no single assessment
method can provide all data required for
judgment of anything so complex as the
delivery of professional services by a successful
physician.
George Miller, 1990

The “Competent” Program: Phase 2
The “Competent” Program: Phase 2
Competencies integrated into curriculum
Assessments aligned with goals and objectives
(identify desired outcomes)
Systematic assessment of competencies using tools
that provide accurate, dependable results (determine
which outcomes have been achieved)

The “Competent” Program: Phase 3
Competencies integrated into curriculum
Assessments aligned with goals and objectives (identify desired
outcomes)
Systematic assessment of competencies using tools that
provide accurate, dependable results (determine which
outcomes have been achieved)
Residents’ performance data used to improve the program
Prepare residents for life-long maintenance of competencies

Citations: RRC Examples
Implementation is insufficient, must demonstrate
effective plan to assess resident performance and
utilize results to improve performance
No formal evaluation process for resident, faculty,
program
No final written evaluation verifying sufficient ability
to practice competently and independently
No semi-annual evaluation

Citations: RRC Examples (cont’d)
No system to document resident experience in procedures
Logs not discussed as part of semi-annual evaluation
All faculty do not provide adequate verbal feedback;
feedback is sporadic and attending dependent
Minimal development of the faculty or the program for
implementation of the competencies
Residents were not aware of the six competencies

Timeline
Phase 1: Initial
response
Phase 2: Sharpening
the focus
Phase 3: Full Integration
with learning and clinical
care
Phase 4: Develop
models of excellence
Role of the GMEC

Outcome Project:
Phase 3 Expectations (7/06-6/11)
Full integration of competencies and their assessment with learning
and clinical care:
Program/Institution Focus
Use resident performance data to improve the
program (provide evidence for accreditation review)
Begin to use external measures (e.g., clinical
quality indicators, patient surveys, employer evaluations of
graduates, national or specialty standardized measures) to
verify resident and program performance

The “Competent” Institution
• Patient Care
• Medical Knowledge
• Practice-based Learning
and Improvement
• Interpersonal and
Communication Skills
• Professionalism
• Systems-based Practice
Institutional Requirements, I.A
The purpose of GME is to
provide an organized
educational program with
guidance and supervision of
the resident, facilitating the
resident's ethical, professional
and personal development
while ensuring safe and
appropriate care for patients.

The “Competent” Institution
• Patient Care
• Medical Knowledge
• Practice-based Learning
and Improvement
• Interpersonal and
Communication Skills
• Professionalism
Institutional Requirements, I.B.2
A Sponsoring Institution must
be appropriately organized for
the conduct of GME in a
scholarly environment and
must be committed to
excellence in both medical
education and patient care.
• Systems-based Practice

The “Competent” Institution
• Patient Care
• Medical Knowledge
• Practice-based Learning
and Improvement
• Interpersonal and
Communication Skills
• Professionalism
• Systems-based Practice
Institutional Requirements, I.C.1
A Sponsoring Institution must be
in substantial compliance with the
Accreditation Council for
Graduate Medical Education
(ACGME) Institutional
Requirements and must ensure
that its ACGME- accredited
programs are in substantial
compliance with the Institutional,
common and specialty-specific
Program Requirements.

The “Competent” Institution
• Patient Care
• Medical Knowledge
• Practice-based
Learning and
Improvement
• Interpersonal and
Communication Skills
• Professionalism
• Systems-based
Practice
Institutional Requirements, I.C.3
A Sponsoring Institution and its
ACGME-accredited programs
must be in substantial compliance
with the ACGME Manual of
Policies and Procedures for GME
Review Committees (ACGME Web
site, www.acgme.org). Of
particular note are those policies
and procedures that govern
"Administrative Withdrawal," an
action that could result in the
closure of a Sponsoring
Institution's ACGME-accredited
program(s) and cannot be
appealed.

The “Competent” Institution
Accountable for what
Administration
Organization
Education
Work Environment (including quality of patient
care)

IRC Common Citations: Examples
Residents not participating in learning the general
competencies
No participation in evaluating program/faculty
No participation on institutional
committees/councils
No opportunities for resident participation in
research

IRC Citations: (10/20-21/04)
It is not apparent that all internal reviews are consistently
addressing all of the common numerical duty-hour
program requirements or the six general competencies.
The institution and program are reminded that the
ACGME is now in PHASE 2 of the Outcome Project
which is “Sharpening the focus and definition of the
competencies and assessment tools,” and to…

IRC Citations: (10/20-21/04-cont’d)
+ …provide learning opportunities (as needed) in all six competency
domains;
+ improve evaluation processes as needed to obtain accurate
resident performance data in all six competency domains; and
+ provide aggregated resident performance data for the program’s
GMEC internal review (ACGME Web site, Outcome Project,
Timeline.)
+ Programs and internal review reports should begin to address and
document these efforts.

Two Educational Concepts
The informal curriculum happens every time the student
or resident is not in a class or on rounds, making it
virtually everywhere and unavoidable, two
characteristics that account both for its power and its
complexity.

Two Educational Concepts (cont’d)
The hidden curriculum includes the hidden transmission of
the dominant culture during formal classes. (Hundert, 1996)
The hidden curriculum is…the curriculum of rules,
regulations, and routines, of things teachers and students
must learn if they are to make their way with minimum pain in
the social institution called the school. (Jackson, 1968)

IRC Citation (10/20-21/2004)
The Sponsoring Institution and its ACGME-accredited programs must
provide an educational and work environment in which residents may raise
and resolve issues without fear of intimidation or retaliation. This
requirement is not being met in that in radiology, consultations, while
readily available by experts, is unpleasantly provided. The radiologist
apparently demeans the residents somewhat which makes the residents
reluctant to ask for help. Members of the faculty are supposedly treated in
the same way. In addition, there are turf battles between the Departments
of Obstetrics-Gynecology and Anesthesiology over procedures. The
institution and programs are reminded that two of the six general
competencies are professionalism and interpersonal and communication
skills defined as follows…

IRC Citation (10/20-21/2004-cont’d)
Based on the above comments about the work
environment, it does not appear that either competency
is being taught by the programs, that program directors
possess them, or that residents are being evaluated in
them. In addition, it does not appear that the GMEC
closely monitors the residents’ work environment or that
there are resident evaluations of theirs programs and
experiences given to the GMEC.

Conversations about Duty Hours
Conversations about Duty Hours
ACGME – “We will tell you exactly what to do to
reform duty hours.”
Program Directors – “That won’t work for my
program.”
ACGME – “Every program must do the same thing.”
Program directors – “You’ve got to be kidding.”

Conversations about Competencies
Conversations about Competencies
ACGME – “We invite you to respond to the challenge
of assessing the competence of your residents.”
Program Directors – “What would you like us to
do”
ACGME – “We don’t really know. Do something and
we’ll let you know if you did the right thing.”
Program Directors – “You’ve got to be kidding.”

How can we know what we think…
…until we see what we say
Karl Weick

Assessment Methods Reported
Typical Methods (80% - 38% of programs)
Global, clinical performance ratings
In-training exams
Focused observation (but not concurrent evaluation)
Multi-rater/360° evaluation
Evaluation Committee or meeting with PD
Review of case/procedure logs (Surgical)

Assessment Methods Reported (cont’d)
Other Methods (22% - 11% of programs)
Formal oral exams
In-house written exams
Review of patient charts or records
Structured case discussion
Resident project report (portfolio)

Assessment Methods Reported (cont’d)
Other Methods (10% - 5% of programs)
Review of patient outcomes
Standardized patients
Anatomic or animal models
Practice/billing audit
Other Portfolio
Stimulated Chart Recall
OSCE

How Are Results Being Used (% prgs reporting)
Method
Clin Perf
Rtgs
Written
Fdbck
Oral
Fdbck
Educ
Plan
Assess
Prgm
Curric
Change
75% 90% 78% 56% 50% 41%
ITE 70% 75% 33% 62% 74% 66%
Focused
Obs
Decisions
Multirater
Eval
Cmte
Formal
Oral
OSCE
69% 90% 52% 58% 36% 26%
68% 89% 47% 44% 37% 26%
71% 85% 74% 58% 46% 35%
65% 87% 38% 43% 53% 36%
63% 80% 25% 40% 47% 44%

RRCs’ Activities
Pediatrics, Physical Medicine & Rehabilitation:
revising Requirements around the competencies
Urology: developing national web-based evaluation
system (with ACGME Research & Information Technology
staff)
Emergency Medicine: identifying clinical quality
indicators to use as measures of GME quality (with
ACGME Research staff—funding from RWJ; Susan R.
Swing, PhD, Principal Investigator)

RRCs’ Activities (cont’d)
Radiation Oncology: refining specialty-wide assessment
tools (with ACGME Research staff)
Internal Medicine: pilot project to move from process to
an outcome-based accreditation model
Medical Genetics: developing curriculum resource
guide (with ACGME Research staff)

RRCs’ Activities for the Future
Agreement about citations
Identify clinical quality indicators to use as
measures of GME quality
Develop an outcome-based accreditation model
IRC – information included in internal review report;
GMEC actions based on findings
Annual data reporting
+ patient care; clinical experiences
+ programs, institutions

Ruminations from the Road
The competencies as organizing principles
New and “improved” conversations
Attention to basic principles
“Grudging” acceptance