The Effectiveness of

Competency-Based Education

Stephen R. Smith, M.D.

Associate Dean for Medical Education
Brown Medical School
Courtesy of Ronald Harden
Courtesy of Ronald Harden
Courtesy of Ronald Harden
Courtesy of Ronald Harden
 Calls for Reform
• AMA Future Directions in Med Ed (1982)
• GPEP Report (1984)
• Macy Conference Report (1989)
• Bloom (1988)
• Robert Wood Johnson Foundation Program
in Medical Education (1989)
 Flexnerian Model
➀ ② ③ ➃
Define Teach the Test for Hope for
“Fundamental Fundamentals Knowledge of the Best
Knowledge” Fundamentals

?
 Competency-Based Model

Design
Develop Define the
Measures and
Learning Successful
Standards of
Experiences Graduate
Performance
 Brown’s 9 Abilities
Effective Communication
Basic Clinical Skills
Using Basic Science in the Practice of Medicine
Diagnosis, Management, and Prevention
Lifelong Learning
Professional Development and Personal Growth
Social and Community Context of Medicine
Moral Reasoning and Clinical Ethics
Problem Solving
 ACGME Outcome Project
General Competencies for Residents
• Patient Care
• Medical Knowledge
• Practice-based learning and improvement
• Interpersonal and communication skills
• Professionalism
• Systems-based practice
 The Scottish Doctor
• What the doctor is able to do
– Clinical skills
– Practical procedures
– Patient investigations
– Patient management
– Health promotion and disease prevention
– Communication
– Medical informatics
 The Scottish Doctor (2)
• How the doctor approaches practice
– Attitudes, ethical understanding, and legal
responsibilities
– Decision-making skills and clinical reasoning
and judgement
– Basic, social and clinical sciences and
underlying principles
 The Scottish Doctor (3)
• The doctor as a professional
– Role of the doctor within the health care system
– Personal development
 Definition
Lifelong Learning

The competent graduate is aware of the limits

of his/her personal knowledge and experience
and has an intellectual interest in general
education and medical science. The graduate
actively sets clear learning goals, pursues
them, and applies the knowledge gained to
the practice of his/her profession.
 Criteria
1.Recognizes personal limits in knowledge
and experience
• initiates steps to rectify gaps in knowledge
• researches, reviews, and extracts data from
multiple sources
• confers with knowledgeable colleagues,
advisors, or experts
 Levels
• Beginner: demonstrate proficiency in a
structured setting (e.g., medical school
class)
• Intermediate: demonstrate proficiency in a
clinical or research setting
• Advanced: submission for publication or
acceptance of abstract, poster, or oral
presentation at an academic meeting
Performance-based Assessment
• Authentic
• Increasing level of challenge
• Formative feedback
• P=KxA
 QuickTime™ and a
DV/DVCPRO - NTSC decompressor
are needed to see this picture.
 Evidence of Effectiveness
• Carraccio et al. (2002) reviewed 469 articles
• “revealed little scientific evidence evaluating the
outcomes of competency-based education”
• “What evidence that does exist clearly favors
competency-based education over the current
structure- and process-based model”
 Clinical Skills

A competency-based curriculum
introduced into three medical schools in
China rapidly improved the clinical skills
of students compared to the traditional
curriculum.

From Stillman PL. Teaching and assessing clinical skills: a competency-based programme in
China. Med. Educ. 1997; 31:33–40.
 Neurosurgical Training
40
35
30
25
20 Traditional
15 CBE

10
5
0
Laminectomy Spinal fusion Pterional Suboccipital
craniotomy craniotomy

From Long DM. Competency-based Residency Training: The Next Advance in

Graduate Medical Education. Acad. Med. 2000; 75:1178–1183.
 Epidemiological Proof
• Strength of the Association
• Dose–Response Relationship
• Consistency of the Association
• Temporally Correct Association
• Specificity of the Association
• Coherence with Existing Information
(Biological Plausibility)
 Educational Proof
• Theory and Principle (Educational
Plausibility)
• Presence of Desired Effects
• Absence of Adverse Effects
• Qualitative Assessment
• Near-term Quantitative Assessment
“The only way to get somewhere, you know, is
to figure out where you’re going before you go
there.”
——John Updike, Rabbit Run, 1960

“It is a highly questionable practice to label

someone as having achieved a goal when you
don’t even know what you would take as
evidence of achievement.”
——R. F. Mager, Preparing Educational
Objectives, 1962
 Desired Effects
• Changes in learning behavior in desired
directions (e.g., clinical skills)
• More explicit guidance for students
• More authentic and relevant teaching and
assessment
 Avoid Adverse Effects
• No diminution in USMLE/NBME scores
• Theoretical criticisms not realized
– Competencies not representative of tasks
– Reductionistic, trivializing, decontextualized
– Devalues knowledge
– Constrains teaching and creativity
• Not too expensive or resource intensive
 Qualitative Assessment
• Concept valued by students
• Initial skepticism by faculty, but willingness
to go along
• Credibility gap created when only lip
service paid to competencies
• Resistance to jumping through more hoops
• Submerged into the culture
Ability VII Fulfillment
 Independent Studies

72.5
80
70
60
50
33.7 Brown
40 U.S.
30
20
10
0
 Evaluation Approaches
• Near Term
– Qualitative assessments
– Intermediate goals
• Long Range
– Comprehensive studies (à la Tamblyn)
– Long-term goals