Evidence Based Medicine

&
Critical Appraisal
Dr Richard de Ferrars
October 2009

Realistic Goals
What is realistic in 1 session?
What does EBM mean?
Why (and how) do we
read medical papers?
How to spot papers that
matter

What Does it Mean?

There are lies,
damned lies
and statistics
Mark Twain

Can you see through the statistics and spot the lies?

Why Bother?
There must be a better way than
being told what to do by
protocols & guidelines
Learning how to think for
yourself and develop skills in
critical appraisal
Learning how to link this in with
the care of your patients
Learning not to be a Lemming

Where Did EBM Come From?

Archie Cochrane
Scottish epidemiologist who, in 1972 wrote a book:
Effectiveness and Efficiency: Random
Reflections on Health Services
Gradual increasing acceptance of the concepts behind
evidence-based practice

Term "evidence-based medicine" first appeared in the

medical literature in 1992 in a paper by Guyatt

What About Critical Appraisal?

Critical

(adjective)

skillful judgment as to truth & merit

Appraisal

(noun)

the act of estimating or judging the nature or value of

something or someone.
More than just knowing how believable the contents of
some research & papers are.
Also an element of judging the value & relevance, in
practical terms, for your own work.

Evidence Based Medicine

Exercise What is good quality?
1. I read a review in the BMJ last week that showed that showed
thiazides gave better outcomes that ACEi in the elderly
2. My GPR (who has just passed his MRCGP) says thiazides are
good anti-hypertensives
3. The new NICE guidelines say first choice BP drug for over 55s is
thiazide or CCB
4. Most people that I see in my surgery who have high BP seem to be
on medication that includes a thiazide
5. That drug rep last week showed my some impressive graphs for his
new CCB-thiazide combination drug

Rate these 5 fonts of wisdom in order of good & sound

Evidence Based Medicine

Good Studies & Bad studies Study Hierarchy
Meta-analysis & systematic review
aggregation of several similar studies
Double-blind randomised placebo controlled studies
Larger, generic products, paid for by neutral body
Double-blind randomised placebo controlled studies
Smaller, branded products, paid for by the manufacturer
Observational studies
Case reports
Anecdotal experience

Big is Beautiful!

Evidence Based Medicine

Big is Beautiful!

Who Says?
Strength of Recommendation:
1++ Meta-analysis of randomised controlled
trials.
1- Randomised controlled studies
2++ Systematic reviews of case-control &
cohort studies
2+/- Case-control studies & cohort studies

A Based directly on category 1+

evidence.
B Based directly on category 2++
evidence
or extrapolated from category 1

Non-analytical studies
(comparitive studies, case reports)

C Based directly on category 2 +/evidence

or extrapolated from category 2++

Expert opinions, clinical experience of

respected authorities

D Based on category or
extrapolated from category 2+

Who Says?
The hierarchy of studies is reasonably obvious
But the source of information is probably
equally important
Where do you go to get information, advice &
recommendations?
What sources & resources do you use?

Who Says?
Compromise between quality of information and
accessibility (ranked in quality order)

CKS (Prodigy), SIGN, Cochrane

Publications - Journals (BMJ),

Daily Mail?

Local Guidelines PCT,

hospital, department

Your team
Your patients????

Accessibility

Quality and believability

National bodies NICE, NPC,

Problems with EBM

Great in theory
What about the real world?
What are the main problems & issues with
putting EBM into practice?

Problems with EBM

RCTs now might be unethical
Open heart surgery, thyroxine

Under-researched groups

Women, racial minorities, multiple problems

Gold standard" DB-RCTs are expensive

Drug companies fund studies looking at their drugs
Public authorities fund preventive medicine studies

Published studies may not represent all studies

Unpublished studies publication bias
Non-publication agreement if not declared at outset

Problems with EBM

The quality of studies performed varies
Trials world and real world are different:
Reality - does not have exclusion criteria
Reality - patients have multiple pathologies
Reality - patients problem wont match EBM

EBM changes slowly, medical advances are rapid

Example trials that compared Clopidogrel to angioplasty in
preventing reinfarction used doses of Clopidogrel that would now
seem too low

I Hate Statistics
Medical Statistics
Cannot escape it
You must have a basic
level of understanding
Tested in the AKT
Talking statistics with
patients may be
required in CSA

How to Read a Paper

Trisha Greenhalgh
Must buy & must read
Medical statistics
EBM
Critical appraisal

Medical Statistics - Taking Risks

Taking Risks
The annual stroke risk in high risk AF:
Taking aspirin
12%
Taking warfarin6%
1.
2.

Is it fair to say that Warfarin halves stroke risk?

How many people need to have Warfarin rather than
Aspirin for a year to stop one stroke?

Taking Risks
Absolute risk (AR)
statement of how often the event happened

Relative risk (RR)

risk of the event in one group relative to another group

Absolute risk reduction (ARR)

statement of how much less the risk is in one group cf. the other

Relative risk reduction (RRR)

How much your relative risk has fallen

Making Sense of Risks

Annual stroke risk in high risk AF patients:
Warfarin 6% Aspirin 12%
AR of a stroke on warfarin is 6% per year (0.06)
AR of a stroke on aspirin is 12% (0.12)
RR of a stroke on warfarin compared with aspirin is 50%
(calculated by: 6 / 12)
ARR from taking warfarin rather than aspirin, for one year,
is 6% (AR reduced form 12% to 6% )

Lies, Damned Lies & Statistics

You could say using warfarin instead of aspirin for a year, your
risk is reduced by 50% (relative risk)
Or you could say using warfarin instead of aspirin for a year,
your risk is 6% less (absolute risk)
Both are true but one is a distortion.
Drug companies normally sell you relative risks!
The word risk means absolutely nothing unless prefaced by
relative or absolute
Big relative risk changes mean little if absolute risk is small.
Make a small risk smaller and you still have a small risk

Turning ARRs into NNTs

Number of patients who would need to be treated for
one to benefit.
How many ARRs make 100%?
Taking warfarin rather than aspirin, for one year:
ARR is 6%
NNT is 16 (as 16 x 6% = 100%)

Longer-term medications & interventions usually

quoted as NNT over 5-10 year period

Numbers Needed to Treat

Number of patients who would need to be
treated for one to benefit.
NNT of under 10 is pretty good
NNT 10-30 is useful
NNT over 50 are stretching resources a bit

Useful EBM Resources

National Library for Health (www.nlh.nhs.uk/)
an online library for NHS staff, patients and the public

National Prescribing Centre (www.npc.co.uk)

supporting high-quality & cost-effective prescribing within the NHS

Cochrane Library (www.thecochranelibrary.com)

contains high quality, independent evidence to inform healthcare decision

making

Clinical Knowledge Summaries (put CKS into Google)

up to date source of clinical knowledge on common conditions for healthcare

professionals and patients

GP notebook (www.gpnotebook.co.uk)
an online encyclopaedia of medicine

Professional Reading
Browsing
Flicking the BMJ, BJGP, Pulse, GP
Has value, BUT restrict to 2-3 journals per week

Targeted Reading
Using the EBM resources to answer a particular
question

Critical Appraisal
How valuable is an interesting paper when (or if!) you
do come across one?

Critical Appraisal
Knowledge of EBM & statistics:
Knowing how believable the contents of
research & papers are.

Developing critical appraisal skills:

Judging the value & relevance, in practical
terms, for your own work.

Critical Appraisal - READER

Relevance
1-5 How relevant to your practice?

Education
1-5 How likely to change your behaviour/ practice?

Applicability
1-5 How easy to apply & implement changes?

Discrimination
1-10 How good is the quality of the paper?

Evaluation
4-25 Total the score

Reaction
4-15
16-20
21+

Forget it
Think about it
Do it

Using READER
Practice using the READER model to
critically appraise journal articles

The End