Understanding and Reporting

on Journal Articles
Useful mathematical tools for
journalists
www.stats.org
Rebecca Goldin, Ph.D.
January 20, 2016
National Press Foundation

A couple words about our organization:

Principal aim is to increase statistical savvy in
the media
Provide workshops, public lectures
Write critical pieces at STATS.org and
occasionally other venues
Advisory Board of statisticians to help journalists
Foundation funded. No corporate sponsorship
permitted.

Statistical concepts in
scientific journal articles
Mean, median, mode
Standard deviation
Confidence intervals
Orders of magnitude
Confounding factors
Percentages
Absolute vs. relative risk
Scientific methods
Study Design
Causation versus correlation

In the beginning: a press

release
Press
Releases
Dont Tell the Whole
Story

Designed to get attention by the

press
Present the results in the rosiest

terms possible
Dont put the results in context of

other research
Shy away from concrete descriptions

Abstracts Dont Tell

the Whole Story

How were subjects recruited?

What was the design of the

experiment?
What methods were used to analyze

the data?
What are the weaknesses of the

conclusions?

Overview of scientific
journals and their value to
scientific consensus
How many scientific journals are there? PubMed is the
web interface of the National Library of Medicine. It has just
under 45,000 titles.
How do I know if a journal is peer reviewed? Check the
journals website. You can also check on Ulrich's Periodicals
Directory, but this requires an institutional subscription.
Where are the authors from? Authors list their affiliations
and funding sources at the beginning of papers.
How do I know if a journal is good? Metric is citations.
Science Citation Index (and websites that mirror it) will give
you rankings so you have a sense. You can google S Not
listed? Not cited!

Overview of scientific
journals and their value to
scientific consensus
Whats the process of publication?
Peer reviewed publications go through a peer-review
process. If theres no review, anything goes.
Peer review can be time-constrained/weak. Many
papers retracted.
Researchers should report their affiliations and support
(industry, government, conflicts).
Is there value in non-peer reviewed science at a
prestigious conference? Sure, but there has been no
examination of the quality. A poster can report extremely
preliminary and optimistic findings.

Moral and statistical

collision
Abortion (and Cancer)
Obesity (and Cancer)
Nursing vs. Bottle

Feeding
Smoking (and Cancer)
Food/Alcohol (and
Cancer)
Natural versus
Chemical (and Cancer)
Pollution (and Cancer)
Crime

Causation or Correlation

Its easy to be fooled

Height correlates with reading skills in children under

10
Height correlates with cancer
Chocolate consumption is correlated with Nobel Prizes
Digit ratio (2D:4D) and prostate cancer
Income correlates with success in college
Parking availability correlates with driving
Facebook correlates with poor grades
Facebook correlates with good grades
Doing heroin correlates with doing marijuana
Alcoholism correlates with less gray matter in the
prefrontal cortex

Cancer breakthroughs that cost

too much and do too little
Newsweek and The Daily Beast, March 2013
Article on targeted therapies, cancer therapies that try to

target the tumor of patients and are often used as part of a

last effort to beat the cancer, or to delay death. Perjeta is a
drug used for late stage breast cancer patients.
Perjeta gives the average woman only about six months more of
calm before her disease starts to stir again. Given the limited
benefit, the price was startling. For most women, a full course of
the drug combination will cost $188,000.

Article is about cost compared to average increased life

span.
But who is the average woman? The studies refer to the
median length of increased life. Half of women die within 6
months, and half of women die more than 6 months later.

Cancer breakthroughs that cost

too much and do too little
Newsweek and The Daily Beast, March 2013
Someone who lives 5+ years more

(considered cured) would count the same

in this calculation as women who live 6.1
months more.
The median increased lifespan tells us
almost nothing about the drugs
effectiveness.
6 months

Time

5 years

Doctors opinion on median life

extension: The New York Times
Op-ed by 20 prestigious oncologists: A Plan to Fix Cancer Care
Complained that .5 percent of cancer patients are consuming 5

percent of the countrys health care spending.

Pointed to the fact that the best of these recent therapies
increased the lifespan of cancer patients by a median of 6
months. Two only increased lifespan by 4-6 weeks.
Older drugs as good as newer drugs costing >$5,900 per month.
Whats wrong with this reasoning?
No one would balk at the price if these drugs really worked.
But with no sense of how long the better half lives, how can
anyone determine how good these drugs are?
If a drug were to cure 25% of people on cancer (and do nothing
for the others), it would be a miracle, but could fail this test.
Favoring older drugs may ignore realities about which people they
work for, versus their average rate of help.

(Easy) questions to ask while

reading a journal article
What is the context and strength of the results? The

abstract will tell you the big punch, but context is

everything.
How are the people in the study recruited? In
particular, would the recruitment method itself bias the
results by involving people who might not be typical with
regard to the thing measured? If the results dont
generalize to a broader population, they arent very strong.
For example, women with a family history of breast
cancer may be more likely to get a mammogram. Since they
also have a higher risk of cancer, the rate of cancer
detection among women choosing mammograms may be
higher than the rate would be if every woman were to get a
mammogram.

(Easy) questions to ask while

reading a journal article
How is data collected? Is there bias? (this is especially

important in survey, opinion, and food studies)

Are the results statistically significant? (more on
significance in a bit)
Are the results clinically significant? Results that
claim significance are not necessarily going to impact
our lives that much.
If a study speaks in increased risk, what is the
absolute increased risk?
Why might there be a correlation without a causal
relationship?

Jumping from correlation to cause

You dont always have to know why it may not be

causal. Be wary of any claims of causality.

Some common reasons that a correlation could look
causal when its not, include: not adjusting for
confounders, misunderstanding the mechanism,
having an unknown confounder.
A causal relationship might be reasonable to suspect
when the statistics are
Overwhelming
Observed in many different contexts
Repeated tests show the same effect, on large

numbers of people
Double blind case-control studies.

What the heck is that p-value?

A p-value is often reported in an

abstract, claiming the results are

statistically significant, with p<.05.
You flip a coin many times, and you
think this coin is biased, because you
arent getting close to heads and
tails. How
can
quantify
your
Whats
theyou
probability
the coin
is
You cant answer thequestion:
fair?
suspicion?
You can answer this: If the coin is fair, whats the probability I

would see that data or more extreme? This is the p-value.

If you flip a fair coin 1000 times, the chance of getting 520 heads
or more is just under 10% or .10. The chance of getting 550 heads
in 1000 flips is only about .001. Unlikely if the coin were fair!
The biomedical community generally accepts p=.05 (5%) or lower
as a standard for when you can reject the notion that the coin was
fair.

A Graphic on Coin Flipping

Probabilities

The normal curve

What happens in the lab:

Experiments Galore...

What the
rest of the
world
sees

Hard to get our heads around

randomness
My Magic Penny

How I found it.

Predicts the presidency: H for

Democrats, T for Republicans

It was correct 15 out of 17 of the

previous elections.
Just .1% chance of that happening

by chance, so it truly is magic.

If it werent a coin, but were some

indicator like the economy, the

party of the incumbent, June
approval ratings, you would
probably think theres a reason for
success.

I asked 1000 of my best friends to flip

a penny 17 times, writing H or T for

each trial. Then I compared to the list
of correct presidents.
Then I picked the best coin.
In a room of 1000 flippers, there is a

70% chance of finding a magic coin.

Do you see any

patterns?
CHOICE A:
H H T T T T H H T H T H H T T H H H T T T H T T H T

TTTHHTTHHTTTTHHTTTTTHHHTHHT
HHTHTTHHTTTHTTHHTHTHHTHHHH
THTTHTHHHTHHTHHTTTHHH
CHOICE B:
T T T T T H T H T H H H H H T H H T H H H H T T T T
HHHTHTHHHHTHTHHHTHTTHHTTTT
TTTTHHTTTHHTHHTHHHTHTTTTHH
HHTHHHHHTTHHHTTTHHTTTH

Which is random?

Some flip data

20
18
16
14
12
10
8
6
4
2
0

There is an 81% probability that a

sequence of 100 flips will have a run of 6
or more Hs or Ts in a row.

Cancer clusters

Cancer clusters are

real: suspected clusters

are geographical
regions where a higher
than expected number
of cancers occur.
Over 1000 possible cancer clusters are reported each year

to state health departments.

Cancer clusters can occur by chance. How to tell if its not?
Statistics help: what is the expected number of cancers?
Just like coin flips with many in a row: even if its all
random, there will be places with more cancers than
expected.
Series of other (non-statistical) factors involved, such as
consistent types of cancer, sufficient number of cases, and
biologically plausible mechanisms.

Texas Sharpshooter Fallacy

Basic Advice for a Journalist with

Limited Time and Ideas
If you have 30
minutes
Read the summary, the

abstract, and the conclusion.

Many writers ignore conclusions.

If you have 60
minutes
Check for significance. Are the p-

values small? Are the differences

clinically meaningful?

The abstract will tell you the

Ask the authors questions. Clinical

result. It hardly ever hints at

limitations.

impact? Generalizable to which

populations? Weaknesses of methods?

The conclusion or discussion

Get another professional opinion.

often shares caveats. The

conclusion is typically at the end of
the paper.

What is the quality of this study? How

does this work fit into the body of
research? Look to local universities
and professional organizations in the
field to find experts.

Avoid most conclusions of

causality. Read as a skeptic at all

times.
Look to quantifying the risk.

What is the actual risk? What does

it mean for most peoples lives?

Use PubMed.gov. Has other

literature been published on the

topic? Is there consensus?

Research is routinely plagued

Research is plagued

What can a
journalist do?

Low levels of significance

Multiple testing
No acknowledgement of

randomness in research design

Lack of context/repeated

experiments
Scientists dont know how to talk to

journalists. You can help by asking

good questions and reiterating
what you think the person
said/sending quotes for edits.
But if you are looking to find one

scientist willing to talk, you may not

get the mainstream opinion.

Write about the levels of significance,

bias, caveats
Ask the researchers about multiple

testing. Did they adjust for them?

Write about absolute risks.
Look for a body of research rather

than one specific paper

Cite your sources!
DONT INDICATE CAUSATION

WHEN A CORRELATION HAS BEEN

SHOWN!

How to get help

Ask the corresponding author and disinterested parties for

information like:
Whats the clinical significance? (Examples include: How many lives will

be saved in a group of 1000 people? Does the improvement described

in a paper mean that peoples lives will improve in a practical way?
Does the measurement that changed when people are exposed to a
toxin matter for their lives?)
Do you see any possible bias based on data you collected or study
design?
What other peer-reviewed research by other laboratories fits in with
what you have done?
STATS.org
On our website, theres a link for journalists to get help
Advisory board consists of statisticians who are familiar with the

challenges journalists face they answer quickly! Typically, within 2

hours.
Call on professional and charitable organizations, and universities,

to find experts.
Want more formal statistics? Khan Academy, iTunes U

Thank you!