Choosing the Right Statistical Test

Key Steps in Statistical Analysis of a Biological Experiment

1. Specify the question 6. Choose best test based on variables/hypotheses

2. Define the null & alternative hypotheses 7. Do power analysis and determine sample size
3. Determine the relevant variables 8. Do the trial/experiment
4. What type of variables are they? 9. Test for assumptions for chosen tests
5. Design experiment (control confounding variables) 10. Analyse the data, interpret, communicate results
Source: http://udel.edu/~mcdonald/statstep.html
Parametric or non-parametric?
There are two ‘families’ of statistical tests – parametric and non-parametric. Histogram for weight~cvd=0

Parametric tests (including t test, ANOVA) assume data comes from a normal 80

(Gaussian) distribution, also when from larger datasets (e.g. at least 2 dozen in each 60

group). In other words the data plotted on a frequency histogram will resemble the 40

bell-shaped curve of ‘normal distribution’. Non-parametric tests are typically used 20

where sample sizes are small, or where normality is not assumed.

0
42 54 66 78 90 102 114 126
36 48 60 72 84 96 108 120
Mid-points for weight~cvd=0

One or two-sided P-value?

Many tests offer both, so a decision has to be made: a one-sided P value is appropriate where you can state
confidently (before collecting data) that there is no difference, or there is a difference in one direction – i.e. you
can specific which group will have the larger mean. If the outcome is uncertain, then a 2-sided P-value is
appropriate.

Type of investigation?
 Differences – one measurement across two samples, so see if they vary. Easily summed up in a bar chart.
 Similarities – is there a relationship between the two samples?
- Correlation: does one measure change as the other varies – strength of correlation visualised in a scatterplot
- Association: two categorical variables, perhaps on two bar charts, so shown in a contingency table

Type of data variables?

 Measurement (or continuous, or interval) variables
‘Real’ numbers measured on a continuous scale (height, weight, blood sugar, temperature etc), with consistent
intervals or scales between measurement points (degree, mm, mg).
 Ranked (or ordinal) variables
Categories that can be ranked in order (income level, educational attainment, satisfaction) but where the
divisions between them are not mathematically consistent (or even meaningful).
 Nominal (or categorical) variables
Two or more categories, but no intrinsic order (e.g. gender, ethnicity, hair colour). you cannot ‘do maths’ on the
numbers.
 Ambiguous variables
A measurement with a small number of variables may be considered a measurement or a nominal variable – the
choice may depend on the null hypothesis (present/absent=nominal, larger/smaller=measurement).
 Ratios
BMI could be treated as a single measure and so a liner regression could test association with LDL, or you
could do multiple regression for weight and height against LDL, which is more complicated and advanced.

Sample size calculation

Four critical quantities: type 1 and type II error rates ( ά and β), fixed at 5% and 20% respectively, variability of
data (σ2) and minimum expected (or minimum clinically important) effect size δ. Larger effect sizes can mean
smaller sample sizes (i.e. easier to detect).
Choosing the Right Statistical Test
Type of Data
Goal Measurement Rank, Score, or Binomial Survival Time
(from Gaussian Measurement (Two Possible
Population) (from Non- Outcomes)
Gaussian
Population)
Describe one Mean, SD Median, interquartile Proportion Kaplan Meier survival
group range curve

Compare one One-sample t test Wilcoxon test Chi-square

group to a or
hypothetical Binomial test
value
Compare two Unpaired t test Mann-Whitney test Fisher's test Log-rank test or Mantel-
unpaired (chi-square for large Haenszel
groups samples)

Compare two Paired t test Wilcoxon test McNemar's test Conditional proportional
paired groups hazards regression

Compare three One-way ANOVA Kruskal-Wallis test Chi-square test Cox proportional hazard
or more regression
unmatched
groups
Compare three Repeated-measures Friedman test Cochrane Q Conditional proportional
or more ANOVA hazards regression
matched
groups
Quantify Pearson correlation Spearman correlation Contingency
association coefficients
between two
variables
Predict value Simple linear regression Nonparametric Simple logistic Cox proportional hazard
from another or regression** regression regression*
measured Nonlinear regression
variable
Predict value Multiple linear regression Multiple logistic Cox proportional hazard
from several or regression regression
measured or Multiple nonlinear
binomial regression
variables
Source: www.graphpad.com

For 2x2 tables, use Fisher’s Exact Test or Chi-Square Test. Fisher’s Test is preferred as it gives an exact P-
value; Chi-Square is easy to calculate, but provides only an approximate P-value and should be avoided when
numbers are below 6 in any cell.
Choosing the Right Statistical Test
Decision making flowchart for Group Comparison and Association Statistical Tests:

Variance known 1 sample z test

One group
Variance unknown 1 sample t test
2 sample (Independent)
t Test for equal
variances

Non-paired data F test

2 sample (Independent)
Compare two groups t Test for unequal
variances

Paired data Paired Sample t Test

Normal

One level 1-way AOV Multiple comparison

(post hoc) Test
Compare more than More than one level 2-Way AOV Multiple comparison
two groups (post hoc) Test
Hierarchical levels Nested AOV Multiple comparison
Interval or ratio data

(post hoc) Test

Normality Test

Association between Pearson’s correlation

two groups

One group Wilcoxon 1-Sample Median Test

Non-normal

Compare two groups Mann-Whitney U test

Compare more than Kruskal-Wallis H test

two groups

Association between Spearman Ranked Correlation

two groups

Source: http://webspace.ship.edu/pgmarr/

Errors, biases and other things to avoid (or watch out for in others)

Bias: a systematic error in data collection or analysis, caused by inadequate procedures

 Sources of bias:
Representative sampling, selection bias, limitations of single-centre studies, treatment bias
 Errors of methodology:
Insufficient statistical power, ignoring measurement error – and the reliability and validity of the
measurement process, making multiple comparisons (without adjusting P value for each one)
 Errors of interpretation:
Confusing effect size with significance, confusing precision with accuracy, causality and causal
interference, graphical representations

Source: http://my.execpc.com/~helberg/pitfalls/
Choosing the Right Statistical Test
Statistics Software
 SPSS is standard high-end software – each Trust library has a copy of SPSS 11.5 on a single PC (a 2002
version) capable of running all standard tests). Support limited to one or two books in the library
 Statsdirect is easy to use software that includes all the standard statistical tests. Loaded on three PCs in
each East Kent Hospitals library. Textbook and helpsheet is main support. (www.statsdirect.com)
 OpenEpi is free open source online (or downloadable) software for epidemiological statistics. Clear
examples, online calculators etc. (http://www.openepi.com/)
 KT Clearinghouse online statistical tools (http://ktclearinghouse.ca/cebm/toolbox)

Other Online resources

 RevMan: Software from Cochrane to conduct a meta-analysis, generate forest plots, present data, as well as
construct a properly formatted systematic review (http://ims.cochrane.org/revman)
 Statistical Packages: online statistics resources (http://www.statsci.org/free.html)
 Internet Glossary of Statistical Terms: (http://www.animatedsoftware.com/statglos/statglos.htm#index)
 StatPages: Huge list of online statistics books, tutorials, software, and online tools (http://statpages.org/)

Further reading
 PMean: monthly newsletter covering statistical topics, as well as examples of common errors and
misinterpretation (http://www.pmean.com/news/)
 BadScience: Guardian journalist Ben GOldacre’s column & blog debunking statistical incompetence in
journalism and public life (www.badscience.net)
 UnderstandingUncertainty: web site and blog looking at statistics in an accessible way
 Medical Statistics At A Glance: companion site to book of the same name – interactive multiple-choice
questions for each section of the book to test your statistical knowledge (http://www.medstatsaag.com/)
 Comprehensive guides on Using SPSS and Using Statistics in Research from East Midland RDS.
Download from http://www.rds-eastmidlands.nihr.ac.uk/resources/cat_view/13-resource-packs.html

Statistical Support
Staff with a project registered with R&D can access free advice from the Statistical Helpdesk provided by the
University of Kent on statistical issues & on studies/projects, data analysis, sample size calculations & help
with grant applications and ethics submissions. This is provided monthly on a pre-booked basis through the
R&D Office. Contact sandra.tapsell@nhs.net for further information.

Wildcard suggestion (at your own risk!): Go to www.elance.com, www.freelancer.com or

www.peopleperhour.com and invite bids from freelancers to undertake your data analysis. Or Google ‘freelance
statistical analysis’ etc...

And in the library? Look on shelf WA950

Mark Kerr, Clinical Librarian mark.kerr@nhs.net v1.2 6.4.14