Instructional messaging selectively improves student perceptions of a non-

traditional question format

Joanna K. Hubbard, Christian D. Wright, & Brian A. Couch

SABER 2017 | Poster supplement

Figure S1. Data from exam 1 shown to students for the grade-based justification at A) Institution
A and B) Institution B.

A) B) 100 MC section
MTF section 82 84

Percent correct
80 73
60 60
60
38
40

20
0-60% 60-80% 80-100%
Total exam score



Table S1. Comparison of mean incoming ability and mean exam performance between
sections within schools. We assessed differences at each school separately using Students t-
tests.
Incoming Ability Std.
School Section n t df p
(ACT score) Dev
Knowledge-based 26.3 3.5 139
Institution A 0.226 297.63 0.822
Grade-based 26.2 3.9 161
Knowledge-based 26.1 4.2 99
Institution B 1.818 200.39 0.071
Grade-based 25.1 4.0 157
Performance on Exam Std.
School Section n t df p
1 MTF Questions Dev
Knowledge-based 78.9 12.5 141
Institution A 0.526 299.45 0.599
Grade-based 78.2 12.8 167
Knowledge-based 74.7 10.9 106
Institution B 1.949 231.12 0.053
Grade-based 72.0 11.5 172


Table S2. Comparison of demographics between sections within schools. We assessed whether
gender, ethnicity, or class level were associated with section at each school separately using
chi-square tests (df = 1 for all tests).
School Section Female Male 2 p
Knowledge-based 101 43
Institution A 0.315 0.574
Grade-based 127 47
Knowledge-based 66 40
Institution B 0.146 0.702
Grade-based 111 61
School Section Non-URM URM 2 p
Knowledge-based 110 32
Institution A 3.301 0.069
Grade-based 116 54
Knowledge-based 87 8
Institution B 0.0001 0.992
Grade-based 153 14
School Section First-time Freshman Other 2 p
Knowledge-based 96 48
Institution A 0.191 0.662
Grade-based 120 54
Knowledge-based 51 55
Institution B 2.978 0.084
Grade-based 101 71



Table S3. CFA factor loadings and alphas for MTF perceptions survey. CFA was done on the
first survey prior to the experimental treatment. Here we show the results of the final CFA after
two items were removed due to loadings <0.3 in the initial analysis.
Factor
Item
loading
Comfort, alpha = 0.89 (survey 2 alpha = 0.91)
Q1 I often feel like the instructor is trying to trick me with MTF exam questions. 0.497
Q2 I enjoy answering MTF exam questions. 0.802
Q3 I feel confident when answering MTF exam questions. 0.762
Q4 I feel like I understand how to go about answering MTF exam questions. 0.695
Q5 I am often confused about how to go about answering MTF exam questions. 0.674
I would prefer that MTF questions be removed from exams because I dislike
Q6 0.835
answering them.
Q7 MTF exam questions cause me to be anxious. 0.701
Q8 I tend to overthink my answers to MTF exam questions. 0.638
Grade, alpha = 0.76 (survey 2 alpha = 0.83)
I would prefer that MTF questions be removed from exams because they hurt my
Q9 0.836
grade.
I like MTF exam questions because they allow me to earn points, even if I don't
Q10 0.479
completely understand a concept.
My overall exam score would be better if the entire exam consisted of MTF
Q11 0.587
questions.
Q12 MTF exam questions take me too much time to answer. 0.566
Q13 MTF exam questions are graded fairly. 0.556
Q14 I have a good chance of guessing the right answer to an MTF exam question. 0.419
Knowledge, alpha = 0.77 (survey 2 alpha = 0.78)
I often guess when answering MTF exam questions because I don't know all the
Q15 0.493
answers.
Q16 MTF exam questions allow the instructor to test a broad range of ideas. 0.628
MTF exam questions allow me to demonstrate my understandings, even if I don't
Q17 0.650
completely understand a concept.
Answering an entire MTF exam question correctly requires that I have a complete
Q18 0.325
understanding of a concept.
Q19 MTF exam questions help identify when I have incorrect ideas about a concept. 0.610
Q20 MTF exam questions accurately represent my level of knowledge. 0.739
Knowing the exam would include MTF questions encouraged me to study main
Q21 0.525
ideas and broad concepts.
Knowing the exam would include MTF questions encouraged me to study
Q22 0.364
definitions and specific terms.



Table S4. Results from MANCOVA examining the effect of the MTF justification treatments
at institution A while controlling for demographics, motivation, and performance on exam 1.
We also report the results of univariate ANCOVAs to show which dimensions were
influenced by the treatment.
Multivariate Test Univariate Test
Pillais
F3, 292 p Dimension F1, 294 p
Trace
comfort 0.106 0.745
Treatment 0.049 4.981 0.002 grade 4.036 0.045
knowledge 2.097 0.149
comfort 0.007 0.933
Gender 0.006 0.619 0.603 grade 0.019 0.889
knowledge 1.400 0.238
comfort 0.284 0.594
Ethnicity 0.003 0.255 0.858 grade 0.040 0.841
knowledge 0.606 0.437
comfort 0.262 0.609
Class level 0.006 0.624 0.600 grade 0.168 0.682
knowledge 1.780 0.183
comfort 0.689 0.407
Extrinsic
0.003 0.264 0.851 grade 0.169 0.682
motivation
knowledge 0.045 0.832
comfort 9.280 0.003
Intrinsic
0.033 3.3 0.021 grade 2.982 0.085
motivation
knowledge 1.071 0.301
Performance on comfort 18.239 < 0.001
exam 1 MTF 0.061 6.329 < 0.001 grade 11.851 0.001
questions knowledge 8.903 0.003



Table S5. Results from MANCOVA examining the effect of the MTF justification treatments
at institution B while controlling for demographics, motivation, and performance on exam 1.
We also report the results of univariate ANCOVAs to show which dimensions were
influenced by the treatment.
Multivariate Test Univariate Test
Pillais
F3, 252 p Dimension F1, 254 p
Trace
comfort 23.246 < 0.001
Treatment 0.164 16.503 < 0.001 grade 48.653 < 0.001
knowledge 2.877 0.091
comfort 1.555 0.214
Gender 0.006 0.549 0.649 grade 0.883 0.348
knowledge 0.576 0.449
comfort 0.033 0.857
Ethnicity 0.024 2.082 0.103 grade 4.036 0.046
knowledge 0.025 0.876
comfort 0.033 0.857
Class level 0.011 0.941 0.421 grade 0.383 0.537
knowledge 2.416 0.121
comfort 0.000 0.988
Extrinsic
0.015 1.282 0.281 grade 2.319 0.129
motivation
knowledge 0.002 0.965
comfort 2.015 0.157
Intrinsic
0.023 2.011 0.113 grade 2.031 0.155
motivation
knowledge 5.646 0.018
Performance on comfort 14.224 < 0.001
exam 1 MTF 0.057 5.067 0.002 grade 3.040 0.082
questions knowledge 0.660 0.417



Figure S2. Summary of qualitative results from open-ended questions about changes in study
habits. A) In what ways did you study differently for this exam? B) Why did you choose to
change your study practices?