TipsforWritingTestItems

Spring2017
Objectives
Discussthereasonswhyeffectivetestitemwritingisimportantand
ingeneralwhatshouldbetested.
ConstructeffectiveSelectAllThatApply(SATA)andOneBest
AnswerMultipleChoiceQuestions(MCQ).
Identifytechnicalflawsthatcandecreasetheeffectivenessoftest
questions.
Whatisthepurposeoftesting?
Communicatestostudentswhatmaterialisimportant
Motivates studentstostudyandlearn
Identifyareasofdeficiencyinneedofremediationorfurtherlearning
Assesses studentlearning
Validatesthelearningprocess(formative)
Measuresstudentaccomplishment(summative)
Evaluatescurriculumandinstruction
Familiarizeslearnerswithtesting
Whatshouldbetested.
Examcontentshouldmatchcourse/clerkshipobjectives
Importanttopicsshouldbeweightedmoreheavily thanlessimportant
topics
Thetestingtimedevotedtoeachtopicshouldreflecttherelative
importanceofthetopic
Thesampleofitemsshouldberepresentativeoftheinstructionalgoals

Evidenceoflearningwhen.
Newinformationislinkedtowhatthelearneralreadyknows
Thelearnercanusenewinformation,notjustrememberfacts
Effectivetestquestionsshould.
Addressimportantcontent
Donttestontrivia

Bewellconstructed
Itemsthatattempttoassesscriticallyimportanttopicscannotdosounless
theyarewellconstructed

Avoidingflawsthatbenefittestwisestudentsandavoidingirrelevant
difficultyareprerequisitesthatmustbemet
Constructingtestquestions
TECHNIQUESFORDEVELOPINGSATAANDOBAMCQs.
SelectAllThatApply(SATA)
Askthestudenttoselectallappropriateanswersfromalistofpossible
options.

Fortheseitems,theexamineemustdecidewheretomakethecutoff to
whatextentmustaresponsebetrueinordertobekeyedastrue.

Thistaskrequiresadditionaljudgement(beyondwhatisrequiredin
selectingtheonebestanswer)

CHALLENGE:Toooften,examineeshavetoguesswhattheitemwriterhad
inmindbecausetheoptionsarenoteithercompletelytrueorcompletely
false.
SATAitemsshouldsatisfythefollowingrules:
Stemsmustbeclearandunambiguous. Forexample,shouldnotinclude
whichofthefollowingisMOSTappropriate:SelectAllThatApply

Imprecisephrasessuchasisassociatedwith;isusefulfor;isimportantand
wordsthatprovidecueingsuchasmayorcouldbe;andvaguetermssuch
asusuallyorfrequentlyshouldbeavoided.

Optionsmustbeabsolutelytrueorfalse;noshadesofgrayare
permissible;avoidphrasesandwordsnotedinthefirstitemabove.

Shouldhaveatleasttwocorrectitemsandoneincorrectitem(i.e.avoid
onecorrectorallcorrect)
Exampleofastructurallyacceptable(SATA)
Whichofthefollowingis/areX Theoptionscanbediagramedas
linkedrecessiveconditions? follows.
SelectAllThatApply 2and4 TotallyWrongOptions
1.HemophiliaA(classic 1and3 TotallyCorrectOptions
hemophilia)
2.Cysticfibrosis
3.Duchennesmuscular
dystrophy
4.TaySachsdisease
 ExampleofanunacceptableSATAitem
Truestatementsaboutcystic Options1,2,and3cannotbe
fibrosis(CF)include:(SelectAll judgedasabsolutelytrueorfalse;
ThatApply) agroupofexpertswouldnot
1.TheincidenceofCFis1:2000. agreeontheanswers.
2.ChildrenwithCFusuallydiein InthinkingaboutOption1,note
theirteens. thattheincidenceisnotexactly
1:2000;expertswouldwantmore
3.MaleswithCFaresterile. information:IsthisintheUSA?Is
4.CFisanautosomalrecessive thisamongallethnicgroups?
disease. SimilarissuesarisewithOptions2
and3,whileOption4isclear.
UnclearStem(actualquestion)
Whilewritteninjest,this
Thewaytoamansheartis true/falseitemillustratesa
throughhis(SelectAllThat commonproblem.
Apply)
Dependingonyourperspective,
1.aorta Options1,2,and3mightbe
2.pulmonaryarteries true;alternatively,1,2,and3
3.pulmonaryveins mightbefalsewhile4istrue.
4.stomach
UnclearStem
Inchildren,ventricularseptal
defectsareassociatedwith:
(SelectAllThatApply)
1.systolicmurmur
2.pulmonaryhypertension
3.tetralogyofFallot
4.cyanosis
Theflawsinthisitemaremore
subtle.Thedifficultyisthatthe
examineehastomakeassumptions
abouttheseverityofthedisease,
theageofthepatient,andwhether
ornotthediseasehasbeen
treated.
Differentassumptionsleadto
differentanswers,evenamong
experts.
 TheOneBestAnswerFamily
Theyconsistofastem (eg,aclinicalcasepresentation)andaleadin
question,followedbyaseriesofoptions,typicallyonecorrectanswer
andfourdistractors.
Stem: Leadin:
A32yearoldmanhasa4dayhistoryof Whichofthefollowingisthemost
progressiveweaknessinhisextremities. likelydiagnosis?
Hehasbeenhealthyexceptforanupper
respiratorytractinfection10daysago.
Histemperatureis37.8C(100F),blood Options:
pressureis130/80mmHg,pulseis A.Acutedisseminated
94/min,andrespirationsare42/minand encephalomyelitis
shallow.Hehassymmetricweaknessof
bothsidesofthefaceandtheproximal B.GuillainBarr syndrome
anddistalmusclesoftheextremities. C.Myastheniagravis
Sensationisintact.Nodeeptendon
reflexescanbeelicited;theplantar D.Poliomyelitis
responsesareflexor. E.Polymyositis
TheOneBestAnswerFamily
Even though the incorrect answers above are not completely wrong, they
are less correct than the keyed answer.

The examinee is instructed to select the most likely diagnosis; experts

would all agree that the most likely diagnosis is B; they would also agree
that the other diagnoses are somewhat likely, but less likely than B.

As long as the options can be laid out on a single continuum, in this case
from Most Likely Diagnosis to Least Likely Diagnosis, options in one-
best-answer questions do not have to be totally wrong.
 BasicRulesforOneBestAnswerItems
Eachitemshouldfocus onanimportantconcept,typicallyacommon orpotentially
catastrophicclinicalproblem.

Focusonproblemsthatwouldbeencounteredinreallifeandbestpractices.

Avoidtrivial,tricky,oroverlycomplexquestions.

Theitemstems mayberelativelylong butshouldposeaclearquestion;theoptions

shouldbeshort.

Whenwriting,covertheoptionstodetermineifthequestionisfocused andclear
 Distractors
Theyshouldfallintothesamecategoryasthecorrectanswer(eg,alldiagnoses,tests,treatments,prognoses,
dispositionalternatives).

Avoidusingdoubleoptionsunlessthecorrectanswerandalldistractorsaredoubleoptions.Rewritedouble
optionstofocusonasinglepoint.

Alldistractorsshouldbeplausible,grammaticallyconsistent,logicallycompatible,andofthesame(relative)length
asthecorrectanswer.

DistractorsshouldrepresentunsafepracticesorcommonlyheldmisconceptionsANDshouldbeplausible.

Whentestingcanthestudentdefineanitem,becertaintheyarefamiliarwithalldistracters

Order theoptionsinlogicalorder(eg,numeric),orinalphabeticalorder.

Avoid usingoppositestothecorrectanswerinthedistractors(i.e tachycardiaandbradycardia)

Ifdistractorsarenumerical then,placeinascendingordescendingorder,keeprangesequalanddontoverlap
values.
Technicalflaws
Theprobabilityofanswering aquestioncorrectly
shouldrelatetotheexamineesamountofexpertiseon
thetopicbeingassessedandshouldnot relatetotheir
expertiseontesttakingstrategies.
PrimaryTechnicalItemFlaws
Testwiseness
Makingiteasierforsomestudentstoanswerthequestion
correctly,basedontheirtesttakingskillsalone.
Commonlyoccurinitemsthatareunfocusedanddonotsatisfy
thecovertheoptionsrule.

IrrelevantDifficulty
Flawsrelatedtoirrelevantdifficultymakethequestiondifficultfor
reasonsunrelatedtothetraitthatisthefocusofassessment
Testwiseness flaws
 Grammaticalcues
oneormoredistractorsdontfollowgrammaticallyfromthestem
A60yearoldmanisbroughttotheemergencydepartmentby
thepolice,whofoundhimlyingunconsciousonthesidewalk.
Afterascertainingthattheairwayisopen,thefirststepin
managementshouldbeintravenousadministrationof
A.examinationofcerebrospinalfluid
B.glucosewithvitaminB1(thiamine)
C.CTscanofthehead
D.phenytoin
E.diazepam
 Logicalcues
asubsetoftheoptionsarecollectivelyexhaustive
Crimeis
A.equallydistributedamongthe
socialclasses
B.overrepresentedamongthe
poor
C.overrepresentedamongthe
middleclassandrich
D.primarilyanindicationof
psychosexualmaladjustment
E.reachingaplateauof
tolerabilityforthenation
Inthisitem,OptionsA,B,and
Cincludeallpossibilities.The
testwise studentknowsthatA,
B,orCmustbecorrect
Often,theitemwritersaddD
andEonlybecausetheywant
tolistfiveoptions.
Thisflawiscommonlyseenin
itemswithoptionssuchas
Increases,Decreases,and
Remainsthesame.
 Absoluteterms
alwaysorneverareusedinoptions
Inpatientswithadvanceddementia, Thetestwise studentwill
Alzheimerstype,thememorydefect eliminateOptionsCandD
A.canbetreatedadequatelywith aspossibilitiesbecausethey
donepezil arelesslikelytobetrue
thansomethingstatedless
B.couldbeasequelaofearly absolutely
parkinsonism
C.isneverseeninpatientswith
neurofibrillarytanglesatautopsy
D.isneversevere
E.possiblyinvolvesthecholinergic
system
Longcorrectanswer.

Secondarygainis Inthisitem,OptionCislonger
A.synonymouswithmalingering thantheotheroptions;itisalso
theonlydoubleoption.
B.afrequentproblemin
obsessivecompulsivedisorder Becauseweareteachers,wewrite
longcorrectanswersthatinclude
C.acomplicationofavarietyof additionalinstructionalmaterial,
illnessesandtendstoprolong parentheticalinformation,
manyofthem caveats,etc.
D.neverseeninorganicbrain
damage
Wordrepeats
awordorphraseisincludedinthestemandinthecorrectanswer
Theprocessbywhichoral Sometimes,awordisrepeated
medicationsaregreatlymetabolized onlyinametaphoricalsense,
bytheliverbeforeitreaches eg,astemmentioningbone
systemiccirculationis: pain,withthecorrectanswer
A.Metabolism beginningwiththeprefix
osteo.
B.Absorption
C.Distribution
D.Elimination
Convergencestrategy
correctanswerincludesthemostelementsincommonwiththeother
options
Localanestheticsaremosteffectiveinthe
A.anionicform,actingfrominsidethe
nervemembrane
B.cationicform,actingfrominsidethe Thisitemflawisless
nervemembrane obviousthantheothers,
butitoccursfrequently
C.cationicform,actingfromoutsidethe andisworthnoting.
nervemembrane
D.unchargedform,actingfrominsidethe
nervemembrane
E.unchargedform,actingfromoutsidethe
nervemembrane
ConvergenceStrategy:Discussion
Theunderlyingpremiseisthatthecorrectansweristheoptionthathasthemostin
commonwiththeotheroptions;itisnotlikelytobeanoutlier.

Forexample,innumericoptions,thecorrectanswerismoreoftenthemiddlenumber
thananextremevalue.

Indoubleoptions,thecorrectanswerismorelikelytobetheoptionthathasthemost
elementsincommonwiththeotherdistractors.

Thisflawoccursbecauseitemwritersstartwiththecorrectanswerandwrite
permutationsofthecorrectanswerasthedistractors.Thecorrectansweris,therefore,
morelikelytohaveelementsincommonwiththerestoftheoptions;theincorrect
answersaremorelikelytobeoutliers astheitemwriterhasdifficultygeneratingviable
distractors.
IRRELEVANTDIFFICULTYflaws
 Optionsarelong,complicated,ordouble
Thestemcontainsextraneousreading,but,moreimportantly,theoptionsareverylongand
complicated.

Requiresasignificantamountofreadingbecauseofthenumberofelementsineachoption.This
can shiftwhatismeasured byanitemfromcontentknowledge/comprehensiontoreading
speed.

Pleasenotethatthisflawrelatesonlytooptionsnotstems.
EXAMPLE
PeerreviewcommitteesinHMOsmaymovetotakeactionagainstaphysicianscredentialstocarefor
participantsoftheHMO.Thereisanassociatedrequirementtoassurethatthephysicianreceivesdueprocess
inthecourseoftheseactivities.Dueprocessmustincludewhichofthefollowing?
A.Notice,animpartialforum,council,achancetohearandconfrontevidenceagainsthim/her.
B.Propernotice,atribunalempoweredtomakethedecision,achancetoconfrontwitnessesagainsthim/her,
andachancetopresentevidenceindefense.
C.Reasonableandtimelynotice,impartialpanelempoweredtomakeadecision,achancetohearevidence
againsthimself/herselfandtoconfrontwitnesses,andtheabilitytopresentevidenceindefense.
 Numericdataarenotstatedconsistently
Followingasecondepisodeof Shouldbelistedinnumeric
infection,whatisthelikelihoodthat orderandtheoptionsshouldbe
awomanisinfertile? listedinasingleformat.
A.Lessthan20%
B.20to30% Confusionoccurswhenformats
C.Greaterthan50% areinconsistentandwhenthe
optionsarelistedinanillogical
D.90% order.
E.75%
 Frequencytermsintheoptionsarevague
Severeobesityinearlyadolescence
A.usuallyrespondsdramaticallytodietary
regimens
B.oftenisrelatedtoendocrinedisorders
C.hasa75%chanceofclearing
spontaneously
D.showsapoorprognosis
E.usuallyrespondstopharmacotherapy
andintensivepsychotherapy
Researchhasshownthatvague
frequencytermsarenotconsistently
definedorinterpreted,evenby
experts.
Thesetermscauseconfusionwhen
theyareusedinthetextof
examinationitems.
Whenoptionsarenotabsolutelytrue
orfalse,examineesrelyontheir
personaldefinitionoftheambiguous
termsortheirguessesaboutwhat
thesetermsmeanttotheitemwriter.
 Languageintheoptionsisnotparallel;optionsare
inanillogicalorder
Acommonflawinwhichtheoptionsarelongandthelanguagemakesitdifficultandtime
consumingtodeterminewhichisthemostcorrect.
Carefuleditingcancorrect.
Inthisparticularitem,theleadincanbechangedtoForwhichofthefollowingreasons
cannoconclusionbedrawnfromtheseresults?Theoptionscanthenbeedited

Inavaccinetrial,2002yearoldboysweregivenavaccineagainstacertaindiseaseandthen
monitoredforfiveyearsforoccurrenceofthedisease.Ofthisgroup,85%nevercontractedthe
disease.Whichofthefollowingstatementsconcerningtheseresultsiscorrect?
A.Noconclusioncanbedrawn,sincenofollowupwasmadeofnonvaccinated children
B.Thenumberofcases(ie,30casesoverfiveyears)istoosmallforstatisticallymeaningful
conclusions
C.Noconclusionscanbedrawnbecausethetrialinvolvedonlyboys
D.Vaccineefficacy(%)iscalculatedas8515/100
Noneoftheabove
ThephraseNoneoftheaboveisproblematic initemswherejudgementisinvolvedand
wheretheoptionsarenotabsolutelytrueorfalse.

Ifthecorrectresponseisintendedtobeoneoftheotherlistedoptions,knowledgeable
studentscanbefacedwithadilemmabecausetheyhavetodecidebetweenaverydetailed
perfectoptionandtheonethatyouhaveintendedascorrect.Theycanoftenconstructan
optionthatismorecorrectthantheoneyouintendedtobecorrect.

Useofnoneoftheaboveessentiallyturnstheitemintoatrue/falseitem;eachoptionhasto
beevaluatedasmoreorlesstruethantheuniverseofunlistedoptions.

Itwilloftenbepossibletofixsuchitemsbyreplacingnoneoftheabovebyanoptionthat
meansroughlythesamethingbutismorespecific.Forexample,inanitemaskinganexaminee
tospecifythemostappropriatepharmacotherapy,replacingnoneoftheabovebynodrug
shouldbegivenatthistimewilleliminatetheambiguityofnoneoftheabove.
 GeneralChecklistforTestItems
Writingtheanswers/distractors:
Overall:
1. ForOBAMCQsisthereonlyonecorrectanswer?
1. Isthetopicrelevant?
2. ForSATAisthereatleasttwocorrectanswersandone
2. Isthewordingclear? incorrect?
3. Arethestemandanswerswrittento 3. Aretheanswerchoicesunambiguous?
reducerequiredreadingtime?
4. Areansweroptionsofsimilarlengthandparallelin
4. Aretheoptionsshortandsuccinct? grammaticalstructure?
5. Doestheitemfollowthecoverthe 5. Arenumericaloptionsplacedplaceinascendingor
optionsrule? descendingorder,withequalrangesandnooverlapping
values?
Writingthestem: 6. Arealldistractoritemsincorrectbutplausible?
1. Isthestemwrittenclearlyandfree 7. Arethedistractorsfreeofoppositeanswers?
ofambiguity? 8. Dorecognizablekeywordsthatappearinthecorrect
2. DoesitavoidImprecisephrasessuch answeroptionalsoappearinsomeorallofthe
asisassociatedwith;isusefulfor;is distractors?
importantandwordsthatprovide 9. Areextremewordsthatoftenindicateincorrectanswer
cueingsuchasmayorcouldbe;or optionavoided(e.g.,all,always,never)?
vaguetermssuchasusuallyor
frequently? 10. Arevaguephrasesavoided(usually,typically,maybe,)?
11. IsNoneoftheaboveusedifanadditionaldistractoris
needed?
References.1.CaseSM,SwansonDB. ConstructingWrittenTest
Questions for BasicandClinicalSciences, ThirdEdition.Philadelphia,
PA: