Documentos de Académico
Documentos de Profesional
Documentos de Cultura
v;,::
J. -.. '
DOCUMENT RESUME
- ~'. . .,
'
TM 014 423 Facione, ~eter A. Critical Thinking: A Statement of Expert Consensus for Purposes of Educational Assessment and Instru~tion. Research Findings and Recommendations. bmtlr1can Philosophical Association, Newark, Del. l12p., Some pages have broken type. Reports - Research/Technical (143) MF01/PC05 Plus Postage. Cognitive Ability, *Critical Thinking, ~Curriculum Development; Cu. r rcurun Evall1ation~ ~Delphi Technique; ~Educational Assessment; Qualitative Research ~Experts; *Panel Consensus Technique
90
SPONS AGENCY PUB DATE NOTE PUB TYPE EDRS PRICE DESCRIP'l'ORS
IDENTIFIERS ABSTRACT
Using a qualitative research methodology, kn~wn as the Delphi MeUlod, an interactj,ve panel of experts was convened to work toward a consensus on the role of critical thinking (CT) in educational asse~sment and instrUction. In Delphi research, experts participate in several rounds of ~uestions that require thoughtful and detailed responses. Panelists work toward consensus by sharing reasoned opinions and reconsidering the opinion~ with regard to comments, objections, and arguments offered by other experts. A total of 46 scholare, educators, and leading figures in CT theory and CT assessment res~arch were gathered for the panel meetings. About half of the ranelists were primarily affiliated with philosophy departments: the others were affiliated with education, social sciences, or physical sciences. Recommendations resulting from the discussion rounds address the cognitivp skill dimension of CT, the diSpositional dimension of CT, and specific recommendations on CT instruction and assessment, including development of a CT curriculum. A discussion of commercially available CT assessment tools, a bibliography With an emphasis on assessment, and a set of letters which chronicle the progcess of the Delphi research group are appended. (TJH)
*~~~~~~*~~*~~~~~~~~~~*~~~~~~~~~*~***~*~*~**~~~*~~~*~~~~**~~*~*~~~~~**** Reproductions supplied by EDRS are the best that can be made *
:):;;i:~;;~·.,.-,
-
·,,'
•• ,,,-,
•••
'"'1
\.
:f'~'i~
.. ,:
Office of Educa',onal Aaseareh and Improvemen, eDUCATIONAL RESOURCES INFORMATION CENTER (ERIC)
-;.•'o "
._!
u.s.
DEPARTrtIIHT
OF EDUCATION
r Tills
r Monor
•
0"11,"8,,"g "
~1(:.~
8 . ;'/±CIOtv[
FOlnts of vIew or oponlonsstated In tlll$ cocument 110 nO' nflceSSIl"ly represent ofloclal OERI pOsItion or poliCy
CRITICAL TIIINKING:
A STATEMENT
OF EXPERT CONSENSUS
Research Findings and Recommendations Prepared for the Committee on Pre-College Philosophy
of the
by
(c)
1990
P.
A.
Pac-ione
~,
:.~~ .
,
;"~.t:
t-
' .••
y1> .•..
....
',
.::~~~h~:= .~.,
,
-'.:.:./'~
.Q_ONTENTS
I II III IV
The Critical Thinkillg Movement and CT Assessmer",t Research Methodology and Purpose
1 4
8
The Dispositional Dimension of Critical Thinking 20 Procedural, Laudatory and Normative Uses the Term "CT" ..21 Dispositions of the Good Critical Thinker 27 Further Recommendations The CT Curriculum. The CT Assessment Tool
e
The CT Goal
•••••••••••••
e.
••••••••••
0 0 •••••••••••••••
00
••
eo
••••
'fhe CT Instructor
30 30
32
28
II
•••••
VI
•••••••••••••••
••••••••••••••
33
TABLES
TABLE 1 TABLE 2 TABLE 3 1ABLE 4 TABLE 5 TABLE 6 TABLE 7 Consensus Statement Regarding Critical Thinking and the Ideal Critical Thinker Project History Consensus List of CT Cognitive Skills and Sub-Skills Consensus Descriptions Affective Dispositions
0 ••••• 0
•••
3 7
12
Consensus Statement on Teaching and Assessing CT Skills.29 Participating Critical Thinking Experts 35
APPENDICES Appendix A: Commercial CT Assessment Tools Appendix B: CT Bibliography with Emphasis on Assessment Appendix C: The Delphi Research Letters
00 ••
36 40 51
),.',', ~.~...;.
~
:
V·t·_--'··
"
_': .•
CRITICAL
THINKING:
As~essment
educational education at
pedagogical
50
and asses$ment
all levels
as to foster associated
in students
those
cognitive
of inquiry that
thinking. thinkers
case
educatinQ students
the student~
themselves
(Ennis, 1962,
19~B; Shef'fler,
StE:!rllberg, 1985; Ruggiero, 19a5; Paul, 1988 (a) and (b); etc.). The arguments After accor-d that advocates of inquiry, disjointed decades for critical thinking have been successful. the eighties exactly witnessed a growing
the heart
always said it
t ,
and senescent
information.
~ r
movement gained momentum throughout papers courses. lesson led to the development In elementary
Conferences
and position
of college
critical
and secondary
schools
(K-12) teachers
plans to incorporate
CT objectives. CT staff
In the span of a few years development programs major success when became
and offering
4
"':..
'_I'_~;:-~~::JZ~~:::';:-'I':"._.,;-~,,::::._r. __ -:~;
~f;;:···
,
~, '
."_. .
universities
general education
decade's end CT could no longer be cha£loacterized as a cottage industry. With success come questions: Not: new ones necessarily, but, because
of the e.:pectations which have been ri!lised and the investments being proposed, vel(ing ones. IntUitively, CT instruction should focus on how CT pedagogyshould dispositions
students approach a. qLlestion and reason about it. develop in sbJdents those cognitive skills which characterize the good critical
and affective
thinker.
to targeting whether a given answer is correct, target arriving the qLlali t y of the critical at that answer. Thuf:;,fer
find they mL(stccmtinue to address some fundamental academ:lc concen,s. Whnt exactly are those skills What are someeffective
if and
dispositions whic:hcharacterize
CT'?
ways to teach CT? And how can CT, particularly or statewide requirement, Ue
it becomesa campus-wide,district-wide
assessed?
When these academicquestions are asked by the individual professor ar teacher ~eeking to introduce CT into her own classroom, they are difficult political district ':-f"lough.But t:he questions take on social, fiscal, and
dimensions when asked by cempuacurriculum committees, sc:hool offices, boards of education, and the educational testing This is not to say that and
publishiwJ
indLl~tr'es. i
a.ndNewJersey have established .... ays of curr-icular frameworks and statewide testing
"
- '->.
. :;'::;:.\
.....~..
-::--::'< - ,
'"'\'>:'~
:·t
programs. enrolls
State
University
system,
which fer
hundreds of thousands
of students, general
a process
the approval
played by philosophers
in art.ic:ulating
value, both indi vidual clnd social, in designing college efforts that to introduce
lev6l1 academic programs in CT, and in assisting CT into the K-12 curriculum, Association, it is little its
wonder
01'
through
Committee
Pre-College
to make a systematic
and CT assessment.
TABLEt
CONSBNSUS STATBMBNT RBGARDIN"G CRITICAL THINKING· AND THB IDEAL CRITICAL THINKER We undentand critic:al thinking to be purposeful, self-regulatory judgment which results in interpretation, analysil, evaluation, aad inference, as weU as explanation of the evidential, conceptual, methociologic:ai, criteriological, or conteztuai considerations UPOD which that judgmeDt is based. CT is essential as a tool of inquiry. As such, CT is a liberating force in education and a powerful resource hi ODe'. personal and civic: life. While not synonymous witb good thinking, CT is a pervasive and self-rectifying human phenomenon. The ideal critical thinker is habituaDy inquisitive, weU-informed,lrusUul of reason, open-minded, flexible, fair-minded in evaluatioD, honest in facing persoD8l biases, prudent i.a making judgments, wiI1iDgto reconsider, cleu about issues, orderly in complex malten, diligent ill seeking relevant informatioa, reasonable in the selection of criteria, focused in inquiry, and penistent in seeking resu'ta which are 81 precise u the subject and the circumstances of inquiry permit. Thus, educating good critical thinken means working toward this ideal. It combines developing CT skills with nurturing those dispositions which consistently yield useful irudghts and which are the basis of a rational and democratic society.
As Table 1 st.lggests, a panel of CT experts dimensions: cognitive a key result of inquiry is the artiCUlation terms of two Section II of by
of CT it
dispositions.
. :.. - ..
';'-,"
this
the
Delphi research
methodology_
III
skill
dimension Research
IV the
of CT. text
presented
throughout
report,
in the
and in tab'.llar
form.
presented
in Sections
rationale.
-. ..
can be reloted
with their to
specifically
CT instruction
V.
II --
Researr:JJ_ Methodology
and PUr:I3ose
The Committee
on Pre-College
several
persons
with
as
special plJrt
might contact
of the
at the
employ
heart the
of the
professl.on's
power"ful q~lalitative
known as
formaticn
be willing
must
r~solution
of
~~perts
to nominate
persons, special
shapp.
In all forty-six
coUeagLles ,l\ssessment project.
widely recognized
to have or
theory, were
made tt.e
not
partj~lpate effort,
If it
is
f(.)r their
il1ve'5.tiga.tor
" repor"t
e:.:tremeiy
dPprec:ia.tivp~, the
consensus
exprp.ssed
in thi.s
cou ld :lot
h~VE~been
eHperts
and detailed
responses.
the
Delphi Method is
not a matter
"...
quantitative their
data.
panelists
work
by sharing
reasoned
by other
once an expert
....
carising from any given expert·s p.-cfessional is initiated through that by the p.r:.;ject director
...~.
Q·f questions
coordinated entire
director
responses,
of their
and
shape the line of inquiry. which respond· to the toward director useful
director
endeavors
to frame questions
and alerts
them to other
As areas
cf accord or disagreement of
are presented
findings or crucial
director
Delphi
findings als~ include descriptions statemp.nts A clear of minority opinion. and ciccurate
o.f residual
and
conceptualization
essential CT
! ; ~.
..
:':'.\~'
.......
--
..
,' ."
...........
assessmf~nt efforts,
early
most worthwhile ~Qntribution could be the articulation correct conceptualization toward that of CT. end. The expert panelists
....
educators
in CT assessment clarity,
or
accuracy and
serious
the experts
asked
educatad
level to
thinker
the typical
the experts
articulating
It may be the
and sub-skills
experts
cultivated critical
thinker.
than in others.
than that
community is that
it should serve
as a rich and
_",~::~\~>:·:~·.t . '..
.
.,';., .
.i .••••
"
~:~.~
_,_,
_.
__
..;.....o..___'__""""""".
"T""~~
""___
_.~
••
TABLB2 PROJECT HISTORY Round 1 (P"b. 11, 1988) and Round 2 (Mar. 14, 1988) initiated the Delpbi process. In both roundB panelists were invited to nomb18t~ other experts to join in this research project. The experts reached consensus on tbe working assumption that "the CODcept f cr could be made operational to the e~eDt that important parts of CT o could be auessed ¥&lidly and reliably." The experts agreed to be!iD their analysis 01 CT by "identifying the core elements of cr which might reuonabl, be expected
cr
at tbe freshman and sophomore general education college level." The rationale for this decision was tbet the college level theoretical c:orustmctof CT could reasonably be used to guide wbat might be said about cr at the K-IZ level. Also the panelists Doted that most of the participatibl ezperts had greater ezpedence at the college level than in Ie12 education. Round 3 (MI.'Y 1988) wu an opea·-ended invitation for experts to write their own 4, list of the operations which they conceived ~f,Jf central to CT. The first syntbesis of this as input was presented for expert review in Round 4 (Sept. 23, 1988). This synthesis focused on the skill dimension of CT. Round 4 invited responses regarding each skUI and sub-skill identified, a proposed [and ultimately rejected] input/output model of CT operations, a list of closely related cognitive operations which might or might not be
distinguished from cr, a general statement regarding what a skiU is and how one is taught, and ill list of caveats and cautions regarding CT instruction aud assessment.
Rountl SA (Feb. 28, 1989) reviewed the definitions and classification of CT cognitive skiDsin the light of expert responses to Rouad4. RDuud S8 (also Peb.28, 1989) proposed sblltements regarding the dispositional dimension of CT and about its possible normative Cf)nnotations. RoundSC (Mar. 10, 1989) aslced forspeclfic recommendations regarding (:f instruction ana assessment, and offered a revision of the general statement (tn teaching and assessing 8 cogYlitive skill. RdUDd S included several quotatioDScuiled from the panelists' earlier responses and invited comments and reactions. The f~Xperts' comments regarding the "arious quotations included in each round added greatly to the project directors understanding of the experts' overaU views. From these and the responses to specific Round SA, SB and SC questions, the project director assembled u draft report of all Delphi findinl~, including recommendations. Round 6, (Sept. 25, 1!'89) circulated that draft and gave the CT experts the opportunity to express their views or make comments for inclusion in the final report, which went through its last revisions in Nov. 1989.
,
. ··f
III
--
Dimension of Cnticsl
ThiokiDR
FINDING: As indicated in Table 1, the expert~ find good critical thinking to include both a skill dimension and a dispositional dimensinn. The experts find CT to include cognitive skills in (1) interpretation, (2) analysis, (3) evaluation, (4) inference, (5) explanation and (6) self-regulation. Each of theso six is at the core of CT. Associated Nith each are criteria by which its execution can be meaningfully evaluated. However, no attempt is made here to specify those criterii since ample criteriological discussions exist in the literature.
Concernednot to generate misunderstandings, the experts offer cautions about the analysis of CT in terms of skills experts warn that and sub-skills. many The
to lose sight of the whole while attempting to attend well to its parts.
RECOMMENDATION 1: All CT instruction should aim at developing good critical thinkers -- persons who can integrate suc~essful execution of various skills in the CT enhanced classroom Nith the confidence, inclination ~nd good judgment to use these pawerful tools in their other studies and In their ev@ryday lives. Persons who have proficiency in CT skills but fail to use them appropriately are most unlikely to be regarded as good critical thinkers. RECOMMENDATION 2: Those Nno seek to infuse CT into the educational system to be guided by a holistic conceptualization of what it means to be a 900d critical thinker. That some aspects of CT, particularly features Hithin its skill dimension, are more readily targp.ted by existing educational assessment strategies should not distort the conceptualization of CT nor truncate full-blown CT instruction.
The exper·t.scharacterize CT s~dlls.
certain
cognitive
skills
as
central
or core
adept at CT.
The experts
at every skill
be
.,
Considering the panel's pl~"poses and of necessary a.ndsuf ficient Thus, in view of the
IIlC'thorJology,
to an~lyze CT in terms
11
~~:!~
·-t .
..
the
panel,
p,."oc.:ess,dec:ided to strive
to Rounds
4 and 5A reveal
the experts
to be virtually
as
evaluatic:m,
and inference
expert
it
str·or.I:Jly
was
dissented
prop~rly
regarding a pdrt of
interpretation,
c:ommunication,
~ot CT.
that
repor-t, overlaps
difficulties for
r'C:lise bvious o
as on~ alt~mpts
to make finer
differenliations
CT and
e~<pert said,
teaching.1I
In response
as compar"ed
that,
of
the
others,
In
self'-r-etJulc,tion self-r-eguldtion
dppear-s to be a skill
one c'pplies
or:~'s
a. differ"ant
the
other
CT skills This
to
'"i'xc:.'\mple, t?vci\lL\c:sUfll;J
o\~n inferences.
gives
r-,c;.!Ct.'t·',;i,,~Lhd.r"~I.:b:w.
e>:pert noted,
~Sf'l'?ct of self--rsfJ,-\la.tion
the
~t-.and<::\rd kinds
of
cOlls~nsu<;:)
retJ'-'le.~tion
t~bles
l o CT.
[For-
deta.iled results
for
response
9
: I
-.~ I, ~
.t,
FINDING:
:.
~.'
_.
... -,
There is consensus that one might improve ane's awn CT in several Nays. The experts agree that one could critically tUamine and evaluate one's awn reasoning processes. One cauld learn ho~ to think mare objectively and logically. One could expand one's repertoire of those mare specialized procedures and criteria used in different areas of human thought and inquiry. One could increase one's base of information and life experience.
It was readily apparent that the experts do not regard CT as a body along
of l::nowledgeto be delivered with others. applications The panel sees in all areas
to students
CT, like reading and writing, as having and learning. And, as several pointed
aT life
out, CT instruction,
like reading and writing, can occur in programs rich content or in programs ~Jhichrely on the events for developing one's CT.
with discipline-specific:
One implication the experts draw from their analysis of CT skills is this: "while CT skills themselves transcend specific subjects ar disciplines, exercising them successfully in cert~in contexts demands domain-specific knowledge, same of which may concern specific methods and techniques used to make re~sonable judgments in those specific contexts.
II
FINDING
transcend,
in
subjects
in many contexts
requires
domain-specific
demain-specific principles
specific-contexts.
mention of "evidential,
I.:r'j, t.er"iologiLcd,
or c:onte)ttuC\l"considerations
of logical operations
10
...... ...',
13
of the nec:essity
of robust
c:onc:epts of both CT
RECOMMENDATION 31 Since becoming adept at CT involves le~rning to use CT skills effectively in many different contexts, the experts insist that "one cannat overemphasize the value of a solid liberal education to supplement the honing of on~'s CT skills and the cultivating of one's CT dispositions."
The experts c:lassified c:aution that CT skills c:an usefully be grouped and sub-
ways.
whic:hresulted
Delphi research
sub-c:lassification,
own sub-classific:ations.
There are
in the c:lassifit:ation
system which emerged from the eac:h skill and sub-skUl simply to forc:e eac:h
Delphi r-esearch.
to bec:ome c:onceptually disc:retlrt from all the others nor useful. In prac:tic:al c:ontexts the exec:ution of others. Thus, order of the
skill~ or sub-skills
may presuppose
Delphi Hsting is nClt intended to imply the endorsement of any psychologic:al, logical or epistemological prescribing any educational the skills or-der or skill-sequenc:e, nor as
e~;hal.lsts the concept of CT in either inclu!iion in CT, ma.nyof the skills valuClble, if not vital, for other
a.nd sub-skills
identified
important
ac:tivities,
suc:h as
Also CT skills
or if Iter-personal skills
11
14
,r:O ..
>
••
. ~.
·:11
concerns
developing a figure
winninc;J sales
aut what might be wrong with his car. mean by characterizing is also fair particular success these CT skills
In part
as pervasive
to say that
a particular
sub-skill,
in a given endeavor,
expert.s are not concerned that used. other It is not a problem that endeavors.
the skills
might be essential
On the contrary,
they were not, since t.he case depends un eT's utility The experts across
system
of life
are clear
not every
TABLB3
CONSBNSUS LIST or CRITICAL THINKING COGNlTIVB SKILLS AND SUB-SKILLS
t. Interpretation
2. Analysis
• Bxamiaingldeu
3. Bvaluation 4. Inference
• Assessing Caims • AssessiDg Arguments • Queryinl Bvidence • Conjecturing Alternatives • Drawina Conclusions
S.Expianation
• Justifying Procedures
• Presentia& Arguments 6. Self- RegulatiOJl. • Self-examination
• Statial Results
• Seif-eorrec:tioD
-/)"':.-,'.0; .
.t5
should
be thought
of as
CT.
Not every
valuable
is
CT is one among a faPail y of clasel y rela ted along with, for e>:ample, problem~sol"ing, Unfortunately the the
forms
thinking,
thinking. ,ollsnongll a
conceptual
vari.ol.~s forms
of higher-order However
n ,
be examined
satisfactorily.
that
does
not
imply that
the
one cannot CT --
develop
a careful
conceptualization
of
target,
a conceptualization
to '~trpCse,
CT assessment on the
on
to acccrd consensus
the
descriptions
sub-·:;kills.
as':>c.Jciatt;ld
These
descriptions
a.re presented
i:\re
The e:·,a.mples
Some
intended
of possible
or initiate
strategies.
development
conversations
implications.
and sl.lb-skill
H'.lwever',
Lo do with the
skill the
not
necessarily eHtend to
examples.
------------------TABLE 4
CONSENSUS DESCRIPTIONS 1. INTERPRETATION: OF CORE CT SKILLS AND SUB-SKILLS
To comprehend and express the meaning or significance of a wide variety of experiences, situations, data, events, judgments, conventions, beliefs, rules, proced~res or criteria. to apprehend or appropriately formulate categories, distinctions, or frameworks for ~nderstanding, describing or characterizing information. * to describe experiences, situations, beliefs, events, etc. ~o that they take on comprehensible meanings in terms of appropriate categorizations, distinctions, or frameworks.
For ~~impl~: to retognize
d
1.1 CATEGORIZATION;
~roblem
and
defin€
its
character
13
." -: '\_
...
16
. .'~:~'~;{_i~.
: ,~.
(
~"
,;. ~. . ;_:
without prejudice to inquiry; to detfrmine a useful way of sorting and subMclassifying information; to make an understa,'dable report of what one experienced in a given situation; to cla~sify data, findings or opinions using a given tlassification schema.
to deteot, attend to, and describe the informational content, affective purport, directive functions, intentions, motives, pur~oses, social significance, values, views, rules, procedures. cr.iteria, or inferential relationships expressed in convention-based oommunication systems. suoh as in language, social behaviors, drawings, numbers, graphs, tables, charts, signs and symbols.
For example: to detect ~nd describe a person's purposes in asking a given question; to appreciate the significance of a particular facial expression or gesture used in a given social situation; to discern the use of iro~y or rhetorical questions in debate; to' interpret the data displayed or presented using a ~articular form of instrumentatlon.
to paraphrase or make explicit, through stipulation, description. analogy or figurative expression, the oontextual. conventional or intended meanings of words, ideas, concepts, statements, behaviors, drawings, numbers, signs, charts, graphs, symbols, rules, events or ceremonies. * to use stipulation, description, analogy or figurative expression to ~emove confusing, unintended vagueness or ambiguity, or to design a reasonable procedure for so doing.
~or example: to restate what a person said using different words or expressions while preserving that person's intended meanings; to find an example which helps explain something to someone; to develop a distinction which makes clear a conceptual difference or removes a troublesome ambiguity.
1.3
CLARIFYING HEANIN'G:
2. ANALYSIS: To identify the intended and actual inferential relationships among statements, questions, concepts, descriptions or other forms of representation intended to express beliefs, judgments, experiences, reasons, information, or opinions. to determine the role various expressions play or are intended to play in the context of argument, reasoning or persuasion. to define terms. to compare or oontrast ideas, concepts, or statements. to identify issues or problems and determine their component parts, and also to identify the conceptual relationships of those parts to each other and to the whole.
* * *
For example: to identify a phrase intended to trigger a sympathetic emotional response which might i~duce an audience agree with an opinion; to examine closely related proposals
to
14
17
I
,'_: .'
regarding a given problem an~ to determine their rOints of similarity and divergence; given a complicated assignment, to determine hOH it might be broken up into smaller, more manageable tasks; to define an austract concept.
* given a set of statements, desoriptions, questions or graphic representations, to determine whether or not the set eKpresses, or is intended to express, a reason or reasons J.n support of or contesting some olaim, opinion or point of view.
For example, given a paragraph, determine whether a standard reading of that paragraph in the context of how and where it is published. would suggest that it presents a claim as well as a reason or reasons in support of that claim; given a passage from newspaper editorial, detprmine if the author of that passage intended it as an expression of reasons for or against a given claim or opinion; given a commercial announcement, identify ~ny claims being advanced. along with the reasons presented in their ~upport. 2.3 ANALYZING ARGUMENTS:
2.2
DETECTING ARGUMENTS:
given the expression of a reason or reasons intended to support or oontest some claim, opinion or point of view, to identify and differentiate: (a) the intended main conolusion, (b) the premises and reasons advanced in support of the main conclusion, (c) further premises and reasons advanced as backup or support for those premises and reasons intended as supporting the main oonolusion, (d) additional unexpressed elements of that reasoning, such as intermediary conclusions, unstated assumptions or presuppositions, (e) the overall structure of the argument or intended chain of reasoning, and (f) any items contained in the body of expressions being examined which are not intended to be taken as part of the reasoning being expressed or its intended
background. For example: given a brief argument, paragraph-sized argument, or a position paper on a controversial social issue, to identify the author's chief claim, the reasons and premises the author advances on behalf of that claim, the background information used to ~upport those reasons or premises, and crucial assumptions implicit in the author's reasoning; given several reasons or chains of reasons in support of a particular claim, to develop a graphic representation which u s e+u l l y ch er e c t er i z e s the inferential flow of that reasoning.
representations which are acoounts or descriptions of a person's perception. experience, situation, judgment, belief, or opinion; and to assess the logical strength of the actual or intend inferential relationships among statements, descriptions, questions or other forms of representation.
",
•f''':,.
.
_. I':~:~\,: •• •
18
.:,01.'
';~/~.'~~ .
·.,
.'
degree of credibility to ascribe tQ d source of information or opinion. to assees the contextual relevance of questions, information, princi.ples, rules or procedural directions. to asseES the acceptability, the level o~ confidence to place in the probability or truth of any given representation of an experience, situation, judgment, belief or opinion.
* *
For example: to recognize the factors which make a person a credible witness regarding ~ given event or credible authority on a given topic; to determine if a given principle of conduct is applicable to decidin~ what to da in a given situation; to determine if a given claim is likely to be true or false based en what one knows or can reasonably find out.
to judge whether the assumed acceptability of the premises of a given argument justify one's acoepting as true (deductiy~ly certain), or very probably true (inductively justified), the expressed conclusion of that argument. to antioipate or to raise questions or objections, and to assess whether these point to significant weakness in the argument being evaluated. to determine whether an argument relies on false or doubtful assumptions or presuppositions and then to determine how crucially these affect its strength. to judge between reasonable and fallacious inferences; to judge the probative strength of an argument's premises and assumptions with a view toward determining the acceptability of the argument. to det·ermine and judge the probative strength of an argument's intended or unintended consequences with a view toward judging the acceptability of the argument; to determine the extent to which possible additional information might strengthen or weaken an argument.
* *
g i Ii en:· p 1 rl 1 2 ~I,
For example: given an argument to judge if its conclusion follows eith~r with certainty or with a high level of confidence from its premises; to ch~ck for identifiable formal and informal f~llacle~l given an objection to an argument to evaluate the l~ ~;:~1 ferce of that objection; tc evaluate tn~ qua!l~~ ~nj applicability of analogical argumerl!.sl 'r: ~'.ldCJe the log.i;:?1 strength of uguments based C,' r~·· ...0 Q t ~~t 1 C c11 's 1 t '.1 e1 t ion s C' r causal rea -:;c n i n 9 ; l 0 j LI d 9 e if a given ~rgument !~ relevant or dpplicable or has implirations for the s i t ue t i on clt t:and; to determine h o a possible new data mlyh~ lead logic~lly to the further confirmation or distonfirmation of a
reasonable conclusions; to form conjectures and hYrotheses; to cOIlsider relevant information and to educe the consequences flowing from data, statements, principles, evidence, judgments, beliefs, opinions, concepts, descriptions, questions, or other forms of r~presentation,
16
19
...
'
in particular, to recognize premises which require support and to formulate a strategy for seeking and gathering information which might supply that supPort. in general, to judge that information relevant to deciding the aooeptability, plausibility or relative merits of a given alternative, question, issue, theory, hypothesis, or statement is required, and to determine plausible investigatory strategies for aoquiring that information.
For example; when attempting to develop a persuasive argument in support of one's opinion, to judge what background inform~tion it would be useful to have and to develop a plan which will yield a clear answer as to whether or not such information is available; after j~dging that certair. missing information would be germane in determining if a given opinion is more or less reasonable than a com~eting opinion, to plan a search which will reveal if that informatiun is available.
* to formulate multiple alternatives for resolving a problem, to postulate a series of suppositions regarding a question, to project alternative hypotheses regarding an event, to develop a variety of different plans to aohieve some goal. to draw out presuppositions and projeot the range of pO~5ible consequences of decisions, positions, policies. theories, or beliefs.
For e~ample: given a problem with technical, ethical or budgetary ramifications, to develop a set of options for addreSSing and resolving that problem, given a set of priorities with which one mayor may not ~gree, to project the difficulties and the benefits which are likely to result if those priorities are adopted in decision making.
to apply appropriate modes of inference in determining what position. opinion or point of view one should take on a given matter or issue. given a set of statements, descriptions, questions or other forms of representation, to educe, with the proper level of logical strength, the~r inferential relationships and the consequences or the presuppositions which they support, warrant, imply or entail. to employ successfully various sub-species of reasoning, as for example to reason analogically. arithmetically, dialectically, scientifically, etc. to determine which of several possible conclusions is most strongly warranted or supported by the evidence at hand, or which should be rejected or regarded as less plausible by the information given.
*
*
to carry out experiments and to apply appropriate inference techniques in order to confirm or disconfirm hypothesis; given a controversial issue to examine
17
informed opinions, consider various opposing views and the reusons advanced for them, gather relevant information, and formulate one's own considered opinion regarding that issue; to deduce a theorem from axioms using prescribed rules of inference.
5: EXPLANATION: To state the resul ts of one 's re.asoning; to justify that reasoning in terms of the evidential, conceptual, methodological, criteriological and contextual considerations upon which one's results were based; and to present one's reasoning in the form of oogent arguments. 5.1 STATING RESOLTS: to produce accurate statements, descriptions or representations of the results of one's reasoning activities to analyze, evaluate, infer from, or monitor those results.
so as
For example: to state one's reasons for holding a given view; to write down for one's own future use one's current thinking about an important or complex matter; to state one's research findings; to convey one's analysis and judgment regarding a work of art; to state one's considered opinion on a matter of practical urgency.
to present the evidential, conceptual, methodological, criteriological and oontextual considerat~ons which one used in forming one's interpretations, analyses, evaluation or inferences, so that one might accuratelY record, evaluate, descrihe or justify those processes to one's self or to others, or so as to remedy perceived deficiencies in the general way one executes those prooesses.
For example: to keep a log of the steps followed in working through a long or difficult problem or scientific procedure; to explain one's choice of a particular statistical test for purposes of data analysis; to state the standards one used in evaluating a piece of literature; to explain how one understands a key concept when conceptual clarity is crucial for further progress on a given problem; to show that the prerequisites for the use of a given technical methodology have been satisfiEd; to report the strategy used in attempting to make a decision in a reasonable way; to design a graphic display which represents the quantitative or spatial information used as evidence.
to give reasons for accepting some claim. to meet objections to the method, conceptualizations, evidence, criteria or contextual appropriateness of inferential, analytical or evaluative judgments.
For ex~mple: to write a paper in which one argues for a given ~osition or policy; to anticipate and to respond to reasonable criticisms one might expect to be raised against one's political viewsl to identify and express evidence and counter-evidence intended as a dialectical contribution to one's own or another person's thinking on a matter of deep personal concern.
* *
18
21
,. ..: .'
8: SELF-REGULATION: Self-consciously to monitor one's cognitive activities, the elements used in those activities, and the results educed, particularly by applying skills in analysis and evaluation to one's own inferential Judgments with a view toward questioning, confirming, validating, or corr~cting either one's reasoning or one's results. to reflect on on~'s own reasoning and verify both the results produced and the corr~ct application and execution of. the cognitive skills involved. * to make an objective and thoughtful meta-cognitive self-assessment of one's opinions and reasons for holding them. to judge the extent to which one'e thinking is influenced by deficiencies in one's knowledge, or by stereotypes, prejudices, emotions or any other factors which constrain one's objectivity or rationality. to reflect on one's motivations, values, at~itudes and interests with a view toward determining that one has endeavored to be unbiased, fair-minded, thorough, objective, respeotful of the truth, reasonable, and rational in coming to one's analyses, interpretations, evaluations, inferences, or expressions.
8.1 SELF-EXAMINATION:
For example: to examine one's views on a controversial issue with sensitivity to the possible influences of one's personal bias or self-interest; to review one's methodology or calculations with a view to detecting mistaken applications or inadvertent errors; to reread sources to assure that one has not overlooked important information; to identify and review the acceptability of the facts, opinions or assumptions one reliej on in coming to a given point of view; to identify and review one's reasons and reasoning processes in coming to a given conclusion.
where self-examination reveals errors or deficiencies, to design reasonable procedures to remedy or correct, if possible, those mistakes and their causes.
For example: given a methodological mistake or factual deficiency in one's work, to revise that work so as to correct the problem and then to determine if the revisions warrant changes in any position, findings, or opinions based thereon.
6.Z SELF-CORRECTION:
19
22
cognitive
cognitive
can be said
aptitude using
per'son exper-ts
is not
personal
dispositions
which seem to
characterize
FINDING: Although the language here is metaphorical, one would find the panelists to be in general accord with the view that th~re is a critical spirit, a probing inquisitivenesn, a keenness of mind, a zealous dedication to reason, and a hunger or eagerness for reliable information which good critical thinkers possess but weak critical thinkers do not seem to have. As water strengthens a thirsty plant, the affective dispositions are necessary for the CT s~,ills identified to take root and to flourish in students. RECOMMENDATION 4: Modeling that critical spirit, awakening and nurturing those attitudes in students, exciting those inclinations and attempting to determine objectively if they have become genuinely integrated with the high quality execution of CT skills are, for the majority of panelists, important instructional goals and legitimate t sr get s for e d u cat ion a las s e ssm en t • H 0 1'4 eve r, t ~Ie ex per t s tl arb 0 r n 0 illusions about the ease of designing appr~priite instructional programs or assessment tools. Procedural, Laudatory
The ey.perts dispositions expressed a.re part have
d
consensus
good critical
However, whether
of the
was an issue
which divided
It became evident
20
'::.: ."
.
23
.... !-.•
>",;,;.
that
various
experts
mean different
things when they used the term "eT" in components. two-thirds majority who
to
possible
dispositional
that
meaning a reference
certain
affective
The project
director
counter-arguments
the debate
513, of those
Inaintairl t.hat
the affective
constitute
part
of the meaning of
cognitive
but
e~<pE"rts r.trgue
adept at
CT skills
c.\pprcpr··iatelydisqualifies
Thus,
in addition
LtSe "C'T"
to
sense,
these
panelists
als
l)
in i ts Ieuaecorv
to say, "This
perSc.111 is a cTitical
thinker,
this
and unmoved by reasonable thinker.'1 of hew well a person the parser. because the
canuo t call
him a critical
approval
The laudatory
dpplie'E>
h~r
CT skills
for
pe.... stu·. has the p.... oper ~ffective m.=-.j~Jrity w-as eloqLtsnt regarding .:Jffective dispositions
the importance of finding ways to instill in the final analysis they were una.ble
in students,
21
.
:-~':
.. .
'
..';~ ,,:¥,"
_, ~l'· .'.....
~.:.
~ ~'. ,"~
:;::,",
.,
to persuade the
0
ther
third of their
expert colleagues
to view these
was, however, the
thinker. (See
on using
"eT" in a strict
judgmental of critical
good ct-itical
valuable habits
as well.
If they
appropriately
also have some or all of the affective But those dispositions are not what is is not
mednt by 'leT.'1
a critical
fer deceptive
or sel.'f-iJlter'~sled
The sophist,
thinl-:er --
pr·o,:eduralist,s
sensible
deny that
a person
is a critical
thinker simply
the
person,
LIp
to check the
'Jives
(.Jrr...:;t.lem, lacks confidence in using ,...~asan to approach everyday i'jnores painfu.l facts. These experts hold that such a person,
thinker -skills).
Llse
of those
As <;.uggest,~dabove, lhere
"good critical
to the thinker's
effectiveness
22 "
25
this
person
using
CT?n to
sense
applies
to
morality
"Is this
CT ethical?" to convey,
In order
whic.:hsense to
of
ngood" the
Round 4 asked
panel
respond to
to
a proposal
have
a normative
dimension
in addition
a skill
disposi tional
dimension.
FINDING: The mistaken notion that CT has a normative component is rejected by the expert panelists. It is an inappropriate use of the term to deny that someone is engaged in CT on the grounds that one di sapproves ethi call y of "Ihat the person is doi ng. What "CTn means, why it is of value, and the ethics of its use are best regarded as three distinct concerns.
The majority of experts
nCT.II <527.) forc:efully
reject
the to
proposed say
What
normative use of
someth~ng is, person's
su"ffers skillS
it
is one thing
and another
how it ought
t,e be used.
the
what the)'
c:\re, even if
person
from certain
inadequac:ies.
(17%), argue
in favor
of using all
of
"C'F" in a use
nCT" in
of experts, sense,
whom also
its
commonly
understood
hold that
nor-oms and a
the
true
meaning of
For
a c:er-tain set
social
values.
WI.Juld
be willing to or
of
defense
attorney client
CT
a mistrial title
for
a guilty
be dignified
with the
thinker.
CT skills
contrive
a way
a gullible a critical
appear
Y into
an innocent
psrson
cg.nd
thiph:~r.
lad:: the
valLte truth
both
to
,nor'al
practice
of
their
socially
Import.an t; professions,
should
be accorded
26
.,""" ;""
. e
· :":
"
~~:
thinkers in another
would imp y. 1
way
-.
as well. should
The
(74h in support
this
report
fullest
of the immense
in education,
personal
life,
and as a vital
democ("atic society.
twist could, the'l" argue, lead to unwarranted and b:l unjustifie.\ble ideological being a "thinking" person. restrictions
on open inquiry
specter
spirit described
Dispo~"iti()"s
Thinker
FINDING: To the experts, a good critical thinker, the paradigm case, is habitually disposed to engage in, and to encourage others to engage in, critical judgment. She is able to make such judgments in a wide range of contexts and for a wide variety of purposes. Although perhaps not al~ays uppermost in mind, the rational justification for cultivating those affective dispositions which characterize the paradigm critical thinker are soundly grounded in eT's personal and civic value. CT is known to contribute to the fair-minded analysis and resolution of questions. CT is a powerful tool in the search for kn~"ledge. CT can help people overcome the blind, sophistic, or irrational defense of intellectually defective or biased opinions. CT promotes rational autonomy, intellectual freedom and the objective,
24
.:~\~;:>'".' ..
~"\1'''
••
'--'
••
.<.~(:.; ',.,
: .... :'
<'..,~'~;;
..
.\.
......
reasoned personal
of a very
Nide
range
of
Thus, in addition
thinker of mind.
dispositions
characteristics
ttllnkers, CT in its
fullest. realization.
listed
The majority
the dispositions
CT.
in Table 5 as part
(8::3X favor) in that
of the
conceptualization
thinkers
A consensus eHists
good critical
characterized
• alertness to opportunities to use CT, • trust in the proc:eues of reasoned inquiry, • self-confidence in ODe'. OWD abUity to reason, • opea-mindedneu reprdin& divergent world views, • ftezibiUty ill considering altemaUves and opinions, • understaadiD& of the opinions of other people, • fair-miDdedaess ill appraiaillg reuonin& • honesty in faciag one'. own biases, prejudices, stereotypes, eloc:entrie or sociocentriL tendencies, • prudence in suspendiag, making or altering judgments, • wiUiapess to reconsider and revise views where honest reOection sugests that change is warranted.
• cladty in statiol the question or concem, • orderliness in working with comple:Kity,
• reuouableaess in selectinl and applying criteria, • care in focusiDS attention on tbe concem at hand,
• persisteDc:e thou ... difficulties are encouatered, • prectlion to the depee permitted by subject and circumstances • 2S
• 0
,...
~ ::-
28
...... ~
.....
' '
:'~
or religious
are not atoms each of which is at any moment subject independently. Beliefs form intricately attention
reconsidered systems
interconnected
of thought. those
on any of them,
particular'ly
or fundamental to one's own view of the throughout cne's entire belief system.
reverberations
as many experts
warn that
an over-emphasis
dimension of CT to the exclusion of the affective might have the unfortunate inflexible result of making some students
closti-minded, intellectually
and dogmatic.
RECOMMENDATION 5: Just as with the cognitive dimension of CT, when conceiving of the education or assessment of critical thinkers, it is important to consider ways of developing materials, pedagogies, and assessment tools that are effective and equitable in their focus on these affective dispositions. The cultivation of thesp. dispositions is particularly important to insure the use of CT skills outside the narrow instructional setting. Persons who have developed these affective dispositions are much more likely to apply their CT skills appropriately in both their personal life and their civic life than are those who have mastered the skills but are not disposed to use them. As with the intend that experts
are
listing
of cognitive
skills
ea.rlier,
the
panel does
not
each disposition
characterizing c:rP:icdl
be considered
.a necessary forth
the ideal.
they
In setting
the paradigm
all per
thinker,
intend
to e>:press
might strive.
SOl
a meaSLlr-e of matLlrity
in college
io=t.fld
twelfth
,al development
saphomor-es or
gradt:?(·s.
Yet to u~l<;'IY
embad:ing
en the practices
and disciplines
which
2f
_""
".:';',';
:~.
"
will
lead to
these vir-tL,es
RECOMMENDATION 6: From early childhood people should be t~ught, for example, to reason, to seek relevant facts, to consider uptions, and to understand the views of others. It is neither impractical nor unreasonable to demand that the educational system teach young people the habits of mind which characterize the good critical thinker, reinforce thos! practiceG, and move students well down tre path toward their attainment.
Y -- Further
Recommendations
Severa.l pedagogic:aland assessment implications follow from the dispositional if dimensionof CT, implications which might not be apparent dimensionof CT. The education of students to execute a set of
good critical
For example,in terms of pedagogy,modelinghow to that information which students would normally accept
and encouraging them to do the same can do wonders for CT ability. With this confidence themselves. Just as
inc:ompletepictu.re of someone's
,=,trengt.hs as a critkal
thinker.
The CT Goal
RECOMMENDATION 7: educational, personal aca~emic goal of CT should be furthering skills and affective
Brcause CT helps students wlth a wide range of and CIVIC concerns in a rational way, the instruction, regardless of the educational level, students in the development of their CT cognitive dispositions.
27
TABLB6
question. A second way is to compare the outcomes (if any) that result from elecuting a given skiD apillit some set of criteria. A third way is to query persons and receive their descriptions of the proc:edures aDd judgments they are using as they exercise that skiU, would use if tbey were to perform that skiD, or did use when tbey performed that skill. A fourth w.y is to campa'ie tbe outcomes (if any) tbat result from performing another tuk against some set of criteria, where the performance of that otber task
has been sbown to correlate strongly with exercising the skill of interest. However, that such correlations exist between any otber task and CT, or any of its sub-skiUs, bas yet to be established in the research literature.
Bach of the four ways of CT assessment has limitations as well 85 strengths. No matter which ways are used, it is important to ensure that the assessment conditions foster an attitude in which the subjects are disposed to use their skills as well as they ean, and are not constrained or inhibited from doing so. In our view it is highly advantageous to gatber evidence regarding CT performance in many situations, using several assessment methods, so as to compile a composite picture of the subject and to cross chtN:k the results of an one wa of assessment.
'"
. ":.y ....
,
;).~ ..
"";.'
.~ .. o_
·_' :', r
~: "
.'~.:.'
Either to transform
or to subject: and
utility,
misapprehend its
nature
diminish its
CT can be
from the goal of learning domain-specific two goals can b£' distinguished, ways to learn the
CT is within a
RECOMMENDATION 8: Direct instruction in CT and assessment of CT should be an eKplicit parts of any course granted approval for purposes of satisfying CT requirements, whether that course is a CT course per se Dr a course in a given subject field. The primary academic criterion in the evaluation of a proposed instructional program for purposes of achieving the CT goal should be whether the program will further the development of students' CT skills and dispositions.
The CT Curriculu.
Given that
can be expected to begin to develop However, the value of CT extends inquiry tool. well
CT programs.
CT is vitally A
impor"tant in the
members of society.
RECOMMENDATION 9: Thus, CT instruction should not be reserved only for those who plan to attend college. hor should it be deferred until college, since it is not likely to be effective if it were. RECOMMENDATION 10: Explicit attention to the fostering of CT skills and dispositions should be made an instructional goal at all levels of the K-12 curriculum. The cultivation of CT dispositions and an insistence on giving and evaluating reasons, should be an integral part of elementary school education. In middle schools and high schools,
, '.~
-,
':,..
..
-'
~.-.
_;.:."
..
instruction on various aspects and applications 04 CT should be integrated into all subject area instruction. Specific courses in CT and an advanced placement examination program in CT for college bound students should be developed. Although for gDod reasons at the postsecondary level CT programs are generally associated with departments of philosophy, no academic unit should be restricted in principle from participating in an institution's CT program, provided that the overall institutional program in CT equips stlldents to apply CT to a broad range of educational, personal and civic subjects, issues and problems.
There is growing evidence of economic, of those industrialized the successes, both scientific and
democracies which emphasize demanding educational standards is for career and a key
Assessment that
counts
unquestionably
RECOMMENDATION 11: Thus, mlnlmum CT proficiency eKpectations should be set for each educational level, including promotion in grads, high s:hool graduation, college entrance, and graduate school admission.
assessmeot
strategies students'
from
inferences
about;
CT, in
their
knowledge or other
~cademic: abilities
essential. classroom
CT assessment strategies,
or"
Llse in the
individual at
for
broader
pL~rposes, must
correct
answers.
sense of
would be shameful if
on CT skills
the
d1sposition.:'\1 co-npcmerrt e of
be neglected.
RECOMMENDATION 12: In evaluating the acceptability strat~qy or in~trument one should consider content validity, reliability, and fairness.
30
,.
of a CT validity,
assessment constrLlct
.: ;......
~::~~
< .....
r'.r
f~.;~;
"
./~
i~·
(I) 011
Content Validity:
The strategy
or instrument
an appropriate
conceptualization
responding to that
item is not a
of rote
CT assessment
CT's dispositional
ccgni ti ve skills
(2) Construct
or
qLtestian should have been evaluated correctly responses do so on the basis euoe the result
to insure
students inadequate
of weak or inadequate
CT. Entire
speci fie: item~"i n which good CT leads o right answers, should not be ,-,sed. (3) Reliability: In acceptable
CT assessment that
each task
or question
to insure
good critical
thinkers If
thinkers.
the results,
gr-r:tding essC\ys or judging presentations, judges should be cross-checked r-sliable, that is, generally to assure
consistent
an open question
and with vari.aus dispo'Sitions has yet to be undertaken. lime, due cQLltion should be exercised reliability 3l regarding
this
how to interpret
t..~.:hnic:alflleCl'·.iLtrl~S t:est-form uf
"
.,.....~.
::.~/ ....
'.
~ ··t,;,
"'-.'
••• F" •
"~~."
Fairness:
CT ~ssessment
disadvantage
or domain-
advantage specific
groups of students
conceptual,
considerations, life
or familiarity
or socio-economic status, in cut tLlral assumptions. in some assumed context, Thus, guaranteeing
or fictional.
of their
individual backgrounds, will ceme to the CT equal basis in terms background knowledge, is impossible. However, these
on a perfectly
reading ability,
or· instrument
to be sure that
Althc,ugtl one cannot eliminate the influence of these bet ,alble to neutr-ali.::e or control The fainleo:;s crit.erion discipline-specific for their affects.
applies
a.nd
CT assessment..
discipline-c::.pec:ific:CT assessment
one ttl be fair concept.s,
for
in one's presumptions
the strength
or weakness of the CT. It is worti., noting also makes similar assumptions topic content.
I:hdt disc:ipline-neutnll
CT assessment contexts
regarding
the everyday
RECOMMENDATION 13: CT assessment should occur frequently, and it should be used diagnostically as well as summatively. Different kinds of
a5
-:..t.,
. '':~
..
instruments should be employed, depending en which aspect of CT is being tdrgeted and where students are in their learning the introductory stage, the practice stage, the integration stage or the generalized transfer stage. Although the veteran CT instructor is abl~ to assess studJnts continuously, CT assessment should be made explicit to reinforce its worth in the eyes of the students, their families, and the public. It should be made explicit to support the goals of educators seeking to improve the curriculum. And it should be made explicit to properly inform educational policy formation.
The CT Instructor
RECOMMENDATION 14: Teaching CT is most effective if the instructor models CT dispositions and the proper use of CT skills in the very process of instruct~on. Regardless of the subject area, students should be encouraged to be curious, to raise objections, ask questions, point out difficulties in the instructor's position. These objections and questions should be clarified, interpreted, and examined objectively. Students should be given reasons for doing things a certain Nay, rather than being dogmatically told how to do them. Instruction should bridge the gap between the subject and the student's own experience. In the case of CT instruction, the topics of discussion should not be restricted to factual matters Dr academic subjects, but sh~uld include issues which have normative, moral, ethical or public policy dimensions.
The ideal CT instruc:tor variety of subject subjects areas. as will integrate instruc:tion in CT in a directly skills. to She
tEl
using these
elaborate,
transfer
and generalize
tEl
skills
classroom
She will model CT in her She will provide to learn about, activities her
stUdents to
in social
e>;planations
in CT continuously.
RECOMMENDATION 15: For CT to infuse the K-12 teacher "training" should give way to teacher
curriculum, If teachers
33
...
,
~ .;:......
'
~re to model CT, so must those who have an instructional role in teacher preparation or staff development. In all instructien, and particularly in CT instruction, both faculty and leaders of faculty development should model CT. They should foster the students' confidence in their own powers of reason, rather than dependency on rete learning. They should nurture in students open-mindedness, attention to alternatives, and as much precision of thought as the subject and circumstances permit.
educators
are are
academically
(22'Y.),
with Philosophy (S2X), the others the Social Scienc,es (207.),or the
Physical Sciences
It
participation
in this
research
Thus,
project
implying that
a person
agrees
a minority
hold divergent
views.
Where near unanimity is reported acc.ord wiLh hew t,he specifics reviewing the f(Jrty-si~ draft
a some panelists
are expressed.
explicitly
It is a
speCial
efferts
experts
relating
experience
and knowledge
tOWai""'lj discovering
found.
As
of the eighties
of the
the
in this
hope that
of CT instruction
..
.;.~.~.
.... ~., -
...J ,
'_) ~'1
TA8LB1
J'
PARTICIPATING
CRITICAL THINKING EXPBRTS Ball State University University of Washington Howard Community College, MD
Jonathan Adler
David Annis AmoldAroDl James Bell Barry K. Beyer Charles Blatz Rob Brady Neil Browne
Philosophy
Philosophy Pbysics
Brooklyn CoUege
Philosophy
Bez CemmellSOD Arthur L Costa Stan Dundon Robert H. Ennis James B. Freeman Jack Furlong Eugene Garver H. Scott Hestevold David Hitchcock
CT Assessment
American College Testing (ACT) Sacramento State University Cal. Polytechnic Uaiversity, SLO University of Dlinois
George Mason University University of Toledo Stetson Univenity BowHng Green State University
Stuart Miller Brooke Noel Moore Wayne Neukberger Richard Parker Richard D. Parry Richard Paul Philip Pecorino William Rapaport Pasqual Sc:hievella Zack Seec:b
Anita Silvers Stephen Norris
Psychology
Philosophy
Philosophy Philosophy
GaDaudet University University of C"mcinnati U. Massachusetts, Amherst Towsen State University CSU L'1tico Oregon Department of Education Memorial University of Newfoundland CSU .. Otico
Hunter CoUele, CUNY Transylvania University Saint John'. UDivenity University of Alabama McMaster University Christopher Newport College University of Colorado University of Wmdsor Bowling Green State Univenity Arizona State University Montclair State College
Steven Tigner
Robert Wengert
Social Sciences Computer Science Council of Critical Analysis, Port Jefferson, NY Behavioral Science Palomar College Philosophy San Pranclsc:o State University Northwest Regional Educational Laboratory, Portland Philosophy U. Massachusetts, Boston Philosopby University of Toledo CT Assessment Bducational Testing Service (BTS) Philosophy Sacramento State University Philosophy University of Illinois Institute for CT Montclair State College Education' University of KentuckY
t
Agnes Scott CoUege Sonoma State University Oueensborough C. College SUNY Buffalo
35
.....
~~... .
:,.~
38
••
••
_:f~-·_·:,
APPENDIX A
Commercially Ayailable OT Assessment Tools
on Pre-College
Philosophy
The Education Testing the academic abilities data, all within the of
Service
Test
measures
context
humanities,
social
sciences with
and natural
an optional
critical
sophomore college
year.
J
ETS also thinking has sections of its targeting logical reasoning and analytical (a) Law
on several
such as the
(LSAT) ilLogical
General Test,
subject-matter Progress,
(e)
higher
lr.:lboratory-based
Forei.gn Service
Test,
in-basket
Advanced Test
in Philosophy to
1982.
8r
~c1I_lc\
ETS is
adding a section
of
Critical
Reasoning Questions
(GMAT).
An example is
prepared
B~ard of
Respiratory
Care, Shawnee
36
... }
· ?"
1'1i5sion, I<ansas.
Norris
the
Institute of
for
Educational has
and Development
the
at Memorial University
Observations
subjects made by
1983. are
backdrop
of a common but
asked
judge
the
relative
credibility
various
(:har·ac::ters.
Testing Academic
Pr'ogram
the
Proficiency"
ACT describes
designed
measure
selec1:ed academic
including
reading,
writing,
reasoning.
evaluate,
1ne CT Test
an~ extend encountered questions. in a
mea=Ltr-es the
anJLhlI13I1tS."
"clarify,
analyze,
composed of passages
posts~?condary c:urricLtlLu-o
fallowed by multiple
the
"Valet:t
Inventory
of
CT Abilities" 348.
vol.
commercially Many of
available
these U5t~ful
tests
in print described
c.,long with
reviews
and I'"'esearch
data.
a d r-eviewed
in thE:irCA, 1989.
#769
ThlnkirllJ
Tes t, Levels
X and
Publica.tions, Psychological
Pacific:
Grdve,
CA.
1983
Educat ional
and by
i'teasureme'nts
pp. 1187-1197,
Modjeski
and Michael.
:.:_,,: ~ , ."
:'.\
.'
40
.~~.
,:,.1.
"
..
;:::~.~':' .
. -.
""
#390
"Ennis-Weir
Argumentation
Test,
Level
X: An Essay
Test
of Rational
..
I,
Thinking Ability,"
,.
<Robert
Ennis
and Eric
University
of Illinois,
IL.
by Herbert Critical
State,
in NMMY. for
..
~,
*391 "Ennis-Weir
Thinking <Robert
An Instrument
Testing/Teaching.1I Publications, #1347 "Watson-Glaser and reviewed other #751 "New Jersey Insti tute tt1258 "Test research Test
for'
and Eric
Pacific Critical
instrument. 1983, Virginia Philosophy Council this test for for Shipman, Children .. Educational to
S~dlls," of
Advancement
of Inquiry For
Skills"
junior of
1979, Australian
high grades,
Research.
purports
evaluate skills
a t-ange
research,
study
and critical
thinking
in the
of
sciences. Cognitive
Processes"
Higher
(John
Ross
and Catherine
Ross) 1976-79,
this
Publications.
4-6,
test
inclLtdes
on analogie~, questioning
strategies,
of Cognitive
test
includes sub-scores
reasoning.
on sequencing,
analogies,
and verbal
McGraw Hill.
Included
in the
sading
sub-score
on infer'sllce
is
a sub-score
1984, LinguiSystem 3
·~~i·q .. .::~ .
.. '. -"'f.,
41
'.
critical
determining causes,
Inc.
of on
hypothesis/evidence.
1972-73,
graduCl scientists te
#1010 "PSI Basic Skills Test for Business and Industry" Psychological Services Inc. Includes sub-scores
1981-1982, on problem
Used to tests
persons
inclL-tdessubidea
reasoning,
3'
42
.. . :.1:,::".
A,
~ .
~"
Prepared for the APA Committee on Pre-College Philosophy Delphi Researoh on Critical Thinking Assessment
Adler, H., "Why 'CT' PrograMii:aWon't Work," Education Heek, Sept. 1986.
Annis, David B. and AnniS, Linda, "An Empirical Study of the Impact ~f Philosophy on Students' CT Ability," Teaching Philosophy, v3, pp. 145152, 1980. Arons, Arnold B. ·CT and the Baccalaureate v71, n2, Summer 1985. Curriculum," Liberal Education,
_______ , "Achieving Wider Scientific Literacy," Daedalus, Journal of the A.erican ACide.y of Arts and SCiences, v112, n2, p91-122, Spr. 1983. Azima, Kiavach, and Henry, Rebecca, "Teaching Students to Reason: An Application of Piagetian Psychology to College Teaching No. 76," Learning and Evaluation Service, Michigan State University, East Lansing, MI, 40p, 1980. Sociology and nssessing CT," Teuching Thinking Sociology, Skills: Theory v8, and in
Barun, Joan B., and Sternberg, Robert J., Teaching Practice, W. H. Freeman Publishing, 1987.
Bangert-Drowns, Rober, L., et, al., "Individualized Systems of Instruction Secondary Schoels," RevieN of Educational Research, v53, n2, p143-58, SUlllmer 1983. Beck, ~onald A., A Guide to Criterion-RefereDced Hopkins University Press, 1984. Beyer, Barry K., ftImproving Thinking Delta Kappan, v65, n7, p4R6-90, _______ , MImproving Thinking Skills Kappar" v65, n8, Apr. 1984. Test Construction, the Problem," Phi Delta Johns Phi
Approache5,~
_______ , "Practical Strategies for the Direct Teaching of Thinking,Y in Veveloping Hinds: A Resource Book 10r Teachi~g Thinking, Arthur L, Costa, ed., Association for Supervision and Curriculum Development, Alexandria, VA, 1985. _______ , "A Suggested Format for Testing Record, v24, n i , p3-5, Spr. 1987. _______ , Practical Bacon, 1987. Strategies Thinking Skills," Social Boston, Science Allyn and
of Thinking,
4C
43
~:~~
;J-'
Blatz, Charles V., "Contextuali~m and CT: Programmatic Educational Theory, v39, n2, 1989.
Investigations,"
Block, R. A., and TaylClr, S. V., "Cognitive Skills: Enhancement and Assessment Issues, Presented to the American Psychological Association, Toronto, Canada, 1984. Blumberg, Fran, et. aI, A Pilot Study of Higher-Order Assess.ent Techhiques in Science and Hathelatics Tested Tasks -- Part II, Final Report, National Progress, Princeton, NJ, Nov. 1986. Thinking Skills -- Part I and PilotAssessment of Educational Evaluations and v44, p67-70, 1986.
Bransford, J. D., et aI, "Teaching Thinl:ing: Evaluating Broadening the Data Base," Educational Leadership,
Branson, Stimmann Margaret, "CT Skills -- A Continuum for Grades 3-12 in History/Social Science," Social Studies Revie~, v25, n2, p24-32, Winter 1986. Brandt, Ron, "On Philosophy in the Curriculum: A Conversation Lipman," v46, Educational Leadership, p 38, Sept. 1988. with Matthew
Braungart-Bloom, Diane S., "Assessing Higher Thinking Skills through Writing," lOp. Presented at the Annual Meeting of the American Educational Research Association, San Francisco, CA, April 1986. BrONn, J. lo, "On Teaching Thinking Skills in the Elementary Schools," Phi Delta Kappa», v64, p709-714, 1983. and Middle of
Bryden, David P., "What Do Law Students Learn? A Pilot Study," Journal Legal Education, v34, n3, p479-506, Sept. 1984.
Carlson, E. R., "Implications of Cognitive Theory and Research for Teaching CT, Presented to the Annual Meeting of the American Psychological Association, Toronto, Canada, 1984. Chaffee, John, "Viewing Reading and Writing as Thinking Processes," 9p, Presented to the Annual Meeting of the American Educational Reoearch Association, Chicago, IL, Mar-Apr. 1986. Chance, Paul, Thinking in the Classrool: College Press, 1986.
A Survey
of Programs,
NY, Teachers
Cierzniak, Susann~ Lipetska, The Question of CT: An An~otated Bibliography, 64p. Exit Project, University of Indiana at South Bend, ED 260 069, April 1985. The specific focus of this work is CT in tha 5econdary schools. Cornbleth, Catherine, "Assessing Skills and Thinking in Social Studies," po~ition paper prepared for Study Group on the national Assessment of Student Achi evement, and ci ted in Appendi x B of thei r report, "The Nation's Report Card" eTM 870 049), Journal announcement: RIEJUL87 CClsta, Arthur L., ed. Developing Hinds: A Resource Book for Teaching Thinking" Association for Supervision and Curriculum Development, Alexandria, VA, 1985.
. .\ ,.' .
.~,:~-.~
41
,-
- , .-.:'
,~,.' ....
_______ , "Thinking: How Do We K~cw Students are Getting Better at It?" ms., Dept. 04 Education, Sacramento State University, CA., 1989. Cost.a, Arthur lo, and Marzano, Robert J., "Teaching £ducati(Jnal Leadership, v45, p29, Oct. 1987. D'Angelo, Edward, The Teachirlf1 11f CT, Amsterdam, the Language
of Th i nk i nq ;"
B. R. Gruner
N. V.,
1971.
Deshmukh, M. N., "Teaching the Unteachable: Some Pedagogical of Creativity," PsychQ-Lingua, v15, n1, p33-40, 1985. Dewey, John, HON He Think, Drake, James A., Teaching Publishers, 1976. D. C. Heath and Co., Baston, CT, Danville, 1933. Printers
Considerations
Il, Interstate
and
Duck, Lloyd E., "Seven Cardinal Principles for Teaching Higher-Order Thin~:ing,1I Social Science Recorl'J, v24, n i , p3-5, Spr. 1987. Elman, Sandra E., and Lynton, Ernest A., IIAssessment in Professional Educition," 24p. Presented to the National Conference on Assessment Higher Education, AAHE, Columbia, SC, Oct. 1985. Ennis, Robert H., 'A Concept
1962.
in
Educational Educational
_______ , IITesting for CT: state of the Art," American Assoc., San Francisco, CA, 1968.
_______ , "Rational Thinking and Educational Practice," in Philosophy and Education, Jonas Soltis (Ed.), Eighteenth Yearbook of the National SOCiety far the Study of Education, Chicago, NSEE, p143-183, 1981. _______ , "Goals for a CT/Reasoning Curriculum, Illinoi~, Champaign, IL, 1984. _______ , "Probl~ms in Testing Informal Logic v6, nr , p3-9. 1984. _____ -_, IIA Logical Basis for Measuring v43, p44-4B, 1965. Illinois CT Project, Ability," U. of Infor.al
Educational
Leadership,
_______ , "A Logical Approach to Measuring CT Skills in the Fourth Grade,lI Illinois CT Project, Champaign, IL, draft, April 1985. _______ , "A Taxono~v Of CT Dispositions and Abilities," in Teachi~9 for Thinking, Joan Baron and Robert Sternberg (Eds.), Freeman, Ne~ York, NY, 1987. _______ , "A Conception of CT with SORle Curriculum Suggestions," He"Jsletter on Tea,hing Philosophy, American Philosophical Assoc., p1-5, Summer 1987. 42
!~~ "-),[:7' :
. '.','"
," -".
,;.-~.::.,"
,'
J .••.;~
.'
~'.'
~-:.; .
<:
___ ..
, "A Bibliography of Testing CT,'! CT He~s, Center for the Reasoning Arts, CSU Sacramenta, v'o, nl, Sept.-Oct. 1987. Class Reasoning Test,
Ennis, Robert H, Gardiner, William L, et aI, Cornell University of Illinois, 1964, .......... Cornell Conditional __ , IL, (date ';') Reasoning
Ennis, Robert H., and Millman, Jason, Cornell Publications, Pacific Grove, CAl 1985. _____ .. Cornell _, CT Test Level
Ennis, Robert H., and Weir, Eric, ennis-Heir Publications, Pacific Grove, CA, 1985.
Ennis, Robert H., and Norris, Stephen P., "CT Testing and Other CT Evaluation: status, Issues, and Needs," in Issues in £valuation, Algina, James, (Ed.), Ablex Press, New York, NY, 1988(7). Facione, Peter A., "Toward a Theory of CT," Liberal Fall 1984. _______ , "Testing College Level CT," Liberal 1980. ___ .. , "Teaching abuut Fallacies," Summer 1987. Teaching £ducation, v70, p253-261, Fall
£ducation, Philosophy,
v72, p221-232,
Regarding
HOTs Assessment,"
v7, n2/3,
_______ , "Assessing Inference Skills," £RIC Clear;nghouse Heasurelents and £valuation, TM 012917, Mar. 1989.
------_. ------_.
-------,
CT He~s v7, n4, pl, 7f, 1989. v7, n4, p12, 1989.
_______ , "Strategies for Multiple Choice CT Assessment," in CT at Colleges Universities, David Hitchcock, (Ed.), Vale Press, Newport News, VA., forthcoming. Follman, John, "Contemporary CT Bibliography," CT HeNS, Center for the Reasoning Arts, CSU Sacramento, vo, n2, Nov.-Dec. 19~7. Fraser, Berry, Test of Inquiry Skills, The Australian Measurement, Hawthorn, Victoria, 1979.
Frederiksen, Norman and Ward, William B., "Measures for the Study of Creativity in Scientific Problem Solving," Applied Psychological
43
'."f,... ;
.~
-
.. ,':,,'.....
46
~: ..
\
. '
~', . ..
".
:- ' ..
Heasurelerlt,
v2, pl-24,
19?B.
II
Fr itz, Pau 1 A., and We aver , ·Ri ch a r d L. II, Tea chi n 9 C T S ki 1 lsi nth e P II b 1 ic Speaking Course: A Liberal Art5 Perspective," 36p, Presented at the Annual Meeting of the Spee~h Communication Assoc., Chicago, IL, Nov. 1984. Gardner, H., Frales of Hind: Socks, NeN York, 1983. The Theory of Multiple Intelligences, of CT, Teachers Alerican Ithaca, NY, Basic College,
in the Develop.ent
Glaser, R, IIEducation and Thinking~ The Role of Knowledge," Psychologist, v39, p93.-104, 1984. 8rinols, Ann Bradstreet, (ed.) CT: Reading Cornell University Press, 1984. Across
the Curriculu.
Haars, Venant J., and Mason, Emanuel J., "Children's Understanding of Class Inclusion and Their Ability to Reason with Implication," International Journal of Behavioral Develop.ent, v9, n1, p45-63, March 1986. Haney, Walt, "Testing Reasoning and Reasoning Educational Research, v54, n4, p597-654, Harnadek, Anita, CT Troy Mich., MidNest about Testing," Winter 1984. 1976. of Speech HA, Nov. Revie~ of
Publications,
Hey, Ellen, liThe CT Movement and Its Implications Communication,1I paper to Speech Communication 1987. Hermstein, R. J., et aI, IITeaching Thinking v41, p1279-12B9, 1986 Hitchcock, David, (Ed.), CT at Colleges Press, Newport News, Virginia.
Skills,"
Psychologist, Vale
and Universities,
forthcoming,
Howard, George S., and Englehardt, Jean L., "Teaching Rival Hypotheses in Experimental Psychology,1I Teaching Psychology, vll, n1, p44-5, Feb. 1984. Hunter, Jacqueline; Jones, Lester; Pre-College Program," Journal
e t , al., IITeaching Cognitive Skills of Learning Skills, vl, n2, p24-26,
in a 1982.
Kean, Michael H., "Testing Future Challenges, Future Respol"ses,1I Presented at the Annual Heeting of the National Ccuncil of Measurement in Education, San Francisco, CA, April 1986. K~arney, C. Philip, et al, Assessing Higher Order Thinking Skills,lI ERIC Clearinghouse on Tests, Measurement, and Evaluation, Princeton NJ, pB6 Apri 1 1986
II
Kneedler, Peter E., "Testinq CT at Grade Eight,1I Social n2, p7B-89, Winter 1986.
Studies
'.'
in California,1I
Social
Scier/ce
Lawson, Anton E., liThe Effects of Causality, Response Alternatives, and Context Continuity on Hypothesis Tssting Reasoning,1I Journal of Research in Science Teaching v20, n4, p297-310, April 1983. Lee, Braham, and Oakhill, Jane, liThe Effects of Externalization on Syllugistic Reasoning,1I Quarterly Journal of Experi.ental Psychology HUlan £xperilf'ntal Psychology, v36A, n3, pSI9-30, Aug. 1984. Lipman, Matthew; Sharp, Anne Margaret; and Oscavan, the Classrool, Universal Diversified Services,
- .... 1
in
(ed.> G,o~ing
Up ~ith
Philosophy,
-------,
Lohman, David F., "Predicting Mathemathanic Effects in the Teaching of HigherOrder Thinking Skills," Educational Psychologist, v21, n3, pI91-208, 1986. Mandic, Peter, and Stojakovic, Peter, IIContemporary Yugoslavia: A Review of Research", Conte.porary v9, n3, p207-13, July 1984. Educational Educational Psychology in Psychology,
Martin, David S., "Restructuring Teacher Education Programs for Higher-Order Thinking Skills," Jour~al of Teacher Education, v40, p2, May-June 1989. Marzano, Robert J, and Jesse, D., IIA Study of General Cag,itive Operation on Two Achievelllent Batteries and their Relationship to Item Difficulty,1I Unpublished paper, Mid-Continent Regional Educational Laboratory, Aurora, Colorado, 1987. Marzano, Robert J, et. aI, Dimensions of Thinking: A FrameMork for Curriculum and Instruc~ion, Assoc. for Supervision and Curriculum Development, Alexandria, VA., 1988 Marzano, Robert J, and Costa, Arthur L., "Question: Do Standardized Test5 Measure G~neral Cognitive Skills? Ans~er: No," Educational Leadership, v46, pp 66-71, May 1988. McKee, Saundra J., IIImpediments to Implementing p443, Oct. 1988. McPeek, John, Critical NY, 1981. Thinking and Education, CT," Social St. Martin's Education, v52t
Mebane, John S., IITeaching Interpretive Skills Through Testing Cla5s," ExerCise Exchange v32, nl, p7-10, Fall 1986.
Modjeski, Richard B., and Michael, William B., "An Evaluation by a F'anel of Psychologists 0,' the Reliability and Validity of Two Tests of eTn Educational and Psychological Heasurelent, v43, n4, p1187-97, Winter 1983. (The test5 reviewed were the "Watson-Glaser CT Appraisal" and the
48
._
~};~.~/.
c .«,
•• '0:". '..t.,'
;~;.:..
~',
-..
"Cornell CT Test."] Morante E. A., and Ulesky, ~. "Assesument Leadership, v42, p71-74, 1984. of Reasoning Abilities,H Educational
Moss, Pamela A., and Petrosky, Anthony R., "A Proposal for Heas~ring Critical Thinking," A revision of paper p,'esented at the Annual Meeting of the American Educational Research Astoc., Montreal, Canada, April 1983. Pittsburgh board of Public Education, PA, Report No: T~a3-2. 23p, Sept. 1983. Moore, Brooke, and Parker, Richard, "T~e Written Debate -- A Technique Honing CT Skills," HeNsletter ~n Teaching Philosophy, American Philosophical Association, p8, Sumt!r 1987. for
Morgenstern, C. F., and Renner, J. W., "Measuring Thinking with Standardized Science Tests," Journal of Research in Sci~nce Teaching, v21, pt39-648, 1984. Meyer, Chet, Teaching Bass, 19B6. Students to Think Critically, San Francisco, JosseyEducational
Nickerson, R. S., -Kinds of Thinking Taught in Current Programs," Leadership, v42, p26-36, 1984. Nickerson, R. S.; Pprkins, D. N., and S.ith E. E, The Teaching Lawrence Erlbaum Associates, Hillsdale, NJ, 1985. Norris, Stephen P. Test on Appraising Observations, Research and Development, Memorial University
of Thinking,
_______ , "The Choice of Standard Conditions in Defining CT COlpetence," Educational Th~ory, v3S, p97-107, 1985. _______ , "Evaluating CT Ability," p13S-146. Spr. 1986. History and Social Science Teacher, v21, n3,
_______ , "Controlling for Background Beliefs When Developing Multiple-Choice CT Tests," Educational Heasure.ent , v7, n3, p5-11, Fall 1988. _______ , "Research Needed on CT," Canadian 37, 1988. ~ournal of Education" v13, p12S-
_______ , "Verbal Reports of Thinking and Multiple-Choice CT Test Design," Technical Report No. 4.17, Champaign, IL: Center for the Study of Reading, University of Illinois. (ERIC Doc. No: ED302826.) _______ , "Effect of Eliciting Verbal Reports of Thinking on CT Test Perforlance," Journal of Educational Heasure.ent, v27, nl, 1990. _______ , "Informal Reasoning Assessment: Using Verbal Reports of Thinking to Improve Multiple-Choice Test Validity," in Infor.al Reasoning and Education, D. N. Perkins, J. Segal, and J. F. Voss (Eds.), Lawrence Erlbaum Publishing, Hillsdale, NJ, in press.
46
".
i·'
):·\:1'·,:
,"'~'. < •
.. . ~.t,";.
<
'.!~ •.
------_.
Norris, Stephen P., and Enni~, Robert H. Evaluating Pacific Grave, CA, 1989.
Norris, Stephen P., and King, Ruth, "ObservAtion Ability: Determining Extending its Presence," Inforlal Logic, v6, n3, p3-9, 1984.
Norris, Stephen P., Phillipe, L. M., "Explanation of Reading-Comprehension: Schema Theory and CT Theory, Teaching College Reading, v89, n2, p281-306, 1987. Norris, Stephen P., and Ryan, James, "Designing a Test of Inductive Reasoning," in Proceedings of the International Conference on Argulentation 1986, Van Eemeren, Frans H., and Grootendorst, Rob (Eds.), Foris Publications, Dordrecht, Holland, 1988 (7). Passmore, J., "On Teaching to be Critical," The Concept of Education, Routledge and Kegan Paul, London, p192-~11, 1967. Patrick, John J., "Core Content, CT, and Civil Values: Issues on Education in the Social Studies," in Trends and Issues in Education, 1986. Flaxman, Erwin, (Ed.), Council of ERIC Directors, ERIC Clearinghouse on Urban Education, New York, Jan. 1987. Paul, Richard W., "Teaching CT in the Strong Sense: A Focus on SelfDeception, World Views, and a Dialectical Mode of Analysis," Infor.al Logic, May, 1982. _______ , "The CT Movement: 1985. _______ , "CT Research: v42, p46, 1985.
A Historical
Perspective,"
National Educational
Farul,
Winter
A Response
to Stephen Norris,"
State UniverSity
_______ , "DialogiLal Thinking: Critical Thou~ht Essential to the Acquisition of Rational Knowledge and Passions," Teaching Thinking Skills, Perkens, et. aL (eds.), Lawrence ErlbaulII Associates Inc., 1987. _______ , "CT in North A~erica: A New Theory of Knowledge, Learning, and Literacy," 115. Center for CT and Horal Critique, Sonoma State U, Rohnert Park, CA., 1989. _______ , "35 Strategies for Infusing CT Principles and Applications," ms. Center for CT and Moral Critique, Sonuma State U., Rohnert Park, CA, Jan. 1989. _______ , "CT: What, Why, and How," ms. Center for CT and Moral Critique, Sonoma State U, Rohnert Park, CA, April 1989. (Draft for CT: Educational l.perative HeN Directions for Co •• unity Colleges, Jossey-Bass.) Pecorino, Philip,
"cr
Bibliography,"
HeNsletter 47
on Teaching
Philosophy"
50
American
Philosophical
Association,
Per kin s, D. N., II Po 5 t - Prim ary'Ed ucat ion Has Li ttl elm p ae ton Infor m aI Reasoning," Journal of Educational Psychology, v77, p562-571, 1985. _______ , "Reasoning As It Is and As It Could Be: An Empirical Pe~spective," Presented to the American Educational Research Assoc., San Francisco, CA, 1986. _______ , KnoNledge as Design: Critical and Creative Thinking Learners, Hillsdale, NJ, Erlbaum Assoc., 1986. for T!achers
and
Quellmalz, Edys S., "Designing an Integrated Instructional and Assessment System for CT SKills," Presented to the American Educational Research Assoc., Montreal, Canada, 1983. _______ , "Needed: Better Hethods for Testing Higher-Order £ducatioDJl Leadership, v43, p29-35, 19R~. Ramirez, Paul H., "Valett Inventory v41, p34B, Dec. 1987. of CT Abilities," Thinking Skills," Teacher,
The Reading
Raths, Louis Edward, et. al., Teachi~g for Thinking Theory and Application, Columbus, Ohio, C. E. Herrill Books, 1967, (2nd ed.) 1986, Resnick, L. W, Education ind Learni~g to Think, Issues National Academy Press, 1987. Lawrance Erlbaum Uorld,
and Agendas,
Ross, Elinor P., "Teaching--Not Just Testing--The v24, n1, p84-BB, Oct. 1984.
Ross, John A., and Haynes, Florence J., Teaching Problel-Solving The Ontario Institute far Studies in Education, Ontario, Canada, 20Bp, 1982. _______ , "Development of a Test of Experimental Problem-Solving Skills," Journal of Research in Science Teaching v20, p63-7S, 1983. Rud , A. G• Jr., Educational
u
CT an d p~. -ColI e9e Ph ilos 0 Ph Y r " Pre sen ted tot he Am eric an e Research Assoc., San Francisco, CA, 1986. Thinking Across the CijrriculuI, Harper and RON, New National
Sabini, John, and Silver, Maury, "CT and Obedience Forul, p13-17, Winter 1985,
to Authority,"
Scherer, Donald, and Facione, Peter A., "A Pre/Post Test for Introductory Logic Courses," Hetaphilosophy, vB, n4, p342-347, Oct. 1977. Schievella, P. S., Critical Analysis: Press, NeN York, NY, 1968. Language and Its Functions, Humanities Station NY.
'.'
5 ~I
4t
=,«:: ~~ ..
'
Scriven, Michael, .
~Critical
for Survival,P
National
, ~Multiple-Rating Items," draft ms., U of Western Australia, Western Australia, Oct. 1988. Sheffler, lsrael, Reason and Teaching, Bobbs-Merrill, 1973.
Shemesh, Michal, and Lazarowitz, Reuven, "The Interactional Effects of Students' Cognitiv~ Levels and Test Characteristics on the Performance of Formal Reasolling Tasks," Research in Science and Technological Education, v6, n1, p79-89, 1988. Shipman, Virginia, Ne~ Jersey Test of Reasoning Skills, Institute for the Advancement of Philosophy for Children, Montclair state College, NJ, 1983. Siegel, Harvey, "CT Nov. 1980. _______ , Educating
as
an Educational Rationality,
Ideal,"
The Educatio~al
Forul,
p7-23, 1988.
Reason:
Routledge,
Smith, E. L., "Effects of Training and Use 'f Specially Designed Curriculum Materials on Conceptual Change Teaching and Learning," paper to National Assoc. for Research in Science Teaching, Washington DC, 1987. Smith, Charles W., "Verbal Behavior and Cla$sroom Practice," 36p, Presented the International Conference on Thinking, Cambridge, MA, Aug. 1984. Sternberg, Robert J., Press, 1982. Handbook of Hu~an Intelligence, Cambridge University at
____ - .., "Teaching CT Part I: Are We Making Critical _ Kappan, v67, p194-198, 1985. _______ , "Teaching CT Part II: Possible p277-280, 1985. _______ , Beyond UniverSity Solutions,"
Mistakes?1I Phi Delta Phi Delta Kappan, Cambridge 19, act. v67,
IQ: A Triarchic Theory of HUlan Press, New York, 1985. of CT Programs,1I Education
Intelligence,
Heel Commentary,
_______ , "Teaching CT: Eight Easy Ways to Fail Before You Begin," Phi Delta Kappan, v68, p456-459, 1986. ______ , (Ed.) Teaching Thinking Skillsl Theory and Practice 1987. Institute of
_______ , "CT: Its Nature, Measurement, and Improvement," Education, Wc?,shington, DC, 37p, 1986.
National
Sternberg, Robert. J., and Baron, Joan B., "A Statewide Approach to Measuring CT Skills," Educational Leadership, v43, n2, p40-43. act. 1985.
4£
,'.I.'
..
..., :~'
.
""".
.. 'i.:
Sternberg, Robert J., and Bhana., K., "Synthesis of Research on the Effectiveness of Intellectual Skills Programs: Snake-Oil nemedies Miracle Cures," Educational Leadership, v44, p60-67, 1986. Sternberg, Robert J., and Wagner, Richard K., Understanding Intelligence: Mhat's in It for £ducators, National Commission on Excellerice in Education, Washington, DC, 82p, July 1982.
or
Ste~art, B. L., "Testing for CT: A Review of the Resources," Rational Thinking Reports HUlber 2, UniverSity of Illinois, Champaign-Urbana, IL, 1987. Stovel, John E., "Document-Based Questions Record, v24, n1, p11-12, Spr. 1987. and Testing for CT," Social Science
Swartz, Robert, "What is CT and Hc~ Can We Teach It?, ms. U. Hass. Boston, June, 1989. Talbot, Jan, "The Assessment of CT in History/Social Science Through Writing," Social Studies Revie~, v24, n2, p33-41, Winter 1986. Thompson, B., and Melancon, Skills," Psychological J. G., "Validity of a Validity Measure of CT Reports, v60, p1223-1230. 1987.
Tucker, Carol B., "Examples of Question Types Used to Test Reasoning," Educational Testing SerVice, Princeton NJ, Apr. 1986. _______ , "Some Multiple-Choice Formats Used to Test Reasoning," Testing Service, Princeton NJ, Apr. 1986. _______ , "Working Paper on Defining CT for Purposes Testing Service, Princeton, NJ, Summer 1988. of Testing," Educational Educational
Van Der Bogert, H. Rebecca, "The Evolution of the Concept of CT: A Literature Review," Ms. Harvard University, 1988. Watson, Good~in, and 8laser, Edward M. Hatson-Glaser CT Appraisal, Psychological Corporation, San Antonio, TX, 1980. The
Weddle, Perry, "CT in California: The Department of Education Testing Program in Social Studies," History and Social Science Teacher, v21, nJ, p14714Q, 151-154, 156-157, Spr. 1986. Whimbey A., and Lochhead, J., "You Don't Need a Special 'Reasoning Test' to Implement and Evaluate Reasoning Training," Educational Leadership, v43, p37-39, 1985.
50
Over course of this research sent to the experts participating College Philosophy Delphi Research
the specific
Tlle eight others were for purposes of planning, providing information, and sustaining
procedures,
involvement.
11, 1988 1, 1988 14, 1988 14, 1988 4 , 1988 18, 1988
28,
ROUND 1 Information ROUND 2 Plans and Procedures ROUND 3 Procedu.re e and Involvement Information Procedures ROUND 4 Plans and Involvement Information ROUND 5A ROUND 5B ROUND 5C Information ROUND 6 and Involvement and Involvement and Involvement and Information and Plans
June
Sept.
1988
1, 1988
Mar. May
10,
9,
Sept. 25,
51
~ :-'~ .. ::-
- . ,,~~.r '
--:Lrj,
.1
Dear Colleague, Gary ~!atthews, Chair of the APA Committee on Pre-College Philosophy, asked me to head up an ad ~oc comrnitt.eeon testing critic~l thinking. To get the project rolling. he .suggested severa names of people interested in the question of ~~w to validly and r~liably test critical thinking skills. I adQe~. a few more. Here are some of the particulars. 1) By using a modified Delphi approach, I think the necessity for actual meetings can be largely, if not entirely eliminated. Co~mittee members' contributions will involve sending their reasoned and timely responses to question~, given their particular background and expertise. 2) Thare are, no 'doubt',a great number of other people who are interested in the question of testing critical thinking and who have valuable expertise which would help us in in dealing with this question. They should also be invited to participate. 3) Since our charge is rather vague, I propose that among the fi~st things we should do is agree on priorities. To do this, using the Delphi process, let me lead off with some assumptions anct questions: First assu~ption: Most of the members of our group will come at the issue of testinq critical thinking with the orientation of philosophers or logicians who teach at the post:-secondary level. rather than as K-12 educators, psychologists, or personnel directors (all of whom also have legitimate theoretical and practical interests in assessing critical thinking). Given the interests of the Amer~can Philosophical Association, this ~ acceptable. Second assumption: Critical thinking can be defined operationally to the extent that it can beco~e a dependent variable in a valid and reliable assess~ent tool.
QUESTIONS:
2. To which educational level (from Kind~rgarten through postBaccalaureate> should the committee give priority? Why so? 3. After looking the attached preli~inary w~uld yo~ recom=end be added to our co~mittee?
rhe Ca"'orn.a Slale l.Jnl~erS'I)'
Appendix
I •
C: Delphi
"~
....~ .
I •. _-,·C
.;;:~:~. . ,) ..
,~
For the Delphi process, which is very interactive, to function optimally, reasonably quick turn around is needed. Let's target a Feb. 29 post~ark. Is th~t possible for you? Please send responses, suggestions, comments, etc. to: Testing
Ad Ho~ APA committee on Critical Thinking c/o Dr. Peter A. Facione Professor of Philosophy and Education Dept. of Philosophy . California State University, Fullerton Fullerton, CA 92634
Like math and composition, at ~any colle~es throughnut the country CT is being built into the curriculum. For example, the California State University system, which enrolls hundreds of thousands of undergraduates, has implemented a system-wide critical thihking requirement as part of its general education package. !f those of us who teach critical thinking were able to agree on a way or ways it could te tested, what a positive contribution that could be to the quality of that curriculum. I sincerely hope you will agree to ~acome an active participant in·what ~rornises to be a most in·t~resting and important effort. I've Ln c Lude d some reference m(lteria~.from Tha r~inth Mental Heasure~an~ Yearbook regarding published instruments which purport ~easure critical thinking and/or related cognitive skills. to
Pete Facione ce. G3ry Matthews, University of Massachusetts, Steve Tigner, University of Toledo, Ohio ATTCHMENTS: Amherst
Preli~inary List of Committee Members Quick Survey of Published Instruments Bibliography on Testing Critical Thinking
TO BE DEVELOFED:
Append1x
'
""
c:
PAGE53
-.
... . ,.
\',
Dopa~ment of
(714) 773-3611
Dear Colleague, tarl¥ :esponses to the first Delphi round are very encouraging! Some of you know what I'm talking about, others, new to the effort, need to be brought up to speed. Let me back track a bit and explain. I am writing to you because you were nominated by one or more of your colleagues as a person intereste~ in the question of testing critical thinking. You are being invited to participate in the work of an ad hoc American Philosophical Association SUbcommittee concerned with the problem of testing critical thinking. In its boldest form, our aim is to find ways to validly and reliably test critical thinking, or find solid reasons why such a goal is not achievable. Using the Delphi process, I will serve as coordinator of the effort. At the moroent we are very near the beginning of Phase 1. Please review the preliminary plan outline~ below and decide to beco~e actively involved in what promises to be an intriguing effort to shed light on an important pedagogical and profession concern. ad hoc APA Sub-Committee on Testing Critical Thinking Plan
Draft Preli~inary
Phase 1: Start the Inquiry The five objectives of this phase are: (a) Initiating the Delphi process. This is a method of achieving reasoned consensus among a group of experts with regards to a given problem or issue. The core of the strategy is to maka inquiries, g~ther each expert's responses and their reasons, then summarize and share those with the group. After "hearing" what other experts think, people ha1e the opportunity to refine their responses or defend those responses. The interactive rounds continue until reisoned consensus is achieved (or communications· break down). (b) Developing the "List of Experts" who will take part in this inquiry. Many of you were nominated by those who replied to my first letter, (2-11-08). That letter was sent to an original group of ~bout twenty-fiv~ experts and interested persons suggested by the APA comnittee that conceived of this project and asked me to coordinate it. A~ lny time if someonE is no long~r int~rested in continuing, just drop ~e a note. I plan to send out ~pdated rosters of participants periodically. Ve are now up to fifty.
5 \. 4
1-'1
rt ....
It.
"I
',lOt •.
· ,.;~~.' .,
'.1
.
(c) Developing a bibliography on testing critical thinking. Work on this is progressing well. Many of you have sent me items to include, and I appreciate that. I will send you a copy of the bibliography later this semester. (d) Planning the subse~~ent phases in our process of responding to the general question of whether or not critical thinking (whatever CT is) can be validly and reliably tested at some educational level or levels. If the consensus is "Yes, at level X," then we will focus on the question: How? If the consensus is "No, at It:.::;t not at level X," then we will focus on, "Why not?" (e) Agreeing on basic assumptions •. In the 2-11-88 letter two assumptions were put to the group: Assumption 11: "Most of t~~ members of our group [of experts participating in the Delph1~process] will come at the issue of testing critical thinking with the orientation of philosophers or logicians who teach at the post-secondary level, rather than as K-12 educators, psychologists, or personnel directors (all of whom also have legitimate theoretical and practical interests in assessing critical thinking). Given the interests of the American Philosophical Association, this [is an] acceptable [orientation]." Assumption #2: "Critical thinking can be defined operationally to the extent that it can become a dependent variable in a valid and reliable assessment tool." People were asked (1) if they agreed with the two assumptions as stated, or whether they would reject them or rephrase them somehow. Naturally, people were invited to explain why. They were also asked (2) to identify the educational level (K-post baccalaureate) to Which our committee should give priority, and why they would recommend that level. Both questions have gener~.ted controversy, as you will see in my next letter. If you haven't had the opportunity to respond to these questions, you will be invited to respond when the first round of the Delphi is raported back to you. Delphi is not about vote-counting, it ai~s at reaching agreement on the basis of reasons and common assumptions. In the Delphi method people are supposed to share their premises, not just their conclusions. As conceived at the moment, our work can be divided into 4 phases. These are not in stone! I welcome your suggestions, a~endments, alternatives, atc. WE WILL USE THE DELPHI PROCESS TO AGREE ON OUR PLAN OF InQUIRY. Because the plan should be. a~ended as a result of your input, only goals, not detailed objectives, have b~en developed so far for the next three proposed phases.
......
58
Appendix C: Delphi Research Letters,
,..,~..-:: '::"~f~,~,: ..
PAGE65
,f,;'r';'-':
...
.
~ ..
,-"-,'
".
Phase 2: Define "Critical Thinking" The goal of this phase is to define "critical thinking" with sufficient clarity and precision to ask and answer the question of whether or not CT can be tested. Is CT fundamentally a set of skills, concepts, procedures, attributes, behaviors, outcomes, dispositions, aptitudes, or what? Even if we canno~ reduce CT to an equivalent operational definition, how might we express what CT is with sufficient operational precisions to permit us to justifiably infer things about the relative CT abilities of students? Phase 3: Recommendations The goal of this phase is to communicate our findings about what CT is and whether there is an adequate way of characterizing CT operationally so as to permit its being tested as some educational level. Depending on our results in Phase 2, we will recommend either that programs aimad at testing CT be abandoned, or:that they be focused in certain ways. If this is the dire~1ion Phase 1 takes, then we will also try to come to consensus on recommendations regarding the relative importance of different kinds of CT sub-skills and possible strategies for accessing and measuring these subskills. Phase 4: Design and Validation of Model Testing strategies Contingent on the results of earlier phases, the goal, if it were considered achievable in principle, would be to construct and evaluate different approaches to testing CT at some appropriate educational level or levels. Ve might find ourselves breaking into sub-committees to achieve this goal, although all work will have to be guided by the agreements reached in earlier phases and as well as by the special e~pertise of those who understand the intricacies of designing, piloting, norming and validating educational tests at specific educational levels. As I mentioned, you have been nominated as person who might be interested in this project and could make a strong contribution to the work of this ad hoc sub-committee. I hope you will agree to participate actively, because, 'as you must know, ti,e quality and utility of our effort is directly related to the invol~e~ent of concerned persons like yourself. '
Sincerely,
~,......:;...---
Pete Facione
5a
Appendix
'-.:.\ . ,<~:~;.~!.:,~ ,
C:
Delphi
Reeearch
Letters,
PAGE 56
'.' ,
.~ .
, ".'. ',' ',"
...•.
.'
'.
"
'
..
~'~'\: ,
. >
.~"
Ii
IIfIIE:Ie
pHS·Wt
3A
::::t=
98'=
California
Sta te University,
92634
Fullerton
Dear Testing
CT Colleagues,
t;'"....a:;!' __
shotl
Thanks for your responses to the first round of questions. Nineteen of the twenty-six or so who received the original 2-11-88 lette: for Round 1 were able to respond. Round 2 invites everyone (which now includes just over 50 people) to review the results of Round 1 and con~ent on the agreements and controversies that are emerging. . ,
, M
Round 1 focused on three issues: (1) The composition of our ad hoc committee in view of the interests of the A3erican Phllosophical Association, (2) the assumption that critical thinking can be operationally defined, and (3) the educational level to which we should give priority. In regard to the composition of our committee, we are in decent shape, particularly since our group has been greatly expanded as per your recom~endations. In regard to an operational definition of CT, we generally agree on the possibility but many would add various caveats. In regard to the educational le7el to which we should give priority, we have disagreements. The following pages cover each of the three questions in turn. You'll find restatements of the original questions and several representative quotes and su~maries of your comments. AfTER EACH SUMMARY, A SECOND ROUND QUESTION WILL BE PUT TO YOU. The new questions take the for~ of stating a position and as~ing your opinion, now that you have had an opportunity to consider what our colleagues have to say. In all, there are three ne~ questions. IF YOU COULD GET YOUR RES PONS ES TO HE \11THHI TEN DAYS OF RECEIVIUG THI S LETTEIh. THAT lJOUL,D ~1 GREAT! (I wish we all had electric Qail, or unli~~ted phone budgets, but •.. ) Several people noted that our task was huqe, yet were willing to give it a try. In contrast, one pernon wrote a major critique of the entire enterprise. This person argued that trying to test CT was a serious mistake. s~ that his opinion is ~ot lost in the shuffle, at the end of this package I have provided extensive quotations from his letter. If you fin~ yourself in agree~ent with his v i evs , t he n let me know and we will take up any "prior questions" we must • .If you don't agree, then we will press on along the path we are charting for our s e 1'I es .
Th 1n kyo
IJ 1n
ad
'J -l n c e
~ pat ion •
,,60,
t' ..,1.
l ·'I~·I
...
'J. ,. ..
1''/
PAGE57
,,, .. ,'tJ.
',r, r_ , ~'I,t:. •
,
".
~.(~
, ,:: ..
',.1"
..
PHASE I, ROUND 2, FIRST ISSUE CURRENT STATUS: CLOSE TO CONSENSUS CONPOSITION OF THE AD HOC COMliITTEE: lie vere asked if we a!/reed v i th the assumption that, al though most of the membe:s of our ad hoc commi ttee would come at the issue of testing with the orientation of philosophers or logicians who te~ch at the college level, this orientation still would be acceptable in terms of the interests of the American Philosophical Association. Almost everyone agreed, however some qualified their responses in terms of our collective professional interests and abilities, or in terms of educational level to which we should give priority. Here are representative responses: "I agree." "I have no problems with this assumption."
"I don't see why this is a problem. First we're concerned about students acquiring the thinking skills required for college work, ••• Second, we're concerned that they learn the standards of good reasoning: I do not believe we need to know a lot about psychology to;achieve this purpose." "I agree, with reservations. We need to avoid tunnel vision. It is acceptable that most members be philosophers, but there should be a generous sprinkling of 'outsiders' for the insights they will bring and to give our findings greater credibility outside the APA." "We ar e what we are! This is an appropr.iate place to begin. lie are starting from what we know best and with what we can deal with most easily. This is not to suggest that we shall forever ignore other orientations, or that we really know that we can define all aspects of CT operationally." "I agree, this is acceptable; but it is unnecessarily many of the tests are created by cognitive and educational think some of them should be included •••" narrow. Since so psychologists, I
"It should not be too quickly assumed that those who t.each at the postsecondary level are therefore knowledgeable and competent with regard to testing at the elementary school level." "I agree, but we should make a r-erious effort to inform ourselves of approaches to CT in pre-colle~e and non-academic settings ••. Assuming our primary focus is everyday reasoning skills, we should not allow college CT instruction to be fundamentally different from pre-college CT instruction nor to become idiosyncratically colored by our own traditions. One person disagreed but did not give a reason. proof-reader!" And one urged "Pete, get a
..
ROUND ~L QUE?TIOtl OrIE: In view of the above co~ments, and in view of the additional names added as the result of your recommendations, can we ~gree that th~ ad hoc comaittae, as listed on the att3ch=ent, is sufficiently w~llconstituted for us to move on with our main task? As you can see, it still has its original orientation toward philosophers teaching at the college level, but it also includas se~aral people from other relevant disciplines backgrounds, including psychology and education.
PAGE58
'~~~,
.,:.
'.
PHASE I, ROUND 2, ISSUE 2 CURRENT STATUS: AGREEMENT SEEMS ACHIEVABLE OPERATIONALIZING CRITICAL THINKING: We were asked if we agreed witb the assumption that CT can be defined operationally to the extent that it can become a dependent va.riabl in a v'llid and reli.able essessaen t tool. e Here, too, most people agreed and were ready to get on with the work. some crucial ambiguities, concerns and caveats emerged. Here are some responses. Yet,
"I expect we will argue about the details of any definition proposed, but I do not object to the assumption that we shall need some such instrument if we are to get any comparisons of interest." "This is a tautology because of the to the extent phrase. how we should leave it •••" Perhaps this is
"Sounds ok. •••I'm not a st a t i st i c i an. so I'm not quite sure what dependent vari~ble means -- but if you'are asking whether CT can be tested, then, yes, I agree!" "I don't understand what this assumption is supposed to meanl"
"I would agree only if we amended it to say at least some components at CT can be defined operationally to the extent that they can become dependent variables in a valid and reliable assessment tool •.• I do not accept as analytic the proposition that CT can be defined operationally •.• I think some important aspects of CT, such as making judgment calls and weighting nuances may resist operational definition." "As the term oper'ltional definition is generally used by philosophers and edu~ation researchers, I do not think CT can be operationally defined •.. but I do think that part of the operational spirit can be employed in formulating reduction santences (that do not reduce!)." "I agree, but there will probably have to be a variety of sub-definitions because CT is not one thing, but many. It somewhat resembles IQ in that.
II .:',',
"Thera are several definitions of CT themselves more to operational definition anywhere, we will have to become clear in distinguished from other kinds of thinklng "Is this a normative, definitional,
floating around ..• Some lend than others •.•. If we are to get our own minds as to how CT is to be •••" or planning assumption?"
conceptual,
ROUliD ~ QUESTION T'.IO: W'ithout hanging ourselves up on the word "operations," can ·,.,e agree tha t: (1) Even if CT cannot be reduced en t i rely to an equ i valent set of opers t ions (or performances, behaviors, processes, outcomes, or skills,) (2) it is possible to conceptually anJlyze CT so as to describe a set of relevant and important CT operations, such that (J) ~sing these descriptions, (competent) investigators could, on J consistent basis, gather sufficient evidence to draw conclusions, with high degrees of confidence, regarding the relative CT abilities of a group of people, [everything else being equal, of course) •
,~ u
')
"wi
PAGE59
........
~.
. ,v~~.
.
PHASE I, ROUND 2, QUESTION 3 STATUS: CONTROVERSY
"
';
PRIORITIES: We were asked which educational level (kindergarten through post-baccalaureate) should be the priority for our committee. And why so? Responses were split. Here are some representative examples:
A person with considerable experience in the area of C'l testing wrote: "I think we should concentrate on high school at first. Since t~is is a sUbcommittee of the pre-college committee, all [levels] above that [are] ruled out. Furthermore, the younger the population, the more difficult the problems. Let's start with the easiest ones first -- and they are very difficult." By contrast, it was argued, "Priority should be given to the post-secondary level. One should examine end-products first, and then work backwards if needed. Find out fi,rst if the car doesn't run before attempting to determine whp.re the problem is. If a good test of CT revealed no CT deficiencies on the part of graduating seniors -- (no doubt a counter-factual assumption) -- then I would think the APA might not wish to pursue the issue down in K-12." Noting that we are a sub-committee ot the APA pre-college committee one person argued: "\Ie must give priority to K-12; that is our mission." However, the person who will assume the chair of the APA pre-college committee for the next three years wrote: "It makes sense to start by playing from the APA membership's greatest experience and strength, which is surely college freshman level logic." Taking note of the interests of the AFA, one person argued, "Since our ad hoc committee is convened within the structure of the APA, our focus should perhaps be primarily on the improvement of post-secondary education .••" However, this person also suggested, " ••• that our assessment tool should be usable in secondaz:y schools as well as at the post-secondary level. ••
II
Some people did not offer an opinion, but did note important distinctions. For example: "There are really two areas. One is the whole K-12 integration of thinking skills into the curriculum. The other is the single CT course, typically the approach followed in post-secondary education. The single college CT course offers exceptional opportunities for measuring gains in thinking skills, while the effort to incorporate thinking skills into the [K-12] curriculum may offer much greater potential for actually improving student skills." Another person, experienced in the pre-college arena, wrote: "It may be necessary to think of four tests, one for grades K-3 (one should not have high expectations for reliability at this level); one for grades 4-8 <the level at which testing might have the maximum impact, even though the maximum impact for the teachi.'lg CT might be at K-3); one for grades 9-12, and one for 13-16." of Some were tan t at Ive : "Perhaps we should gi';e priority to CT at the college level, at least to start with, since the large ~ajority of APA members telch at the college level. Later we might wish t o broaden our focus."
t)J
"t ~::. ,
00
'.' ,~'.~:-
.'
.;.~.
..
Some were direct, "College and university level." others were focused, bUL concerned not to overlook anything important, "College level -- but someone ought to look at the high school level." One person declared for the college freshman level and ~rgued against going any higher saying, '''rhereare few if any thinking skills pos ses+ed by people beyond this level not also possessed by well-prepared college freshman. Graduate school and professional life chiefly consists in the ability to persistently apply these skills in more and more recondite subject matters. Some narrowed the range, but still left us with a choice: grades 9-12 and freshman/sophomore level in college." "I would say
Another argued we should give priority to the introductory baccalaureate level saying, "First, it is the area where most philosophy departments have numerous classes actually being taught. Second, it is taught at a level which will have the nest connections in other areas and at other levels." To get started let's give priority to the college Do you agree? If not, is your disagreement based on pedagogical and theoretical concerns or on concerns relating to our charge as a sub-committee of the APA committee on pre-college philosophy?
freshman/sophomore level.
1'hanks.
6~i
Appendix C: Delphi Research Letters, PAGE 61
. r\
.:".~-"' : -,
_-
."
".'.'
....
,~
·,UI· ._ , .. ~ ..
."';'.1".
"
PHASE I, A CHALLENGE RAISED AGAINST THE ENTERPRISE One person wrote: "I would like to comment that teaching CT skills ••• is a far mote significant matter than testing for them. Since CT is not an inherited trait, as is intelligence, the quality and extent of the CT is dependent on and proportional to the degree to which chi'dren have learned to, or have been taught to, think critically about their experiences and knowledge. "Our educational system ••• is an abysmal failure. Most students ar e unable to recognize assumptions, not alone question or examine them .•.• Yith rare exceptions, they show an almost total absence of recognition of even the simplest of 10gical/mathematical/linquisti~/philosophical/scientific facts and concepts needed to be able to think critically. They have been so nurtured in a world of superficial "education" ••• that to think of testing them on the basis of that "education" is an exercise in futility or at best an attempt to determine how inadequately they think critically as opposed to how much -- which if taken literally amounts to the same thing. "I am willing to contrirute my expertise to the teaching of CT. I have been doing so for over thirt' years. But until I see considerably more evidence of students being able to think critically without such teaching, I s~p. little point in testing them for the insignificant amount of critical acuity they may have acquired haphazardly. " ••• Teaching CT must precede and supersede testing for CT. Testing for CT cannot be cons~dered to be an enterprise separate from teaching it. Testing for such skills and concepts presumes prior teaching of them. Psychologists and K12 teachers as well as other educators show interest in CT. The problem remains, however, that most of them have only superficial, naive, and conflicting concepts, of what critical thinking entails. Even we philosophers can't agree on what it is ••• "From my understanding of the term, testing for CT means testing based on what I teach CT is •••• Any tests that I would, and have designed, are predicated upon the version of CT I have taught. "My co~ments will undoubtedly reflect those which you will receive from my colleagues equally concerned with the problems facing our educational system, particularly as they relate to the teaching of CT skills and concepts." REMIUDER: If you believe there are issues (suggested by the above or otherwise) which our ad hoc sub-committee must address before we can m01e ahead, please let me know. On the other hand, the above challenge may represent a view which is not widely shared, or may raise questions which, in your view, do not fall within the scope of our work or do not warrant our attention at this time. If that is the case, then, for the present, no response on this item is necessary. Thanks again for 10ur participation .
,. ,-t..(e
-
/ »-"
. .1. '
»: -...
.....
PAGE62
65
•, _,.'-'. ~ 11·'-'
,.,:,,~.: ...-,.'
!'In .
...,
Apr i I 14,
1988
Dear Colleagues, Thanks for your help with the addresses. I also appreciate the notes, suggestions and other helpful comments many of you are sending. So far 17 responses to RJund Tvo have been rec eived . It would help if we had more, particularly since it there may still he a split over Question Humber 3, the one:about which le~el to pursue first. Drop me a note, with your reasons and opinions. Thanks. One of the responses to Round Two proposed an alternative to actually trying to come up with our own CT assessment instrument -- a goal some of us hope to achieve, but others of us are extremely skeptical about. This person suggastad that we articulate the best list of CT skills we can, then let people go their own way with regard to building testing instruments. I'll expand on that idea when I sum~arize Round Two responses. I Mention it here because perhaps we all should be thinking ahead and trying to chart the most reasonable path for ourselves. Some of you ha~e electronic mail. I don't, at least not yet. So, Illl be using snail mail and talephona to try to reach you for particular questions or clarifications. If you want to phone in your views on the questions in Round T~o you can reach rne at 714-773-3742 (office) 08:30-10:00 HIJ or 09: 30-11: '0 TTh (PDT). If those ti~es are incon~enient, call the depart~ent secretary at 7l~-773-3611 and lea7e your phone number. 1111 get back to you. I ~ill be attending the First National Conference on Assessing Thinking in Baltimore on May 6 and 7. This confarcnce is sponsored by tha Maryland State Department of Education and the Association for Super~ision and Curriculu~ Da~elop~~nt. Although plans are to discuss all educational levels. the ~articipating organi:ations, (over 35 professional associations. centers and qo~ernment agencies) ~re concerned pri~arily with K-12 education in so~e way or anothe~. I'll report on what pro~i5~s to be a ~ost interesting gathering. ~lthin th~ ~aek you should racei~d two 1te~s intsr~3t. One is a plrtial bibliography CT with emphasis on t~sting CT. the other 15 an upd a t ed listing of sone of the ex i s t i ru t-:sts vt.xch purport to =easur~ CT or closaly r~llt~d r~asoning s~llls.
s i nc e r ' ".
2Z~~
",
i.,J
-".'
•..
~.'
....,.
E __
p:
#4
#4#*'96'
_ide±;
:d5J'f::£!IJUIiI:IJ:aAMNf>$!UE
'u,
'''I'
California
Fullerton,
CalifornIa
RO(/NI>
May 4, 1988 Dear Colleagues, I hope this letter finds you happy and well. It's time for Round 3 of our Delphi -- the round where we finally get to the heart of the matter: What is CT? Also, ;,or those who are new to our effort, this letter includes a brie~ overview of who we are and what we are about. The last three pages summarize our Round 2 results. Please send Round J respon~es by June 25. As a result of nominations in Round 1 and Round 2, sixty persons, inclu1ing some of the most eminent names in the field, are now invite~ to participate in this effort. By way of bQckground, in January the American Philosophical Association Committee on Pre-College Philosophy asked me to chair an ad hoc sub-committee on testing critical thinking. Beginning with an initial group of APA nominees and asking them for additional recommendations, the "sub-committee" has grown to include people from a variety of academic disciplines and professional affiliations. Our unifying concern is in testing CT. However, we do not necessarily share the same conceptualization of what CT is nor do we necessarily agree on how it might best be tested. It is to resolve precisely these two things that we have undertaken the Delphi process. In Round 1 (Feb. 11, 1988) and Round 2 (Mar. 14, 1988) the focus was on establishing group membership and agreeing on preliminary working assumptions -such as the assumption that CT could be operationalized to the extent that valid and reliable assessments of importa~~t and relevant CT skills could be made. Starting with Round 3 we will focus on what those CT skills are and eventuall~ we will decide on recommendations regarding testing, based on any Delphi consensus we achieve. To assist with the conceptual work that must be done, I developed and circulated two items, a list of existing CT tests and a CT-Testing Bibliography (Apr. 19, 1988). If you want copies of any of materials mentioned or if you wish to have a copy of the mailing/membership list, just drop me a line. Last week I presented a workshop on testing CT at Sacramento State. At that time Perry Weddle agreeci to publish the CTtesting bibliography and the list of CT tests in a fall issue of CT News. So, please get any corrections, additions, or deletions to me as soon as you can . AUD llOW OU TO ROUliD 'l'HREE! Appendix C: Delphi Research Letters, PAGE 64
",
****
QUESTION:
ROUND THREE
****
Response requested ~
Jun£ 25.
The sole task for Round 3 is for each of us to come up with that list of operations [or p~rformances, behaviors, processes, outcomes or skills] which we understand to be at the core of the concept of Critical Thinking. Although many of us have published on this question, since there are potentially sixty in our group and since I have no assistants, it will be most helpful to me if you would take the time to distill your views and send a list of what you interpret the central CT operations to be, fell free to indicate which are the more gelleral and which are the sub-operations. Naturally you are welcome to include justifications for the items on your list. In thinking about this, please keep in mind that in the two preliminary Delphi rounds w~ hav~ narrowed our focus for now to CT '~opera tions unders tood as performances behaviors processes outcomes ~ sk~lls which could be tested validly and reliably at the college freshman/sophomore level. But keep in mind that we very likely will extend the question downward to K-12 later.
II
I I
In Round 4, which r will initiate in sept., you will be given combined lists and invited ~o comment on the wisdom of excluding, retaining, or amending the descriptions of specific items. If the results of Round 3 are clean enough, Round 4 will also invite you to begin rating items in terms of how ~ore or less important, crucial, central, integral etc. they are to the concept of CT. If you do not intend to respond to Round 3, for whatever reasons, please drop me a lin@. so that I can keep track of participation levels. Matt Lipman suggested that we might have -an excellent chance of working with the APA and the Assn. fer Infor~al Logic and CT tc secure the use of the Wingspread Conference Center. Please let me know if you think it would be productive to get together in that setting. How might a conference be organized to most effectively use our valuable time? What kinds of issues, problems, tasks might we address? What kinds of solutions or desirable results might we achieve by meeting which couldn't be achieved (at all or as well) using the Delphi? since Round 3 asks the "big" question, to respond. please take the time
Sincerely,
-;:;~~
PAGE65
68
SUMMARY OF RESPONSES TO ROUND TWO Overall response rate: 23 of a possible 51. Question! 9i Ro~nd ~ asked if cur sub-committee was sufficiently well constituted to move on with our task. As you can infer, the consensus answer was "Yes." A small number of additional names from psychology and education were recommended and strong cases for adding ther were made. That is how we moved up to sixty members -- presuming the new nominees agree to join in. I'll send you an updated list soon. Question ~ of Round l asked if we agreed with this claim: Even if CT cannot be reduced entirely to an equi valent set of operations [or performances, behaviors, processes, outcom~s, or skills,] (2) it is possible to conceptually analyze CT so as to describe a set of relevant and important CT operations, such that (3) using these descriptions, [competent] investigators could, on a consis tent basis, gather suff icient evidence to draw ecncl usions , with high degrees of confidence, regarding the relative CT abilities of a group of people, leverything else being equal, of course].
II
:.:~
(1)
II
There was sufficient consensus on this to move ahe~d. The m3jority of responses (18 were strongly to moderately positive, 2 were negative and 3 did not respond to this question.) However, to ?ovoid mis unders tanding, let us keep i.. mind tha t the above statement should not be interpreted to imply that constr~ct validity can be determined strictly in an a priori manner (by simply coming up with our list). Nor should the words "relevant and important" be interpreted to imply that we can come up with an exhaustive list. Nor should the purpose in (3) be interpreted to mean that we have set our sights on actually writing a CT test, or, for that matter come to any agreement about what recommendations regarding testing we are likely t~ ~ake. Most of the positive responses (13 of 18) were Vf;.ry short, "Yes," or "Agree," or "Yes, this is an acceptable working hvpothesis." Here are two of the longer positive responses I found interesting: "I c1gree with the statemcant offered. I am compelled to note ttl'l' the tasks described in the statement are going to be quite difficult to c~mplete. There will nnt be universal acceptance. There will be criticisms (legitimate and illeqitimate) cf the results tor a long time to come and many will .un a course similar to criticisms of attempts to defirle and measure intelligence." 'I(J) V~S. pvpn thnuMh ~.~ -- ~~~~:h ! ~~li:~~ ~: :~= ~:~i:7~ ~ ~roadly sa tisfying reduction... (2) agreed here, (J) agreed here. Theref ore, yes. And rather than get hung up on this questivn, I'd prefer to jump right in and see whether we can do it. If we can, terrific! If we can't, well, then the doubting T's will have a field day; but I'm prepared to take that risk ••." One of the n"g..ltiveresponses was expressed this way: "I'm. sorry, but I can't help but qet hung up on 'operations' -- the term so psychologizes ard jargonizes ••• the question. Why not go for Iprinciples'? ••" The strongest negai;ive w~.;. registered by a Derson who said, "I fear this
Appendix
:'':''
',.:
< ;~~ "
c:
69
PAGE66
<,»,
.. ...
•
",:',\.
,:'1>
,~
..
,,(,>
. ~! II
proposal to define CT operationally may have the effect of ruling out, by definition, one major position on CT, a position with which many in the CT field mali' in sympa thy, at leas t to some degree." This' per con was concerned be that defining CT as a list of skills would focused on weak-sense-CT and missed stronq-sense-CT which relates to a person's character -- "being self-critical, sPJkinq to overcome blind spots, biases, prejudices .•• , [being] critics of une's society, ... [seeking] what is of value in auother's position •.•" The person asked, "Can these character traits be defined operationally?" In con tras t, note this rnspcnse t "Yes, (1 agree wi th the statemen t] particularly if we confine ourselves to abilities and ignore dispositions ..•" Even though agreeing that "A subset of the processes that constitute CT can be assessed using the multiple-choice format that I assume is being sought," one person expressed serious concern saying, "I fear the creation of an instrument promising more than it can deliver -- an instrument touted n~t for what it is, an assessment device measuring certain important, but rudimentary, CT activities, but rather as a valid and reliable assessment tool for CT." He maintained that "As is so commonly true when discussing assessment, the instrument and its characteristics would then dominate the social construction of what is being assessed, in ~his case CT. The initiation of CT activiti~s, generation of appropriate CT strategies, and defense of a tentative reasoned judgment are.not susceptible to the type of assessment legislators are willing to finance or faculty are willing to undertake. The extensive writing or oral argument required to demonstr~te CT, as I understand it, are not practical inclusions in an assessment instrument." Four who agreed with the approach mentioned important factors which relate to co~struct validity. One mentioned the role of background knowledge in CT, another the role of divergent assumptions, another the relationship of CT skills to reading skills, and a fourth spoke to the need to validate any list of CT operations we might agree. There is much in the research literature about these problems. Steve Norris, in particular, has been working on strategies to respond to precisely these kinds of problems. Although I mentioned all the negatives, the positive responses were far more numerous than the negatives. A consensus to move ahead exists. But we must not forget the warnings and concerns of our colleagues. A great deal depends on what we ~ome ~ with when we actually sit dowll to an§wer the question for ~ound ~ since both the positives and the negatives were based on our ideas about what CT i~. Rou . ~ question I asked if people would be willing to agree my proposal that to get started by giving priority to the college freshman/sophomore level. The responses ran: 14-yes, 3-no, 2-both, 3-abstain, and one that I could not figure out. Since the question was about priority and was not intended to exclude working at the K-12 level, which is, after all, what the APA Precollege Committee is charged with doing, I believe we have sufficient consensus to focus initially on the lower division post-secondary level. Here are some "Yes, give priority to the frosh/soph level" comments: "Most if not all of the CT we teach is directed at this level" "I doubt philosophers should take the lead .••when it comes to K-12." "I'm still unpersuaded to reverse my form.erlv expressed views -Appendix C: Delphi Rese&rch Letters, PAGE 67
"0 ., ,
'">:
.. \:
'~'
\,j
c
,10
',,-,:.:._~_
-.._.'S
":.'-
~E--//:
- -." I,~_:
--
~~.~,
, ,~
intro. undergr.aduate level first, until we get square there, where we live." "I feel rather strongly that we should begin on tho post-secondary level. I agree with those who say that that is where the strength of our membership lies and that that is where the vast majority of our members teach." "Yes. That's where most of the pedagogical action is; that's where the students we're interested in testing and have relatively easy access to are to be found. Later, if we succeed at all here, we can extrapolate to other levels." I agree in the Iigh t of wha t was said in Round 1." "Why not? It's what we know best, and we can always move on to other levels later."
II
'On the negative side: "[Your] recommendation seems arbitrary and not consistent with the fact that this is a pre-college committee." "No, based on the name of the committee." "I would prefer to begin at the K-J level. My posi tion and opposition is based upon pedagogical and theoretical concerns which I assign a higher priority to than to political concerns related to the officers and membership of the APA •••• I am willing to accept that the sub-committee begin with the college level but the project will have to be extended downward then •••" Two people expressed the concern that this was a difficult question to answer until one knew the purposes for testing. [A point well taken.] Another suggested that our goal should be to make "contrihutions to the criteria for a college test ••• [but that] individuals should be encouraged to make up their own [assessment inst~uments] and try them out, obtain correlations with other tests and with outcomes, and then subsequently co~pare notes with one another as to what worked and what didn't."
Thanks to all who responded to Round 2. '{our letters were most interesting and thoughtful. Don't forget -- Round 3 by June 25!
·~tK~ ..
.
;-(.-~-
71
~.\
to
:f~.~~·;. :.\
;"
.~
May 18, 1988 Dear Colleagues, I hope this letter finds you well, happy and looking forward to a restful mentioned Conference as well a~ productive su~~er. In my last letter I
in Baltimore
The conference
report is attached. the current list of persons invited to We will have to freeze
to try to add people who have not the benefit of With the circulation that point . of tha
. Many thanks to those who have already sent there responses to Round 3. Don't panic if you haven't yet, though. The target
Sincerely,
.....
.. ,
'7'~ i:.
• •.• • • _ •. ,. -_ •. f'
. . .... -'
.
California State UniveraJty. Fullerton
Fullerton. california 92~
_ ,-'-;r .
: v-
-,.;--
~~~;'::~".
.:
Department of PhilosophV
(714) 773-3611
Dear Colleagues, Thanks for the large number of interesting responses to the Round 3 of our critical thinking Delphi process. Judging from
the length and sophistication of what many of you sent. it should take me a fair amount of time to organize and synthesize the material and t.hen to frame fruitful questions for our next round.
r expect to
3.
some reason you haven't had,the opportunity to respond to Round please know that your ideas are more than welcome. Have a good summer and thanks again for so much high quality participation -- it's very encouraging.
Sincerely,
Pete Facione
, .:..c-: ..~
a-
=;» ... ~-
-';'.'
· .-_J.
nia 92634
Onpartmeut of Philosophy
(714) 773-36'11
1.-
s e p t , 1,
1988
Dear
Oelphi I'm
Colleagues, through skills. Right play letter the The now them 25 responses quality analyzin&, for two your weeks. is to drop whatsoever Dolphi she rounds. not persons after ~ff the Delphi to Round and care 3, which evident a~kcd fer your responses the
working ~ore CT
list
of
in your and
organi4ing
rectification.
the
Round
to at
sustained here
silence
wishes shared.
to be It (c) lists
is somethin~ Logic
Texts,
Logic and
Texts,
to CT,
(~) Bell, of
OrganiZations. College,
want
21044
Columbia,
Maryland
Maryland,
~~
!~~ igr
how of CT Is
Scientists.
Includes backgrounds
a wealth and
material lists
seen and
from
different
disciplinary
a number
ideas
resources. Thanks ogain for so much be high hearing quality from partiCipation me soon. in Round 3 -it's
very
encouraging.
You'll
Appendix
The California
"-
c:
Pete
Facione
PAGE71
Slale Uniyersily
",::~,~-
74
L")epartmentof Philosophy
ROUND ~
. .--,
l ,
Dear Delphi Colleagues, Round 4 seeks verification of a list of CT skills and sub-skills. Please accept, reject, amend, and comment on the group's responses to round 3. Remember, the goal in this phase of our project is to arrive a't; n accord regarding the skills we understand to be central to CT. a Your responses to round 3 yielded 200+ ~ages. Some sent previous publications, some sent lists and commentaries, some wrote new pieces of clearly publish~ble quality. ,You tended to approa9h the question of identifying core CT skills and sub-skills four ways: (a) by appeal to your own experience and understanding, (b) by citation and comment on what other CT authorities (including others in our Delphi list) have written or said, (c) by describing the key characteristics of persons who have internalize CT, and Cd) by consideration of what should reasonably be taught or included in a CT curriculum. In addition to differences of opinion, there were' variations in disciplinary orientation, vocabulary, and emphasis. There were also differences in the specificity, depth, and scope of responses. Some were extremely general, others very specific. Distilling your 'opinions, positions, views, ide~s, lists, descriptions, explanations, examples, counter-examples, caveats, . credos, and course outlines was one of the most intellectually interesting and stimulating experiences I've ever had the pleasure of attempting. Although I've been '~lching and researching CT for two decades, I noticed that my ~wn vi~ws on the range and character of CT expanded greatly as a result what you contributed in round 3. In naming and describing CT skills for Round 4, I intend to rely on standard English usage and to avoid technical or disciplinespecific vocabulary. Your responses emphasized generic skills. Distilling your responses, I name and describe six generic CT skills and give two or three sub-skills under each. Clearly additions or deletions might be needed. To avoid prejudicing your responses at this crucial time, I do not indicate the numbers of persons who may have agreed on any given point. Areas of agreement and controversy will come out in round 4. In this round you are invited to make a number of kinds of responses to a variety of questions. However, because organizing the material and framing the' issues was such a delicate and complex task, I urge you to read the whole package and get an overview of the terrain prior to starting to reply. Thanks again for the high quality responses to Round 3. To insure we are talking· about the same things when we refer to CT skills, we'll need maximal participation in round 4. If you could consider this material and reply in 15-20 working days that would be wonderful. If you need more time, or want to discuss any aspect of this project, call me at CSUF (714) 773-3742, [office] or 372-3611 [dept.], or 993-1356 [home]. SincE:rely,,........-} [J '1 Appendix C: Delphi Research Letters. PAGE 72
_"',"'~-"
>~;)';.
"_
-_
::'
..~.~.
".
,"
.. ./ ....../.' ..........>. , .
. ..
.... ~"_
.....
--._.
,...... , ""'"'-'
.....
'
RESEARCH
IN PROGRESS
-- ~qr ~
PUBLICATION
PAGE 2
are five parts to Round 4: (1) The listing of CT sldlls and sub+e k i l l s • (2) A model diagramming the flow of CT skill input (3) A list of things some of you said CT is not. (4) A description of what is meant by "s k i l l ". (5) A list of caveats and comments you shared. Round
and output.
11 s
Com men t : I" e ' v e all see n e ): amp 1 e s, 1 i' e the d u c I: - r a b bit " 0 f how com pre 11ens ion I~ combines imposing an order on reality as well as discovering an order inherent in reality. Applying that lesson to the problem at hand, there are many ways CT skills and sub-skills could be organized. Even among those of you who essentially agree, the variations in your responses to round 3 illustrate this. After considering a large number of ways of organizing things, I've decided on the configuration you will find beloH. True, the skills and subskills could have been named using other labels or grouped in other ways. To this add that so me sub - 5 Id lIs ma~' c·om e i n top Ia yin m 0 ret han 0 n e .9 en era 1 area, some can operate on the products of others, some presuppose others, some are almost always employed while others may not come into play except under special circumstances, and you have an even more conceptually complex situatio~. Now add that some of us might· exclude one of the more generic groupings, to say not~ing of how we might want to add, su~tract, amend or rearrange the sub-skills, and the complexity of our task takes on greater magnitude. There were other ways to "see" the Round 3 data. So, beside asking yourself if each skill and sub-skill belongs in the list, and if anything central is missing, another question to ask is, like the duck-rabbit, can you see CT this Hay? Ins t r u c t ion 5 : Be l a N y ~IU irJ 1lIT j n d a· 1 l $ t n a 0 I n 9 and d e So c r l b j r, 9 U..!. '0' e C T skill~ and sub-skjlls. Read and cons11er the e~tjre Ijst, Then, ( 1) Co f1 sid e r i.!Lct ~ c h ~l: 1 1 uu: $ u b - s k 1 1..L,.. tM.r...L. tl.~.l!.::t_e..§: ti. ~9.J:L ylliJ.1.. . j Lt te. Ql. 2. ~U U 5 I: ill 2.!.. sub - s I: i.ll. and ~ Lf.. ~ I'f 0 LII dar 9 LI e ~ e l< r. 1LI d e lL... §.!i\te ~oLlr r..g_~ for ~luding 9111.. marl:ed :liB..::... ill "I a 1!1_ Qll2£.B.. 5 LI b s tan ti "e CI men d m e n t s tQ. the des c r J p tin. n s Qi.!!!.}:_ sub!k i..i.LL 2!1_Q.. e:~ 1..ll!l. tt!L'i. the .ch CI n q e i..§.. nee d e d..!.. dd !!l:t.. m iss i n q LU b -! i. J II! p A n am e_ Ws. ~ 5 c: r i Ite.. lli 1 0 cat e_ U. "I i t h in gru.. Qi the s i.lLU 5 I: ills. ~.!l.S ide r in g tb.. ~ ty tt - s k.111 $ \II i ~.bilL e CI. c h.. Q..i. t h e ?jlL U a I: ills! i n die ate H h i c: h sub - s I: i 11 5 LL g.!l.Y...1. a!lP.lli...~.~g_ ~.2.:!e d ~B.. aP...tt 0 tlL~!.5 I: i l.L_ ( 3) AIn e ns. the Li~i. Qi. silL Q. ! k i 11 ! • !.i YJUL del eli. ! s I: ill! i n die B t e ~c:,1. to. d q_ w iJl!.. its sub - s U IJ..L.. !i. ~9..!:!.. add s.. m iss i n g rr s I:i II! ~. and ~!?ic_ri beLl\.. QMlt c3 n d. des c: rib e Ui? u b - 5 k ill S, e :( 1 a i n h 0 1'1 2 1 see n t r a 1 tQ.. C T.. h 0 ~I il. d iff e r 5 fro m 2!!L Qi. the ail:.. 5 I: ill s Q.!l the cur r en t 1 i s t! and ~ \.. L.t m e r its b e iM 1 i s ted.. tl. the illtl. Pi. 2. ? I- ill rat her t han ~ 5 U b - s I:: ill. ill ~n ~j de r th ~. ~I g_ reg a r din 9 the r ole and £..Q..!n.P0 5 i ti 0 n gJ_ ~ hum an· s 0 aT I: n ~I PI 1 ssss: QiJ.1...L.. v e r i f L.. a men d \.. com men t QlL 9..!li:. asp e t: t Q.i t his. 1.§l_ l'Iak_g_.n:L needed edi torial chan~ e 1.6 ) C q m men t, u.~. LI 1'1 i.~_Ql!. the en t ire !J...a1. Q.i ski lIs, its 0 r g ani ;: t ion 0 a
I
u..
\:LliJ
it
'I
llil.U.
b.ll§.. ~
1 urge ~'ou to read the entire list of skills and sub-skills before beginning to respond to any specific item. Thanks!
'18
<>:
...
..'~'-,.
'
RESEARCH
IN PROGRESS
Core
-- ill
f].R.
PUBl.ICATION
Skills
PAGE :3
PROPOSED:
Critical
Thinking
TAXONOMY
~:;.E:,:prr-=,ssi ng 6. l'lord"t or i ng
R~gul~ting,
Conj e c t LI r i r. g, L> r i;WJ i n q Cor If': 1 LI S ion !-,,s • Loc a tin 9 (~'''IJL\fIlen t 5, F','oU' sin 9 (:",. urnen t~:;. g VElrifying Clc:\ims, Assessing Logic"l Strength. St ~ tin 9 n e S u 1 t s , Df.~ S c rib i n q F'r"0 C E.Hj I.W ~ S •
(.!LH~r
Observing,
yin g
Dscoding,
Clarifyinq.
Reviewing,
Correcting.
DESCRIPTIONS
YES/NO
1, IN T E R PRE.TIN G: T" C (I m pre ben d the s i 9 n i i i c a fl ceo l' a .. d e v a r i e t y 0 f Ii experiences, situations, judgments, beliefs, rules, procedures and criteria. 1.1 OBSERVING: To detect, attend and correctly perceive experiential input with particular focus on input that ~onveys or is intended to convey data, information, or inferential relationships. 1.2 DECODING: To detect, attend to and torrectly percelve the informational content, rules, procedures, criteria, and inferential relationships expressed in various c~nvention-based communication systems, such as language, social behaviors, dra~ing5, numbers, signs and symbols • • 1.3 CLARIFYING: Ta make explicit, through stipulation or description, the contextual, conventional andlor intended meanings of ijords, ideas, concepts, statements, behaviors, draWIngs, numbers, si~ns or symbols; to remove confusing vagueness and ambiguity; to facilitate communication.
2: INFERRING:
deteraine the
T~ secure infere~tial
aDd tQ
irol
statements,
descriptions or representations. 2.1 QUERYING: At any,point in the CT process, to r e c o qn i z e the need for evidence or information of some kind, and to formulate and execute a strategy for see~ing and gatherIng that eVIdence or' information. 2.2 CONJECTURING: To formulate alternatIves, to develop hypotheses, to postulate suppositi~ns, 2.3 DRAWING CONCLUSIONS: Given a set of statements, descriptions or representations, to educe their inferential relationships and to educe the consequences Hhich they support, warrant, imply or entail.
7~i
• ,:...;f1'; -. t~'
'.:'".
APA Delphi
Project
RESEARCH
IN PROGRESS
It
-- ~
trui PUBLICATION
PAGE 4
,.'
NOT ~: A 5 man y 0 f y 0 L' a r 9 LI ed, all C T s I: ill s, but par tic u 1 a r 1 y sub s ki 1 1 5 1ike 2. 1, 2. ~ and 2. 3 pre s un e u ~ n 0 I~1 e d q_e- bas e • A hum an' s knowledge-base i~ c~mposed of at least these things: (a) a ~I0 rId If i fHI \,1 h inc Iud e son e 's u nd e rs tan din gsa f hie ~Ihat i s rea 1 (a met a p h Y 5 i C 5), • .
*'
* how knowledge is gained and refined (an epistemology), • what is important or valuable (a value theory); (b).a lli..s.. b a s e , including one's cp i n i cn s , beliefs, experiences, ~tc. as filtered through the world view; (c) an inferlt!J..U.engine which includes * general rules for drawing logical inferences (a logic), * sets of procedures and criteria appropriate for making reasonable judgments within specific areas of human thoLlght and inquiry (discipline-specific ~ule;.)
Executing sub-skills 2.1, 2.2, and 2.3 as well as 4.1 and 4.2, 'more effectively can be achiev~d by learning how to think logically, by expanding one's repertoire of sets of procedures and criteria used in different areas of human thought and inquiry, and increasing ene's base of relevant data. An implication of this analysis of CT skills and sub-s~ills is that they transcend specific disciplines, but executing them demands background knowledge, some of which is specific to how one goes about making reasonable judgments in different realms. Becomin~ adept at CT ,involves learning CT skills and learning to use those CT skills more effectively in different contexts -- hence the importance of a liberal education to go along with one's CT ability.
*
J. ANALYZING: Tu ide~tlfy
the inierentlaJ ,~JationshJPs betNeeh st,tements, descriptions or representations which express experi~nces, situations, judgments, beliefs, or opinions. 3.1 LOCATING ARGUMENTS: Given a set of statements, descriptions or representations, to determine whether it does e~press or was intended to express a reason or reasons in support of some claim, opinion or point of view. ~ 3,2 PARSING ARGUMENT5~ Given a the e~pression of a reason or reasons in support of some claim, opinion or point of view, to identify: (a) the intended conclusion, (bl the premlses and reasons advanced in support of that conclusion, (e) additional unexpressed elements of that reasonIng, such as intermediary conclusions, unstated assumptions, and ld) for ehclusion, any items contained in body of expressions being parsed which are not intend to be ta~en as crucial to the reasoning being expressed.
4. EV~LUATIHG: representations;
10 assess
the
,redJbJlity
of statements,
and
tu asse~s
the Jtrength
.'
....
'18
..,
_,
, • _oo. ~
. ·;~7(:
~ - '. 1
RESEARCH
IN PROGRESS -- ~
~BLICATION
PAGE 5
:'
.'
such statements, descriptions or representations. 4.1 VERIFYING CLAIMS: To assess the degree of confidence to place in a given statement, description or representation. 4.2 ASSESSING LOGICAL STRENGTH: To de t er ni ne the n a tur e and quality of e~pressed inferential relationships; to judge whether the assumed truth of the premises of a given argument justify one's accepting as true, or very probably true, the expressed conclusion of that argument.
betNeen
5: EXPRESSING:
the results
or others prOdUCJDq
t ties«
r e s ul is ,
5.1 STATING RESULTS: To produce accurate statements, descriptions or representations of the results of one's CT activities so as to analyze, evaluate, infer from, monitor or remember those results, or so as to communicate them effectively to others. 5.2 DESCRIBING PROCEDURES, To produce accurate statements, descriptions or representations of hO~1 one applied and executed any CT skill or sub-skill so as to evaluate or monitor one's proficiency, or so as to co~municate to others about how one went perfcrming a given CT skill or sub-skill.
0: !'IONITORING: TL' regulate s l l e s s e c t s (If o u e ": ONrl CT a c t r v r t re s , the • eleaents used in those activJties, a~d the resul:s produced by those actJvities, particularly by applying the skills of analyzing" and
evaluating validating
to one's own inferring with a view toward confirming, and/or correcting the results. 6.1 REGULATING: To sequence one's execution of CT s~ills and sub-skills. 6 • 2 R E IJ IE 1'1IH G: Toe x ami n eon e "s 0 1'1 n C T act i 'I i t j e 5 and v e r 1 f Y both the result~ produced and the correct applIcatIon and e~ecutl0n of each CT Sl;i~l and s ub+s k i l l involved. 6.3 CORRECTING: Where errors are found in one's own CT e c t i v i t i e s , to c cr r e c t those errors and remedy their causes.
PAGE76
•• :r .t.
*
ROLmd 4t
RESEARCH
2,
IN PROGRESS
-- NOT ~
Model
PUBLICATION
of
PAGE 6
.Part
An Input/Output
CT Skil,ls
Comment: "nlt?I"e'r:_':, no dmlyincJ thC\t ·the model 1 'va c omo LIp is hecwi ly I n f 1 Lle~11 c (~d, ·f or (j ood or i 11. by my r es~c:.r ch inc: (JcnPLI (?I- SC i ell c F.? cn("l d t a r i~., of i c i,:-d. i n t f? 1 1 i 9 f"J n C E • j The ~ r r Q ~'Jsin cJ i c ate t h P. U ire c t i o n 0 of t h r~ + 1 CH'J 0 f i 11P u t n d ClLIl:. P LIt as i t c i r' c: I_I i.l t e s b Q t vJ e e nan dis 0 per-ate d LIP0 n 1 by t; h t; s i:-: C T ~~ i 1 .1 s , her e 'c on c e i ve'd 0 f a s ~~ k LIfH: t ion s •
i.'
OLltPLIi.: in the form o+ in1~ornlc:\ti(::m'fr-onl the knr.::n·ller.fge basr~, the resLIlts of ol:hetCT fltn(::'l:icJI'H:', i.~ncl Lt s c s elf t·.he sb:;ps pc;r"fonnf::d dur-t n q other CT +unc t i cris , I':: .:p '" t:.:1!::~:;i n C.J l; "1f} 1'\ of 0 r' III ,;;,I: s t; h r.:1 Lou t; P LIt of 0 r- LIS e by ttl r.:' n e :.:t; C'I of U II C t i o I I , : +or' ~~ 1.:0'" ~1(]f? in l: h t;: kn ow l edC:lc-:--b se, a or' for tr an s;mi ss ion out; 0+ tt1E~ C, c:yc:l e. nlater-ial +r om e:·:prE?s~Jing ~.:md, Ins t.h~~ regLllc?tin~l and self-correc::ting +unc t i on , deterrnine~:s Idhere it shc.:tLIld C;}O nen:t •. It can r ou t e things i n t o c:,r ou t of iU1Y of the fiveC) 1:I'H:? I'" C T ski 1 1 s a r- 'l: tHo? !< n 0 vLl e d q e - b ~ sse • For e :.:a In P 1 e , rn 0 nit 0 r" i n q C c?11 loop materi al b a c k t hr ouqh ariv ski 11. ThLI5, it c ari send the resLIl t: t:>f r.Jnf.~'s own in"f_cgl.:cLr:l_9_ for revie\'1 by r ou t i nu them to ~Y.5"l.~1L~.~_to..g. e+or e b all o",d ng ~=11Jdr.'!?-Jf~~j_D.9. to trr.lnsmi t them to others. tJr. it c an r out e infcH'"ma'l:iol1 f.rom ths knol.-lled9fJ bas~~ to ~y_~.t~l.r!.t;.A.D.fl to help it ver"ify a c:1 i~ i III? 0 r t; 0 i.ni.. .[E..Og t 0 h f? 1 pit ~ t cj r a ~'J en c em c 1 LI i em LIsin q c r i t En- i 1:\ s p (: c i ·f i c ·c 0 ~ 9 i v end i sc i p 1 i n t~ •
In
t.his
model,
§:l~iE.r_g_??jn.g.receives
l'Jithin
thf.?
CT cvc l e , t!.!..Q.QJ.tgr-i.nr.trec:eives
CT skill~~ lI\c:\ny o·f t.1:i c:r.~E!nd so rmic h tllTle .. h e Lp i nn ~tLlLl£-.'fll~". b(:?C::(~lll1(? pl"o+ic:i£~rlt i~l., n~l1Iel'l ~_L'.~~::r':I~.t"_gtl..n_(:l.\. ~~n§ty'~-:j.l"..q,. t..~Y..c:3~1~~,:1.l:p~lll c.1!.1.t1. •. ~nJ".~~r. t"1 q r ec f.?i 'It? nti.\ l: t.?ri ~ r cu ted tot .. rj. 1 h em by th ~ (lion i t. or in q ;: LHICt i C.'r1 • They op(:'?r-c.~t(?011 'l:.l1i S rrtlatsri c.\l. And then th(~y s~nd tI1',: resLIl ts tr..) t11l:~ e:.: ,- U'.:~in g f LU'I t.. ion t o b E~ f 0'- mcd: ted f IJr del i 'II:,I rye 1Sf.H·j £:l r e • ·1I\f:? Sl~ p s c h skil.l!s In150 ou t pu c r.\ r·.~{:ord o+ t.lle steps thl;~y pf.:?rfor·lned in corni nc to those rt?~.;l.11tc:::. Th i u rE~cor-~ is:, cr uc i e l if the monitor-ir'tq +unc t i on is 1:0 ~'~r:Jr-k c(.J,"r-tJctly, s i ric a it mLl~t check not only ~'Jlh.~t ~'j~S C\chievr:d t.)I_ti.:
"'hc~ fC.Hlt"' ..
how it
w~s
achieved.
ThE? L!Jg.ttJ.!-"r_.ISJ~ R.~~~7U?. is a storehoLI.se +r cn. \'jhic:I-, interprf...?t:i.nq~ c:.mc.,.ly::.in<;.1 , ~'1r.l.lu,;.\tinq arid lnferrll1(;J t.lra~~ reSOLlr(:es. it st.cwes tl1f: output ct any CT function, wh&rr directed to do 50 by tIle monitorlng +unc c i on , The L.!J.f!.\.:!.~_rg.(~_q1l Q_~?..!-~: ~150 flJ.i.:t:?rs ra~'j ex t er n aI i rip u t; and. t h LIS. i n + 1 l.1E:::" '\ c.:i-J'= t h e ;'.0 t e 1 eo _i..!l9.. 0 f ~j h c:\ t ~'I e cu- e 0 b 5: eo v i rt 1':1, dec 0 d l. n 9 t ro r c J. t:\ r- .i. f Y i. n q •
r.!2L.
I nstrLlcti ens:
!;,Q, ~C'.H:!?
§.i..!" S_g'_ ~lf\D.Y R~.qpJ_§'. t.i.Q.c.J. Ri_c;:.t.9_r_i_~l_ ~J_qd_E?L=. t1 f:?..l_p.!'l:-Il . in ~lns~!.~~.:s...'': .. ::1ndj..Dq ~J~.U)P'.,l_,~~).L C t:.~.l.f:~ t:.i..Qf].?hlll.s ..s, t:;§!.1 yg.l:!. s9~.9 '?.~1;. 1.:l~t~·1 t.t!.i. ~ C?!' ~~ WtC;Ltl:t- 9~:.t!l.lpr.9Y~:9. ~-:'L.'1. ~_f \':R.L.~ r:f-:..~:_O..!!!!!2.llrrQ. 9.t~.~~.~:.sJ_to_CJi_t...\. r;_C\_f'!. 'y~q~l. .r..DP.~~S.~ .. P mO_ fo'\l~.§.r 1~~:~i.Y..@.? 1
"_-'-('.,
• ->',~
•
_-
80
..
~,'
AN
INPUT/OUTPUT CRITICAL
FLOW
MODEL SKILLS
OF
THINKING
....
•,---_
I
..,
I --~
I
..,
o
If)
"'0
w.
.-.
"
t1"
EXPRESSING
~
'g
K
N 0
5. ~.
W
L
o
II
E
0
G
'8.
a
~ ~
...
if
i' ....,
1 I \ .:
"
l>
ee m r.n m
.....--........ ..
c")
:r:
~;.,
INTERPRETING
E
B A S E
~'A
A II A L VZ 1 N G , )
INFERRING
'. ..-...:-_/
~
<+ (1)
~ r.+
,.
----.
MONITORiNG
I~'
16
l:tl
."
..
'1 CD
--------
c::
tt1
~ 0
tI:l
Q)
...
-.2
100'4
.....
n :D ~
.....
\.I Z
1. C
81
82
';:~;;,'._'
:('.~~
RESEARCH
ROl.md
IN PROGRESS
~: F'ar-t
PAGE S
:':~l;
i~
Not.
,'_ . j
Nany of Yf.1L.1 di sti nqu i shed CT ski 115 from oth~r c I o:;·f?ly 1"(~lci\ted t.h i nq s , 8elCl~1 i~) c:. list c+ l'Jhat various p er aon s sc:dci ~Ias n9~~.CT. ~.r1.l5t beci:\l.ISe s ome t h i nq is on this 1 isi: dl"JEH.' not ltlt::'iHl a P€uOc;HJi·, (H'HH1 n'c:d:. \.ISS cr in d o i n q the thin(~ nor· tI1c\;:' C\ P(=I""son rn i qb l; not do th~ thing befure, after or during CT. Instructions: ~;.g.ftJl£lJ]DJ;,.9.U
Comment:
t~.?. ~i;_i
~_~r:j.. x: i
i.~._,.~.fll~Llg, iL..
~.o.d
YES/NO
CRITICAL
I, Sen sin
q, ( 5e e i n q ,
THINKING
IS NOT
etc.)
TOL.lchi n q ,
He a t- i n g.
2. Reading.
Listening,
3. Speaking 4. Motivating,
or writing,
persuading, s~lling, petitioning, world around~
S
I:)
5. Interrog~ting, ~.
7.
PhysicaLly
1 v i nq ,
0
~:: ere i 5 i n l)
n G~• s "d. 1 1 •
b j P. C t i ve s ,
Ii\
c rib i ngam
0
e ~n i n 9 t IJ r bel i e f,
r t,
1 i 'ff?)
C i\
. 0 rev
ent s ,
1'"'
...
D C' -t end i n q
on n
pin
ion
a r- g u i n 9 a
= eo,
Ot··
t·lr~naql.nCJ,
admil1istratinq,
cr' gavurning
per£;':Jn~)
t h i ric s ,
13. Philosophizing,
14. Conducting
research
within
any
q,
particular
0
disciplin~,
1. 5 • E>: per i e~ c i n c , 11
16. Comrnun i c e t i n q
fer:] 1 i n q , u ar nq
ern 0 tin
1 anyL.laqe.
'e~'
PAGE79
.,-
:f" .
:;\r,,
83
'0\""""_ ',,',"_ ., ...... ' ........ r , "
' :'.~.
..
~:: ~
.·X~~{\APADel phi
Pr c Jec l
*
ROll
RESEARCH
IN PROGRESS
-- ~
trua PUBLICATION
PAGE 9
':~',<
Comment: It became clear from your responses that it would be useful to have an understanding about what a skill is. Although there may be very little disagreement about this, 'ome of you mentioned subtleties others mayor may not accept. Based on your contributions I 've written a little narrative. Ins t rue t ion s: B.! yo 5 e, e d iJ_ and com ole t e t h ~_ f 0 11 0 ''I in 9 n ar r a ti ve.J_ i A $kill Nhat i~ the ability
to do something
Nell.
Having That
a skill
knoNing
judqih9
those
or adept of Nays
car, I
to,
laking Skills drill,
training. Nhen
the procedures
cah
and shoNJng
a cOlbinatioh
and hON
be learned
ahd self-,orr~ctJI in
a
Persons
can
proficient as it
JS
9iven
skil;.
The first
to observ~
A second
a given ski:l
result
against
~ay is to
and receive
their
of the proced~res ~ould that sklils avaJlable use skill. if they Si~ce
skill,
~~ cannot
directly skills,
observe only
the perfor~ance
and
the Nay
the second
thJrd
to those bec~use
CT.
The second
to the thlrd
provided
that •••••••••••
84
~~.
'.":
.~)
»; , ,Y•.,;' .»~.
...
:~t'
,-.," ~ • -.e-;~"~:; .
,':
,T0
: .:f~··" .
_.0.-::,0'"
~ •••
*
Round
RESEARCH
4: Part
IN PROGRESS
5,
PAGE 10
Caveats
Comment A number of you sent comments. Some were intended as caveats or cautions, other as encouragement. I appreciate them all. I thought some might be good to sharei In many cases to save space I have paraphrased. might agree or disagree with what your coll~agues have ~aid. Or, reading 'list might prompt you to pass along a contrary view. Let's find out. Instructions agreement reflections.
QL
You this
AGREE/DISAGREE in mind that our goal in defining CT is to do some assessment. But you dor.'t just start testing people. As~essment needs focus and purpose •. 2 • I a g r e e I,d th 1'10r I: i n gat the col leg e I eve 1 0 n I y b e c a use I want to have an idea about what He ~hould be doing with kids in K-12. 1 think we can use what we say about CT for college frosh/soph to guide curriculum development in K-l. CT. Of course, we will havp to adjust reading levels, background knowledge expectations, and lots of other thing5 •. 3. Skills are not the same as operations. Resist behaviorism! 8ehaviors give evidence that a person has a skill or ability, but a skill is not a set of behaviors. 4. Don't trap yourself into using the jargon or vocabulary of anyone discipline (especially philosophy) when y'ou describe CT. S. AI tho ugh n 0 H 0 r d H ill esc ape e r i tic i Sin, don' t use d e du c t ion 0 r in d u c t ion. A v ai d s e ID ant i c spa ts • b. If CT is a set of attitudes as well as a set of skills that poses no problem for assessment because we can just develop ways to assess the CT attitudes, too. 7. C Tis th i n I: i n 9 5 k i I Is. Say in geT i s a 5 etc f attitudes may be a way of describing what people who are good dt CT ar~ like, but it is not a way of describing critical n' king itself. 8. Even if lie agree on ~Ih~t CT is, I'lestill have to face the problem that any student might get the right answer on a CT test but for the wrong reason, or might get a problem wrong but have done a good Job of CT, 9. When assessing CT He should not ~uplicate efforts with areas already well tested by existing instruments, such as covered by reading or intelligence tests. 1 (J. You don' t h a vet 0 t est ,? 'I er yin 9 1 e C T sub - s L ill t 0 decide that ~ person i~ qood at CT. 1 1. I I 0 0 I: e d a t that lis t 0 f e:~ per ;. ~1 and y 0 II h a v e all the big names I can think of, but you can't possibly exp~ct those people to agree. If they did agree, even on what CT is, that would really be something. Good luc~!
II II II II
1. Keep
.)
THIS MATERIAL
WITH COLLEAGUES,
...
~.. .. ~.'"
. .,."\
\ '_"
DeM'T REPRESENT ANY OF THIS LETTER AS THE OPINION OF OUR DELPHI Appendix c: Delphi Rese~oh Letter8, PAGE 81
the
.
,;
.~
.',
i~
Ene:
tl
Thanl:s!
l~...
. ."
~').~'.; :. '.
...•
'
..
85
...
,,'....
.. ,i.
Fullerton. California
92634
During this Thanksgiving season and I want to express my gratitude to you for your generous participation in this research during 1988. The many responses to the very long and difiicult ROUND 4 have been most gratifying. There won't be any more "ugly-long" rounds like that, I promise. Let me pass along this quick review of what we accomplished in 1988. First, working under the auspices of the Amerl~an Philosophical Association Committee on Pre-College Instruction in Philosophy, we built the Delphi list of expertg. By your recommendations, during Rounds 1 and 2 (Feb. 11 & Mar. 14) we expanded the original APA list of about ten names to a working group which numbers around 45 active partici~ants and which, I am proud to say, includes many of the most important people in CT research today. During Round 1 and 2 we agreed that whatever CT is, we would be possible for us to make the concept operational to the extent that important parts of CT could be assessed validly and reliably. We also agreed to begin by identifying the core elements of CT expected at the Frosh/Soph. general education college level. Ve agreed to use this college level theoretical-construct of CT to guide what is said about CT the K-12 levels. Round 4 (Sept. 23) sought to verify the concept of CT which emerged from Round 3 (May 4). A quick look at the results of Round 4 is most encouraging! Along the way we shared journal articles, lists of existing CT tests, CT bibliographies and other items of mutual interest. 1he Delphi, however is not a SUbstitute for the fine work being done by journals, newsletters and the many centers for CT that have emerged in recently. While I work on analyzing the results of Round 4, I invite you to consider where do we go from here. Last spring I outlined a four phase Delphi project. When '~e achieve consensus on the core list of CT skills expected at the lower 'livision college level, we will have completed the two of the four phases. Originally phases 3 and 4 were described this way: Phase 3: Recommendations The goal of this phase is to communicate our findings about what CT is and whether there is an adequate way of characterizinq CT operationally so as to permit its being tested at some educational level. Dependinq on our results in Phase 2, we will recommend either that programs aimed at testing CT be abandoned, or that they be focused in certain ways. If this is the direction Phase 3 takes, then we will also try to come to consensus on recommendations regarding the relative importance of dif~erent kinds of CT sub-skills and possible ~trategies for ~ccessing and measuring those sub-skills. Phase 4: Design and Validation of Kodel Testing strategies Contingent on the results of earlier pbases, the goal, if it were considered achievable in principle, would be to cons truct and evaluat~ different approaches to testing CT at some appropriate educational level or levels.. \fe might find ourselves breaking in'~o'sub-commi ttees to achieve this g~al, although all ~r.k will
•.
'.
-a.' ..
,'
...•
..1 .....
'
•••
•• _._
.. _··.
"~"'!.
~I).'"'_'
:~h(
••••
'
"
_.
_ •• :.
_.
, -. ':
'
have to be guided by the agreements reached in earlier phases and as well as by the special expertise of those who understand the intricacies. of desig:'ing, piloting, norming and validating educa tional tes ts at 'spe ci f ic educational leve ls. To date I've done little regarding preparing to communicate findings. Two CT newsletter~ inte~ested in publishing something our results have contacted me. Also, the Pre-College Committee session at the March 1989 APA meetings at UC Berkeley. At that I'll be outlining our Delphi process and what we have agreed on time. More suggestions are most welcome. our regarding scheduled a session by tnat
For many reasons I am extremely skeptica! about actually developing a good CT test using a Delphi process, Once we declare consensus regarding the theoretical construct of CT for the general education (lower division) college level our choices include at ieast these three, and maybe more. (1) We could move on to consider questions like these: Given what we understand CT to be at the college level, what does CT m~an a~ different grade levels in K-12? What is the relative importance of thE skills or sub-skills i~ our college-level CT construct in terms of testing, say jJnior high school students? How might one write a question which assesses a given sub-skill in, say 5th graders? (2) Having declared consensus on a conceptualization of CT for use at the college level, we might recommend that test makers at all educational levels be guided by our conceptualization. But we, ourselves, might decide to leave the matter of writing specific tests for specific age groups to others, better qualified than ourselves for developing and valida~inq such instruments. Those of us interested in specific grade levels could be put in contact with one another. (3) When we reach consensus on the CT concept as it applies at the ~ollege level, we might recommend examples of how questions framed to address these skills and sub-skills in college students or K-12 students. We could share these ~xa~~le questions and evaluate them. Those which we think a priori might be good ~o assess certain skills or subskills, could be included in recommeneat~ons we mak~ regarding CT assessment. These questions would not be a CT assessment tool, At best we might think of them as models of how to c~nceive of questions that might be included in a CT assessment. Nota: Even with questions which a priori se~m to address the proper concept of CT and avoid other difficulties (like relyifig on special background knowledge or esoteric vocabulary), there is still the problem of a posteriori verification. Steve Norris has done important work on how to overcome the "cons truc t -val ida tion problem, namely de: termi n ing, for any given test item, if students get right answers because of good CT s}~ills and wrong answers because of inferior CT skills.
II
Give the issues of what do to next some thought. I would welcome hearing from you on this. Have a joyous hvliday season and thanks again for contributing so generously during phases 1 and 2 of this project. Sincerely,
"
.' ..
,/ "'/~-.,
'
...
, ..-
....~---,
----
L.,..,_.
'-;':'.-~~,..
PAGE. 3 8
.....
_'.'
..
'
87
lito .•.
. '"
•• ·-~.t -,.:
,\.
California
Fullorton,
Department of Philosophy (714) 773-36~1
..
California
..
Feb. 7, 1989
Dear Heroic Uelphi Colleagues,
34 people responded to Round 4. A great retur~, our biggest so far -both ill numbers of persons and in numbers of pages! 1 ..i i l l tally, analyze, distill and share "dlat people s s i d on each c hur ': of Round 4. lhal ..,ClY you will havQ the b~nefit of an overview of what others in the Delphi are thinking. But, intending lo spare all of us any more horrendously bur' dell S 0 mer 0 un d s, .., hen i teo mest 0 for mU 1 a tin CJ the n ext set 0 f que s t ion s , I ' 1 I h e e p t h (~RI a 5 S h 0 r t a 11d f CJ c: LIsed asp 0 s sib 1 e •
1 pclrticularly wanl to t h ank the I!'any people who sent me d e t a i l e d , thoughtful, (and even fu~tnoted) r~sponses. Several ran near ten page~ single spaced. ~ot inten~ing to diminish the value of the ~~ief "Yes/No" res p CIn s e s a s w e a p pro a c: h C tI n 5 ens "I SOli C rue j alp 0 i "t s , I mu s tat I: nOl'I 1 e c.J 9 e and co~vey my appreciation for the many extra hours of wor~ several of you i:i rep LIt t i (I gin. Al s c , l' vel e c1 r ned a 9 rea t d e a 1 fro m you r sen sit i v e and sen sib 1 e com IT,e Il t S • Yes, 1'It? are approaching consensus. Ny first two r e ad i nq s of the ir,put on koulld 4 is that we have a great deal Of accord on the list of I.:T 5~4ll~. a 1 tho LI g h l her e ..Ii lib esc 0 res 0 fad jus t f.\ e n l san dam'.? n d me 1I t S t 0 tJ emit de. Also, your comments on whdt a s~111 is and hON a cognitive s~ill can be assessed suggest l'Ie ar e c l o s e there too. But I'll be sumlllar.izing all that and more for you very soon. 1 n l II a "fI >: t f e~, ..I a e I: s I' lib e sen din g y c u a 4 e w qui c k s h 0 r t r 0 u 11d s • 1'1 y ~ 1 it n i 5 l 0 4 0 c use a c h tI r i e fIe t t e ron 0 ned i s ere tea s pee t 0 f 0 u r Ii0 r L \·1 (? '= h C,I U 1 d l r y t IJ com p l e t e s e 'I era 1 min i - r 0 u n d s t his s p r i n 9 • T II a l way l' 1 I b (? a b 1 I:! t o r ou 9 h 0 uta fir s t d r a flo f 0 u r rep 0 r t tot h e A P A f' r e - Colle 9 e Philosophy CUrTlmitlee during the SLimmer. lhinl:ing about recommendalions.
ve
I ' "t? l' n c: 1 "S l! u ~1 1 i s l CJ t t tr e p e 0 p 1 ~ who are par ti c i pal i n gin 0 urI) e 1p h 1 r f! S ~ C\ r c II ~I I" 0 j ~ l. Tile" R 1" " R2" .. R3 " and c kif" s YITI b 0 1 sin d 1 cat e the r 0 L\n d 5 t~ which the p~rson has ~~nlributed. The "l" symbol me~n5 the person c cnnun i c e t e d an interest to be involved, but has not r e sp on d e d yet to any rounds. Cur r e n t l v there are 56 !lames an the list uf people being invited t (1 r e 51p 0 n d • A f e ~I h a v e n eve r res p 0 n d e din d n y way. 5 0, '" hen 1 teo mest. 0 mJ I: i n y o u r fin i:i 1 rep 0 r t , 1 e x p [! C t the ret 0 b ear 0 II n d 5 0 0 f u S 1 fI the 9 r 0 up.
II
etc.
so me min i - r 0 u n d s 5,
(.,
f'eter
: .'::l~::::
):
•. "
. ",
88
.,,'
_",;2"·
U
: .' . '_ ;,:
California
Fullerton,
California
Department of Philosophy
(714) 7"73·3611
De.ar Delphi ColleagueG, One of you wrote, "J'm beginning to think that to have done all that [we have doneJ and not have tried to define CT may turn out to be a mistake.
1I
Also in response to ROUND 4 three or four of YOlo' commented like this: "I have no major quarrel with any parts of your organization, [butJ your emphasis leaves out a major component of CT -- the dispositional component and the set of values i'1herent in being a critical thinker ••• I think it is a good working model of CT skills, but an incomplete picture of being a cri tical thinl~er.1I In view of the many positive responses to ROUND 4, yet sensitive to the concerns raised by comments like the abcva, ROUND begins by building on 5 our success in articulating a dec.ent first draft list of CT skills. In this letter we start right in on the question: For purposes oT ner ral education assess~ent at the college lo~er division level, ~hat do ~e experts reco ••end be included as a core critical thinking skill? In addition to asking YOLlrendorsement of a revised draft of the skills dimension of CT, this letter also shares some key ROUND results 4 and some of the many useful comments you sent. The next letter, ROUND 5-B, works on the two other aspects of CT you commented on in your ROUND responses 4 -- rHlmely eT's dispositional and the normative dimensions. Some of this also finds its way into the revised sJdlls statement -- see 6.1! RaG, J 5-C pic:ks up the remaining pieces of ROUND and asks your approval of an outline of our report to 4 the AF'ACommittee on Pre-College Philosophy. The table at the end of this letter shows that over 85'l. of us, (23 of 26), could lJe described as fundamentally in accor-d with our first listing of CT skills. The second draft you are now being asked to consider endorSing was prepared in view of the many helpful comments and suggestions you sent in. I am very optimistic about the revised statement of CT skills, first hecaLLse ROUND 4'~ dr'aft was approved by such a solid plurality, and second because your suggestions helped me substantially strength~n and enrich tha.t statement. Since we are very close on so many things, your approval or disapproval of the expressions of our views presented in ROUND should clarify 5 things enough for me to start putting together our report to the APA Committee on Pre-Collegp Philosophy. Where we have consensus our report will say so. ,Where we diverge, it will say that as well. I truly appreciate all that vou have already contributed, and I realize you you are all very busy folks. Yet I beg vcur' cont inued indulgenct'? Please respond to the three round 5 letters with all reasonable dispatch. All responses are welcome, no matter how brief or seler:ti vee Wih sincerest t
. 'J
;:?.i":~t-~'L'~~{ ',.,
;;}~~~t. \'~
C: Delphi
"
Research
.
~t".\~,: ,._:_;_~:.).
"
. .-:'
._
, t, ...
..
,.
:.....
.V_.....~., ~ . ....
._ .,._
"
Letters, PAGE,·85
gratitude,
,j
,..
." ,
,·.:--.h
_
.
89
..
....
...
".
"
",,,._,";'
......,',
*.''';'
. ..
'
~ .:
Half of this letter shares commenls regarding defining and testing CT made in response to ROUND 4. Before digging into our revised statement of CT SkIlls, you might jump to page 6 and look through the comments or examine the tabular results of our earlier Hork on page 10. I learned, for example, that not listed in our original statement was a skill the majority feels is part of CT -- arguing. And, given what this Delphi project is all about, how on earth could I have emitted the CT skill of analyzing an idea from the first draft? ROUND ~ PART
.L:.:.
rr SKILLS
INSTRUCTIONS: Consider the following a~ended description of CT skills along with the accompanying statements. Starting with the title and preamhle, make any needed changes, deletions, or addi~ions (editorial or substantive). After working through the descriptions and statements you will be asked specific questions regarding endorsement. Please respond to those question as well. Skills Dimensions ~ CriticJI Thinkin~
For purposes of general education assessment at the college, lower division level, we understand Ci to include the cognitive skills of interpreting, inferring, analyzing, evaluating, expressing and monitoring. Because of our collective conviction regarding their centrality to CT, He urge those persons interested. in assessIng the stills dimensions of CT focus on these six abilities. However, since CT can be subclassified in a number of legitimate ways, our subclassification should not be interpreted as an educational taxono~y nor as implying or presupposing any psychological, logical or episte~ological order or sequence of skills. While including those s~ills He take to be central to CT, we do not claim' that our list is exhaustive in either breath or detail. Critical thinking involves actively interpreting one's experiences and self-consciously ~aking and expressing one's analytical, evaluative ~nd inferential judg~ents regarding what to believe or do. As .such, critical thinking is a pervasive and multi-dimensiohal human phenomenon involving both dispOSitions and skills. Without di~inishing the vital i~portance of cultivating CT di~positions throughout the K-12 and post-secondary educational proce~s, He have here chosen to focus our attention on listing and describIng CT abilities. As a goal statement of .Hhat a generally educated college level critical thinker should be able to do~ He hope our cons~nSU5 description of CT Skills will assist in CT assessment and CT curriculum development both at the college and the K-12 levels. Among the ~any ways one ~ight improve one's CT arp. by reflecting on one's reasoning processes and learning hOH to think ~ore analytically, objectively and logically, by expanding one's repertOire of those· ~ore spcrialized pro c e d u res a d c r i t ~i· i a use din d iff ere n tar e a 5 0 f hum ant h 0 ugh tan din qui r y , and by increasing one's base of information and experience. An i~plication of our analysis of CT skills, however, is that CT skills per se transcend spp.cific disciplines, yet executing them successfully in certain contexts deman~s background knowledge, so~e of which ~ay be specific to hON one makes reasonable judg~ents in that context. Since beco~ing adept at CT involves learning to use CT skills effectively in many different contexts He cannot overemphasize the value of a solid liberal education to supplement the honing Df Dne's CT skills and the cultivating of on~'t CT dispositions.
I'
-::;~,.
::: .'.:\;\.J'.:; "." .... .., ..... <
__ •.. •
Le"vt:l&.8,
PAGE86
90
.1
.
Hales of Core C1 Skills ahd Sub-~kills
(Oo.t.:-
Categorizing, Investigating, Decoding, Clarifying. Analyzing Ideas, Id~ntifying Arguments, Analyzing Arguments. Assessing Clai~e, Assessing Arguments. QUQr~ing, Conjecturing, Concluding, Developing Rea50ns. Stating Results~ Describing Procedures, Stating Arguments. SElf-examination, Self-correction.
of Core CT Skills
Descriptions
1. INTERPRETING: To comprehend the meaning or explain the significance of a ' wide variety of experien:es, situations, data, events, judg~ents, conventions, beliefs, rules, procedures and criteria. 1.1 CATEGORIZING: To formulate categories, distinctions, frameworks or questions, and to describe experiences, situations, beliefs, events, etc., so that they take on comprehensible significance or meaning, as for exa~ple to recognize a problem and define its cnaracter without prejudice to inquiry. 1.2 INVESTISATIN6: To actively seek, attend to, discriminate and describe experiential input relevant to a given situation, problem or cencern; to gather input that conveys or is intended to convey data, information, or inferential relationships, as for example to gather evidence relevant to solving a proble~ in the light of how that problem is defined. 1.3 DECODINr,: To actively detect, attend to and correctly understand, the informational content, tffective purport, directive functions, intentions, purposes, symbolic significance, values, views, rules, procedures, criteria, or inferential relationships expressed by others in convention-based com m u n icat ion s y stells , s u e has i n I an g u age, soc:i alb eh a vi Drs, :'"' d n g s , u, number5, signs and symbols. 1.4 CLARIFYING: To explain, paraphrase or ~ake explicit, through stipulation, description, analogy or figurative expression, the contextual, convention~l or intended ~eanings of Hords, ideas, concepts, 5tate~ent~, behaviors, drawings, nu~bers, siqns, sYBbols, rules, events or cere~onies; to an extent proportionate with the purposes at hand, to use stipulation, description, analogy or figurative expression to remove confusing, unintended vagueness and a~b~guity, or to design a reasonable rro:edure for so doing. 2. ANALYZINS: To identify the intended inferential relationships a~ong statements, questions, concepts, descriptions or other for~s of representatlon intended to express beliefs, judgments, experi~nces, information, opinions. 2.1 ANALYZING IDEAS: to identify expressions used in co~munication and determine the role they are intended to play in arguing or persuasion, as for example to identify a phrase intended to trigger a sympathetic emotional response and induce an audience to agree ~ith an opinion; to identify related judgments, views, or concepts and to deter~ine the conceptual si~ilarities and differences between them; to identify issue$ or problems, determine their component parts, and identify the conceptual relationships of those parts to edch other and to the whole. 2.2 IDENTIFYING ARGUMENTS: Giv~n a set of state~ents, descriptions, qu~stions or representations, to deterline whether it does express or was intended to express a rea~on or reasons in support of or conte~ting so~e claim, opinion or point of vieM. 2.3 ANALYZING ARGUMENTS: Given the expression of a reason or reasons intended to support or contest some claiB, opinion or point of vieH, to Appendix C: Delphi Research Letters,
l.J
PA3E 87
..
_".
91
l~-
:.~.~t :.,.l. __
..
~ .. ~
'.
-;
... '
.....
identify: (al th~ intended main conclusion, (b) the premises nnd reasons advanced in support of the main conclusion, (c) further premises and reasons advanced uS backup or support for those pre~jses and reasons inlended as supporting the ~ain conclusion, (dl additional uneKpressed elements of that r e e s on i ne s u c h Jr. i n .. .... ......J c ..... ,t..w""... o .... un c t c t c d ...... ..t i cn s or "' "'dl' cnc l u i '·r. ~"'''''''~w presuppositions, (e) the overall structure of the argument or int~nded chain of reasoning, and (f) any items contained in body of expressions being examined which are not intend tc be taken as part of the reasoning being expressed or its intended bac~ground.
~,_...... .. ._.101.. 'WI
11_,
W•
.JW
.. '"
,
•
3. EVALUATING: To assess the credibility of statements, descriptions, questions or other representatio~s expressing experiences, situations, beliefs, judgments, or opinions; and to assess the strength of the expressed inferential relJtionships among su~h statements, descriptions, questions or other forms of representation. 3.1 ASSESS1N6 CLAIMS: To assess the degree of credibility to ascribe to a source of information or opinion; to a: .. s r the relevance " of questions, i n for mat ion, p r in ci p 1 e s , r u 1e s 0 r pro c e d v ,: e c t ion 5 t 0 a 9 iv e n iss u e D r cencern; to assess the truth or the level oJ" " idence to place in any given representation of an experience," situation, judgment, belief or opinion. 3.2 ASSESSLNG ARGUMENTS. To determine the nature and quality of expressed inferential relationships; to judge whether the assumed truth of the premises of a given argument justify one's accepting as true, or very probably true, the exprp,ssed conclusion of that argument; to anticipate and raise questions and objections, and then to assess whether these point to significant Heakness in the argument being evaluatedl to determine "hether an argument relies on false or doubtful assumptions or presuppositions to Judge how crucially these affect its strength; to Judge between reasonable and fallacious inferences; to judge the probative strength of an argu~ent's premises and assumptions with a view toward deter~ining the acceptability of the arguBent; to determine and Judge the probative strength an argument's intended and unintended implications with a vieH toward judging the acceptability of the argument; to judge the extent to Hhich additional information would strengthen or Heaken an argu~ent.
I :"
4: INFERRING: To identify and secure elements needed to make inferences and to deter~ine the inferential relationships between or flowing from statements, descriptions, questions, or other forns of representation on the basis of which inferences can be drawn. 4.1 QUERVING: to recognize the need for evidence or information of some kind, in particular to recogni:e Hhic~ statements, including those offered as premises, need justification, and to for~ulate and execute a reasonable strategy for seeking and gathering that evidence or infor~ation. 4.2 CONJECTURING: Glven a problem, question or point of view on an issue, to formulate multiple alternatives, develop hypotheses, or postulate suppositions, and to design reasonable st.rategies for deter~irting their plausibility, viability or rqlative merit; to objectively draw out the presuppositions and the consequences of decisions, positions, beliefs or vieHs Kith which one lIight agree or disagree. 4.3 CONCLUDING: GAven a set of statements, descriptions, questions or other forms of representation, to educe with the proper level of logical strength, their inferential relationships, both deductive and inductive, to educe the consequences or the pr~suppositions ~hich they support, warrant, imply or entail; to successfully emplo~ iar'ous sub-species of inductive or Appendix
• :":l.;~".'"'t:':.~-,;\,"-"-"_~"".,,,,.
: .::~~'f.-\~"
. ,.'
<~. ,l,'.·
C: Delphi
Research
Letters, PAGEB8
"
.'
92
. "(f,','
... G ...
:.. ~".~ J ..
','! '.~';l.' •
•. ~"
,~I
deductive rea~oning, as for example to reason analogically, arith~etically, dialectically. scientifically, etc. 4.4 DEVELOPING REASONS: Given a question to be answered or a position on an issue, use appropriate inductive or deductive modes Ot inference to articula~e reasons for answering the question one way as opposed to another, or for supporting or for opposing the position.
5: EXPRESSING: To state, describe or repre$ent to one'~ self cr to others the results of ene's reasoning and the way on~ went about producing those results. 5.1 STATING RESULTS: To produce accurate statements, descriptions or representations of the results of one's reasoning activities so as to analyze, evaluate, infer from, or monitor those results, or so as to accurately and effectively recall or represent those results to one's self or to others. 5.2 DESCRIBING PROCEDURES: To represent as cl~arly as possible how one cam e too ne •5 i n te r pre tat ion s , a n a 1y s e s , e val u a ti 0,"~~ r i n fer e n c e s , sot hat 0 ne 0 might accurately record, evaluate, describe or justify those processes to one's self or to others, or so as to re~edy perceived deficiencies in the general way one executes those processes. 5.3 STATING ARGUMENTS: To present argu~ents which communicate one's ground~ for accepting some claim, their logical force in supporting that claim, and, as nece5~ary, ~eeting objections to the pre~ises one relied on or the reasoning one employed.
To self-consciously regulate one's co~nitive activities, the elements used in those activities, and the results produced by those activities, particularly by applying analyzing and evaluating to one's(own inferring with a view toward confir~ing, validating, correcting or questioning either one's reasoning or ene's results. 6.1 SELF-EXAMINATION: To reflect carefully on one's own reasonIng and verify both the results produced and the correct application and execution of the cognitive skills involvedl to make a thoughtful meta-cognitive selfassess~ent; to reflect on the extent to wh:ch ~ne's thinking is in~luenced by deficiencies in one's knowledge, or by stereotypes, prejudlces, emotions or any other factors which constrain one's objectivity or rationality; to reflect on one's ~otiyations, values, attitudes and interests with a view toward deter~ining that one has endeavored to be unbiased, fair-~inded, thorough, uujective, respectful of the truth, reasonable, and ratienal in '-:oalingto one's interpretations, analyses, evaluations, inferences, or expressions. 6.2 SELF-CORRECTION: Where self-examination reveals errors or deficiencies, to design reasonable procedures to remedy or correct, if possible, those ~i5takes and their causes.
tttttttl
0; MONITORING:
ENOORSEMENT~
1) Do you endorse the above statement as useful for purposes of assess i ng tb~ skills dimension of CT at the lower division coLl ege level? 2) Would you be willing to l'ave yo~r name listed in association with the above description of CT skills as a contributing ~ember of the Delphi research project which generated it?
*****
Lettere.
PAGBB9
f,·~,
.. I
''/'"i·:'
93
The responses to ROUND 4 included so~e telling ob~ervatiDns, thoughtful objections and tlell f o cu s e d c r i t i c i s ns , Sharing these WIth others In tlla lJQlphi qr o up is e s s en t i a l to t h e DL'lphi pr cc e s s , L'v e i nc l u de d l1r:, nJ an y 115 I S f Q iI sib I e and u ~) C u n bell n cJ P. r 5 too dOLI t sId o t h Po call t C! l: l af whatevr.r ROUND 4 item mt'lY hay£! prOr.Jpted t h e e , nather than use of your time reading positive comm~nts, of Hhich there were many, I've stuck chiefly to the critical one$. INSTRUCTIONS: Read and consider what our colleagues are t~lling us. In addition to the adjustments already incorporated into the above draft statements regarding CT and CT assesscent, Nhat other re5ponses and improvements should we make? In the jight of these comments and other concerns that come to ~ind, what specific rer.om~endations should we include in our report to the APA CO~Dittee on Pre-College Philosophy?
$ f L £ r T El'.. R £ S PO It ~~f S Q.!i.
£l T f S T S d1iQ. T F: S T 1 H GeT
• "A test cannot be considered in the abstract, without Horklng out its intended use and intended ~sers, the specific population to be tested, and the discrimlnations the test would be required to ~ake.,. The domain the test covers is governed by this conte~t, and concepts that constitute s~b~lassiflcations of the domain are arranged in different ways fro~ the Hays in ~hlCh they mIght be al ranged for a test WI th the s a e a name but a dl f ferent purpose... CT can be sub c l a s s i f I £?d in a nu~ber of leglti~~te Hays, WIth ~any of the same ele~cnts recurrIng in dlfferent plac~s jn dIfferent claSSIficatIon scheDcs ••• "
hardJy anything "There's tests. 1 've used ••• in pre-post best 1 knoH of • (Su t ••• J
II
He now testIng
CT th e
• " ••• to ~ake sense of CT ~e nust thInker. The cognitive ~usl be dIscussed and both ~U5t be discussed in relatIon world. Curriculu~ and assessment r.ust be ••• There is no one right definItion of never confuse testing for ~lcro-skills Host CT tests are ~Icro-skill tests only. in a qualified Hay. The Delphi project confusing the part with the \1hole,u
I
lake sense of the critIcal in relation to the affective, to thelr roles in the real put into some broad context • CT, and in testing He should WIth testIng for CT itself. They are valuable, but only see~s Hell on its Hay tONard
"The [Round 4 list of skIlls] see~s fine to ~e. lhere are definite liAlts to ar~chair analYSiS, and untJl s~.e~ne actually ~tarts trying to .easure these thJngs, It i~ dIffIcult to kn~~ just ho~ t~ revise the list of CT skills and sub-s~ills." uPlease interpret ny responses r.autlously. ~uch time trying to definH CT. My business assess~ent once you 9uys arrive at clear and usei~l
I
i5
trying deflnltlons!"
0
1 have
II
We
tJ
us t f 0 c U son
the
au d I ~ nee
fan
'; 1
Pl~GE 90
94 \
.. .-'._:~:~I
. ~~.'";
that way can we fine tune our definition "theoretical construct" Of CT. II
of
CT,
or
"Rather than limlt our conc:eption of CT to achieve a certain kinti of test, why not simply make more modest claims about the test? Why not say that you are tasting dSp'Cts of CT? This ~~~~~ ~ore justified. Otherwise the test becomes the tail that wags the dog ••• Be honest about what ~~ are a~d are not testing by multiple choice tests. L~t's not reduce rich and complex realities like CT to that which can be directly tested in th~ multiple-choice format."
f [Some of your descriptionsl are diificult to te~t without openended items ••• [For examplel it's difficult t.o t£st {or~ulation of strategies, He e~ams focus more on recognition of best ~tr~tegy,
liThe categorization syste~ ••• gives little guidance on what pr ec i s e l v to teach and test. For instance, under "evaluating" is lito assess the credibility ~f state~ents.N But what should be taught when doing this, and Hhat ~hould b~ tested when trying to find out what skills st~dert~ have? If teachers ~re supposed to act based on what He produce, I believe we need to incl~de criteria for assessing N credibility. Again I refer you tOt ••
t
conn~cted in those individuals Hho are successful critical thinkers to a pervasive self consciousness ~bout one's OHn thinking and reasoning processes. Such self consciousness should be deliberately cultivated in our students and should therefore be included somewhere in the overall description -- it is an intrinsic part of the (CTl process. It monit, s the selection, applitation and interlillkage of the va~jous relevant processes."
t
"eT is deeply
A H ~ D f F 1 H 1 H 6'
rL
IIUntil we have a theory of reasoning (a cc~bined nor~ative theory of infor~al logic along Hith a descrlptive theory of cognitive processes) He will not be able to spell out CT skil!s non-arbitrary Hith ccnp Ie t ene s s and precision .... We ar e procfucjng ... a frallJeHork, [that isl a list of concepts used for underst~nding a do~ain. If that i5 so, it Hill play hell out of our attelDpts to assess CT skills. 1 doubt that we will be able to manage construct validity for any ccnventio"~l MC test with our [list of CT skills]. Vet, Hhat we are coming up Hith is p.~tre~ely valuabl~ if we focus on performence ass es s a\ e :1t • Our f r a II e Ii 0 r k g i v e sus a pas s a b 1 y g cod set 0 fer i t eria by which one would judge goud perfor~an,e on CT tasks. The criteria are devel~ped by experts -- us -- and we are currently judging whether He accept the~ or reject them, another step in the process of developing a goo~ perforlDance assess~ent. The next step would be to distribute typical essays, (good, bad and ugly ones) and aim at some consensu~ in telling the good ones from the bad ones."
I
"We
should
resist
the assertion
that
CT is dOl'llain dependent."
donel What
beginning to think that to have done all that [Me have and not have tried to define CT may turn out to be a mistake ••• lakes [the list ofl itlportant, indeed basic, intellectual skills
tlI'~
95
. ,~./;.;:
••
_ ....
_, __ '''._ ••
__ ,...• ' .
('.";?~,~ ;-;,~..
_._T
-_
[ in R 0 u n d 4 1 ali s t o)f cr i tit; a J t h ink i Il g 5 j.: i 11 s? I teo u 1 d ; ~ s t il S well be a list of rational thinking ski lIs, or logical thinking skills or h rg her 0 rd er cog r.it i ve ski 11 s • • •• I fa i 1 to see h 0 H th i 5 1 is t captures the force of the word "e r t t i e al ", [EtY'Dologicallyl the meaaninQ of "cr t t i c ar " is judging, evaluating, estimating the worth of something ••• A critical thlnker is so~eone ~hu renders an oplnion on an intellectunl product ••• by assessing the strengths and weaknes$es of that product, •• Doing s~ requires the capacity to elicit and apply standards, principles and crlteria, None of the [Round 4] list is really this s~ill. If we as~ed for a list of problem solving skills, Nould we get the same list [as in Round 411 If S~, then either there is no conceptual difference between the two, which 1 think is wro~g, or else the list fails to capture what is distinctive about C~ s~il1s. CT also connotes IIcrucial". tHere) the (Round 4J list fares better, because ••• these six ski 11 s are cruci al --i. e., essential for intellectual survival. The problem is that the list is ~o bread and wide-ranging that it is not clear ~hat intellectual skills have been excluded.1I If 0 n e ide n ti fie 5 C T as that w hie h m a I: sac e r·itJ s: alp er sen to be Hhat he is, t~en [yourJ narrow concept of CT is inadequate. CT is", an anSHer to the goneral proble~ of conformity, prejudice, narrow-mindedness, ~nd irrationality in the world. CT is what one does to achieve autonomy and independence of thought, to lesson one's prejudices, to broaden one's perspective, and tu become ~ore fitional,»
I
II
liThe ~ain overall Horry is that the categories are Iuch too When He get down to testing it will not be for something like "assessing loqical strength or IIclarifying" but very specific skills such as "rer:ognt~ing whether so~ething is a necessary or sufficient condition; recognizing the difference between if p, then q's and if q the p's etc. The lines of de~arcation are very unclear e.g. between psychological and logical (epistemological) criteria con~erning say observati on. II
t
broad.
ll
t III have no major quarrel with any parts of your organization, [butJ your emphasis leaves out a major compcnent of CT the dispositional component and the ~et of values inherent in being a critical thinker.,. I think it is a good working model of CT skills, but an incomplete picture of being a critical thinker.1I
"There is no attention ••• to the dispositions of CT. These... are as essential to CT as are information used in the ~~ocess~s.~
I t
••• the
suggest
I
are at least
as i e.p c r t e n t as the
IICT works by recognizing and criticizing sourcec; of inforllation, by drawing i~plications irc~ given ~aterials, identifying assumptions, noticing relationships of consistency, inconsistency, i~plications and contradiction, inferring interesting consequences, recognl~lng, analyzing and evaluating argu~ents and c~nstructing them as Mell. Of course, there's a lot ncr a to it.·
PAGB 92
.,: ·.... 01
96
..~{:i'~r
.'
')
\',
"
-~~
'.-'.'~' ..
.... ,',c. J:
,·:V:
"I
."
f
.-:
"Arguing
is not li5te~
as a separate
h
f
. <
would
* "I di~agree with the list of CT skills as described. Interpreting is obviously a cognitive operation. But it is not a CT operation. Expressing is essentially ION level communicatiun, not g~nerative in the sense of CT. Monitoring is 'meta-cognitive. The problem is that this description is so broad "critical" thinking gets lost in all the other kinds of thinking. This blurs the nature of CT beyond ~ecognition... CT is "judging the werth, accuracy or sigr.i·[jcance of something."
"There are several items I mess overall. They may be subsumed in some of the processes you have listed, but very few people will be conscious of them unless they are brought out explicitly. One is the capacity for arithmetical reasoning with ratio and di\ision it begins Hith word problems in 5th and 6th grade arithmetic and carries up to exactly similar reasoning with concepts such as density, composition, contraction, rates of change, in more sophisticated settings. It includes the ration reasoning that goes with scaling areas, volumes rates, etc. This capacity is profoundly important in any CT that involves nu~erical inforBation (whether it be scientific, economic, sociological, psychological ... I'ID talking about a."jthletic and not sathe.atics at the level of calculus or even algebra. CA second capacity to include is] "correlational reasoning." Finally I miss er.plicit inclusion of the process of translating symbols (e.g. graphs, numerical data, hjstogra~s) into words or Hords into corresponding sYlDbols. Such tr anslation is essential to lIuch CT."
f f
"CThe concept of CT should also include] discriminating betHeen the fact~al or experiential input and the inferences identifying gaps in available infor~ation and identifying or superfluous infornation, and it should include the to consider so~e situation in the abstract and, by applying governing pr1~ciples or constraints, arrive at reasonable and conclusions about the outcomes that would result fro~ the of so~e change -- hypothetico-deductive reasoning,-
the "Include so~ething about ~pposlti~hal rpasonlng -- taking hypothetical part of one with whom one disagrees. Also include reas~ning -- reasoning fro~ suppositions and hypotheses,
t "Taxo~omy" is not a good word fer ~hat we've got; it's more like a list. H~vJny a taxonomy in biology and education is to have a hierarchical set of categories such that each subsequent step in the h1erarchy subsumes t~p. steps below it. We don't have that here.
U[ConjecturingJ
is creative seeking
thinklng
NCRegarding
expressing,J
PAGE93
';;h,. ~
,
"~.'
;'..;, ._.-. .
97
, ':.>:<:,i
.
\1; .
,.- ....
.:,"
..
extending
II
the focus
to communicating
the results
of CT.
* Non e 0 f [y 0 u r s i :: 1 i s ted i t EJ m s are s ~ ill 5 - - 1 eta J Ion e subs~ills. They are general categories into Nhich many (at least do:ens of) distinguishable skills may be lumped. This ~s important.
TABLe
QL R£SPOHS£
1- INTERPRETING
2.
3. 4. 5. 6.
Observing Decoding Clarifying INFERRING Querying Conjecturing DraHing Conclusions Analyzing Locating Arguments Parsing Arguments Evaluating Claims Verifying Assessing Strength Logical Expressing Stating Results Procedures Describing Honitoring Reg u 1 " ti n g Reviewing Correcting
ReSPOHSES
.I.~
.,26 ., ..
.I.~
24 20
2 s 3
..
24 23 24 23 23 23 24
2S 26 21
., ..
1
"-
.,
3 1 1
3 2 3
., ..
,)
3
.)
2:S
23
21
21 i1
..
3 .. 3 .,
'" 3
3
24
'"
.,
of on
r.Jl..
'UHAT
To help delineate CT through co~parisons and contrasts, I offered a list a~tivitles that bore so~e family resemblances to CT. Each of the~ depends CT. Sut whether any of Has CT per se was ~he issue. Here's what you said: Isn't (Seeing, Touching, Hearing, etc.) 2. Reading, Listening, 3. Speaking or ~riting, 4. Motivating, persuading, selling, S. Interrogating, cross-examining, petitioning, 6. Fhysically investigating the world around, 7. Trouble-shooting, problem-solving, pu:zle solving, 8. Decision-making, selecting, choosing, exercisin~ will, 9. Planning, defining goals and objectives, 10. Finding or ascribing a ~eaning to art, life, or events, 11. Defending an opinion or belief, arguing a case, 12. Ma~aging, ad~inistrating, governing persons or things, 13. Philosophizing, 14. Conductlng research within any particular discipline, 15. Experiencing, feeling, emoting, or empathizing, 16. Co~cunicating u5i~g language.
1. Sensing,
£L ll. HOT"
U
16
13 14
1 .;
U.
4 3 4 4
4 3 7
5 7
ll.
2
Pu
rr 5
6 6 5
6
t!.1..
12 13
13
14
10 13 16
6 6 6
6 8
12 12
15
3 3 3
2
8 8 5
6
15
PAGB94
9.8
~.~.'..
--~
" "
.....
"
California
Fullerton. California
State University,
92634
Fullerton
This letter Tollows the outline in the Round 5-A letter -questions Tirst, background information second. The focus here is on fleshing out our. conceptualiza tion oT CT beyond the revised list of CT abilities presented in Round 5-A. Specifically this letter responds to your comments regarding those dispositional and normative dimensions many include when describing CT. The implica tions for K-12 and college level assessment, curriculum development and pedagogy oT including either of these dimensions are crucial for our efTort. You're asked for your ideas about this, too. To get things started, on the ne>:t ;lage you'll find a draft statement r'egarding the dispositional dimension oT CT. Following ".hat is u druft sta:ement regarding the normative dimension. These two draTt statements are based on your comments regarding ne~ding a fuller conc~ptualization of CT and an analysis OT commonly referenced concepts of CT which appear in the literature. Have at those draft statements. Amend, edit, comment, uccept, reject -- whatever- you think will help us be able to present an intellectually credible and educationally useful conceptualization of CT. Coming soen: Round 5-C focuses on our report to the APA Pre-College Philosophy Committee. It will include a proposed outline of that: report, showing how the various pieces of the assessment puzzle we've wor~:ed on for over a year now will be incorpora ted. It also picks up the two pieces from Round 4 which haven't been addressed yet, namely the Inpu t-output model (which we rather roundly rejectedJ, and the statement of what a cognitive sJ:ill is and how cognitive skills might be asses~ed. I'll be speaking on March 24 at 1:00 p.m. al the Pacific Division meetings of the APA in Oakland CA, sharing a little of what we've been doing and the direction things seem to be taking. If you happen to be in the neigtlborhood, please .stop by 50 we can visit. Knowing you are very busy, I beg mercy and ask you to reply as as is reasonably possible. A quidde note with general comrnerrt a is helpful, if you won't have time to go through things in careful detail. Thanks for your continuing support and involvement. sincerely, soon
Yours
~
e_,.
Pete Facione
99
Part:, !l The Dispositional Dimension e£ CT INSTRUCTIONS Re-flect on the -following statement. Edit, amend, or revise as you see "'it. Some backgr·ound information is presented in the second half of t.his letter. Please respond to the following questions: 1) Should our- final report include a statement on the dispositional dimension of CT? Why, why not? 2) What implications or- recommendations for K-12 ,and college lower division level assessment, curr~culum dev~lopment and pedagogy follow from inc:.luding a dispositional element in our conceptualiz,3tion of CT?
CT -- The Dispositional
skills traits use. co~ceivin9 perso~al
Dimension
In addition c09~itive crucial ~bilities i~structional laterial~, dispositions. ilportant instructional and personal $kills tha~ discipline disposJtions, ~ seeki~g ~ curiosity ~ eagerness ~ openness • trust ~ honesty ~ willingness ~ prudence ~rle's position,
~ ~Jsdol
to its cog~itive and successful of CT, ~hen in CT, strategies the use Perso~s listed beloN ~ho knoN
CT also
involves ~hich
an
certai~ are
di~po;itions, to its broad diDensio~ progral teaching in insuring sftti~g. traits are those
virtues
or
of CT assessaent ~hich
it is i3portant and assess~ent of CT cognitiye of C1 abilities Nho have are ~uch the skills
I~re
CT
The cultivation
dispositions
is particularly
CT
virtues their
CT
The crit~cal
personal
traits Itatelent
or problels,
in elploying
CT abilities, inquiry, i~ keeping ONn credible and one's reasoning sources, in taking cha~gin9 and to the
divergent
~ perseverance,
judglents
in the use of appropriate, reasonably the subject and to relain per.its, issues
defensible relevant
relevant basic
the extent
in the treatlent
100
I •.
t/;. :'"
- . --,. '.:~ .
~·-·r:
Part 2: The Normativ~ Dimension of CT INSTRUCTIONS Reflec:t on the following statement. Edit, amend, or revise as you see fit. Some background information is presented in the second half of this letter. Please respond to the follo.~ing questions: 1) Should our final report include a statement on the normative dimension of CT? Why, why not? 2) What implications or recommendc:\tions for ~~-12 and college lowt=~r division level assessment, curriculum development and pedagogy follol.~ from including a normative element in cur conceptualization of CT7
CT -- The Hor~ative In addition also involves Understanding descriptive lust be I~re that analysis, than values used, rather used, should used, used, to its cognitive certain norlative in laking ~e judge of CT. training Education ~hich insure freedol the blind, defective those sound rational
any
Dilension and dispositional ~hich 90vern this Ne are going and rational than dilensions, use. a purely cDlpanent education of a set of of those qenera~ions. a~( resolutio~ of vie~s in defense civic fairbeyond noraative society
~T
its proper
this
it vital
to include
include be passed
and personal
the herita9~
CT co~tributes
to be intellectually
and intellectually
process,
and analyses Properly the objective Pr~perly equal abusive. Thus, assesslent developing the nor.ative respect
freedol,
and
investiqation
issue
CT treats
or ~ithout open
sensitivity
and ~ith
coercive.
to consider tools
of CT ~ill
be developed
in students,
in addition
CT skIlls
and di$posit,ons.
Appendix
C: Delphi
Research
Letters, PAGE 97