J. Behav. Ther. & Exp. Psychiat.

48 (2015) 66e74

Contents lists available at ScienceDirect

Journal of Behavior Therapy and

Experimental Psychiatry
journal homepage: www.elsevier.com/locate/jbtep

The utility of case formulation in treatment decision making; the

effect of experience and expertise
Robert Dudley a, b, *, Barry Ingham b, Katy Sowerby a, Mark Freeston b, c
a
b
c

Doctorate of Clinical Psychology, Newcastle University, UK

Northumberland Tyne and Wear NHS Trust, UK
Institute of Neuroscience, Newcastle University, UK

a r t i c l e i n f o

a b s t r a c t

Article history:
Received 7 July 2014
Received in revised form
17 January 2015
Accepted 22 January 2015
Available online 18 February 2015

Objectives: We examined whether case formulation guides the endorsement of appropriate treatment
strategies. We also considered whether experience and training led to more effective treatment decisions. To examine these questions two related studies were conducted both of which used a novel
paradigm using clinically relevant decision-making tasks with multiple sources of information.
Methods: Study one examined how clinicians utilised a pre-constructed CBT case formulation to plan
treatment. Study two utilised a clinician-generated formulation to further examine the process of
formulation development and the impact on treatment planning. Both studies considered the effect of
therapist experience.
Results: Both studies indicated that clinicians used the case formulation to select treatment choices that
were highly matched to the case as described in the vignette. However, differences between experts and
novice clinicians were only demonstrated when clinicians developed their own formulations of case
material. When they developed their own formulations the experts' formulations were more parsimonious, internally consistent, and contained fewer errors and the experts were less swayed by irrelevant
treatment options.
Limitations: The nature of the experimental task, involving ratings of suitability of possible treatment
options suggested for the case, limits the interpretation that formulation directs the development or
generation of the clinician's treatment plan. In study two the task may still have limited the capacity to
demonstrate further differences between expert and novice therapists.
Conclusions: Formulation helps guide certain aspects of effective treatment decision making. When
asked to generate a formulation clinicians with greater experience and expertise do this more effectively.
Crown Copyright 2015 Published by Elsevier Ltd. All rights reserved.

Keywords:
Cognitive therapy
Formulation
Treatment
Expertise

Case formulation is the process of blending the theoretical

framework and scientic knowledge the clinician brings with the
unique experience of the client to help understand the presenting
issues, and to select the optimal treatment (Kuyken, Padesky, &
Dudley, 2009; Mumma & Mooney, 2007). Consequently, formulation is considered to be at the heart of effective Cognitive Behavioural Therapy (CBT, Butler, 1998). However, the status afforded
case formulation is somewhat mismatched with the relatively scant
evidence base (Bieling & Kuyken, 2003).

* Corresponding author. Doctorate of Clinical Psychology, Ridley Building, Newcastle University, Newcastle Upon Tyne, England, NE1 7RU, UK. Tel.: 44 191 222
7925.
E-mail address: r.e.j.dudley@ncl.ac.uk (R. Dudley).
http://dx.doi.org/10.1016/j.jbtep.2015.01.009
0005-7916/Crown Copyright 2015 Published by Elsevier Ltd. All rights reserved.

Most of the previous research on formulation has examined the

extent to which clinicians agree with each other or with an expert's
formulation (e.g. Dudley, Park, James, & Dodgson, 2010; Kuyken,
Fothergill, Musa, & Chadwick, 2005). Clinicians demonstrate
modest levels of agreement (see for example Dudley et al., 2010;
that demonstrated most elements of a formulation were not agreed
on by over 70% of clinicians); although this is better for more overt
presenting issues (where agreement was often over 70%). Also,
clinicians with advanced training, more clinical experience, and
accreditation as a CBT therapist, produce more reliable formulations (Kuyken et al., 2005; Persons & Bertagnolli, 1999).
Research considering validity has shown that therapist expertise (Kuyken et al., 2005) and training in case formulation
(Kendjelic & Eells, 2007) improve the quality of case formulations

67

R. Dudley et al. / J. Behav. Ther. & Exp. Psychiat. 48 (2015) 66e74

produced and lead to higher quality treatment plans (Eells &

Lombart; 2003; Eells, Lombart, Kendjelic, Turner, & Lucas, 2005).
Research into the utility of formulation has considered the
relationship between CBT case formulation and therapeutic
outcome and has produced mixed results (Chadwick, Williams, &
Mackenzie, 2003; Ghaderi, 2006; Schulte, Kunzel, Pepping, &
Schulte-Behrenburg, 1992). These studies have a number of methodological limitations one of which is that the quality of the
formulation has generally not been evaluated (Mumma, 2011). So
whilst formulation is afforded a central role in CBT the evidence for
its reliability, validity and utility is mixed and at present it is not
clear how formulation affects treatment outcome (Kuyken, Beshai,
Dudley, Abel, Gorg, Gower et al., 2015).
Of course, multiple factors potentially impact on treatment
outcome. Owing to the complexity of linking any one aspect of
therapy to outcome it may be more helpful to investigate the
impact of formulation on an intermediate feature such as treatment
planning (Eells et al., 2005). The focus of the two studies reported
here is whether formulation guides certain aspects of treatment
decision making and planning.
There were two aims to this research. The rst was to establish
whether formulation guides the ratings of treatment interventions
that do or do not t the formulation. The second was to consider
whether greater training and experience leads to more effective
use of CBT case formulations when making these ratings of
treatment relevance. To examine these questions two related
studies were conducted both of which used a novel paradigm
using clinically relevant decision-making tasks with multiple
sources of information. The rst study examined how clinicians
utilised a pre-constructed CBT formulation to make treatment
decisions. The second study utilised a clinician-generated formulation, rather than a pre-constructed one, to examine the process
of formulation development and the impact on treatment planning and decision making. Both studies considered the effect of
therapist experience.
1. Study one
If formulation guides treatment, then formulations that differ in
key features should lead to different interventions (Butler, 1998).
Beck, Epstein, and Harrison (1983) identied two dimensions
important in the development of depression: autonomy (i.e.
stressing the importance of independence and freedom of choice)
and sociotropy (i.e. stressing the importance of afliations and
interpersonal relations). Therefore, a formulation of a person with a
sociotropic depression would differ from that of someone with an
autonomous depression, even though both people would meet
the criteria for major depressive disorder.
Previous research has indicated that greater experience in CBT
leads to better performance on formulation tasks (Kuyken, Padesky,
& Dudley, 2008; Mumma & Mooney, 2007). Hence, greater experience in CBT should increase recognition of such features (i.e. autonomy and sociotropy) and guide clinicians towards treatment
focussed on the key dimension rather than less pertinent features.
Conversely, less experienced clinicians may be more distracted by
less pertinent, or plausible but actually unhelpful or irrelevant
treatment options.
The primary hypothesis is that the content of a CBT case
formulation would help the therapist rate as more appropriate
formulation matching interventions rather than less pertinent or
mismatched interventions.
The secondary hypothesis was that more experienced clinicians
would endorse more formulation matched interventions than the
novice clinicians and endorse less the less pertinent or mismatched
treatment options than novice clinicians.

2. Method
2.1. Design
A mixed (between-within) design was used. The withinsubjects (formulation type with two levels, sociotropy or autonomy) manipulation was used to test the main hypothesis that CBT
case formulations have an effect on ratings of treatment options.
The between-subject manipulation (experience; two levels, novice
or experienced) tested the secondary hypothesis that differences in
clinician experience would account for variance in performance.
2.2. Participants
Two groups (labelled as novice and experienced practitioners)
were recruited. The novice group consisted of 23 clinicians who had
an introductory training in CBT and a limited amount of practice in
the use of CBT (e.g. rst year trainee clinical psychologists).
The experienced group included 20 clinicians with extensive
experience and training in CBT. The sample came from a variety of
backgrounds but were clinicians who had supervised others in
training and practice of CBT (e.g. experienced clinical psychologists,
psychiatrists, nurse specialists in CBT). The demographic information for the participants is shown in Table 1.
2.3. Measures
A novel task was developed to assess use of CBT case formulations to plan treatment. Participants were presented with two
prepared case formulation vignettes followed by multiple-choice
options of potential CBT treatments. Participants made judgements as to which treatment planning options were the best t for
the presented case formulations.
Information about the case was presented sequentially to the
participant. This consisted of; a referral letter from the client's GP
relevant history from childhood to present day including early
experiences and problem history, the results of assessment measures relating to emotional distress, a completed thought record
and a completed activity schedule. Then an initial psychological
conceptualisation (described as developed after two assessment
sessions) outlined an early working hypothesis for the development and maintenance of difculties. Finally, a full formulation was
provided. The model for the CBT case conceptualisation diagram
(see Fig. 1 for an example) was based on the cognitive model of
depression (Beck, Rush, Shaw, & Emery, 1979) and featured a diagrammatical representation of the longitudinal (which incorporates information about early experience and how this
predisposes a person to emotional problems owing to their core
beliefs, rules and assumptions) and cross-sectional (also described
Table 1
Demographic information (including professional experience, CBT-related qualications, further/higher education) within clinician experience groups.
Novice (n 23)

Experienced (n 20)

Gender (n, % female)

Diploma in CBT trained
BABCP accredited
Supervised on Diploma
level CBT training

N
N
N
N

N
N
N
N

Mean

Sd

Mean

Sd

Age (yrs)
Further/higher education (yrs)
Months of clinical experience
Months qualied in profession

31.1
4.1
14.3
41.8

8.3
1.4
14.3
16.2

41.8
8
117.2
175.4

6.5
2.4
64.2
86.4

17, 74%
0, 0%
0, 0%
0, 0%

11,
15,
10,
18,

55%
75%
50%
90%

68

R. Dudley et al. / J. Behav. Ther. & Exp. Psychiat. 48 (2015) 66e74

Fig. 1. CBT case conceptualisation of Jess's difculties.

as a maintenance formulation which emphasises what factors lead

to the perpetuation of distress, such as the Padesky ve factor
maintenance model, Greenberger & Padesky, 1995) case formulations (see Dudley & Kuyken, 2013; Kuyken et al., 2008).
To establish the extent to which the formulation inuences
judgments of relevance of treatment options the two vignettes
differed in content. The rst formulation vignette (Jess) was based
on a published case of depression (Blackburn & Twaddle, 1996). The
second vignette (Gerald) was based on the same conceptualisation
model, but differed on a key theoretical dimension. Hence, Jess's
vignette was high autonomy and low sociotropy, and Gerald's was
high sociotropy and low autonomy. Information consistent with
sociotropy or autonomy featured at the longitudinal (e.g. in the
early experience and core beliefs) and cross-sectional (e.g. forming
the basis for maintenance cycles) systems of the CBT case
formulation.

The two cases (and their formulations) had to be equivalent in

their accuracy, complexity, coherence, parsimony and overall clinical credibility. If they differed except on the key dimension of
sociotropy and autonomy, then it would not be possible to
demonstrate that different performance was not owing to some
confounding variable. To help ensure these criteria were met, one of
the cases was derived from a published case study and served as a
template for the second case that was based on the clinical work
undertaken by the authors.1 Then, three experienced CBT clinicians
acted as an expert reference group and completed a rating sheet

1
The rst case material was derived from a published chapter. The second was
an amalgamation of several cases the authors had worked with. In addition, the
cases were anonymised with identiable information removed, and or disguised to
ensure that the person/people they were based on where not identiable.

R. Dudley et al. / J. Behav. Ther. & Exp. Psychiat. 48 (2015) 66e74

69

Fig. 2. An example treatment planning question following from Jess's formulation.

that provided feedback on the consistency and coherency between

and within vignettes. The experts were asked to assess each set of
materials for i) the coherence of the assessment information and
conceptualisation; ii) the quality of the case conceptualisation; iii)
how realistic the cases were; and iv) the coherency of the treatment
planning options in relation to the case. Each was rated on a scale
from 1 to 10 (with 10 being high quality, coherence etc.). The experts endorsed all the items above 7. Hence, the case materials were
considered to be equivalent. The formulations were also endorsed
as differing on the key dimension of sociotropy or autonomy. The
reviewers also completed a quality of case formulation measure
(Kuyken et al., 2005) and both of the experimenter provided formulations were rated good enough on the quality of case formulation measure.
Two sets of CBT based treatment planning tasks (with 14
questions in each set) were developed that followed from Jess and
Gerald's formulation vignettes respectively. Participants had to
judge how good a t those options were with the formulations (see
Fig. 2 for an example of a treatment planning question). For each of
these questions, three options were presented. Each option was
either a good match (pertinent), mismatched (not pertinent) or a
plausible but irrelevant option (ared herring). The treatment
planning questions were taken from CBT manuals (e.g. Beck, 1995)
and included problem list development (1), goal setting (2),
behavioural experiments (2), use of thought records (2), activity
scheduling (1), role plays (2), continuum method (2), relapse

prevention planning (1) and identication of potential therapeutic

barriers/problems (1). For each of the two vignettes (Gerald and
Jessica) the questions were equivalent (e.g. the same number of
behavioural experiment, or cognitive restructuring interventions
were presented). Within each question for each of the options
participants were asked to rate (on a scale of 0e10, where
0 doesn't t and 10 denitely ts) how good a t that option
was to the CBT case formulation.2
It was important that the treatment options really were regarded as matched, mismatched or as irrelevant. The match treatment
options were taken from the case study as published for Jess, and
for Gerald were based on a round of feedback and discussion with a
group of experienced therapists before being presented to an
expert in CBT. Then the materials were piloted with participants
representative of the potential sample (i.e. novice [n 3] and
experienced [n 3] clinicians). A further round of feedback from an
expert clinician was then completed prior to full experimental
testing.
In keeping with the rst hypothesis it was anticipated that all
the participants would rate the match option higher than the
mismatch and irrelevant options. Consistent with the second hypothesis the experienced group was expected to endorse the match

2
Copies of all the materials including treatment options are available from the
corresponding author on request.

70

R. Dudley et al. / J. Behav. Ther. & Exp. Psychiat. 48 (2015) 66e74

option more than the novice group. The novice group was expected
to less strongly endorse appropriate treatment options and more
strongly endorse the red herring and irrelevant options.
2.4. Procedure
Participants were recruited from local psychology and CBT
training courses and were provided with an information sheet, and
signed a consent form. The participants were told to imagine that
they were the therapist and that they need to plan treatment based
on the assessment information and the case conceptualisation.
They were instructed they needed to decide how well the treatment options t in relation to the case. An example unrelated to
either of the two subsequently presented cases was provided to
help familiarise them with the task. Each participant was presented
with both of the case vignettes. The order of presentation was
counterbalanced. After reading through the case information and
formulations, participants worked through the treatment planning
task. The order of the treatment planning questions and options
was also counterbalanced throughout.
2.5. Review procedures and ethics
The work was subject to independent peer review, was registered with the Research and Development Department of the local
NHS trust and received a favourable opinion from a Local NHS
Research Ethics Committee.
3. Results
Preliminary analysis revealed no missing data. Extreme responses were identied by boxplot and winsorised. Data analysis
was completed using SPSS 20 for Windows (SPSS, 2012).
Ratings on the response to treatment planning tasks are outlined in Table 2 that shows the ratings across the two groups and by
vignette.
A mixed ANOVA with a between subject variable (experience
with two levels [novice/experienced) and two within subject variables (vignette type with two levels [Jess/Gerald], and response
type with three levels [match/irrelevant/mismatch]) examined
differences between the ratings for responses to treatment planning tasks. The analysis was also run with Order as a between
subjects variable. However, it had no bearing on the results and is
not reported further. The main effect for vignette type across
treatment planning options was not signicant (F(1, 41) 1.26,
p 0.27, h2 .03, CI .00, .18). There was a main effect for response
type (F(2,40) 388.01, p < 0.001 h2 .90, CI .84, .93). Mauchley's
test of sphericity was signicant so the lower-bound Epsilon
correction was used when determining F values. Planned simple
contrasts showed that match responses were signicantly higher in
rated t to the vignette and provided case formulations than both
irrelevant (F(1,41) 221.85, p < 0.001 h2 .84, CI .74, .89) and
mismatch (F(1,41) 516.85, p < 0.001, h2 .93, CI .88, .95), and

irrelevant responses were signicantly higher than mismatch

response types (F(1,41) 365.62, p < 0.001, h2 .90, CI .83, .93).
This is consistent with the rst hypothesis.
Within the above ANOVA, there was no effect for experience (F(1,
2
41) 1.35, p 0.19, h .04, CI .00, .19). There was also no signicant interaction effect between experience and response type
(F(2, 40) .10, p 0.76, h2 .00, CI .00, .09). Experience did not
have an effect on treatment decision making hence hypothesis two
was not supported.
4. Discussion
This study considered whether a predetermined formulation
guided treatment planning decisions and whether CBT experience
effected these decisions. As expected, participants rated match
responses signicantly higher on t to the formulation than the
other options. This supports previous case formulation research
proposing a link between formulation and treatment plans (Butler,
1998). However, note that the match option was not endorsed at
ceiling, and mismatched and red herring were not at oor, so
overall the treatment plans, which clinicians may develop, would
include a proportion of matched elements they would also include
less than optimal elements.
There was a second hypothesis that clinician experience would
effect how CBT case formulation was used to inform treatment
decisions. It was predicted that novice clinicians would more often
endorse the red-herring/irrelevant and mismatch options. However, there was no difference between the novice and experienced
practitioners. This may be owing to limitations within this study
outlined below and that are addressed in the subsequent study.
This study indicates that clinicians who are provided with a case
formulation and a list of treatment planning options are able to
choose options that t with the formulation, regardless of level of
training. Perhaps knowledge of CBT techniques provided through
basic CBT training is sufcient, at least in this type of task to plan
treatment. Previous studies have shown that more experienced and
expert clinicians construct higher quality and more reliable CBT
case formulations (Persons & Bertagnolli, 1999). Hence, the difference in experience may not be in the use of the formulation for
treatment planning but in the development of the formulation.
A number of potential limitations need to be considered when
interpreting the ndings. First, in this study clinicians rate predetermined treatment options from a number of possible options.
Essentially, this was a recognition task. This is easier than when the
clinician generates their own treatment option, and may help account for the lack of differences between the experienced and
novice clinicians. Second, the task involved using a constrained,
prepared formulation and this may have reduced the potential for
difference between the two groups. If clinicians had a greater role
in constructing a formulation then this may lead to differences in
the formulations produced and increase the likelihood of detecting
differences in the resultant treatment plans based on these
differing formulations.

Table 2
Mean ratings for response type across the two clinician experience levels and vignette.
Jess

Novice
Experienced
Total

Mean
Std. Deviation
Mean
Std. Deviation
Mean
Std. Deviation

Gerald

Match

Irrelevant

Mismatch

Match

Irrelevant

Mismatch

7.8
0.8
7.6
0.9
7.7
0.9

5.3
1.2
4.7
1.5
4.8
1.5

2.7
1.6
2.1
1.3
2.2
1.4

8.0
1.0
7.5
0.7
7.9
0.9

4.7
1.3
4.2
1.4
4.3
1.3

2.5
1.3
2.5
1.3
2.4
1.2

71

R. Dudley et al. / J. Behav. Ther. & Exp. Psychiat. 48 (2015) 66e74

Third, the inclusion criteria used to distinguish between the

novice and experienced groups may not have been sufcient to
distinguish the two. Often samples have been composed of experienced rather than expert clinicians, which may make the groups
too similar to be contrasted (Skovholt, Ronnestad, & Jennings,
1997). Furthermore, the tasks used in the research may have been
too simple to differentiate between groups (Skovholt et al., 1997).
These criticisms may apply to study one and may be a reason for a
failure to detect difference between experienced and novice therapists. Study two addresses some of these possible limitations.
5. Study two
This study investigated the role of formulation in treatment
planning and drew on and adapted the method in study one, and
utilised Ericsson and Smith's (1991) Expert Performance Approach
as a model to increase the difference between novices and experts.
This three-stage descriptive and inductive framework is proposed
to underpin the empirical analysis of expert performance; arguing
for i) ecologically valid tasks, ii) process-tracing methods to
measure processes underpinning performance, and iii) a detailed
understanding of the individual's background to aid classication
of expertise.
Following this framework led to three key changes to the task
outlined in study one. First, in order to increase the ecological
validity, therapists were asked to generate their own formulation
when treatment planning. This change increased the potential
differences between the participants. Experts have been shown to
generate better quality (Kuyken et al., 2005) formulations than
novices. Hence, it was predicted that the formulations of experts
would be of a higher quality than those of novices. Specically, it
was hypothesised that expert formulations would contain more
essential (and less extraneous) content from the vignette and thus,
they would be more parsimonious and more internally consistent
than those of novice clinicians.
Second, in this study an attempt was made to explore or trace
the processes underpinning performance during each stage of the
formulation generation (Bennett, 2008; 2010; Checkel, 2008;
Einhorn & Hogarth, 1981; Einhorn, Kleinmuntz, Klein & Muntz,
1979). In other domains, experts, as compared to novices, have
been shown to perform faster, whilst making fewer errors,
demonstrating frequent self-monitoring and adjustments of their
work (Glaser & Chi, 1988). Hence, it was predicted that experts
would make fewer errors than novices.
Previous research has indicated that novices are often unaware
of their lack of skill and hence are overcondent in their estimations of task performance (Davis et al., 2006; Kruger & Dunning,
1999). Hence it was predicted that novices would report a higher
level of condence in key aspects of the formulation than the
experts.
Third, in this study particular effort was made to recruit expert
rather than experienced therapists. As with study one, it was
hypothesised that experts would choose better treatment planning
options (endorsing more the match option, and rating as less
relevant to the formulation the mismatched, and irrelevant options) than novice clinicians as it was assumed that the expert
produced formulations were of a higher quality than the novices
and would better direct treatment planning.
6. Method

Table 3
Demographic information for each group.

Gender (n, % female)

Diploma in CBT trained
BABCP accredited
Supervised on Diploma level
CBT training
Age (yrs)
Months of clinical experience
Months qualied in profession
Research publications
Number of workshops delivered
Number of CBT cases seen

Novice (n 31)

Expert (n 16)

N
N
N
N

N
N
N
N

31, 94%
0, 0%
0, 0%
0, 0%

16, 56%
2, 13%
8, 50%
1, 6%

Mean

Sd

Mean

Sd

27
28.7
0
0
0
4.1

2.7
17.5
0
0
0
4.9

49
271.2
231.7
12.3
20.8
251.9

11.4
63.5
64.2
18.9
19.7
154.0

6.2. Participants
Two groups (different to those in study one) were recruited. One
group consisted of 31 novice therapists all in their rst year of
clinical psychology training. The second group consisted of 15
expert CBT therapists who met at least two of three criteria: a
minimum of ten years of experience; evidence of continuous
reection regarding formulation and/or CBT techniques through
research, publications/conference appearances on related topics,
supervision of other CBT therapists; or evidence of commitment
through delivery of training and continuing professional development (CPD) events on the topic of formulation (Skovholt et al., 1997).
Table 3 indicates that the expert group were very highly experienced (with a mean of 271 months of experience). Whilst these
participants were more experienced than the group of experienced
therapists in study one they were less likely to have completed a
diploma or equivalent in CBT, or to supervise on diploma level
training but were just as likely to be BABCP accredited. However, in
terms of publications, books, and other esteem indicators the experts differed substantially.3
6.3. Measures
The assessment and formulation materials (Jess) developed in
study one were utilised within study two. However, participants
were not provided with the provisional or nal completed formulation, instead they were asked to generate their own which was
recorded on to a blank formulation template. They then used this to
answer the treatment planning questions.
A manual4 used to score the quality of the participants formulations was utilised in this study. It was based on the manuals
developed by Kuyken et al., (2005) and Eells et al. (2005). Four
components were chosen as important measures for the current
study namely; parsimony (correct items divided by total items
entered), internal consistency or coherence of the formulation
(which was scored out of a maximum of 11), exibility (number of
changes made as new information was provided) and errors
(number of mistakes in either providing material that experts had
not seen as appropriate in the formulation or entering the correct
information in the wrong section of the formulation).
The quality manual was developed and tested on the formulations produced by a separate group of 30 novice, and experienced
but not expert clinicians (this data was not used within the main
analyses as it was used for manual development) who undertook
the same task as the novice and experts in this study. Initially ve of

6.1. Design
A mixed between groups (novice versus expert) with within
subject (three levels treatment option) design was utilised.

3
Further information about the expert participants is available on request from
the corresponding author.
4
A copy of this is available from the corresponding author.

72

R. Dudley et al. / J. Behav. Ther. & Exp. Psychiat. 48 (2015) 66e74

these formulations were used to generate content examples for the

manual. Two members of the research team then independently
scored a further ve formulations, both were blind to the experience level of the participants. At this point the two researchers
achieved 67.3% consistency in scoring, indicating the usability of
the manual. Discrepancies were discussed and resolved and the
manual was updated. A further ve formulations were scored in
light of these revisions. Eighty-two percent agreement was achieved. The two researchers then scored the 15 remaining formulations and agreement increased to 86%. The manual was used to
score the expert and novice formulations by the same two raters
who were blind to group membership with a similarly high rate of
agreement between raters (Cohen's kappa greater than 0.8 for experts and novice ratings and overall).
6.4. Procedure
The novices were recruited via Clinical Psychology training
courses. Experts were identied if they published on the topic of
formulation, and or were regular contributors to national and international conferences in CBT. They were invited by email to
participate. Individuals were provided with the task instructions; a
complete set of assessment materials from the case vignette from
study one (referral letter, relevant history, assessment scores,
thought record, and activity schedule); a blank formulation template based on the format in Fig. 1; four extra maintenance cycle
templates (as in the lower part of Fig. 1); treatment planning
questions; and a set of coloured pens.
6.5. The formulation
The task was structured to allow a formulation to be built up
step-by-step from the sequential presentation of the ve pieces of
assessment material. Each piece of assessment material was colour
coded and for each a corresponding coloured pen was provided.
Individuals were asked to start with the referral letter and using the
corresponding coloured pen note down emerging ideas about the
case on the blank formulation template. Every time new information was added therapists were asked to rate their certainty
(0e100% certainty) regarding the contribution of it to the emergence or maintenance of Jess's difculties. Therapists were asked to
follow the same procedure for each piece of assessment material. In
order to explore the way in which judgments and decisions change
as more information comes to light therapists were asked to neatly
score through information they later believed to be irrelevant or
erroneous, including changing condence ratings.
6.6. Treatment planning
Once all ve pieces of assessment material had been seen and
the formulation template was completed participants were asked
to consider the treatment planning questions (taken from study 1)
in light of their own clinician-generated formulation.
6.7. Review procedures and ethics
The work was subject to the same review processes as study
one.
7. Results
7.1. Formulation quality
The Quality of formulations was scored using the manual.
Condence intervals, means and standard deviations for parsimony

and consistency are summarised in Table 4. The experts were more

parsimonious in that they reported more correct information
relative to the irrelevant information (t(44) #7.7, p < 0.01, Cohen's
d 2.61), and produced formulations that were more internally
consistent and coherent (t(44) 3.1, p < 0.01, Cohen's d 0.73)
than the novices.
7.2. Process measures
It was not possible to record how quickly each individual performed the task as some of the testing sessions were undertaken in
groups, so a proxy measure of speed of processing was to record
the stage at which participants detected the key dimension within
their formulation, and the stage at which they were at least 70%
certain that this contributed to, or maintained, the client's difculties. Errors, changes made and legitimacy of changes were also
recorded.
Fifty-seven percent of novices identied the key dimension of
autonomy by stage one, 32.1% by stage two, and the remaining
10.9% between stages three and ve. Similarly 35.7% of experts
identied it in stage one, 14.3% in stage two and the rest between
three and ve. More novices (33.3%) than experts (18.2%) were
condent about the importance of the key theme by stage one,
most experts reached the condence threshold by stage two or
three. None of these trends were signicant. However, they imply
that the experts were more cautious than the novices in the early
stages of formulation. This would be consistent with the hypothesis
that novices are relatively overcondent.
Experts made fewer changes (t(44) 3.02, p < 0.005, Cohen's
d 1.1, M .4, sd 0.6) than novices (M 3.45, sd 3.75) and
fewer errors, with novices making 35 errors between them and
experts making only two. This supports the hypothesis that experts
would make fewer errors. Many of the errors made by the novices
involved either placing incorrect material in the formulation (i.e.
core beliefs that were not related to the client formulation) or
placing information in the wrong part of the formulation (so
labelling core beliefs as rules or assumptions or as thoughts, or
perhaps placing trigger/precipitant information in the early experience part of the formulation). Overall it appeared that the experts
waited until certain and then did not need to revise their
formulations.
7.3. Treatment planning
Preliminary analysis on the dataset identied four missing
values within the treatment planning questions. Data was replaced
with the mean of the person's rating on the other items. Parametric
assumptions were met. Subsequently, a mixed ANOVA with a between subjects variable of Group (2 levels) and within subjects of
Table 4
Parsimony and coherence of the expert and novice generated formulations.
Quality component
Parsimony (%)
Mean (SD)
Condence intervals
Coherence/Consistencya
Mean (SD)
Condence intervals
Flexibility (changes made)
Mean (sd)
Condence intervals
Errors made
Total
a

Novice (n 31)

Expert (n 15)

81.7 (5.2)
79.8e83.6

93.0 (3.2)
91.3e94.8

7.9 (.9)
7.6e8.3

8.7 (1.3)
7.9e9.4

3.4 (3.8)
2.1e4.8

0.4 (0.6)
#0.01e0.7

35

Coherence is scored out of a maximum of 11.

R. Dudley et al. / J. Behav. Ther. & Exp. Psychiat. 48 (2015) 66e74

Table 5
Mean ratings for response type across the two clinician experience levels.

Novice
Expert

Mean
Std. Deviation
Mean
Std. Deviation

Match

Irrelevant

Mismatch

8.9
1.5
7.9
2.9

6.9
1.6
4.5
1.3

4.4
2.0
1.9
1.2

response type (3 levels) was undertaken with planned comparisons

to compare responses to the treatment planning tasks between the
groups.
Ratings on overall response to therapeutic prediction tasks are
outlined in Table 5, these provide the summed responses from the
task. As in the rst study, these were either match, irrelevant, or
mismatch.
An ANOVA comparing experts versus novice clinicians on
response type (match, mismatch, irrelevant) demonstrated a signicant main effect of response type (F (2, 42) 137.92, p < 0.001,
h2 0.76). Mauchly's test was violated (c2 (2) 21.61, p < 0.001)
and hence the Greenhouse-Geisser correction was applied. Planned
simple contrasts showed that match (M 8.46, sd 2.25) was
signicantly higher than irrelevant (5.77, sd 1.45) and mismatch
(3.19, sd 1.6), and that irrelevant/red herring was signicantly
higher than mismatch (all p values <.001). There was a main effect
of group (F (1, 43) 12.1, p 0.001, h2 0.23) with the expert group
(M 5.27, sd 0.42) endorsing all options less than the novices
(M 6.73, sd 0.28). There was no signicant interaction between
the response patterns of the two groups (F (2,42) 3.43, p 0.05,
h2 0.07), although it was approaching a signicant value. Given
the predicted differences in performance by the groups on the task
these were explored as these pertained to the hypotheses. There
was no signicant difference between experts and novice clinicians' ratings on match questions (t(44) 1.52, p 0.14,
CI #3.27e23.22, Cohen's d 0.43). Experts rated the irrelevant or
red herring treatment planning questions as a lower t to the
formulation than the novices (t(44) 4.91, p < 0.001,
CI 14.14e33.85, Cohen's d 1.65). Also, the experts rated the
mismatch option as less relevant to the formulation than the
novices (t(44) 4.18, p < 0.001, CI 12.79e36.61, Cohen's d 1.52).
Whilst these were planned analyses owing to the lack of interaction
effect in the omnibus test we applied a Bonferroni correction and
the latter two analyses remained signicant.
8. Discussion
As predicted Experts generated higher quality formulations than
novices. Expert clinicians included less inappropriate or supercial
information, and provided more internally consistent and coherent
formulations than the novices. Both groups identied the important features, however, the experts seemed to be able to do so more
elegantly.
A second hypothesis was that novices would be overcondent in
their formulations relative to experts. Novices reported the key
theme earlier and felt more condent in it earlier than expert
therapists but not to a signicant degree. As predicted, experts
were shown to make far fewer errors than novices. In general the
experts demonstrated a more cautious approach. This ts with
Glaser and Chi's assertion that experts require more information
than novices before making a decision.
It was further hypothesised that clinician experience would
affect treatment planning choices. Experts rated inappropriate
treatment plans as less relevant than novices, although there was
no signicant difference in ratings of the matched treatment options. It seems therefore that both experts and novices can identify

73

important treatment options, but experts are less likely to rate or

endorse less-relevant or less-appropriate interventions.
Previous studies have also suggested that experience and
expertise leads to differences in treatment planning from a
formulation (Eells & Lombart, 2003; Eells et al., 2005). It is plausible
that as individuals move towards expertise they become better at
seeing the key themes, or patterns, ltering out the less salient
information when formulating, and choosing the best of all
available treatment options, in turn becoming more efcient and
hopefully effective. Certainly, experts have been shown to notice
meaningful patterns more readily than novices (Glaser & Chi, 1988).
However, based on Glaser and Chi's (1988) assertion that experts
are more able to self-monitor their performance and adapt
accordingly, it was surprising that experts made fewer changes
than novices when presented with new information. One plausible
explanation (given the small number of errors made by experts)
was that the task was not difcult enough to challenge experts.
Whilst the task may have been too easy to challenge experts, it did
draw out a number of crucial differences between experts and
novices in line with previous research (e.g. Eells et al., 2005). In
study one a comprehensive formulation was provided and there
were no differences in ratings of t treatment options, however
when individuals were asked to generate their own formulation
signicant differences in treatment choices were evident between
novice and expert clinicians. Of course, it is important to remember
that both studies were essentially a recognition task which substantially reduces the ecological validity of the tasks.
The experts were selected based on a broad range of criteria
shown to be fundamental to the development of expertise (such as
involvement in supervising others, researching in the area) and
differed from novices on a range of key variables taken to be evidence of expertise (such as years of experience, evidence of
reection and/or evidence of commitment to Continuing Professional Development). In this way the study moved towards
providing a more comprehensive operationalisation of expertise.
9. General discussion
Study one demonstrated that providing a comprehensive
formulation enabled clinicians of all levels to make similar treatment decisions. In study two, when participants were asked to
generate their own formulations, it resulted not only in novices
generating less parsimonious formulations than experts, but also
negatively impacted upon the appropriateness of their treatment
choices. This has important clinical implications. Butler (1998)
proposed that the formulation provides a plausible explanation
for a person's symptoms and is of central importance as it is
thought to instil hope in the client as well as enhancing the alliance,
and provide opportunities for intervention. A poorer quality
formulation as generated by the novice clinicians is less likely to
full these aims and may even lead to therapeutic ruptures. The
importance of expert supervision therefore, especially at the
formulation stage is crucial, given the nding in study one that
given a comprehensive formulation all groups can plan treatment
equally well, receipt of expert supervision, especially during the
formulation stage, may lead to better treatment plans.
However, it is important to note that experts and novices did not
differ on the correct information within the formulation, or indeed
on ratings of appropriate treatment plans. This indicates that all
clinicians were able to identify important aspects of a client's
presentation on the case formulation generation task as was used
here, but differ solely on the focus and clarity with which this is
developed and used.
Future research may consider manipulating the content of the
cases so that there are simple and more complex presentations

74

R. Dudley et al. / J. Behav. Ther. & Exp. Psychiat. 48 (2015) 66e74

(Dudley, Kuyken, & Padesky, 2011). Such material may challenge

novice clinicians and better reveal the value that expertise brings to
the process of formulation. Finally, using think out aloud methods
may provide a means of understanding the process of formulation
rather than just seeing the outcome of the formulation process.
Acknowledgements
We would like to thank Peter Armstrong, Stephen Barton, Ivy
Blackburn and Willem Kuyken who kindly acted as the expert reviewers of the materials used in these studies.
References
Beck, J. S. (1995). Cognitive therapy: Basics and beyond. New York: Guilford Press.
Beck, A. T., Epstein, N., & Harrison, R. P. (1983). Cognitions, attitudes and personality
dimensions in depression. British Journal of Cognitive Psychotherapy, 1, 1e16.
Beck, A. T., Rush, A. J., Shaw, B. F., & Emery, G. (1979). Cognitive therapy of depression.
New York: Guilford Press.
Bennett, A. (2008). Process-tracing: a Bayesian Perspective. In J. M. Box-Steffensmeier, H. E. Brady, & D. Collier (Eds.), The Oxford Handbook of Political Methodology (pp. 702e721). Oxford: Oxford University Press.
Bennett, A. (2010). Process tracing and causal inference. In H. E. Brady, & D. Collier
(Eds.), Rethinking Social Inquiry: Diverse Tools Shared standards (2nd ed.). (pp.
207e220). Lanham: Rowman and Littleeld Publishers Inc.
Bieling, P. J., & Kuyken, W. (2003). Is cognitive case formulation science or science
ction? Clinical Psychology-Science and Practice, 10, 52e69.
Blackburn, I., & Twaddle, V. (1996). Cognitive therapy in action: A practitioners' case
book.
Butler, G. (1998). Clinical formulation. In A. S. Bellack, & M. Hersen (Eds.),
Comprehensive clinical psychology (pp. 1e24). New York: Pergammon Press.
Chadwick, P., Williams, C., & Mackenzie, J. (2003). Impact of case formulation in
cognitive behaviour therapy for psychosis. Behaviour Research and Therapy, 41,
671e680.
Checkel, J. T. (2008). Tracing Causal Mechanisms. International Studies Review, 8(2),
362e370.
Davis, D. A., Mazmanian, P. E., Fordis, M., Van Harrison, R., Thorpe, K. E., & Perrier, L.
(2006). Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. Journal of the American Medical Association, 296(9), 1094e1102.
Dudley, R. E. J., & Kuyken, W. (2013). Case formulation in cognitive behavioural
therapy: a principle driven approach. In L. Johnstone, & R. Dallos (Eds.),
Formulation in psychology and psychotherapy (2nd ed.). Brunner Routledge.
Dudley, R., Kuyken, W., & Padesky, C. A. (2011). Disorder specic and transdiagnostic case conceptualisation. Clinical Psychology Review, 31, 213e224.
Dudley, R., Park, I., James, I., & Dodgson, G. (2010). Rate of agreement between
clinicians on the content of a cognitive formulation of delusional beliefs: the
effect of qualications and experience. Behavioural and Cognitive Psychotherapy,
38, 185e200.
Eells, T. D., & Lombart, K. G. (2003). Case formulation and treatment concepts
among novice, experienced, and expert cognitive-behavioural and psychodynamic therapists. Psychotherapy Research, 13, 187e204.

Eells, T. D., Lombart, K. G., Kendjelic, E. M., Turner, L. C., & Lucas, C. (2005). The
quality of case formulations: a comparison of expert, experienced and novice
cognitive-behavioural and psychodynamic therapists. Journal of Consulting and
Clinical Psychology, 73(4), 579e589.
Einhorn, H. J., & Hogarth, R. M. (1981). Behavioral decision Theory: processes of
judgment and choice. Annual Review of Psychology, 32, 53e88.
Einhorn, H. J., Kleinmuntz, D. N., Klein, & Muntz, B. (1979). Linear regression and
process-tracing models of judgment. Psychology Review, 86, 465e485.
Ericsson, K. A., & Smith, J. (1991). Prospects and limits in the empirical study of
expertise: an introduction. In KA, J. Ericsson, & Smith (Eds.), Toward a general
Theory of Expertise: Prospects and limits (pp. 1e38). Cambridge: Cambridge
University Press.
Ghaderi, A. (2006). Does individualization matter? A randomized trial of standardized (focused) versus individualized (broad) cognitive behavior therapy for
bulimia nervosa. Behaviour Research and Therapy, 44, 273e288.
Glaser, R., & Chi, M. T. H. (1988). Overview. In M. T. H. Chi, R. Glaser, & M. J. Farr
(Eds.), The nature of expertise. Hillsdale, NJ: Lawrence Erlbaum.
Greenberger, D., & Padesky, C. A. (1995). Mind over mood: Change how you feel by
changing the way you think. New York: Guilford.
Kendjelic, E. M., & Eells, T. D. (2007). Generic psychotherapy case formulation
training improves formulation quality. Psychotherapy, 44, 66e77.
Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: how difculties in
recognizing one's own incompetence Lead to inated self-assessments. Journal
of Personality and Social Psychology, 77(6), 1121e1134.
Kuyken, W., Fothergill, C. D., Musa, M., & Chadwick, P. (2005). The reliability and
quality of cognitive case formulation. Behaviour Research and Therapy, 43,
1187e1201.
Kuyken, W., Padesky, C., & Dudley, R. (2008). The science and practice of case
conceptualisation. Behavioral and Cognitive Psychotherapy, 36(Special Issue 06),
757e768.
Kuyken, W., Padesky, C., & Dudley, R. (2009). Collaborative case conceptualisation.
Working effectively with clients in cognitive behavioural therapy. Guildford Press.
Kuyken, W., Beshai, S., Dudley, R., Abel, A., Gorg, N., Gower, P., McManus, F., &
Padesky, C. Assessing competence in collaborative case conceptualization,:
development and preliminary psychometric properties of the collaborative case
conceptualization rating scale (CCC-RS), Behavioural and Cognitive Psychotherapy, published online 28/01/15.
Mumma, G. H. (2011). Validity issues in cognitive-behavioral case formulation.
European Journal of Psychological Assessment, 27(1), 29e49. http://dx.doi.org/10.
1027/1015-5759/a000054.
Mumma, G., & Mooney, K. (2007). Comparing the validity of alternative cognitive
case formulations: a latent variable, multivariate time series approach. Cognitive
Therapy and Research, 31(4), 451e481.
Persons, J. B., & Bertagnolli, A. (1999). Inter-rater reliability of cognitive-behavioral
case formulations of depression: a replication. Cognitive Therapy and Research,
23, 271e283.
Schulte, D., Kunzel, R., Pepping, G., & Shulte-Bahrenberg, T. (1992). Tailor-made
versus standardized therapy of phobic patients. Advances in Behaviour Research
and Therapy, 14, 67e92.
Skovholt, R., Ronnestad, M. H., & Jennings, L. (1997). The search for expertise in
counselling, psychotherapy and professional psychology. Educational Psychology
Review, 9, 361e369.
Statistical Packages for Social Sciences (SPSS). (2012). SPSS 20.0 for Windows. Chicago: SPSS inc.