Alternative experimental design paradigms in choice experiments and

their effects on consumer demand estimates for beef from endangered

local cattle breeds: An empirical test
Marcos Domnguez-Torreiro

CIFA Cantabrian Agricultural Research Centre, Spain

a r t i c l e i n f o
Article history:
Received 19 September 2013
Received in revised form 24 December 2013
Accepted 26 January 2014
Available online 6 February 2014
Keywords:
Choice modelling
Optimal designs
Consumer preferences
Beef attributes
a b s t r a c t
In this paper we empirically validate the hypothesis of context independent preferences in choice exper-
iments under different experimental design approaches. Two alternative paradigms for constructing
designs are analysed in this study under a simple Multinomial Logit (MNL) framework: optimal orthog-
onal in the differences (OOD) and D-efcient (EFD) designs. Once having taken into account potential dif-
ferences in scale (inversely proportional to the variance of the random term in random utility models),
preference estimates for beef attributes are found to be sensitive to the design generation process.
2014 Elsevier Ltd. All rights reserved.
1. Introduction
In the recent years there has been a growing interest in foster-
ing production of beef from local cattle breeds in Europe
(Hiemstra, de Haas, & Mki-Tanila, 2010). Agri-environmental
schemes in the European Unions rural development policy
framework have addressed this issue so far by means of specic
subsidies included in national and regional rural development
programs (Council Regulation (EC) No. 1698/2005 and Council
Regulation (EC) No. 1974/2006). In many cases, such subsidies
have also been related to others which promote organic beef pro-
duction. According to EU organic production regulation (Council
Regulation (EC) No. 834/2007 and Commission Regulation (EC)
No. 889/2008), local breeds are expected to be more vital, more
resistant to disease and more adapted to the specics of the
different territories, and consequently should be given preference
as an input for organic farming activities. Furthermore, young
calves from endangered local cattle breeds such as the Tudanca
and Monchina breeds in Cantabria, Northern Spain, are also
amenable to extensive farming practises based on grazing, which
contribute to obtaining produce (meat) which scores high in
both organoleptic characteristics (tenderness) and nutritional
attributes (fatty-acids prole) (Humada, Serrano, Saudo, Rolland,
& Dugan, 2012). Accordingly, regional, national and supra-
national public administrations usually consider (and support)
extensive productive systems based on local breeds as a key
factor for guaranteeing the sustained provision of public goods
and environmental amenities in rural areas (biodiversity conser-
vation, landscape preservation, cultural heritage, etc.).
In spite of all above, it must be conceded that one of the keys to
subsistence for those productive systems in the long-run should be
their matching to a signicant demand of differentiated meat
products by local consumers. In order to assess and evaluate the
extent of potential consumers demand for those products, stated
preference methods in general and choice experiments in particu-
lar have been widely reckoned in the literature as a powerful ana-
lytical tool (Bateman et al., 2002; Hanemann, 1984; Hanemann,
1989; Hanemann & Kanninen, 1999; Louviere, Hensher, & Swait,
2000).
In those research contexts appropriate for the implementation
of choice experiments, several decisions have to be made in the
rst instance relating to, for example, the design generation
process, the selection of the attributes and attributes levels to be
considered in the analysis, or even the explicit specication of
the indirect utility functions. Notwithstanding its importance in
the nal outcome of the study, the debate on which experimental
design approach to use in applied research has been sometimes re-
garded as potentially confusing and misleading for researchers,
either because of its own complexity or even due to a lack of
inclination of analysts to deal with experimental design theory
(Louviere, Pihlens, & Carson, 2011). Such caveats have been allevi-
ated (at least partially) by continuous contributions to theory and
http://dx.doi.org/10.1016/j.foodqual.2014.01.006
0950-3293/ 2014 Elsevier Ltd. All rights reserved.

Address: c/Hroes 2 de Mayo 27, 39600 Muriedas, Cantabria, Spain. Tel.: +34
942 25 43 93; fax: +34 942 26 90 11.
E-mail address: marcosdominguez@cifacantabria.org
Food Quality and Preference 35 (2014) 1523
Contents lists available at ScienceDirect
Food Quality and Preference
j our nal homepage: www. el sevi er . com/ l ocat e/ f oodqual
applications within the wider literature, as well as by the advent of
specialised commercial software supporting the development of ad
hoc designs based on efciency criteria.
However, to the authors knowledge there has not yet been sub-
stantial empirical research work specically addressing the com-
parison between two of the most widely used techniques of
experimental design: optimal orthogonal in the differences designs
(OOD) and D-efcient designs. For instance, in an empirical setting,
Burgess, Street, and Wasi (2011) analysed four designs based on no
prior knowledge assumptions; and Bliemer and Rose (2011) com-
pared two Bayesian efcient designs with a (simultaneous) orthog-
onal design. With this paper we aim at providing some additional
insights into this hitherto much neglected issue by means of ana-
lysing the results of a split-sample internet survey. In this eld
application, potential consumers of local meat products have been
confronted with choice scenarios selected from two alternative
experimental design procedures. The research was restricted to
xed (non-random) attribute coefcients Multinomial Logit
(MNL) models. Our results indicate that experimental design issues
might have a signicant inuence on consumer responses.
The remainder of the article is organised as follows. In section 2
we provide a brief description of the relevant methodology, the
questionnaire and the eldwork. The results obtained from the
estimation of the models are presented in Section 3. The nal sec-
tion is devoted to discussion and conclusions.
2. Materials and methods
2.1. Experimental design in choice experiments
The theoretical underpinnings of Choice Experiments (CEs) are
rooted both in Random Utility Theory (Luce, 1959; Manski 1977;
McFadden, 1974) and Lancaster (1966) characteristics theory of
value. In practise, CEs allow us to simulate and analyse markets
for commodities (as well as for non-commodity outputs) in a con-
sumer-friendly trade-off manner. Actual or prospective goods are
described in terms of their relevant attributes, which are bundled
together to form sets of alternative consumption opportunities,
and subsequently presented to a sample of individuals from the
target population, so that they can choose their favourite options
from the choice sets according to their own preferences and budget
constraints (Bateman et al., 2002; Bennett & Blamey, 2001;
Hensher, Rose, & Greene 2005; Louviere et al., 2000; Train, 2003).
The statistical analysis of the responses given by the individuals
to the proposed choice tasks will produce the estimates of the util-
ity coefcients in the indirect utility functions (IUFs) underlying
every discrete choice model. Extreme value type 1 (EV1) indepen-
dently and identically distributed (IID) error terms, usually com-
bined with strictly additive specications for the IUFs, will result
in the closed-form probability expression for the most important
choice model known as the Multinomial Logit (MNL) model, which
has been described as the workhorse of discrete choice analysis
(Hensher et al., 2005; Scarpa & Rose, 2008). We will refer to this
model in what follows.
As stated above, CEs imply the combination of bundles of goods
described in terms of their relevant attributes or characteristics to
create choice sets to be presented to the individuals who partici-
pate in the eld application. Experimental design techniques allow
researchers to select the choice sets to be included in the CE
according to some optimality criterion. We may describe infor-
mally an optimal design as one that gets as much information as
possible from an experiment of a given size (Street & Burgess,
2007). In the literature, two main paradigms for constructing de-
signs have been proposed, which differ in terms of the criteria
and properties they use to judge the performance of the designs:
Orthogonal designs vs. Efcient designs (Choicemetrics, 2010;
Scarpa & Rose, 2008).
1
On the one hand, the orthogonal design approach seeks for
designs that satisfy the property of orthogonality, underlining the
expected benecial inuence of orthogonality on the properties
of the resulting model estimates. The property of orthogonality re-
fers to experimental designs where the attributes of the design are
uncorrelated with one another, thus allowing for the effects (read
parameter coefcients) from different attributes to be indepen-
dently estimated (Scarpa & Rose, 2008; Street & Burgess, 2007).
Optimal Orthogonal in the Difference (OOD) choice designs refer
to a well known type of sequential orthogonal designs -i.e., designs
where correlations are zero within alternatives but, as opposed to
simultaneous orthogonal designs, not necessarily so between alter-
natives (see e.g. Louviere et al., 2000; Rose & Bliemer, 2009; Scarpa
& Rose, 2008)-, grounded in the hypothesis of equally attractive
options (Goos & Grossmann, 2011; Street & Burgess, 2007) and in
the design principle of maximising the differences between the
attribute levels of the alternatives by forcing respondents to trade
on all the attributes in the experiment (Burgess & Street, 2005;
Choicemetrics, 2010; Street & Burgess, 2007; Street, Burgess, & Lou-
viere, 2005). OOD designs have been discussed within the general
framework of MNL models and unlabelled (or generic) choice
experiments (Street & Burgess, 2007). Unlabelled choice experiments
use generic titles for the alternatives, meaning that the between
alternative correlations are not of concern, and also that the imple-
mentation of sequential orthogonal designs becomes appropriate
(Hensher et al., 2005; Scarpa & Rose, 2008; Street & Burgess, 2007).
Upon all these considerations, the level of optimality of OOD designs
is usually expressed as a percentage of the upper bound value cal-
culated for the determinant of the information matrix in the class of
competing designs (Street & Burgess, 2007; Street et al., 2005). In the
remaining of this paper, our reections and empirical work on
orthogonal designs will be located in OOD designs.
On the other hand, efcient designs try to minimise the vari-
ability of the parameter estimates, relying on the fact that the var-
iancecovariance matrix of the maximum likelihood estimates can
only be derived if some prior information on the parameters
included in the utility specication is available to the researcher,
either from previous studies or from taking a sequential approach
to experimentation (Choicemetrics 2010; Street & Burgess, 2007).
However, within the framework of efcient designs, we encounter
many different competing criteria to be used in the design genera-
tion process. According to different authors (see e.g. Kerr & Sharp,
2010; Rose & Bliemer, 2009; Scarpa & Rose, 2008; Street & Burgess,
2007), it has become usual to look for D-efcient designs, which
minimise on average the elements contained within the asymp-
totic variancecovariance matrix (i.e. standard errors and covari-
ances). Apart from D-efcient designs, we might nd alternative
proposals in the literature such as A-, E-, S-, G- and V-efciency
(Bliemer & Rose, 2005; Kessels, Goos, & Vandebroek, 2006; Street
& Burgess, 2007; Sndor & Wedel, 2001); there is also an increasing
body of literature advocating for the C-efciency measure, which
minimises the variance of the ratio of two parameters
2
(Kanninen,
1993a; Scarpa & Rose, 2008); and nally, other proposals favour the
implementation of Bayesian and sequential updating techniques
(Bliemer & Rose, 2011; Ferrini & Scarpa, 2007; Kanninen, 1993b;
1
For purposes of comparison, we limit our study to specic cases of orthogonal and
efcient designs. However, we also note that there exist a number of additional
methods for constructing experimental designs which are not covered by the present
paper (see e.g. Rose & Bliemer, 2009).
2
When a study aims at estimating implicit prices, authors like Scarpa and Rose
(2008) have underlined the relevance of the C-efciency criterion, since it is quite
conceivable to minimise the variance of the ratio of two parameters while at the same
time obtaining a relatively large variance for one or more of the individual
parameters.
16 M. Domnguez-Torreiro / Food Quality and Preference 35 (2014) 1523
Kerr & Sharp, 2010; Scarpa, Campbell, & Hutchinson, 2007;
Vermeulen, Goos, Scarpa, & Vandebroek, 2011).
Since OOD and efciency paradigms differ in terms of the
criteria and assumptions used to assess the overall efciency of a
generated experimental design, applied researchers are sometimes
confused and mislead when it comes to making an informed choice
over what design should be used for any particular study (Louviere
et al., 2011). Taking that into account, it would be helpful to have
more applied research directed to analysing the performance of
alternative construction approaches on real and comparable data-
sets (Louviere et al., 2011). Those comparative studies should focus
on evaluating potential differences in terms of standard errors and
covariance matrices of the estimated preference parameters, but
also on analysing the sensitivity of model t and willingness to
pay measures to such alternative experimental design approaches.
In the published literature on individual attitudes and prefer-
ences there are many recent examples of studies analysing the
impact on consumers demand of credence and experience attri-
butes in general (Bernus, Olaizola, & Corcoran, 2003; Gracia,
Loureiro, & Nayga, 2009; Loureiro & Umberger, 2003; McCluskey
& Loureiro, 2003; Olynk, Tonsor, & Wolf, 2010; Pouta, Heikkila,
Forsman-Hugg, Isoniemi, & Makela, 2010), beef attributes in partic-
ular (Angulo & Gil, 2007; Loureiro & Umberger, 2007; Lusk, Fox,
Schroeder, Mintert, & Koohmaraje 1999; Lusk, Roosen, & Fox,
2003; Realini et al., 2013; Verbeke, Perez-Cueto, de Barcellos,
Krystallis, & Grunert, 2010), or even situational factors such as
the eating occasion (Jaeger & Rose, 2008). In those studies, cre-
dence attributes are described as those for which quality cannot
be assessed by the consumer even after purchase and consumption
(Bernus et al., 2003; Lusk et al., 2003; Olynk et al., 2010), as for
example organic production standards, healthfulness, animal
welfare claims and origin; conversely, experience attributes are
dened as those that can only be evaluated by the individual at
the moment of actual consumption (e.g. tenderness), whereas
search attributes are those that can be perceived prior to purchase
(e.g. price). But despite all the previous work, there is still a lack of
eld applications based on choice experiments that study the
potential implications of the choice made by the analysts between
OOD or EFD designs on the results of their applied research work.
The survey described in Section 2.2 was designed to highlight the
relevance of this issue and to contribute to lling the gap in the
literature.
2.2. Survey instrument and data collection
As with every other stated preference technique, a choice exper-
iment must be framed into a questionnaire to be delivered to a sam-
ple of individuals. The questionnaire developed for this study was
divided into four different sections. The rst two sections were
devoted to questions related to consumption habits, familiarity with
the product, beliefs and factors which might inuence consumers
decisions and purchasing behaviour towards beef products. The
third section presented our generic choice experiment, and the last
section was devoted to recording the socioeconomic characteristics
of the interviewee. Previous to the implementation of the nal
survey, a pre-test comprising 40 sampled individuals was also
undertaken to provide the prior parameter values needed for the
construction of the nal D-efcient design.
Both the D-efcient (D
p
-error = 0.179008, D-optimal-
ity = 71.24%) and OOD(D-optimality = 97.90%, D
p
-error = 0.214161)
designs used in the nal version of the survey were generated using
the Ngene software for experimental designsuppliedby Choicemet-
rics (see Table 1). Since the software was unable to locate an OOD
design in less than 24 rows, an equivalent restriction in the number
of choice sets was imposed when searching for a D-efcient design
in order to allowfor a direct comparison between the designs (Rose
& Bliemer, 2009). As a result, forty-eight choice scenarios were used
in the nal version of the questionnaire (twenty-four OOD and
twenty-four D-efcient choice scenarios), each of them made up
of two different unlabelled purchase alternatives and an opt-out
(i.e. do not buy) option. They were combined in four blocks of
twelve different choice scenarios each, with half of the scenarios
in every block being constructed under the premises of OOD
designs, and the other half under the premises of D-efcient
designs. Every individual in the survey was randomly assigned to
answer one of the four blocks. The order of presentation of the
scenarios within each block was also subject to random rotations
to avoid potential biases and order effects.
A short text was shown to respondents as an introduction for
the simulated market decision environment, stating that:
Table 1
OOD and EFD experimental designs.
a
Option A Option B Block
Choice sets OOD design
0 0 0 2 21 1 1 1 1 18 1
1 0 0 2 15 0 1 1 1 12 1
1 1 1 1 15 0 0 0 0 12 1
0 1 0 0 21 1 0 1 2 18 1
0 0 1 2 12 1 1 0 1 21 1
1 1 1 0 15 0 0 0 2 12 1
1 1 1 2 21 0 0 0 1 18 2
0 1 1 1 12 1 0 0 0 21 2
1 0 0 1 12 0 1 1 0 21 2
1 0 1 1 21 0 1 0 0 18 2
0 1 0 1 21 1 0 1 0 18 2
1 0 0 0 12 0 1 1 2 21 2
0 0 1 0 18 1 1 0 2 15 3
0 1 1 0 12 1 0 0 2 21 3
0 1 0 2 18 1 0 1 1 15 3
0 0 0 1 15 1 1 1 0 12 3
0 1 0 0 15 1 0 1 2 12 3
1 0 1 0 21 0 1 0 2 18 3
0 0 1 2 15 1 1 0 1 12 4
1 1 0 1 18 0 0 1 0 15 4
1 0 0 0 18 0 1 1 2 15 4
1 1 0 2 12 0 0 1 1 21 4
1 1 1 2 18 0 0 0 1 15 4
0 0 1 1 18 1 1 0 0 15 4
Choice sets EFD design
0 1 0 1 21 1 0 1 0 15 1
1 0 0 1 12 0 1 1 2 18 1
0 0 0 1 21 1 0 0 0 21 1
0 0 0 1 12 1 1 1 2 18 1
0 1 1 1 15 1 0 1 0 15 1
1 1 1 0 21 0 0 0 1 15 1
1 1 1 0 18 0 1 0 1 12 2
0 0 1 1 15 1 1 0 2 18 2
1 1 0 2 21 0 0 1 1 12 2
1 0 1 2 15 0 1 0 1 15 2
1 0 1 2 15 0 1 0 0 18 2
1 0 0 0 12 0 1 1 1 21 2
0 1 1 2 21 1 1 0 0 12 3
1 1 1 1 21 0 0 0 2 12 3
1 1 0 0 18 0 0 1 2 15 3
0 1 0 2 18 1 0 1 1 12 3
0 1 1 0 12 1 0 0 2 21 3
0 0 1 2 12 1 1 0 1 21 3
0 1 0 0 12 1 0 1 1 18 4
1 1 0 0 15 1 0 1 2 21 4
1 0 1 1 18 0 1 0 2 12 4
0 0 0 2 18 0 1 1 0 21 4
0 0 1 0 18 1 0 0 0 18 4
1 0 0 2 15 0 1 1 0 15 4
a
Note: Ngene output presents 3-level dummy-coded attributes as if linear to
facilitate interpretation.
M. Domnguez-Torreiro / Food Quality and Preference 35 (2014) 1523 17
Imagine that you enter the place where you usually purchase
beef with the intention of buying beef steaks. Once there you
are presented with two possible options of steaks. Both pieces
of meat have a similar external appearance. However, and
regardless of their similar appearance, both pieces of meat
might encompass quite different attributes and characteristics.
The corresponding attributes and attributes levels that were
considered for the study of consumers preferences are listed in
Table 2. The general layout of the choice scenarios included in
the nal survey is also shown in Fig. 1.
Participants in the survey were recruited from a panel of con-
sumers managed by a rm specialised in internet surveys. All the
interviewees had to comply with the requisites of being resident
in Cantabria, non-vegetarian and above eighteen years old. The
survey was administered online in December 2012. Respondents
answered the full set of questions in the questionnaire in an aver-
age time of about nineteen minutes. As expected in internet sur-
veys based on a panel of consumers, difculties arose with
regards to achieving a broad sample fully representative of the
population of interest. In our case, the nal sample was made up
of 248 individuals who, according to the summary of sample and
population statistics in Table 3, were younger and more educated
than the general population; there was also a higher percentage
of individuals dwelling in more populated urban areas (such as
the regions capital, Santander), and a higher percentage of individ-
uals whose level of family income was below average. Taking that
into account, we acknowledge that our sample is not fully repre-
sentative of the population of Cantabria.
However, as the main objective of this paper was to compare
two different experimental design strategies, the validity of the
results shall not be affected by the degree of correspondence
between the sample and the general population. In fact, the spe-
cic split-sample structure developed for the survey guarantees
that every individual has had to cope with twelve potential choice
situations, half of them designed following the OOD methodology
and the other half following the D-efcient methodology. As a
result, we ended up having at our disposal two equivalent samples
of (the same) 248 individuals responding to both experimental
design approaches.
3. Results
In the present section we analyse the potential impact and
implications on the model estimates of the decision of which
experimental design approach to use. Two MNL models were
implemented for the OOD and D-efcient subsamples. In order to
focus on the experimental design issue, the models presented here
are intentionally simple, since only the previously dened beef
steak attributes and the alternative-specic constants (ASCs) have
been included as explanatory variables. Other potential explana-
tory variables such as socio-demographic characteristics have been
deliberatedly excluded from the analysis.
Firstly, to test the hypothesis of homogeneous preferences
across the two competing design strategies considered in our eld
application, we followed the two-step approach proposed by Swait
and Louviere (1993). This procedure takes into account the fact
that comparing absolute parameter estimates across models is
uninformative due to scale differences. More precisely, in MNL
models the scale parameter is inversely related to the variance of
the error term in the model as well as confounded (i.e. inextricably
related) with the utility parameters, which results in a fundamen-
tal identication problem (see Louviere et al., 2000). To overcome
this problem, potential differences in preference coefcients
estimates and scale parameters are analysed sequentially by
means of nested likelihood-ratio tests. In the rst step, the hypoth-
esis of equal preference parameters while permitting scale factors
to differ between data sets is evaluated. In the second step, the
hypothesis of equal scale parameters in both data sets is tested.
Only if both sub-hypotheses are not rejected can the hypothesis
of preference regularity be retained at a given condence level.
Testing the rst sub-hypothesis makes it necessary to compare
the estimation results from the joint scaled
3
data matrix with those
from both independent subsamples (OOD and EFD). As derived from
the log-likelihood values for the individual and scaled pooled models
shown in Table 4, the rst sub-hypothesis of equal preferences
adjusted by scale is rejected for a = 0.05 (the chi-squared statistic
with 5 degrees of freedom
4
is 2[LL_scaled (LL_OOD + LL_EFD)] =
2[2789.405 (1299.076 1484.481)] = 11.696, and the corre-
sponding critical value at the 95% condence level is 11.070). The
rejection of the rst null hypothesis of the Swait-Louviere test makes
it unnecessary to proceed to the second step in the procedure. It also
means that differences in consumer responses and estimated prefer-
ences for beef products assessed using OOD and EFD experimental
design techniques do exist in our data sets, and that they are not ex-
plained simply because of differences in scale factors (i.e. variance of
the error term).
Secondly, when compared to OOD designs, it is expected that
D-efcient designs capture the full extent of information from
survey participants, resulting in improved estimation efciency.
Efciency gains in parameter estimates should thus be reected
in terms of improvements in the absolute values of the t-scores,
Table 2
Attributes and levels of beef steaks considered in the choice experiment.
Attribute Levels Variable
*
Nutritional quality Healthy fatty acids prole NQ
Not healthy fatty acids prole [Base level]
Organoleptic quality Tenderness guaranteed OQ
Tenderness not guaranteed [Base level]
Organic production Organic production OP
Not organic production [Base level]
Origin Cantabrian production from Cantabrian cattle breeds LB
Cantabrian production from non-Cantabrian cattle breeds LP
Produced outside Cantabria from non-Cantabrian cattle breeds [Base level]
Price (/kg) 12, 15, 18, 21 PRICE
*
Note: All the variables have been designed using dummy-coding [0-Base level,1] except the (linear) price attribute.
3
The optimal relative scale factor (l) for the scaled pooled model, premised on the
equality of the two utility preference vectors, was set equal to 1.12 after performing
the grid search procedure described by Swait and Louviere (1993).
4
As recommended by Louviere et al. (2000, p. 361), the location parameters in the
models (i.e. the alternative specic constants, ASCs) have been excluded from the test
of preference regularity.
18 M. Domnguez-Torreiro / Food Quality and Preference 35 (2014) 1523
smaller standard errors and smaller D
p
-error values (Bliemer &
Rose, 2011; Kerr & Sharp, 2010; Rose & Bliemer, 2009; Scarpa
et al., 2007). If we again focus on the results related to the attribute
coefcients and exclude from the analysis the alternative-specic
constants, Table 4 shows in the rst place that all the utility coef-
cients in both data sources present highly signicant t-scores.
According to the bootstrap (1000 replications) percentile interval
approach (Hole, 2006), the analysis of the 95% condence intervals
in Table 5 shows that the t-score of at least one OOD attribute (NQ)
is signicantly higher than its EFD counterpart, whilst none of the
t-scores for the remaining EFD attributes are signicantly superior
to those of the OOD model. Furthermore, a smaller variance of the
coefcient estimates in the EFD model, which should translate into
a positive semi-denite difference matrix between OOD and EFD
samples (Scarpa et al., 2007), is not conrmed in our case study.
In fact, according to the results of the percentile interval approach
also shown in Table 5, only the OP coefcient presents signicantly
lower standard errors in the EFD model than in the OOD model;
conversely, we nd a signicantly lower standard error for the
OQ coefcient in the OOD sample. These empirical ndings are
somewhat unexpected results which clearly go against a priori
expectations. Finally, as the EFD design by denition minimises
the D-error for any particular set of non-zero priors, it has the
best performance in terms of this efciency criterion
(D
p
-error = 0.179008 vs. 0.214161).
Fig. 1. General layout of a beef steak choice set included in the survey.
Table 3
Sample and population (Cantabria) descriptive statistics.
Sample Population
*
X
2
(p-value)
Municipality 0.001
Santander (capital) 40.32% 30.05%
Other 59.68% 69.95%
Size of the municipality 0.000
>25.000 Inhabitants 70.13% 50.16%
<25.000 Inhabitants 29.87% 49.84%
Age 0.000
1834 Years 44.76% 24.82%
3554 Years 50.00% 37.59%
55 Years or more 5.24% 37.58%
Size of the family unit 0.697
1 Person 7.66% 6.84%
>1 Person 92.34% 93.16%
Family income 0.003
High 19.44% 28.20%
Medium 50.56% 50.70%
Low 30.00% 21.00%
Higher education 0.000
Yes 47.58% 24.40%
No 52.42% 75.60%
Gender 0.939
Female 51.61% 51.17%
Male 48.39% 48.83%
No. of individuals 248 593,861
*
Source: INEbase-National Statistics Institute (www.ine.es).
Table 4
Estimated scaled pooled and individual data source MNL models.
Variable Scaled pooled OOD EFD
Coefcient Std. error |t-Score| p-Value Coefcient Std. error |t-Score| p-Value Coefcient Std. error |t-Score| p-Value
ASC 0.54 0.23 2.36 0.018 1.30 0.18 7.36 0.000
ASC_1 0.84 0.14 6.16 0.000
ASC_2 1.05 0.14 7.52 0.000
NQ 0.61 0.05 13.30 0.000 0.81 0.07 11.29 0.000 0.53 0.07 7.25 0.000
OQ 0.88 0.05 17.95 0.000 0.96 0.07 13.29 0.000 0.92 0.08 11.21 0.000
OP 0.29 0.04 6.49 0.000 0.37 0.07 5.33 0.000 0.27 0.06 4.11 0.000
LP 0.45 0.06 7.15 0.000 0.46 0.10 4.77 0.000 0.47 0.10 4.82 0.000
LB 0.90 0.06 14.26 0.000 1.02 0.10 10.61 0.000 0.90 0.10 9.21 0.000
PRICE 0.10 0.01 13.43 0.000 0.10 0.01 7.92 0.000 0.12 0.01 9.60 0.000
No. of observations 2,976 1,488 1,488
Log-Likelihood 2789.405 1299.076 1484.481
AIC 1.87998 1.75548 2.00468
Pseudo-R
2
0.11062 0.16866 0.05586
M. Domnguez-Torreiro / Food Quality and Preference 35 (2014) 1523 19
Thirdly, it is worth noting that observed differences in esti-
mated parameters (b) do not necessarily imply that average mon-
etary measures (e.g. marginal willingness to pay and/or
compensating surplus measures) derived from those parameters
are different. Marginal willingness to pay (MWTP) quanties the
marginal rate of substitution of one non-monetary attribute of a
good or service with respect to the monetary attribute (i.e. price),
while maintaining utility constant for the consumer; compensat-
ing surplus (CS) can be dened as the amount of money taken
away that would make a person as well off as they would be before
a change in their consumption of a good or service, i.e. the differ-
ence in maximum willingness to pay between two consumption
alternatives described by different combinations of non-monetary
attributes (see e.g. Bennett & Blamey, 2001). In fact, the impact of
preference heterogeneity is expected to be less important, for
example, for compensating surplus estimates than for marginal
willingness to pay estimates due to a sort of compensation
between the overstated willingness to pay of one attribute and
the lower value for another attribute (Colombo, Hanley, & Louviere,
2009). Moreover, it is also worth noting that contrasting such mon-
etary measures across subsamples is informative as the scale
parameter cancels out in their corresponding analytical expres-
sions (Colombo et al., 2009; Louviere et al., 2000):
(1) MWTP
j
= b
j
/b
PRICE,
for each j-th non-monetary attribute,
(2) CS = (V
a
V
b
)/b
PRICE
, being (V
a
V
b
) the indirect utility dif-
ference (Bennett & Blamey, 2001) for any combination of
non-monetary attribute levels dening any pair of alterna-
tive products a and b.
To analyse the statistical signicance of the differences in the
estimated (OOD vs. EFD) willingness to pay measures (i.e. price
premiums consumers would be willing to pay for the presence of
individual beef steak attributes or for the consumption of a steak
characterised by a specic combination of attributes), the percen-
tile interval approach was again implemented (Table 5). Following
that approach, non-overlapping 95% bootstrapped condence
intervals for the NQ attribute price premium indicates signicant
differences in the estimated monetary measures. This result is fur-
ther implementation of the graphical approach introduced by
Chambers, Cleveland, Kleiner, and Tukey (1983). According to
those authors, non-overlapping notches in the boxplots in Fig. 2a,
corresponding to the simulated bootstrap distributions of price
premiums to be paid for alternative beef steak attributes, must
be regarded as strong evidence of signicant differences in the
locations of the simulated OOD and EFD distributions. From the
analysis of those boxplots, we may conclude as well that the price
premium values for the beef attributes estimated from the OOD
data set are signicantly higher than those from the EFD data
set. The same conclusion is reached when analysing in Fig. 2.b
the simulated distributions for the difference in maximum willing-
ness to pay resulting from the comparison of a conventional beef
steak (i.e. base level) and an improved beef steak (i.e., health-
ier, tender, organic and locally produced steak from local cattle
breeds).
Finally, goodness of t measures in both models differ substan-
tially in magnitude as well, with a higher pseudo-R
2
value for the
OOD model (0.16866) than for the EFD model (0.05586).
4. Conclusions and discussion
The aim of this study has been to empirically validate the impli-
cations of choosing between one of the two competing paradigms
for experimental design in choice experiments: orthogonal vs. ef-
cient designs. In fact, as opposed to a majority of studies focusing
on some particular practical goal other than to test the impact of
alternative experimental designs (Bliemer & Rose, 2011), our
choice experiment was originally conceived to rigorously test
optimal orthogonal in the differences (OOD) and D-efcient (EFD)
designs in practise. Furthermore, we distinguish our empirical
analysis from that of others like Bliemer and Rose (2011) by the
specics of our split sample approach, where each and every indi-
vidual in the eld application was exposed to choice sets from both
underlying experimental designs under evaluation.
As a rst conclusion of the study, the results from the estimated
OOD and EFD models, in terms of sign and signicance of the utility
coefcients for all the beef attributes considered, clearly support
the idea that productive systems oriented to the provision of differ-
entiated beef products increase the chances for such products to be
accepted and chosen by consumers in the marketplace. Accord-
ingly, those results can also be interpreted in the sense that such
producers will be in a more favourable position to access the mar-
ket and maximise their revenues to the detriment of their non-dif-
ferentiated competitors. Indeed, the highest price premium
estimates in our OOD and EFD models for both organoleptic and
nutritional quality attributes do provide robust evidence that those
attributes have a most decisive inuence on the purchase decision
of potential beef consumers. Notwithstanding, the analyses under-
taken in this study have also demonstrated the existence of signif-
icant differences in model coefcients and estimated monetary
measures between the two experimental design approaches con-
sidered. For example, higher estimated price premiums for attri-
butes and beef products have been found to consistently occur
within the OOD data set.
Secondly, the impact of alternative design stategies on estima-
tion efciency indicators has not always conformed to prior
expectations. As stated in Bliemer and Rose (2011), efcient
designs are expected to produce smaller D-errors, which in turn
should translate into smaller standard errors at a given sample
size. However, this has not been the case in our study, with the
estimated EFD model failing to produce signicantly lower stan-
dard errors, or conversely signicantly larger t-ratios. Further-
more, inspection of our OOD design (see Table 1) shows that it
Table 5
Bootstrapped (1000 replications) 95% condence intervals for standard errors, t-
scores and willingness to pay measures.
OOD model EFD model
Lower Upper Lower Upper
|t-scores|
NQ 9.50 12.84 5.34 8.99
OQ 11.72 14.81 9.37 12.92
OP 3.39 7.13 2.34 6.16
LP 2.73 6.57 3.04 6.63
LB 8.88 12.29 7.31 10.95
PRICE 5.28 11.00 6.19 12.03
Standard errors
NQ 0.069 0.076 0.072 0.075
OQ 0.070 0.076 0.080 0.084
OP 0.068 0.073 0.064 0.066
LP 0.094 0.101 0.096 0.100
LB 0.093 0.100 0.095 0.100
PRICE 0.012 0.013 0.012 0.012
Willingness to pay measures
NQ 6.24 11.69 3.49 5.75
OQ 7.54 13.71 6.60 9.59
OP 2.24 6.01 1.32 3.43
LP 2.71 7.62 2.60 5.70
LB 8.00 14.56 6.08 9.75
Maximal WTP
a
25.67 44.30 19.16 26.95
a
Note: Difference in willingness to pay between a conventional and an
improved beef steak.
20 M. Domnguez-Torreiro / Food Quality and Preference 35 (2014) 1523
does not contain choice sets with clearly dominant (i.e. univer-
sally superior) alternatives (Bennett & Blamey, 2001). This
situation precludes conrming the conjecture made by Bliemer
and Rose (2011) that a trade-off between the statistical efciency
Fig. 2. Bootstrap distributions of price premiums to be paid for individual beef steak attributes and for an improved beef steak. (a) Price premium for individual beef steak
attributes (OOD vs. EFD). (b) Price premium for an improved beef steak (healthier, tender, organic and locally produced beef steak from local cattle breeds) (OOD vs. EFD).
M. Domnguez-Torreiro / Food Quality and Preference 35 (2014) 1523 21
of the design and error variance (Louviere, Islam, Wasi, Street, &
Burgess 2008; Rose 2011) might be due to dominant choice alter-
natives usually found in (inefcient) orthogonal designs. Accord-
ing to their conjecture, the presence of dominant alternatives
would lead to smaller error variances and, consequently (see
e.g. Adamowicz, Louviere, & Swait 1998), to larger parameter esti-
mates. In our study, despite the absence of dominant alternatives,
we did observe that, in general, estimated attribute coefcients
tended to be larger in the OOD model than in the EFD model.
In addition, overall goodness of t has not been found superior
in the EFD model than in the OOD model.
Further, it is important to note that we have developed two
alternative designs (OOD and EFD) for simple MNL models in
which only the experimental attributes were considered in the
model specication. We must also take into account that, as
recently remarked in the literature (Choicemetrics, 2010; Rose &
Bliemer, 2009), an experimental design specically created for esti-
mating a specied model would be sub-optimal if later estimated
under other model forms (e.g. models accommodating preference
heterogeneity). Therefore, even though a comparison of the results
from a simple MNL model with those from alternative models
would be an interesting extension of the present analysis, to
undertake such a comparison it would have been necessary to
enlarge (from the beginning) our split sample design so as to in-
clude specic choice sets accounting for such extended model
specications. Consequently, we must abide by the above acknowl-
edged shortcoming in the original conception of our empirical
study: of all the multiple models that could be specied and esti-
mated, we have chosen one of the simplest to be optimised on in
our design generation process (see also Bliemer & Rose 2011).
Accordingly, we must also acknowledge that further comparison
studies should be encouraged to ascertain the effects of alternative
experimental design options under alternative model specica-
tions in empirical settings.
Regarding the issue of prior values and their importance in the
design generation process, Rose and Bliemer (2009) and Rose
(2011) hypothesise that the construction of more efcient designs
would be primarily an issue for studies involving small nite sam-
ples. When we look at our results, the OOD design based on no
prior knowledge does not seem to be clearly inferior in terms of
estimation efciency to the EFD design based on prior estimates
of parameter values. Accordingly, our speculation is that the sam-
ple used in our empirical study might have been sufciently large
enough to outweigh the expected loss of efciency in a design
based on zero prior estimates. Bayesian approaches -such as the
one implemented by Bliemer and Rose (2011)- would introduce
additional exibility in the construction of efcient designs, since
under such methodology researchers need not assume precise
prior parameter values. This additional exibility is not present
in the xed set of parameter priors used in our study, and conse-
quently we cannot assume for certain that our design should
remain relatively efcient over a range of parameter prior values.
Once more, additional empirical work with Bayesian and sequen-
tial updating approaches to experimental design would be needed
to validate whether prior values are not in fact as important in the
construction of choice experiments in empirical applications as in
simulation studies.
All in all, the results presented in this paper leave the door
open to further discussion on the empirical validity of experi-
mental design issues and on how different design construction
strategies are expected to perform on real data sets. Similar eld
studies confronting alternative experimental design approaches
and/or model forms different from the MNL model in comparable
split-sample data frameworks will surely be welcomed by
applied researchers involved in market research and food policy
analysis.
Acknowledgements
The comments and suggestions made by two anonymous refer-
ees are deeply acknowledged. Also, development of the eld survey
questionnaire has beneted from fruitful discussions with Emma
Serrano and Mara Jos Humada from CIFA. As usual, all errors
and omissions remain the authors responsibility.
References
Adamowicz, W., Louviere, J., & Swait, J. (1998). Introduction to attribute-based stated
choice methods. Final report. NOAA National Oceanic and Atmospheric
Administration, US Department of Commerce. [pp. 144].
Angulo, A. M., & Gil, J. M. (2007). Risk perception and consumer willingness to pay
for certied beef in Spain. Food Quality and Preference, 18, 11061117.
Bateman, I. J., Carson, R. T., Day, B., Hanemann, M., Hanley, N., Hett, T., et al. (2002).
Economic valuation with stated preference techniques: A manual. Cheltenham:
Edward Elgar.
Bennett, J., & Blamey, R. (Eds.). (2001). The choice modelling approach to
environmental valuation. Cheltenham: Edward Elgar.
Bernus, A., Olaizola, A., & Corcoran, K. (2003). Extrinsic attributes of red meat as
indicators of quality in Europe: An application for market segmentation. Food
Quality and Preference, 14, 265276.
Bliemer, M. C. J., & Rose, J. M. (2005). Efciency and sample size requirements for
stated choice studies. The University of Sydney Institute of Transport and
Logistics Studies Working paper: ITLS-WP-05-08 [pp. 124].
Bliemer, M. C. J., & Rose, J. M. (2011). Experimental design inuences on stated
choice outputs: An empirical study in air travel choice. Transportation Research
Part A, 45, 6379.
Burgess, L., & Street, D. J. (2005). Optimal designs for choice experiments with
asymmetric attributes. Journal of Statistical Planning and Inference, 134,
288301.
Burgess, L., Street, D. J., & Wasi, N. (2011). Comparing designs for choice
experiments: A case study. Journal of Statistical Theory and Practice, 5(1), 2546.
Chambers, J. M., Cleveland, W. S., Kleiner, B., & Tukey, P. A. (1983). Graphical methods
for data analysis. Pacic Grove, CA: Wadsworth & Brooks/Cole Publishing Co..
Choicemetrics (2010). Ngene 1.0.2 User manual and reference guide. Available from
http://www.choice-metrics.com/download.html.
Colombo, S., Hanley, N., & Louviere, J. (2009). Modeling preference heterogeneity in
stated choice data: an analysis for public goods generated by agriculture.
Agricultural Economics, 40, 307322.
Ferrini, S., & Scarpa, R. (2007). Designs with a-priori information for nonmarket
valuation with choice-experiments: A Monte Carlo study. Journal of
Environmental Economics and Management, 53, 342363.
Goos, P., & Grossmann, H. (2011). Optimal design of factorial paired comparison
experiments in the presence of within-pair order effects. Food Quality and
Preference, 22(2), 198204.
Gracia, A., Loureiro, M. L., & Nayga, R. M. (2009). Consumers valuation of nutritional
information: A choice experiment study. Food Quality and Preference, 20(7),
463471.
Hanemann, W. M. (1984). Welfare evaluations in contingent valuation experiments
with discrete responses. American Journal of Agricultural Economics, 66, 332341.
Hanemann, M. (1989). Welfare evaluations in contingent valuation experiments
with discrete response data: Reply. American Journal of Agricultural Economics,
71(4), 10571061.
Hanemann, W. M., & Kanninen, B. (1999). The statistical analysis of discrete-
response CV data. In I. J. Bateman & K. G. Willis (Eds.), Valuing environmental
preferences: theory and practice of the contingent valuation method in the US, EC
and developing countries (pp. 302442). Oxford: Oxford University Press.
Hensher, D. A., Rose, J. M., & Greene, W. H. (2005). Applied choice analysis.
Cambridge: Cambridge University Press.
Hiemstra, S. J., de Haas, Y., & Mki-Tanila, A. (2010). Local cattle breeds in Europe.
Wageningen: Wageningen Academic Publishers.
Hole, A. R. (2006). A comparison of approaches to estimating condence intervals for
willingness to pay measures. The University of York Centre for Health Economics
[CHE Research Paper 8, pp. 124].
Humada, M. J., Serrano, E., Saudo, C., Rolland, D. C., & Dugan, M. E. R. (2012).
Production system and slaughter age effects on intramuscular fatty acids from
young tudanca bulls. Meat Science, 90, 678685.
Jaeger, S. R., & Rose, J. M. (2008). Stated choice experimentation, contextual
inuences and food choice: A case study. Food Quality and Preference, 19,
539564.
Kanninen, B. J. (1993a). Optimal experimental design for double bounded
dichotomous choice contingent valuation. Land Economics, 69, 138146.
Kanninen, B. J. (1993b). Design of sequential experiments for CV studies. Journal of
Environmental Economics and Management, 25, 111.
Kerr, G. N., & Sharp, B. M. H. (2010). Choice experiment adaptive design benets: A
case study. The Australian Journal of Agricultural and Resource Economics, 54,
407420.
Kessels, R., Goos, P., & Vandebroek, R. (2006). A comparison of criteria to design
efcient choice experiments. Journal of Marketing Research, 43, 409419.
Lancaster, K. J. (1966). A new approach to consumer theory. The Journal of Political
Economy, 74(2), 132157.
22 M. Domnguez-Torreiro / Food Quality and Preference 35 (2014) 1523
Loureiro, M. L., & Umberger, W. J. (2003). Estimating consumer willingness to pay
for country-of-origin labeling. Journal of Agricultural and Resource Economics,
28(2), 287301.
Loureiro, M. L., & Umberger, W. J. (2007). A choice experiment model for beef: What
US consumer responses tell us about relative preferences for food safety,
country-of-origin labeling and traceability. Food Policy, 32(4), 496514.
Louviere, J. J., Hensher, D. A., & Swait, J. D. (2000). Stated choice methods: Analysis and
applications. Cambridge: Cambridge University Press.
Louviere, J. J., Islam, T., Wasi, N., Street, D., & Burgess, L. (2008). Designing discrete
choice experiments: Do optimal designs come at a price? Journal of Consumer
Research, 35(2), 360375.
Louviere, J. J., Pihlens, D., & Carson, R. (2011). Design of discrete choice experiments:
A discussion of issues that matter in future applied research. Journal of Choice
Modelling, 4(1), 18.
Luce, R. D. (1959). Individual choice behaviour: A theoretical analysis. New York:
Wiley.
Lusk, J., Fox, J., Schroeder, T., Mintert, J., & Koohmaraje, M. (1999). Will consumers
pay for guaranteed tender steak? Research Bulletin 3-99, Research Institute on
Livestock Pricing Agricultural and Applied Economics Virginia Tech. [pp. 120].
Lusk, J. L., Roosen, J., & Fox, J. A. (2003). Demand for beef from cattle administered
growth hormones or fed genetically modied corn: A comparison of consumers
in France, Germany, the United Kingdom, and the United States. American
Journal of Agricultural Economics, 85(1), 1629.
Manski, C. (1977). The structure of random utility models. Theory and Decision, 8,
229254.
McCluskey, J. J., & Loureiro, M. L. (2003). Consumer preferences and willingness to
pay for food labelling: A discussion of empirical studies. Journal of Food
Distribution Research, 34(3), 95102.
McFadden, D. (1974). Conditional logit analysis of qualitative choice behaviour. In P.
Zarembka (Ed.), Frontiers in econometrics (pp. 105142). New York: Academic
Press.
Olynk, N. J., Tonsor, G. T., & Wolf, C. A. (2010). Consumer willingness to pay for
livestock credence attribute claim verication. Journal of Agricultural and
Resource Economics, 35(2), 261280.
Pouta, E., Heikkila, J., Forsman-Hugg, S., Isoniemi, M., & Makela, J. (2010). Consumer
choice of broiler meat: The effects of country of origin and production methods.
Food Quality and Preference, 21(5), 539546.
Realini, C. E., Font i Furnols, M., Saudo, C., Montossi, F., Oliver, M. A., & Guerrero, L.
(2013). Spanish, French and British consumers acceptability of Uruguayan beef,
and consumers beef choice associated with country of origin, nishing diet and
meat price. Meat Science, 95(1), 1421.
Rose, J. M. (2011). The usefulness of Bayesian optimal designs for discrete choice
experiments by R. Kessels, B. Jones, P. Goos and M. Vandebroek. Applied
Stochastic Models in Business and Industry, 27, 193196.
Rose, J. M., & Bliemer, C. J. (2009). Constructing efcient stated choice experimental
designs. Transport Reviews, 29, 587619.
Sndor, Z., & Wedel, M. (2001). Designing conjoint choice experiments using
managers prior beliefs. Journal of Marketing Research, 38, 430444.
Scarpa, R., Campbell, D., & Hutchinson, W. G. (2007). Benet estimates for landscape
improvements: Sequential Bayesian design and respondents rationality in a
choice experiment. Land Economics, 83(4), 617634.
Scarpa, R., & Rose, J. (2008). Design efciency for non-market valuation with choice
modelling: How to measure it, what to report and why. Australian Journal of
Agricultural and Resource Economics, 52(3), 253282.
Street, D. J., & Burgess, L. (2007). The construction of optimal stated choice
experiments. Hoboken: John Wiley & Sons.
Street, D., Burgess, L., & Louviere, J. (2005). Quick and easy choice sets: Constructing
optimal and nearly optimal stated choice experiments. International Journal of
Research in Marketing, 22, 459470.
Swait, J. D., & Louviere, J. J. (1993). The role of the scale parameter in the estimation
and comparison of multinomial logit models. Journal of Marketing Research, 30,
305314.
Train, K. (2003). Discrete choice methods with simulation. Cambridge: Cambridge
University Press.
Verbeke, W., Perez-Cueto, F. J. A., de Barcellos, M. D., Krystallis, A., & Grunert, K. G.
(2010). European citizen and consumer attitudes and preferences regarding
beef and pork. Meat Science, 84(2), 284292.
Vermeulen, B., Goos, P., Scarpa, R., & Vandebroek, M. (2011). Bayesian conjoint
choice designs for measuring willingness to pay. Environmental & Resource
Economics, 48, 129149.
M. Domnguez-Torreiro / Food Quality and Preference 35 (2014) 1523 23