The International Journal of Transitional Justice, Vol.

4, 2010, 315328,
doi: 10.1093/ijtj/ijq023

Editorial Note
Colleen Duggan
Acknowledging and atoning for the systematic abuse of human rights and for violations of international humanitarian law is arguably one of the most controversial,
complex and unpredictable processes undertaken by governments and citizens in
societies transitioning from a violent past. As many scholars and practitioners
have noted, the implementation of transitional justice policies and mechanisms
has become increasingly entrenched in the response of the world community to
peacebuilding and social reconstruction. Yet, how do we know whether transitional
justice works? For those with a stake in the outcomes of transitional justice, the
impact question has come to occupy a position of singular importance. Concern is
growing that the enthusiasm of transitional justice proponents and the eagerness of
the international aid community to finance transitional justice mechanisms have
not been accompanied by commensurate efforts to evaluate the effects that these
interventions are having on the lives of the intended beneficiaries. Does transitional justice help or hinder democratic transition? Does it heal or hurt victims?
Does it contribute to reconciliation or just wake old ghosts?
Over some 20 years of transitional justice efforts across countries and continents,
an evolution has occurred in our understanding and thinking about type, timing,
political and social consequences, local approaches and the relationship of judicial
mechanisms to democratic consolidation. With this maturation comes a call for
greater scrutiny. The world over, individuals involved in transitional justice
government officials, victims, perpetrators, ordinary citizens, scholars and those
in the international aid community are beginning to ask hard questions about
the evidence base that speaks in favor of or against the place that transitional justice
occupies in the public policy arena.
But what counts as evidence in the world of public policy? In my remarks, I
am unapologetically single-minded in my perspective on what constitutes good
evidence. Over the last two decades, I have been an activist, a researcher and a funder
of research for development. As such, my worldview is shaped by my perception
of how research should serve the cause of human rights and social and economic
justice. As an emerging field of study and practice, in many ways transitional justice
is still finding its conceptual and political feet. However, what is abundantly clear
is that because transitional justice mechanisms aspire to catalyze processes of deep
social change at the global, national and local levels, transitional justice, by its very
nature, dwells in the realm of politics and public affairs. When seen through this
lens, a strong case can be made that research on transitional justice theory and

Senior Programme Specialist, Evaluation Unit, International Development Research Centre,

Canada. Email: cduggan@idrc.ca


C The Author (2010). Published by Oxford University Press. All rights reserved.

For Permissions, please email journals.permissions@oxfordjournals.org.

316 Editorial Note

practice is not a simple question of scholarship for the ivory tower. It should be
useful, accessible and able to inform public agendas, particularly for those who will
live with the legacy of transitional justice once policy makers predictably relocate
their interest and resources to new frontiers.
In this special issue of the International Journal of Transitional Justice, which we
have entitled Transitional Justice on Trial: Evaluating Its Impact, we will delve
more deeply into the question of how the field might be evaluated and what constitutes good evidence in transitional justice research and practice. Having spent
considerable time over the years following the literature and policy debates around
transitional justice, I am convinced that transitional justice knowledge production cannot indeed, should not be separated from knowledge dissemination,
translation and utilization.

Making Research Matter

I believe that if research is to play any sort of influential role in effecting lasting social
change, the inescapable conclusion is that utility must emerge as the principal goal
of applied research and practice in the cause of transitional justice. Utility means
that the intended use of the knowledge being produced serves the needs of the
intended user(s). Research utility can be broken down into the following three ancillary dimensions, and these should act as the guiding criteria for judging research
excellence and good evidence in the field of transitional justice: (1) credibility,
understood as research being methodologically strong and based on considerations of scientific merit as assessed in relation to the relevant discipline/field; (2)
relevance, understood as research being appropriate, given its intended audience,
user(s), context and purpose; and (3) actionability, understood as research being
accessible and having the potential for mobilization and uptake into policies and
practices for social change in the organizations, entities and societies where transitional justice mechanisms are in operation. Simply put, the transitional justice
evidence base that we are accumulating through applied research and evaluation
practice should be utilization focused.
In laying out the above framework, I am conscious that there are different degrees
of credibility, relevance and actionability and that not all research will be weighted
equally between the three dimensions. For instance, research can be influential by
being relevant and accessible to policy makers or just by virtue of its high quality.
However, as we are so often reminded, evidence is but one consideration in policy making. It can be used, misused or even ignored, sometimes with disastrous
consequences. International development practice is awash with examples of governments that have been closed to or unwilling to act on evidence.1 At the time of
writing, the international community is marking the five-year anniversary of UN
1

One of most notorious failures to follow evidence was the South African governments delaying
the implementation of antiretroviral therapy. It has been estimated that this led to over 300,000
deaths that would likely otherwise have been averted, even given the difficulties of saving lives with
therapy. Kevin Kelly, The Vagaries of Research and Evaluation Influence in the HIV/AIDS Field,
in Evaluating Research in Violently Divided Societies (forthcoming).

International Journal of Transitional Justice, Vol. 4, 2010, 315328

Editorial Note

317

member states endorsement of the responsibility to protect doctrine, reminding

me of the Rwandan genocide and one of the biggest failures in recent human history
to act on evidence. I am also aware of the trade-offs researchers face in choosing
between producing policy-oriented publications and peer-reviewed publications.
Peer-reviewed publications allow researchers to gain some prestige and build a
name for themselves in their fields, which is less so the case with policy-oriented
publications. Also, peer-reviewed publications rarely make the reading lists of policy makers, creating a troublesome dilemma for grounded researchers who hope
to be recognized as scholars and influence policy for social change on the ground.
Despite these tensions, I believe there are three reasons why the field of transitional justice should adopt a utilization-focused approach that aims to achieve a
credibilityrelevanceactionability balance. First, transitional justice is a field of
practice that strives to prescribe solutions for some of the most tragic experiences
of human interaction. Researchers and advocates have a moral obligation to collect
and analyze evidence of the highest standard that speaks in favor of or against the
consequences brought about by the implementation of transitional justice mechanisms. Second, as the research community continues to point out, some of the core
assumptions that tie the mechanisms and their potential effects to larger processes
of peacebuilding or rule of law are tentative,2 and the evidence base is fragmented.
Despite there being so much we do not know about the long-term effects of transitional justice, including its differentiated impacts upon specific groups such as
women and children, mechanisms continue to be transferred across settings with
insufficient reflection given to contextual relevance and appropriateness. Third, in
the field of transitional justice, scholarship, advocacy and practice have tended to
coexist under the same roof, creating a single feedback loop that can be considered
either virtuous or harmful, depending on ones perspective and disciplinary bent.3
Although this may improve prospects for actionability it means that researchers
need to be particularly vigilant in safeguarding credibility. The opposite is also
true: advocates need to work to underpin activist activities with methodologically
solid research.

Applied Research and Development Evaluation: Sharing

the Impact Terrain
For transitional justice actors scholars, advocates and policy makers evaluation
of the field is usually achieved by two means: applied empiric research or what is
2

See, David Mendeloff, Truth-Seeking, Truth-Telling, and Postconflict Peacebuilding: Curb the
Enthusiasm? International Studies Review 6(3) (2004): 355380; Eric Brahm, Uncovering the
Truth: Examining Truth Commission Success and Impact, International Studies Perspectives 8(1)
(2007): 1635; Jack Snyder and Leslie Vinjamuri, Trials and Errors: Principle and Pragmatism
in Strategies of International Justice, International Security 28(3) (2003): 544; Erin Daly, Truth
Skepticism: An Inquiry into the Value of Truth in Times of Transition, International Journal of
Transitional Justice 2(1) (2008): 2341.
For a more in-depth discussion, see, Leslie Vinjamuri and Jack Synder, Advocacy and Scholarship
in the Study of International War Crime Tribunals and Transitional Justice, Annual Review of
Political Science 7 (2004): 345362.

International Journal of Transitional Justice, Vol. 4, 2010, 315328

318 Editorial Note

commonly known as development evaluation. Here, I will pause to bring some

conceptual clarity to the terms applied research and development evaluation,
as this special issue brings together these two broad categories of knowledge and
practice. Applied research is a process of inquiry that applies scientific knowledge,
improvements or discoveries with the intent of solving a defined problem.4 While
much applied research is descriptive and helps advance understanding of practical
problems and their solutions, the applied research submissions in this special issue
are empiric and ask evaluative questions about understanding, improving or determining the effectiveness of the transitional justice mechanisms being employed
in real-world settings Cambodia, South Africa, Sierra Leone, the Central African
Republic, Bangladesh, Italy, Chile as opposed to in highly controlled, traditional
scientific laboratories.5
This issue also includes submissions that discuss the impact problem from the
perspective of development evaluation. Evaluation is an applied inquiry process
that collects and synthesizes evidence that
culminates in conclusions about the state of affairs, value, merit, worth, significance
or quality of a program, product person, policy, proposal or plan. Conclusions made
in an evaluation encompass both an empiric aspect (that something is the case) and a
normative aspect (judgment about the value of something).6

Development evaluation stems from the field of evaluation science and is a generic
term for evaluations conducted in developing countries, usually focused on the
effectiveness of international aid projects, programs and organizations.7
Applied research and development evaluation have many similarities and a few
differences. Both rely on social science methods and examine multiple facets of a
problem, often using multimethod approaches. Both also collect and analyze data
in order to come to conclusions, and both utilize theory to inform work. While the
two processes of inquiry share more commonalities than differences, two important
features set evaluation apart: judgment, or valuing, and use. The primary purpose
of evaluation is to amass sufficient information to allow an evaluator to assess the
value or worth of something against a set of criteria. Evaluation is not just about
collecting and analyzing information; it is supposed to use data to make evaluative
judgments. These are fed back to a client in order to assist management and decision
making. Without this additional valuing, an evaluation is only a research project
that may increase knowledge but does not help in decision making.8
4

6
7
8

P. Cristian Gugiu, What Is Evaluation and How Does It Differ from Research? (paper presented at
the annual conference of the American Evaluation Association, Evaluation 2006: The Consequences
of Evaluation, Portland, OR, 14 November 2006).
This suggests a need to exercise caution and good sense in viewing transitional justice interventions
as experiments. Papers in this special issue underscore the importance and validity of using multiple approaches to evaluating impacts: experimental, quasi-experimental or non-experimental;
qualitative, quantitative or mixed method.
Deborah M. Fournier, Evaluation, in Encyclopedia of Evaluation, ed. Sandra Mathison (Thousand
Oaks, CA: Sage Publications, 2005).
Michael Quinn Patton, Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use (New York: Guilford Press, 2010).
I am grateful to Rick Davies for this observation.

International Journal of Transitional Justice, Vol. 4, 2010, 315328

Editorial Note

319

Why should we care about the seemingly subtle differences between applied
research and evaluation? In our discussion, these differences matter because they
define how researchers and evaluators are tackling the challenges of assessing
transitional justice impacts. Each seeks to answer evaluative questions about these
impacts for slightly different reasons. An understanding of the strengths and limitations of each field of activity can bring us closer to untying some of the knots
of transitional justice as a hard-to-measure area of inquiry and practice. This is
why it is both positive and timely that analysts of transitional justice bring the two
fields of research evaluation science and applied social science closer together
for greater learning.
As evidenced in this and in previous issues of IJTJ, over the past few years applied
researchers have shown an increasing interest in making use of the principles and
methods emerging from development evaluation to inform and improve applied
research.9 Part of this interest stems from the reality that a number of researchers
and advocates of transitional justice are finding themselves in the position of
acting as evaluators of international aid projects in support of transitional justice.
Similarly, researchers and advocates can find themselves in the uncomfortable
position of being evaluation subjects, that is, of having their research or advocacy
efforts questioned by donors and treated as a unit of analysis in which quality and
impacts are assessed or measured against criteria or standards that oftentimes feel
at odds with original intentions or hypotheses.10

Transitional Justice as a Hard-to-Measure Area of

Social Change
As a field that inspires great emotion and passion, not surprisingly, transitional
justice is increasingly being held to the highest of normative and academic standards. The critiques offered in this issues articles by Oskar Thoms, James Ron
and Roland Paris and by Geoff Dancy allow us to begin wading into the impact
labyrinth, providing a useful conceptual backdrop for grounding further reflections about research utility and the strengths and weaknesses of interpretivist and
positivist approaches.11 The other articles in this issue equip us with further conceptual and theoretical insights and rich learning drawn from empiric scholarship
and advocacy practice. Adding to this collection of issues, I would like to share my
own reflections about the evaluation of transitional justice. What is it about the
9
10

11

See, Phuong Pham and Patrick Vinck, Empirical Research and the Development and Assessment of
Transitional Justice Mechanisms, International Journal of Transitional Justice 1(2) (2007): 231248.
Hence my call for a new research for development framework that prioritizes use and assesses research excellence in terms of conceptual/methodological quality, relevance and contextual
actionability.
In the social sciences, interpretivists believe that we should be concerned not only with quantifying
what actually happens in social phenomena but also with providing an interpretation of events and
phenomena in terms of how the people involved understand their own experience. Positivists believe
that scientific method is the best approach to uncovering the processes by which both physical and
human events occur and the only authentic knowledge is that based on sense experience and
positive verification.

International Journal of Transitional Justice, Vol. 4, 2010, 315328

320 Editorial Note

particular settings in which transitional justice experiences unfold that necessitates

evaluation approaches that are highly context sensitive? How do we get better at
putting in place evaluation practices that reward rather than punish the innovation
and adaptation often needed to ensure that transitional justice mechanisms have
a reasonable chance of success? I believe that scholars, advocates, practitioners
and donors must reflect upon these questions if we are to begin separating what I
consider to be research of questionable use from research that engages with transitional justice in all its complexity and asks meaningful questions about the how
and the why of impacts.
While transitional justice can be defined as a hard-to-measure area for many
reasons, three sets of difficulties stand out and merit further discussion here: conceptual challenges, contextual challenges and practical challenges. The last relates
to the real-world constraints that arise from research and evaluation practice. As
these three areas bleed into each other, I will discuss them in no particular order.
Currently, transitional justice suffers from a particular problem: as a relatively
new and emergent field, we are in the early days of theory building and a number
of the hypotheses that underpin transitional justice initiatives are either untested
or inconclusive. Typical change processes or implicit (and often untested) assumptions about the goals of transitional justice include social healing (through
truth-telling initiatives), reducing recidivism (through criminal trials for human
rights abusers) and facilitating the formation of new identities (through history education reform). As noted by Hugo van der Merwe12 and many others, transitional
justice mechanisms often lack clearly defined goals, which throws up obstacles for
development evaluation. An evaluator, first and foremost, needs clarity on intentionality transitional justice programming for what, for whom and how. In other
words, what are you setting out to do and how do you plan to get there? Being clear
about what a transitional justice process is trying to achieve and how what in
evaluation theory and practice is known as a theory of change and in social science
as a hypothesis is the point of departure of any development evaluation. I have
always been a bit baffled by the question, Does transitional justice work?
The obvious answer is, It depends on what it is trying to do. Transitional justice
mechanisms aspire to achieve many different, varied goals and yet in debates, one
often has the impression that there is one homogeneous, universal and overarching goal of transitional justice. Many proponents of criminal tribunals would
favor utility arguments around deterrence and ending impunity over those for
individual healing for victims. The opposite is true for some proponents of truth
commissions. The point is that in the domain of transitional justice, depending
on the mechanism, we are often faced with a panoply of theories and most often
they are not well articulated by those involved in the implementation process. In
international aid projects, it is often the evaluator who must articulate or reconstruct the theory of change (often ex post) prior to beginning work. A complicating
12

Hugo van der Merwe, Delivering Justice during Transition: Research Challenges, in Assessing the
Impact of Transitional Justice: Challenges for Empirical Research, ed. Hugo van der Merwe, Victoria
Baxter and Audrey R. Chapman (Washington, DC: United States Institute of Peace Press, 2009).

International Journal of Transitional Justice, Vol. 4, 2010, 315328

Editorial Note

321

factor here is that transitional justice has widened to envelop a host of social justice
goals that are usually ascribed to international development,13 thus multiplying
the menu of theories that need to be tested.
Given this state of affairs, there are good reasons for ensuring that when we
discuss impact, we are clear about which theory is being interrogated and who
decided that this theory is the most relevant one to pursue in the first place. In
the aid world, this means that transitional justice projects and programs should
be underpinned by theory-based evaluation.14 As its name implies, theory-based
evaluation can uncover the difficulties or deficiencies in the original theory underpinning a programs or projects logic. In an ideal world, transitional justice
programming would be accompanied by deeper empiric research. However, as
this is most often not the case and as transitional justice projects generally unfold
in limited time frames (for example, an average truth commission may run two
or three years), theory-based evaluation offers short-term possibilities for clarifying some of the most important questions that inform failure or success (for
example, transitional justice for what and for whom).
Transitional justice is also almost always the result of political compromise and
seldom reflects the ideal state of justice. Yet, in transitional justice research and
practice, mechanisms are almost always measured against someones ideal concept
of justice. Culture, ideology and politics also muddy the waters. Funders and those
being funded often have very different perceptions of what constitutes justice.
Most unfortunately, the record of development evaluation indicates that it is often
the donor that determines the broad parameters around what transitional justice
success or failure look like. This is not always a good thing; in some cases, donors
have been complicit in human rights abuse legacies and may be keen to whitewash
their own reputations. Recipients of transitional justice programs (governments,
nongovernmental organizations (NGOs), victims groups) may have good reason
to distrust bilateral or multilateral donors. In such contexts, external evaluation
can understandably be viewed as an extension of repressive tactics. In order to
level the playing field and offset power imbalances, donors should consider using
a combination of external and internal evaluation. The success or failure of a
program in support of transitional justice must pay heed to the equally important
imperatives of vertical accountability to the donor and horizontal accountability
to a wider base of civil society stakeholders.
Similarly, those being evaluated often view the parachuting of external evaluators
into socially volatile environments as an imposition and a liability to be managed,
as they assume that any negative findings will be taken up and exploited by political
adversaries of transitional justice. It is not uncommon to come across this same

13
14

See, Special Issue: Transitional Justice and Development, International Journal of Transitional
Justice 2(3) (2008).
See, Carol H. Weiss, Theory-Based Evaluation: Past, Present, and Future, in Progress and Future
Directions in Evaluation: Perspectives on Theory, Practice, and Methods, ed. Debra J. Rog and Deborah
Fournier (San Francisco, CA: Jossey-Bass, 1997).

International Journal of Transitional Justice, Vol. 4, 2010, 315328

322 Editorial Note

line of argument in the field of human rights advocacy.15 In politically charged

contexts, the potential for doing more harm than good is very real. For example,
the availability of evidence that suggests financial malfeasance in the administration of reparations or that a truth commission has fallen far short of its original
intentions could result in a serious loss of political capital for a transitional regime.
While I am genuinely sympathetic to this argument, there are also reasons why we
need to start examining the unintended outcomes both positive and negative
of transitional justice. What in some circumstances might be understood as failure
in other circumstances may yield evidence to the contrary. The time may have
arrived for us to begin looking for the positive deviants in transitional justice processes those victims who manage to coexist in close proximity with perpetrators
and reconstruct their life projects despite disappointing truth commissions and
undelivered reparations. Admittedly, this will be difficult in the aid world given
the multiple accountabilities at play and the different set of reasons why admitting
failure is less than appetizing to donors. The most immediate problem is that many
development evaluations are never publicly accessible, which is ironic in the case
of publicly funded bilateral and multilateral development agencies.16
Absent or insufficient baseline data also contributes to measurability problems,
although this is not a phenomenon unique to the field of transitional justice. Development evaluators Michael Bamberger, Jim Rugh and Linda Mabry estimate that
at least 75 percent of impact evaluations of development projects and programs are
conducted without access to any systematic information on the conditions of the
project population prior to the intervention and that even fewer evaluations have
access to baseline data on relevant comparison groups.17 In fact, many bilateral and
multilateral development agencies, UN organizations and NGOs have accepted,
explicitly or implicitly, that their impact evaluations are not commissioned until
late in the project cycle and that they cannot access baseline data. Besides establishing a basis for counterfactual comparison, the value of baseline data lies in its
potential to provide us with an intimate portrait of context that the ever-changing
business of international aid might otherwise miss. James Gibson, Jeffrey Sonis
and Sokhom Heans article in this issue, on Cambodias experience with the Khmer
Rouge trials, provides us with an important example of how baseline studies can
contribute to deep contextual understanding. Such insights should act as the analytical starting point of the many projects and programs in support of transitional
justice.
Working in a hard-to-measure area, transitional justice actors face a political and
operational conundrum. Attribution of results or change to a single intervention
or a series of them is something that everyone dreams about, but the evidence is
15

16
17

See, Paul Gready, Reasons to Be Cautious about Evidence and Evaluation: Rights-Based Approaches
to Development and the Emerging Culture of Evaluation, Journal of Human Rights Practice 1(3)
(2009): 380401.
There are notable exceptions and in some agencies, practice is moving toward greater transparency
through public access.
Michael Bamberger, Jim Rugh and Linda Mabry, RealWorld Evaluation: Working Under Budget,
Time, Data and Political Constraints (Thousand Oaks, CA: Sage Publications, 2005).

International Journal of Transitional Justice, Vol. 4, 2010, 315328

Editorial Note

323

usually lacking. Changes in the well-being of intended beneficiaries of transitional

justice can occur before or after a program ends, they may not take the form
anticipated and they may be influenced by the actions of stakeholders who remain
beyond the reach of the program18 (peace process spoilers come immediately
to mind). Proof of causality and attribution of results are even more difficult in
complex environments. Yet, this is exactly what many donors ask recipients of
international aid to do. The attribution obsession in international aid has led to
a tendency to rush toward what I would argue is a singular, unhelpful focus on
impact data, often at the expense of process. Critically important reasons exist for
also focusing on the human dimension of process in transitional justice initiatives.
Transitional contexts are often characterized by intense feelings of distrust at many
levels. The ideological divisions in civil society can be deep and particularly difficult
for an outsider to understand, let alone navigate. Evaluation of programs designed
to serve disadvantaged groups such as victims of human rights violations may
actually be threatening to stakeholders in those groups. Program beneficiaries
have often suffered bad experiences with the management of international aid,
including evaluation. For many of these people, evaluation has been a highly
disempowering experience. Whatever form of evaluation is chosen, it should be
conflict sensitive and not exacerbate tensions, making a fraught situation worse.
Bearing in mind the divided nature of many transitional contexts, we need to give
more serious consideration to development evaluation approaches that embrace
rather than side-step context and that build rather than erode the social capital of
the organizations and beneficiaries involved. Approaches such as self-assessment
and peer review can play a constructive role in the achievement of the longer-term
social goals that underlie transitional justice processes. As Nokukhanya Mncwabes
article on the African Transitional Justice Research Networks experience in piloting
a peer review methodology illustrates, such approaches have the added advantage of
building learning capacity into the organizations of transitional justice stakeholders
as well as strengthening practitioner-based networks. While we often dwell upon
the importance of evaluation for evidence, in societies dealing with a violent past,
it is important to remember the linkages that exist between findings as evidence,
the process-use benefits of evaluation19 and the potential to recuperate lost social
capital. Findings alone are of limited use and rarely (if ever) is an evaluation report
sufficient to support social change. Because a large part (although not all) of
transitional justice programming is about rebuilding broken relationships, donors
and evaluators need to double their efforts to straddle the intersect between the use
of evaluation findings for evidence-based decision making and good participatory
18
19

Terry Smutylo, Outcome Mapping: A Method for Tracking Behavioural Changes in Development
Programs, Institutional Learning and Change Brief 7 (August 2007).
Process use refers to individual changes in thinking, attitudes and behavior, and program or
organizational changes in procedures and culture that occur among those involved in evaluation
as a result of the learning that occurs during the evaluation process. Michael Quinn Patton,
Utilization-Focused Evaluation (Thousand Oaks, CA: Sage Publications, 2008), 155. Process use in
evaluation is akin to the benefits ascribed to action research.

International Journal of Transitional Justice, Vol. 4, 2010, 315328

324 Editorial Note

process that renews strained relationships and builds strong local organizations.
At the end of the day, it is local actors who need to be convinced of the impacts of
transitional justice and be in possession of the necessary analytical skills to make
these sorts of judgments.
While applied researchers can encounter constraints around time, resources
and political conditions, in development evaluation, these factors are experienced
tenfold. As mentioned, evaluation has a client who wants to know something and
is used by funders to make real-time management decisions that are often linked
to project or program funding and continuity. Brandon Hamber, Liz S evcenko and
Ereshnee Naidus article on evaluating sites of conscience in Italy, Bangladesh and
Chile and Patrick Vinck and Phong Phams article outlining their evaluation of the
International Criminal Courts outreach program in the Central African Republic
provide useful portrayals of the tensions that evaluators face in finding evidence
of results for a client, as well as around measuring short or intermediate outcomes
versus long-term impacts the everyday realpolitik of development evaluation.
The sort of long-term impacts that transitional justice mechanisms hope to bring
about (for example, rebuilding civic trust, creating a human rights culture or
generating empathy between former adversaries) are the result of years and years
of investment and can take generations. Not surprisingly, Hamber et al. lament
that showing impacts on attitudinal change should be satisfactory in itself, while
acknowledging that in times of transition, institutions that have clear political
content must be seen to be influencing or effecting wider social change.
This gaping chasm between outcomes20 and impacts21 suggests that transitional
justice is an area where applied, social science research and development evaluation
could collaborate more effectively. As David Backers article on victims attitudes in
South Africa demonstrates,22 longitudinal research that tracks changing attitudes
and perceptions across time can be particularly useful for understanding if and
how transitional justice makes a difference to victims. Of course, the difficulty
lies in making the leap from cognitive change (an outcome that is not readily
evident) to change that suggests forward movement toward greater impact. What
victims say and what they do are often two different things. The world of development evaluation may be able to make a contribution here. Outcome mapping,23 a
planning, monitoring and evaluation methodology that uses behavior change as a
key indicator of social change and was specifically designed to track the effects of
research, has been used by a limited number of actors in the field of transitional
20

21
22

23

The likely or achieved short-term and medium-term effects of the products, capital goods and
services of a development intervention. Organisation for Economic Cooperation and Development, Development Assistance Committee, Glossary of Key Terms in Evaluation and Results-Based
Management (2002).
Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended. Ibid.
See also, James L. Gibson, Jeffrey Sonis and Sokhom Hean, Cambodians Support for the Rule of
Law on the Eve of the Khmer Rouge Trials, International Journal of Transitional Justice 4(3) (2010):
377396.
Sarah Earl, Fred Carden and Terry Smutylo, Outcome Mapping: Building Learning and Reflection
into Development Programs (Ottawa: International Development Research Centre, 2001).

International Journal of Transitional Justice, Vol. 4, 2010, 315328

Editorial Note

325

justice.24 Embedding empiric research initiatives into mainstream transitional justice programs that are funded by the international aid community might also offer
new opportunities for elucidating how transitional justice works (not just if it
works), for improving its effectiveness and for moving away from the templatization of the processes a trend that is worrying many scholars and advocates and
is noted in this issue by Thoms et al.
Good development evaluation should adhere to the universally accepted standards of utility, propriety, feasibility and accuracy.25 Herein lies one of the root
problems of evaluating transitional justice. As a field that brings together multiple
disciplines and is still under conceptual construction, transitional justice can and
does fall victim to poor-quality evaluation. The regulation of evaluation as an
area of professional practice is still in its infancy, with heated discussions about
the pros and cons of accreditation taking place in evaluation associations in the
global North and South. Some argue that accreditation would boost the prestige
of the profession and help weed out unethical and incompetent evaluators. The
counterargument is that accreditation would be one more means of shutting out
evaluators from the developing world. There is a certain ring of truth in this perspective, as evaluation as a field of research and practice was born and has grown
up in the North. With notable exceptions, development evaluation is dominated
by professionals from North America and Europe (few university-accredited programs in evaluation are available elsewhere), which raises serious questions about
who decides what counts as good evidence and whose values shape an evaluation.
Good evaluators who understand what transitional justice is, let alone who can
untangle and measure the fields multilayered and often complex goals, are in
short supply. In this sense, building the field of evaluation in the global South is as
urgent a priority as the need to build a cohort of transitional justice researchers in
countries of the developing world.
There is also a problem of vocabulary and dialogue between applied researchers
and development evaluation specialists. As I read through the articles and peer
reviews submitted for this issue, I was fascinated to see how often individuals of
diverse academic and professional backgrounds accused authors of using jargon
and unintelligible language (a tendency I hope I have not compounded with this
editorial). This was most marked in articles written by development evaluators and
reviewed by nonevaluators. As is the case with all fields, development evaluation
has generated a particular language that can be unclear or inaccessible to those
less familiar with this field.26 This is not to say that evaluators should have to
24

25

26

Experiences have been mixed, however, and I am not suggesting that outcome mapping is appropriate for tracking the effects of all transitional justice interventions. See, Colleen Duggan, Show
Me Your Impact: Evaluating Transitional Justice in Contested Spaces, Journal of Evaluation and
Program Planning (forthcoming).
These are the program evaluation standards adopted by the American Evaluation Association.
Similar standards exist in the other regional and national evaluation associations in both the
developing and the developed world.
I am not the first to remark upon this issue. The American Evaluation Associations New Directions
in Evaluation journal devoted an entire special issue to the question of language in evaluation.

International Journal of Transitional Justice, Vol. 4, 2010, 315328

326 Editorial Note

change their lexicon completely; interdisciplinarity dictates moving beyond ones

default setting and comfort zone. It suggests, instead, that if evaluators hope to be
influential, they need to find ways to bridge this divide, particularly as evaluation
in international aid is laden with power relationships. Evaluators generally hold
the power and, regrettably, not all evaluators carry the scepter with benevolence.
Far too often, evaluators are viewed at best as an annoyance to be endured and at
worst as the grim reaper.
As a field of research and practice, evaluation is frequently maligned and often
misunderstood. This is not surprising given that evaluation in the context of
international aid has come to be associated with the current status quo: wide
application of orthodox, linear approaches that oversimplify causal relationships
and are ill-suited to the particular needs of transitional justice situations. The
dominant approach to development evaluation as currently practiced by most
funders (and as currently experienced by most recipients) follows the dominant
approach to scientific practice. It attempts to isolate for effects and look for evidence
of attribution to a single initiative (usually a project or program that supports the
operation of a transitional justice mechanism). The danger with this evaluation
approach is that both funders and funded inevitably fall into the project trap,
looking for attribution of outcomes to a single project or intervention.
The articles in this special issue leave no doubt that we are getting much better at
tackling some of the conceptual, methodological and contextual issues that bedevil
the evaluation of transitional justice. In its early days, transitional justice research
asked fairly linear questions that sought simple causeeffect linkages, particularly
in the application of a single mechanism (such as a truth commission), to solve
what could arguably be understood as long-term, intergenerational social problems
(such as intergroup violence) that have generally been products of decades of bad
governance and misguided public policy. This obsession with isolating a single
variable and then endeavoring to hold the model constant (in such volatile and
unpredictable terrains as those where transitional justice tends to operate) always
seemed curious to me. Transitional justice proponents are often quick to echo
Michael Ignatieffs observation that, at a minimum, truth commissions can narrow
the range of permissible lies.27 I have always felt the same about research: it only
narrows the range of causality. In the field of transitional justice, we would be
ill-advised to expect empiric research to close that gap completely. Thankfully, the
madness of trying to isolate for independent effects to single mechanisms appears
to have abated. The field has now fanned out, asking more complex questions about
the interplay of mechanisms and interrogating the effectiveness (or ineffectiveness)
of these mechanisms when used in concert with each other. Tricia Olsen, Leigh
Payne, Andrew Reiter and Eric Wiebelhaus-Brahms article, which tables the idea
of a justice balance, explaining why and how truth commissions are more likely

27

Special Issue: How and Why Language Matters in Evaluation, New Directions for Evaluation 86
(2000).
Michael Ignatieff, Articles of Faith, Index on Censorship 25(5) (1996): 113.

International Journal of Transitional Justice, Vol. 4, 2010, 315328

Editorial Note

327

to achieve their human rights goals when accompanied by amnesties or criminal

trials, is an example of how thinking has evolved.

Promising New Directions

So what is on the horizon for evaluating the often elusive goals of transitional justice
mechanisms? Depending on what they are aspiring to achieve, some processes
could be understood as emergent interventions, social innovations that take place
in socially and politically fragile contexts characterized by increasing complexity.
This means that causal relationships are extremely difficult, if not impossible, to
establish; that the change being sought through a transitional justice mechanism
will be nonlinear, the result of multiple interactions by numerous actors; and that
these will be difficult to predict, let alone control. As social interventions, one of
the most critical features of transitional justice processes is that they are nested in
social systems. It is through the workings of entire systems of social relationships
that any changes in behavior or social conditions will be effected. As a result of
contextual heterogeneity, rarely, if ever, will the same approach to a transitional
justice mechanism be equally effective in all circumstances.28 This suggests that
context-sensitive and systems approaches to evaluating transitional justice will be
of much greater utility than the current linear approaches that dominate practice
in the international development industry.
Truth commissions are probably the best example of a transitional justice mechanism that can be considered a social innovation. Interviews with those involved
in such commissions commissioners, staff, witnesses and victims all attest to
the fact that these processes never follow a linear pathway. They elicit unexpected
and unanticipated divergences and upheavals, and they involve a high degree of
creativity and constant adaptation of general principles from previous truth commissions to fit the new context. In his ethnographic study of the effects of Sierra
Leones Truth and Reconciliation Commission on one village, Gearoid Millar alerts
us to the nonlinear nature of truth commissions and the need for the innovation
to be locally informed, rather than a photocopy of the transitional justice field at
a local level through aid agencies and elites.
In keeping with this reality, the most promising newcomer on the scene of evaluation in international aid is developmental evaluation (note, developmental, not
development). As its name suggests, developmental evaluation has the purpose
of helping develop an innovation. It embraces the emergent nature and turbulence of social innovation. It also centers situational sensitivity, responsiveness and
adaptation and is appropriate for adapting effective general principles to a new
context.29 This may bode well for transitional justice, a field of activity that is highly
principle driven. Although the operationalization of developmental evaluation is
still in its infancy, it may revolutionize thinking and practice in the spheres of
28
29

Ray Pawson, Evidence-Based Policy: A Realist Approach (Thousand Oaks, CA: Sage Publications,
2006).
Patton, supra n 6.

International Journal of Transitional Justice, Vol. 4, 2010, 315328

328 Editorial Note

humanitarian and international development thinking and practice, thus trickling

down to subfields such as transitional justice.

Concluding Remarks
These reflections are by no means an exhaustive examination of the promise and
pitfalls of evaluating transitional justice. Evaluation as a field of inquiry and practice
is as varied in its methods and ideologies as is social science research. It can be
empowering or punitive. It can be qualitative, quantitative or mixed method. There
is no perfect evaluation model or approach for evaluating transitional justice, only
choices that need to be made. This editorial seeks to open up the debate around the
myriad ways and multiple places in which evidence of the impact of transitional
justice can be found. That IJTJ received more high-quality contributions than could
be published in this special issue is testimony to the importance that scholars,
advocates and policy makers attach to evidence-based decision making in the
administration of international aid in the area of transitional justice.

International Journal of Transitional Justice, Vol. 4, 2010, 315328