Workshop on JICAs

Evaluation Methods for ODA

Loan Projects
Seminar on Evaluation of Japans ODA Loan
Projects 2009
Tokyo, November 11-12 and 16-17, 2009
Takako HARAGUCHI/ Junko MIURA, GLM Institute
1

Purpose of the Evaluation Method

Workshop
By sharing the evaluation results for the Cheater
Highway Project, seminar participants
consolidate their knowledge of the concept and
procedures of Japans ODA loan project
evaluation.

Workshop Program
Day 1 (Wednesday, November 11)

Module 1: Introduction and Reminder

Module 2: Evaluation of Relevance
Module 3: Evaluation of Efficiency

Day 2 (Thursday, November 12)

Module 4: Evaluation of Effectiveness

Day 3 (Monday, November 16)

Module 5: Evaluation of Impact

Module 6: Evaluation of Sustainability

Day 4 (Tuesday, November 17)

Module 7: Evaluation Feedback

3

Reminder :
Ex-ante Evaluation

Ex-Ante Evaluation of ODA Loans

Monitoring during Project

Ex-Post Monitoring

(For 7 years after completion)

Project

Ex-Ante
Evaluation
(Appraisal)

Mid-term
Review
(5 years
after L/A)

Ex-Post
Evaluation
(2 years
After
completion)

one type of monitoring

5

Ex-ante Evaluation

For ensuring quality of ODA loan projects

with full accountability and transparency
Focused on verification of necessity and
relevance of the project and JICA assistance
Ex-Ante Evaluation Report is prepared/
made public by JICA based on the project
appraisal and discussions with borrower
countries.
6

Tools for Ex-ante Evaluation

Logical Framework (Logframe) for

enhancing logical structure of the project plan
Performance Indicators for ensuring good
steering of the project
*JICA uses Operation and Effect Indicators
Cost Benefit Analysis for ensuring economic/
financial profitability of the project
*JICA uses Internal Rates of Return (IRR)
7

Whats Logframe?

A project summary in a form of 4x4 matrix.

Outlining project plan towards specific
objectives in a specific timeframe.
Management tool to plan, implement and
monitor/evaluate the project.

Use of Logframe for Japans ODA

Loan Projects

The table format is not used for

management of Japans ODA Loan Projects

However, the idea of the logframe is

underlying in the project design and
evaluation design.

Logical Framework (Logframe)

1st column

2nd column

3rd column

4th column

Project
Summary

Indicators

Means of Assumptions
Verification

Overall Goal
Project
Purpose
Outputs
Activities

Inputs
PreConditions

10

The 1st column : Project summary

Overall Goal

Longer-term development goal.

Necessity of the project.
(Impact of the project)

Project
Purpose

Direct effects of the project (positive

changes for the target group/area)

Outputs

Goods and services created by the

project.

Activities

Actions required for achieving

Outputs.
11

The 4 column: Assumptions

th

Important
Assumptions

Conditions important for project

success, but not controllable by the
project.

Pre-conditions

Conditions to start the project

12

Relationship between 1st & 4th

Columns
Increased
income
of farmers.

Overall Goal
(Impact)

Increased cultivation
Project purpose
areas and agricultural
Outcome
production

Assumptions

Price of products
are not decreased.

Irrigation facilities

Outputs

Assumptions

Seeds are available.

Tender, Construction

Activities

Assumptions

Natural calamity not

occurred.

Fund, materials,
human resources

Inputs

Pre-conditions Water at the

13
reservoir is enough.

Vertical Logic of Logframe

(Logic model)
Project
Summary

Indicators

Means of
Verification

Assumptions

Overall Goal

Then

Project Purpose If

And if

Then

Outputs

If

And if

Then

Activities

Inputs
And if

If

Pre-Conditions
Then

If

14

The 2 & 3 Columns

nd

rd

Indicators

Measurement of the performance of

the project objectives and outputs

Means of
Verification

Sources of data for verifying

indicators

Inputs

Physical, financial and human

resources to carry out project
Activities

15

Indicators (example)

Specified evidence as:

[data type ]

Production volume of paddy

[quantity]

45,000 tons 77,000 tons

(baseline target)

[time]

between 2004 2009

In JICA, Operation and Effect Indicators are

the indicators to measure the achievement of project
purpose.
16

Operation and Effect Indicators

Operation/effect indicators are two kinds of

performance indicators for project purpose
(outcome).

Operation indicator: A quantitative measure for

the operational status of the project outputs.
Effect indicator: A quantitative measure for
effects generated by the project.

17

Logic of Operation /Effect

indicators
Goal
Project Purpose Measured by
effect
(Outcome)

Project
Purpose

Use and operation Measured by

operation
of outputs

Outputs

indicators

Project
purpose
indicators

indicators

Outputs

Activities

18

Sample Operation /Effect Indicators

Sector

Operation indicators

Effect indicators

Irrigation

Cultivated area (ha)

Production volume of major

crops (tons)

Electricity
generation

Plant load factor (%)

Net electric energy

production (GWh/year)

Water supply Amount of water supply

(1,000m3/day)

Percentage of population
served (%)

Road

Time saving (hours)

Traffic volume (no. of

vehicles)

19

In the Project Appraisal .

Based on the discussion between the Executing
Agency and JICA:

Operation/Effect indicators are selected from the

sector-wise list named Reference for Operation
and Effect Indicators.
Targets (value & time) are determined based on the
baseline data.
Clearly noted in M/D and P/M.
Reported in Ex-Ante Evaluation Report
20

Sample
indicators
of
ex-ante
with baseline and targets
evaluation
Indicators for a road
Road project
Baseline
Sections

(2001)

Target
2 years after
completion
(2009)

Operation Increase in
indicators traffic
(vehicles/day)

Section
between A
and B

1,088

1,570

Effect
indicators

Section
between A
and B

1.6

1.03

Reduction in
travel time
(hours)

21

When to set and measure

operation/ effect indicators
Monitoring (till 7 years after completion)

Project
Ex-ante
Evaluation
(Appraisal)

Ex-Post
Evaluation
(2 years
After
completion)

Set
indicators
(baseline/
target value)

EA Measures and reports indicators to JICA

22

IRR (Internal Rate of Return)

Indicator to assess economic and financial

profitability of a project, by means of cost benefit
analysis.

Important indicator of project effectiveness in

ODA loan evaluation from ex-ante to ex-post

EIRR: Economic Internal Rate of Return

FIRR: Financial Internal Rate of Return

23

FIRR and EIRR: Which to calculate?

FIRR: calculated for projects that have fee

income (e.g. toll roads, power station)

EIRR: calculated wherever possible(for all

the projects the benefits of which could be
calculated on the monetary basis).

24

Group Task

Share the result of STEP 1 of the

Pre-departure Exercise among the
group.

Outline of the Project

Hierarchy of Objectives

25

Sharing of Group Discussion

Presentation by each working group
Questions and answers
Comments from moderators

26

Reminder :
Monitoring and Mid-term
Review

27

Monitoring and Mid-term Review

Monitoring during Project

Ex-Post Monitoring

(For 7 years after completion)

Project
PSR
Ex-Ante
Evaluation
(Appraisal)

Mid-term
Review
(5 years
after L/A)

Ex-Post
Evaluation
(2 years
After
completion)

one type of monitoring

28

Monitoring of On-going Projects

By whom?

JICA Overseas Offices

HQ Regional Departments (former JBIC
Development Assistance Departments)

What to monitor?

Progress (Progress Reports)

Procurement
Disbursement
Portfolios (Portfolio meetings)
29

Mid-term Review

By JICA Evaluation Department

5 years after the conclusion of L/A
What to review:

Relevance

Is the project still relevant, considering the changes of external

factors?

Effectiveness

Is the operation/ effect indicators likely to achieve the targets?

Is IRR necessary to be recalculated?

30

Mid-term Review (cont.)

What to take as countermeasures

Re-set the targets (value and time) of operation/ effect

indicators based on the present status.
If some indicators are not feasible to monitor,
reconsider feasible ones.
Recalculate IRR, if necessary.
If the change of scope is necessary considering the
external factors (e.g. economic status, necessity),
warn the concerned agencies.
31

Reminder:
Ex-post Evaluation

32

Ex-Post Evaluation of ODA Loan

Projects
Monitoring during Project

Project

Ex-Ante
Evaluation
(Appraisal)

Mid-term
Review
(5 years
after L/A)

Ex-Post Monitoring

(For 7 years after completion)

PSR

(Former PCR)

(within 6months
after completion)

Ex-Post
Evaluation
(2 years
After
completion)

one type of monitoring

33

DAC Five Evaluation Criteria

Criteria

Basic Ideas

Relevance

Relevance of the project objectives and plans both at

appraisal and at present.

Efficiency

Comparison of plans for outputs, terms (duration)

and costs to actual results.

Effectiveness

Comparison of planned and actual figures of

operation and effect indicators and IRR

Impact

Direct and indirect impact on socio-economic and

environmental aspects.

Sustainability

Medium- and long-term sustainability of project

effects.

34

Evaluation Criteria and

Logic Model

Policy
Overall Goal
(Impact)

Impact

Program
Relevance
Project Purpose
(Outcome)

Effectiveness

Project

Sustainability
Output
Efficiency
Input

35

Work flow of ex-post

evaluation of ODA loan
projects

36

Work Flow of Ex-post Evaluation

Evaluation
Design
Data Collection
Data Analysis

1) Terms of reference
2) Evaluation design sheet
3) Questionnaires/ other survey
instruments
4) Answers to the questionnaires/
interviews, documentation,
statistics, photographs, etc.

Group Exercise

5) Evaluation summary sheet

Reporting
6) Evaluation report

Feedback

37

1) Terms of Reference

JICA Evaluation Department prepares TOR

for evaluation, including:
Purposes of evaluation;
Scope of work;
Evaluator; and
Time frame and budget

38

2) Evaluation design sheet

Evaluator reviews available information

and prepares the overall evaluation plan:

Evaluation questions: what the evaluator need to

know = information at the times of appraisal and
evaluation [BEFORE and AFTER]
Points of evaluation/ data collection strategies

All for each of the DAC Five Criteria

39

3) Questionnaires

Questionnaire for the executing agency

Questionnaire for other related agencies:

Planning agencies
Operation & maintenance (O&M) agencies
Local authorities where the project is located, etc.

Beneficiary survey sheets

40

4) Data collection

Documents
Answers to the questionnaires
Hearings from related agencies and other key
informants
Observation/ direct measurement by the
evaluator
Survey results
Etc.
41

5) Evaluation summary sheet

Criteria

Ex-ante

Ex-post

Gaps analysis

Relevance
Efficiency
Effectiveness
Impact
Sustainabilit
y
Lessons/ recommendations
42

6) Evaluation report
Standard format:
1.
Project profile (background, objectives, etc.)
2.
Evaluation results
i.
ii.
iii.
iv.
v.

3.

Relevance
Efficiency
Effectiveness
Impact
Sustainability

Feedback (lessons and recommendations)

43

Reminder:
Points of Evaluation by DAC
Five Evaluation Criteria

44

Relevance
Are the project
objectives and
plan meaningful
in relation to
policy/ program
priorities and
needs?

Levels

Logframe

Policy

Overall Goal
(Impacts)
Program

RELEVANCE
Project Purpose
(Outcome)

Project

Outputs

Inputs

45

Evaluation questions: relevance

Policy level consistency with national

development policies
Program level/ needs consistency with
sector and/or regional development programs
and needs
Project level priority of project objectives
and scopes in relation to policies, programs
and needs.
46

Data sources: relevance evaluation

Policy level and program level policy

and program documents
Project level F/S reports, SAPROF
reports, appraisal documents, MD, PM,
etc.
Needs Demand data, problems of
target areas/ sectors
47

Rating criteria: relevance

Consistency with needs/ policies a

Partial problem in consistency
with needs/ policies ..b
Serious problem in consistency
with needs/ policies ..c

48

Examples: relevance evaluation

High relevance

High priorities in development policies/ programs

Large share in investment
Continuing demands for the targeted services
Project design suitable to needs

Low relevance

Policies/ program priorities do not exist any more

Decreasing demands for the target services
49

Group task (Relevance)

Share the result of STEP 2 of the Pre-departure

Exercise (relevance evaluation) among the group.
1. Similarities among participants answers?
2. Differences? Why different?
3. Difficulties in analyzing information and making
evaluation?
4. Group conclusion on the rating?

50

Sharing of Group Discussion

Presentation by each working group
Questions and answers
Comments from moderators

51

Efficiency
Were the actual
project period
(duration) and
costs appropriate
for producing the
planned outputs?

Logframe
Overall Goal
(Impacts)

Project Purpose
(Outcome)
Outputs
EFFICIENCY
Inputs
Project period
Project cost
52

Evaluation questions: efficiency

Outputs degree of completion, reasons for

cancellation of specific output/ additional
outputs
Project period actual project period,
reasons for delays/ earlier completion
Cost actual cost, reasons for under runs/
overruns
53

Data sources: efficiency evaluation

Appraisal documents
Progress reports
Project Completion Reports (PCR)
Project account records
On-site observation
Direct measurement
Hearings from related personnel
54

Rating criteria: efficiency

1) Sub-rating for project period
100% or less of the original plan........a (3 points)
More than 100%, but 150% or less of the original plan..b (2 points)
More than 150% of the original plan ..c (1 point)

2) Sub-rating for cost

100% or less of the original plan.....a (3 points)
More than 100%, but 150% or less of the original plan..b (2 points)
More than 150% of the original plan ..c (1 point)

3) Overall rating
aa (6 points).Overall efficiency a
ab, ba, ac, ca, or bb (4-5 points) ...............Overall efficiency b
bc, cb or cc ...Overall efficiency c

55

Efficiency: When actual outputs are

different
from
the
original
plan

In case of additional outputs:

Consider the project period and cost that were

spent only for the production of the original
outputs.

In case of cancellation of original outputs:

56

Group task (Efficiency)

Share the result of STEP 3 of the Pre-departure

Exercise (efficiency evaluation) among the group.
1. Similarities among participants answers?
2. Differences? Why different?
3. Difficulties in analyzing information and making
evaluation?
4. Group conclusion on the rating?

57

Sharing of Group Discussion

Presentation by each working group
Questions and answers
Comments from moderators

58

Effectiveness
Was the project
purpose
(outcome)
achieved as a
result of outputs?

Logframe
Overall Goal
(Impacts)

IRR

Project Purpose
(Outcome)

EFFECTIVENESS
Use of
Outputs

Outputs

Inputs

59

Evaluation questions: effectiveness

Operation and effect indicators plan

(target) vs. actual
FIRR and EIRR plan vs. actual
Qualitative indicators satisfaction of users
and beneficiaries, promoting and inhibiting
factors

60

Data sources: effectiveness evaluation

Operation and effect indicators F/S and

planning documents, records of Executing
and O&M agencies, direct measurement
FIRR and EIRR F/S and planning
documents, financial statements of O&M
agencies, economic indicators
Qualitative information Pictures,
beneficiary survey
61

Rating criteria for effectiveness

80% or more of the original plan (target)

... a
50% or more, but less than 80% of the
original plan (target) ..b
Less than 50% of the original plan
(target) ...c
62

Examples: effectiveness evaluation

High effectiveness

High utility rates of project facilities (outputs)

Good performance of indicators
High satisfaction of users/ beneficiaries

Try to exclude effects of external factors !

Good performance of indicators as a result of

other investment than the project
Good performance due to favorable external
conditions
63

Group task (Effectiveness)

Share the result of STEP 4 of the Pre-departure

Exercise, including the re-calculation of EIRR,
among the group.
1. Similarities among participants answers?
2. Differences? Why different?
3. Difficulties in analyzing information and making
evaluation?
4. Group conclusion on the rating?
64

Sharing of Group Discussion

Presentation by each working group
Questions and answers
Comments from moderators

65

Impact
Was the overall
goal achieved as a
result of the
outcome? Any
other positive and
negative effects of
the project?

Logframe
Overall Goal
(Impacts)

IMPACTS

Changes in
economic, social
and environmental
conditions

Project Purpose
(Outcome)
Outputs

Inputs

66

Evaluation questions: impact

Overall Goal plan (target) vs. actual

Socio-economic impacts contribution of
the project to socio-economic development of
the country/ region
Environmental impacts impacts of the
project to natural environment; impacts of
land acquisition and involuntary resettlement;
appropriateness of countermeasures
67

Data sources: impact evaluation

Overall Goal and other socio-economic

impacts National/ regional/ sector statistics,
reports from executing agencies, beneficiary
survey
Environmental impacts Environmental
Impact Assessment (EIA) reports,
environmental monitoring, reports from
executing agencies, hearings from
environmental authority, direct measurement,
beneficiary survey
68

Examples: impact evaluation

Positive impacts

Improvement of macro-economic indicators

Improvement of social indicators
More income, employment opportunities, business
chance, etc.
More convenience, safety, empowerment, etc.

Negative impacts

Disruption of communities
Environmental pollution
Deterioration of security, etc.
69

Group task (Impact)

Share the result of STEP 5 of the Pre-departure

Exercise (Impact) among the group.
1. Similarities among participants answers?
2. Differences? Why different?
3. Difficulties in analyzing information and making
evaluation?
4. Group conclusion on the rating?

70

Sharing of Group Discussion

Presentation by each working group
Questions and answers
Comments from moderators

71

Sustainability
Will the benefits
of the project
continue in
medium and long
term?

Logframe
Overall Goal
(Impacts)

Project Purpose
(Outcome)

Operation &
maintenance

SUSTAINABILITY

Outputs

Inputs

72

Evaluation questions: sustainability

Technical Technical level of operation and

maintenance (O&M) staff
O&M system Decision-making system and
organizational control
Financial Financial status of O&M
agencies, their capacity to bear O&M costs
O&M status Appropriateness of on-going
O&M practices, conditions of the facilities
73

Data sources: sustainability

Technical PCR, O&M staff list, training

record, O&M manuals, etc.
O&M system PCR, organization chart,
official documents on organizational matters,
etc.
Financial PCR, financial statements
O&M status PCR, records of O&M
activities, inventory of spare parts,
observation, etc.
74

Rating criteria: sustainability

Highly satisfactory .. a
Small concern, but no major problem ..b
Major concern . c

75

Examples: sustainability

High sustainability

Adequate number and technical level of O&M staff

Established organizational system for O&M
Secured budget for O&M
Regular O&M practices in accordance with the plan
Good conditions of the project facilities/ equipment

Low sustainability

High turn over of trained O&M staff

Unclear O&M responsibilities
Insufficient budget allocation for O&M, serious deficit
Unavailability of spare parts
76

Group task (Sustainability)

Share the result of STEP 6 of the Pre-departure

Exercise (sustainability) among the group.
1. Similarities among participants answers?
2. Differences? Why different?
3. Difficulties in analyzing information and making
evaluation?
4. Group conclusion on the rating?

77

Sharing of Group Discussion

Presentation by each working group
Questions and answers
Comments from moderators

78

Reminder:
Evaluation Feedback

79

Evaluation Feedback from JICA (1)

For Partner Countries:

Feedback meeting for planning, executing and

O&M agencies of the project
Request for comments to the evaluation report
from concerned agencies
Official notice of recommendations for concerned
agencies
Request for conduct of Ex-post Monitoring
Follow-up facilities (SAF)
80

Evaluation Feedback from JICA (2)

For Japanese Tax Payers:

Annual Evaluation Report

For JICA:

Feedback of lessons learned to new projects

(mostly at ex-ante evaluation stage)

81

Evaluation Feedback from Partner

Country

Sharing of information on evaluation results

with concerned agencies
Taking actions that are recommended from
evaluation
Any others?

82

Lessons and Recommendations

Lessons Learned: suggestions for future or

other on-going projects
Incorporation to planning of another
project
Recommendations: suggestions for
improvement of the evaluated project
Actions to be taken by JICA, executing/
O&M agencies and/or other related
organizations

83

Group task
(Lessons and Recommendations)

Share the result of STEP 7 of the Pre-departure

Exercise (sustainability) among the group.
1. Similarities among participants answers?
2. Differences? Why different?
3. Difficulties in analyzing information and making
evaluation?
4. Group conclusion on the rating?

84

Sharing of Group Discussion

Presentation by each working group
Questions and answers
Comments from moderators

85

Group task (Overall Rating)

Share the result of STEP 8 of the Pre-departure

Exercise (Overall rating) among the group.
1. Similarities among participants answers?
2. Differences? Why different?
3. Difficulties in analyzing information and making
evaluation?
4. Group conclusion on the rating?

86

Reminder:
Ex-post Monitoring

87

Ex-Post Monitoring

Keeping monitoring of projects after expost evaluation.

(7 years after the project completion)

Keeping consistent data collection using

Monitoring Sheet

88

Ex-Post Monitoring of ODA Loan

Projects
Monitoring during Project

Ex-Post Monitoring

(For 7 years after completion)

Project

Ex-Ante
Evaluation
(Appraisal)

Mid-term
Review
(5 years
after L/A)

Ex-Post
Evaluation
(2 years
After
completion)

one type of monitoring

89

Monitoring Sheet

Objective: To confirm the sustainability of project

effects through consistent data collection after the
project completion

Low cost tool for the follow-up of completed

projects

Ownership of borrower countries concerned

Easy measures to explain projects effectiveness

90

Project Effect/ Impact Indicator Monitoring Sheet

91